Nav

DataWeave Memory Management

When processing large files through DataWeave in Mule, there are a few things you can set up to fine-tune how much memory will be used up and when.

Memory vs Disk Usage

DataWeave uses the system’s memory as a buffer while processing a transformation unless a certain threshold is exceeded, in which case it resorts to using the system’s hard disk as a buffer. By default, this threshold is set at 1572864 bytes, but this value may be changed. The value refers to memory usage of each individual Transform component, not to an aggregate of all the ones in the project.

To change the threshold value at which memory is no longer used as a buffer, you must add a system property com.mulesoft.dw.buffersize and assign it the number (in bytes) of your new threshold. System properties may be defined in several ways. See system properties for more details and more ways you can set these.

The value you assign to this property affects your entire Mule application, affecting each instance of the Transform component individually.

Please note that for Mule runtime versions 3.8.3 and older, all indexed readers can process payloads with a maximum limit of 2 GB. Also, the text length for a single field can not be larger than 1 MB (this applies for CSV, JSON and XML).

Immediate vs Deferred Execution

By default, DataWeave processes the transformation of a message as soon as the component is called out in the flow. You can change this behavior so that the DataWeave transformation returns a WeaveOutputHandler, which is only processed when read by another component. This handler is capable of deferring writing the Mule Message’s payload until there is a stream available to write it to. This allows for the DataWeave output to remain outside of the heap as processing continues on other components in the flow.

To set this up, in the XML of your Transform component, add a mode attribute. This attribute accepts the values immediate or deferred.

  • With immediate, the output is an inputStream

  • With deferred, the output is a WeaveOutputHandler

Example: <ee:transform mode="deferred">

We use cookies to make interactions with our websites and services easy and meaningful, to better understand how they are used and to tailor advertising. You can read more and make your cookie choices here. By continuing to use this site you are giving us your consent to do this.

+