DataWeave Memory Management
When processing large files through DataWeave in Mule, there are a few things you can set up to fine-tune how much memory will be used up and when.
DataWeave uses the system’s memory as a buffer while processing a transformation unless a certain threshold is exceeded, in which case it resorts to using the system’s hard disk as a buffer. By default, this threshold is set at 1572864 bytes, but this value may be changed. The value refers to memory usage of each individual Transform component, not to an aggregate of all the ones in the project.
To change the threshold value at which memory is no longer used as a buffer, you must add a system property
com.mulesoft.dw.buffersize and assign it the number (in bytes) of your new threshold. System properties may be defined in several ways. See system properties for more details and more ways you can set these.
The value you assign to this property affects your entire Mule application, affecting each instance of the Transform component individually.
Please note that for Mule runtime versions 3.8.3 and older, all indexed readers can process payloads with a maximum limit of 2 GB. Also, the text length for a single field can not be larger than 1 MB (this applies for CSV, JSON and XML).
By default, DataWeave processes the transformation of a message as soon as the component is called out in the flow. You can change this behavior so that the DataWeave transformation returns a
WeaveOutputHandler, which is only processed when read by another component. This handler is capable of deferring writing the Mule Message’s payload until there is a stream available to write it to. This allows for the DataWeave output to remain outside of the heap as processing continues on other components in the flow.
To set this up, in the XML of your Transform component, add a
mode attribute. This attribute accepts the values
immediate, the output is an inputStream
deferred, the output is a