Contact Us 1-800-596-4880
  1. Homepage
  2. DataWeave (2.8)
  3. Language Guide

  4. Memory Management

DataWeave Memory Management

When processing large files through DataWeave in Mule runtime engine, there are a few things you can set up to fine-tune how much memory will be used and when.

RAM vs Disk Usage

DataWeave keeps small files in memory, but after a configurable threshold uses disk space to avoid running out of memory. The created buffer files are placed in a default temporary directory. If you want to store those files in a custom directory instead, you can specify the directory by using the java.io.tmpdir property.

The buffer files remain in the temporary directory until the streams that reference the files are closed. Closure normally occurs when flows complete their execution, so many buffer files in the temporary folder can remain in use during long-running and concurrent executions.

Three types of DataWeave buffer files are generated:

  • dw-buffer-output-${count}.tmp
    Used to store the output of a transformation when the result is bigger than the threshold 1572864 bytes. To change this threshold value, add the system property com.mulesoft.dw.max_memory_allocation and assign it the number of bytes you want as your new threshold. Mule runtime engine deletes the file when the value is no longer referenced, JVM GC collects it or when the Mule Event finishes executing.

  • dw-buffer-input-${count}.tmp
    Used to store the input of a transformation when it is bigger than the in-memory buffer, which is 1572864 bytes by default. This is analogous to the dw-buffer-output-${count}.tmp file, but for input data.

  • dw-buffer-index-${count}.tmp
    Used to store index information of a value being read. This file helps DataWeave access data quickly. Mule runtime engine deletes the file when the execution of the transformation ends or, in a streaming use case like the foreach loop, when the stream ends (when foreach finishes its execution).

System properties:
Because you can define system properties in several ways, see system properties for further details.

  • com.mulesoft.dw.buffersize
    This system property determines the size in bytes of the in-memory input and output buffers used in DataWeave to keep the processed inputs and outputs. The default buffer size is 8192 bytes (8 KB). This property receives a number (size in bytes), for example: -Dcom.mulesoft.dw.buffersize=8192.

  • com.mulesoft.dw.directbuffer.disable
    Introduced in Mule 4.2.2, this option controls whether DataWeave uses off-heap memory (the default) or heap memory. DataWeave uses off-heap memory for internal buffering. However, this setting can cause problems on machines that have only a small amount of memory. This property receives a boolean, for example: -Dcom.mulesoft.dw.directbuffer.disable=true.

  • com.mulesoft.dw.memory_pool_size
    Since Mule 4.3.0, DataWeave buffers use off-heap memory from a pool up to a defined size and allocates the rest using heap memory. This property determines the number of slots in the pool. The default number of slots is 60. This property receives a number (slots in the pool), for example: -Dcom.mulesoft.dw.memory_pool_size=60.

  • com.mulesoft.dw.max_memory_allocation
    Introduced in Mule 4.3.0, this property determines the maximum amount of memory to be used before it switches to disk. If a payload exceeds this size, it is stored in dw-buffer-input-${count}.tmp and dw-buffer-output-${count}.tmp temporary files. It also determines the size in bytes of each slot in the pool of off-heap memory. The default size is 1572864 bytes (1.5 MB). This property receives a number (size in bytes), for example: -Dcom.mulesoft.dw.max_memory_allocation=1572864.

  • com.mulesoft.dw.buffer.memory.monitoring (experimental)
    Introduced in Mule 4.3.0, when this property is enabled, a message is logged each time a slot from the memory pool is taken or released. Note that it may be removed or change its behavior in future versions. This property receives a boolean, for example: -Dcom.mulesoft.dw.buffer.memory.monitoring=true.

  • mule.weave.script.parser.cache.size
    This system property determines the maximum size of compiled scripts that are being cached. The default value is 10.000.

  • com.mulesoft.dw.charbuffersize
    This system property determines the size (in bytes) of the buffer used by the writter. The default value is 4.096.

See Also