Supported Data Formats
DataWeave can read and write many types of data formats, such as JSON, XML, and many others. Before you begin, note that 2.x versions of DataWeave are used by Mule 4 apps. For DataWeave in Mule 3 apps, refer to the DataWeave version 1.2 documentation. For other Mule versions, you can use the version selector in the DataWeave table of contents.
DataWeave supports the following formats as input and output:
| MIME Type | ID | Supported Formats |
|---|---|---|
|
|
|
|
|
|
|
|
DataWeave Format (dw) for testing a DataWeave expression |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
DataWeave Readers
DataWeave can read input data as a whole by loading it into memory or by indexing it in local storage and, for some data formats, DataWeave can read data sequentially in parts by streaming the input. The indexed reading strategy implements a mechanism to avoid out-of-memory issues when processing large files. The streaming strategy can improve performance but restricts file access to sequential access (which disallows random access) because you can access only the part of the file that is being read and stored in memory.
| Read Strategy | Description | Supported Formats |
|---|---|---|
In-Memory |
This strategy parses the entire document and loads it into memory, enabling random access to data. When using this strategy, a DataWeave script can access any part of the resulting value at any time. |
DataWeave can read all supported formats using this strategy. |
Indexed |
This strategy parses the entire document and uses disk space to avoid out-of-memory issues on large files, enabling random access to data. When using this strategy, a DataWeave script can access any part of the resulting value at any time. When processing a For additional details, see Indexed Readers in DataWeave |
|
Streaming |
This strategy partitions the input document into smaller items and accesses the data sequentially, storing the current item in memory. A DataWeave selector can access only the portion of the file that is getting read. For additional details, see Streaming in DataWeave |
|
Using Reader and Writer Properties
In some cases, it is necessary to modify or specify aspects of the format
through format-specific properties. For example, you can specify CSV input and
output properties, such as the separator (or delimiter) to use in the CSV file.
For Cobol copybook, you need to specify the path to a schema file using the
schemaPath property.
You can append reader properties to the MIME type (outputMimeType) attribute
for certain components in your Mule app. Listeners and Read operations accept
these settings. For example, this On New File listener example identifies the , separator for a CSV input file:
<file:listener doc:name="On New File" config-ref="File_Config" outputMimeType='application/csv; separator=","'>
<scheduling-strategy >
<fixed-frequency frequency="45" timeUnit="SECONDS"/>
</scheduling-strategy>
<file:matcher filenamePattern="comma_separated.csv" />
</file:listener>Note that the outputMimeType setting above helps the CSV reader interpret
the format and delimiter of the input comma_separated.csv file, not the writer.
To specify the output format, you can provide the MIME type and any writer
properties for the writer, such as the CSV or JSON writer used by a File Write
operation. For example, you might need to write a pipe (|) delimiter in your
CSV output payload, instead of some other delimiter used in the input. To do
this, you append the property and its value to the output directive of a
DataWeave expression. For example, this Write operation specifies the pipe as
a separator:
<file:write doc:name="Write" config-ref="File_Config" path="my_transform">
<file:content ><![CDATA[#[output application/csv separator="|" --- payload]]]></file:content>
</file:write>The following example shows a DataWeave script in which the output directive specifies two writer properties, one for the CSV field separator, another to indicate that there is no header.
%dw 2.0
output application/csv separator=";", header=false
---
payloadWriter properties depend on the format. The following example uses the JSON writer property, skipNullOn:
%dw 2.0
output application/json skipNullOn="everywhere"
---
payloadThe following example shows a DataWeave script that uses an invisible ASCII character as a separator for the CSV output. The separator property value is \u001E. \u is the escape sequence representation and 001E is the hexadecimal representation of the record separator (RS) ASCII character.
%dw 2.0
output application/csv separator = "\u001E"
---
[
{
"Name" : "Tom",
"UID" : 2569,
"ShipOrderID" : "ui-288188"
},
{
"Name" : "Alan",
"UID" : 7756,
"ShipOrderID" : "xj-232142"
},
{
"Name" : "Dan",
"UID" : 7821,
"ShipOrderID" : "uk-259459"
}
]The CSV example produces the following output:
Name UID ShipOrderID
Tom 2569 ui-288188
Alan 7756 xj-232142
Dan 7821 uk-259459Scripts that use the read or readUrl methods can also specify reader properties through the readerProperties parameter. See examples in read. However, note that use of reader properties is more common in message sources, such as the HTTP listener, as shown in the CSV reader example.
It is also possible to use a Mule variable as a value of a configuration property. For more complete examples, see Set Reader and Writer Configuration Properties.
Setting MIME Types
You can specify the MIME type for the input and output data that flows through a Mule app.
For DataWeave transformations, you can specify the MIME type for the output data.
For example, you might set the output header directive of an expression in the
Transform Message component or a Write operation to output application/json or
output application/csv.
This example sets the MIME type through a File Write operation to ensure that a format-specific writer, the CSV writer, outputs the payload in CSV format:
<file:write doc:name="Write" config-ref="File_Config" path="my_transform">
<file:content ><![CDATA[#[output application/csv --- payload]]]></file:content>
</file:write>Starting in Mule 4.3.0, you can set the output directive using the format ID alone,
instead of using the MIME type. For example, you might set the output header directive
of an expression in the Transform Message component or a Write operation to
output json or output csv. You can also use the format ID to
differentiate the format of the output data from the MIME type in the output header, and
you can use a custom MIME type.
For example, you can write JSON data but customize the MIME type to
application/problem+json by using the DataWeave directive output application/problem+json with json.
See Change a Script’s MIME Type Output
for examples that customize the MIME type output of a script.
The with keyword separates the output MIME type from the DataWeave writer for the script’s output. For example, you can specify 'output application/csv with binary' to output the application/csv MIME type when using the binary writer. This setting is necessary in cases shown in Use Dynamic Writer Properties and Use Reader and Writer Properties in DataWeave Functions. It is also possible to append writer properties, such as in the use case output application/my-custom-type with json indent=false.
For input data, format-specific readers for Mule sources (such as the
On New File listener), Mule operations (such as Read and HTTP Request
operations), and DataWeave expressions attempt to infer the MIME type
from metadata that is associated with input payloads, attributes, and
variables in the Mule event. When the MIME type cannot be inferred from
the metadata (and when that metadata is not static), Mule sources and
operations allow you to specify the MIME type for the reader. For example,
you might set the MIME type for the On New File listener to
outputMimeType='application/csv' for CSV file input. This setting provides
information about the file format to the CSV reader.
<file:listener doc:name="On New File"
config-ref="File_Config"
outputMimeType='application/csv'>
</file:listener>Note that reader settings are not used to perform a transformation from one format to another. They simply help the reader interpret the format of the input.
You can also set special reader and writer properties for use by the format-specific reader or writer of a source, operation, or component. See Using Reader and Writer Properties.