Supported Data Formats

DataWeave can read and write many types of data formats, such as JSON, XML, and many others. Before you begin, note that 2.x versions of DataWeave are used by Mule 4 apps. For DataWeave in Mule 3 apps, refer to the DataWeave version 1.2 documentation. For other Mule versions, you can use the version selector in the DataWeave table of contents.

DataWeave supports the following formats as input and output:

MIME Type ID Supported Formats

application/avro

avro

Avro Format

application/csv

csv

CSV Format

application/dw

dw

DataWeave Format (dw) for testing a DataWeave expression

application/flatfile

flatfile

Flat File Format, COBOL Copybook Format, Fixed Width Format

application/java

java

Java Format, Enum Format for Java

application/json

json

JSON Format

application/octet-stream

binary

Binary Format

application/xlsx

excel

Excel Format

application/xml

xml

XML Format, CData Custom Type

application/x-ndjson

ndjson

Newline Delimited JSON Format (ndjson)

application/x-www-form-urlencoded

urlencoded

URL Encoded Format

application/yaml

yaml

YAML Format

multipart/form-data

multipart

Multipart Format

text/plain

text

Text Plain Format

text/x-java-properties

properties

Text Java Properties

DataWeave Readers

DataWeave can read input data as a whole by loading it into memory or by indexing it in local storage and, for some data formats, DataWeave can read data sequentially in parts by streaming the input. The indexed reading strategy implements a mechanism to avoid out-of-memory issues when processing large files. The streaming strategy can improve performance but restricts file access to sequential access (which disallows random access) because you can access only the part of the file that is being read and stored in memory.

Read Strategy Description Supported Formats

In-Memory

This strategy parses the entire document and loads it into memory, enabling random access to data. When using this strategy, a DataWeave script can access any part of the resulting value at any time.

DataWeave can read all supported formats using this strategy.

Indexed

This strategy parses the entire document and uses disk space to avoid out-of-memory issues on large files, enabling random access to data. When using this strategy, a DataWeave script can access any part of the resulting value at any time.

When processing a String with a size larger than 1.5 MB, DataWeave automatically splits the value in chunks to avoid out-of-memory issues. This feature works only with JSON and XML input data.

For additional details, see Indexed Readers in DataWeave

  • CSV

  • JSON

  • XML

Streaming

This strategy partitions the input document into smaller items and accesses the data sequentially, storing the current item in memory. A DataWeave selector can access only the portion of the file that is getting read.

For additional details, see Streaming in DataWeave

  • CSV

  • JSON

  • Excel (XLSX)

  • XML

Using Reader and Writer Properties

In some cases, it is necessary to modify or specify aspects of the format through format-specific properties. For example, you can specify CSV input and output properties, such as the separator (or delimiter) to use in the CSV file. For Cobol copybook, you need to specify the path to a schema file using the schemaPath property.

You can append reader properties to the MIME type (outputMimeType) attribute for certain components in your Mule app. Listeners and Read operations accept these settings. For example, this On New File listener example identifies the , separator for a CSV input file:

Example: Properties for the CSV Reader
<file:listener doc:name="On New File" config-ref="File_Config" outputMimeType='application/csv; separator=","'>
  <scheduling-strategy >
    <fixed-frequency frequency="45" timeUnit="SECONDS"/>
  </scheduling-strategy>
  <file:matcher filenamePattern="comma_separated.csv" />
</file:listener>

Note that the outputMimeType setting above helps the CSV reader interpret the format and delimiter of the input comma_separated.csv file, not the writer.

To specify the output format, you can provide the MIME type and any writer properties for the writer, such as the CSV or JSON writer used by a File Write operation. For example, you might need to write a pipe (|) delimiter in your CSV output payload, instead of some other delimiter used in the input. To do this, you append the property and its value to the output directive of a DataWeave expression. For example, this Write operation specifies the pipe as a separator:

Example: output Directive for the CSV Writer
<file:write doc:name="Write" config-ref="File_Config" path="my_transform">
  <file:content ><![CDATA[#[output application/csv separator="|" --- payload]]]></file:content>
</file:write>

The following example shows a DataWeave script in which the output directive specifies two writer properties, one for the CSV field separator, another to indicate that there is no header.

%dw 2.0
output application/csv separator=";", header=false
---
payload

Writer properties depend on the format. The following example uses the JSON writer property, skipNullOn:

%dw 2.0
output application/json skipNullOn="everywhere"
---
payload

The following example shows a DataWeave script that uses an invisible ASCII character as a separator for the CSV output. The separator property value is \u001E. \u is the escape sequence representation and 001E is the hexadecimal representation of the record separator (RS) ASCII character.

%dw 2.0
output application/csv separator = "\u001E"
---
[
	{
		"Name" : "Tom",
		"UID" : 2569,
		"ShipOrderID" : "ui-288188"
	},
	{
		"Name" : "Alan",
		"UID" : 7756,
		"ShipOrderID" : "xj-232142"
	},
	{
		"Name" : "Dan",
		"UID" : 7821,
		"ShipOrderID" : "uk-259459"
	}
]

The CSV example produces the following output:

NameUIDShipOrderID
Tom2569ui-288188
Alan7756xj-232142
Dan7821uk-259459

Scripts that use the read or readUrl methods can also specify reader properties through the readerProperties parameter. See examples in read. However, note that use of reader properties is more common in message sources, such as the HTTP listener, as shown in the CSV reader example.

It is also possible to use a Mule variable as a value of a configuration property. For more complete examples, see Set Reader and Writer Configuration Properties.

Setting MIME Types

You can specify the MIME type for the input and output data that flows through a Mule app.

For DataWeave transformations, you can specify the MIME type for the output data. For example, you might set the output header directive of an expression in the Transform Message component or a Write operation to output application/json or output application/csv.

This example sets the MIME type through a File Write operation to ensure that a format-specific writer, the CSV writer, outputs the payload in CSV format:

Example: MIME Type for the CSV Writer
<file:write doc:name="Write" config-ref="File_Config" path="my_transform">
  <file:content ><![CDATA[#[output application/csv --- payload]]]></file:content>
</file:write>

Starting in Mule 4.3.0, you can set the output directive using the format ID alone, instead of using the MIME type. For example, you might set the output header directive of an expression in the Transform Message component or a Write operation to output json or output csv. You can also use the format ID to differentiate the format of the output data from the MIME type in the output header, and you can use a custom MIME type. For example, you can write JSON data but customize the MIME type to application/problem+json by using the DataWeave directive output application/problem+json with json. See Change a Script’s MIME Type Output for examples that customize the MIME type output of a script.

The with keyword separates the output MIME type from the DataWeave writer for the script’s output. For example, you can specify 'output application/csv with binary' to output the application/csv MIME type when using the binary writer. This setting is necessary in cases shown in Use a Mule Variable as a Configuration Value and Use Reader and Writer Properties in DataWeave Functions. It is also possible to append writer properties, such as in the use case output application/my-custom-type with json indent=false.

For input data, format-specific readers for Mule sources (such as the On New File listener), Mule operations (such as Read and HTTP Request operations), and DataWeave expressions attempt to infer the MIME type from metadata that is associated with input payloads, attributes, and variables in the Mule event. When the MIME type cannot be inferred from the metadata (and when that metadata is not static), Mule sources and operations allow you to specify the MIME type for the reader. For example, you might set the MIME type for the On New File listener to outputMimeType='application/csv' for CSV file input. This setting provides information about the file format to the CSV reader.

Example: MIME Type for the CSV Reader
<file:listener doc:name="On New File"
  config-ref="File_Config"
  outputMimeType='application/csv'>
</file:listener>

Note that reader settings are not used to perform a transformation from one format to another. They simply help the reader interpret the format of the input.

You can also set special reader and writer properties for use by the format-specific reader or writer of a source, operation, or component. See Using Reader and Writer Properties.

Was this article helpful?

💙 Thanks for your feedback!

Edit on GitHub