Newline Delimited JSON (ndjson) Format

MIME type: application/x-ndjson

ID: ndjson

DataWeave represents the Newline Delimited JSON format (ndjson) as an array of objects. Each line of the ndjson format is mapped to one object in the array.

The following parser strategies are supported by the ndjson reader:

In-memory
Streaming

For details, see DataWeave Readers.

Examples

The following examples show uses of the ndjson format.

Example
Example

Example

This example shows how DataWeave represents a simple ndjson input.

Input

{"name": "Leandro","lastName": "Shokida"}
{"name": "Mariano","lastName": "De Achaval"}

Source

The DataWeave script transforms the ndjson input to the DataWeave (dw) format and MIME type.

%dw 2.0
output application/dw
---
payload

Output

The DataWeave (dw) format outputs the ndjson input into an array of comma-separated objects.

[
    {"name": "Leandro","lastName": "Shokida"},
    {"name": "Mariano","lastName": "De Achaval"}
]

Example

This example shows that the ndjson reader ignores all lines of ndjson data that are invalid if skipInvalid=true.

Input

The input to the DataWeave source includes valid and invalid lines of ndjson data. Assume that the input is from a file myInput.ndjson.

myInput.ndjson:

{"name": "Christian"
{"name": "Mariano"}
{"name": "Tomo"
{"name": "Shoki"}

Source

The DataWeave script inputs the contents of the input file myInput.ndjson, applies the skipInvalid=true reader property, and transforms the input to the JSON format and MIME type.

%dw 2.0
var myInput = readUrl('classpath://myInput.ndjson', 'application/x-ndjson, {skipInvalid=true})
output application/json
---
myInput

Output

The JSON output is an array of the valid objects from the ndjson input.

[
  {
    "name": "Mariano"
  },
  {
    "name": "Shoki"
  }
]

Configuration Properties

DataWeave supports the following configuration properties for this format.

Reader Properties

This format accepts properties that provide instructions for reading input data.

Parameter Type Default Description

Parameter	Type	Default	Description
`ignoreEmptyLine`	`Boolean`	`true`	Ignores empty lines. Valid values are `true` or `false`.
`skipInvalid`	`Boolean`	`false`	Skips invalid records and ignores values that are not valid in this format. Valid values are `true` or `false`.

ignoreEmptyLine

Boolean

true

Ignores empty lines.

Valid values are true or false.

skipInvalid

Boolean

false

Skips invalid records and ignores values that are not valid in this format.

Valid values are true or false.

Writer Properties

This format accepts properties that provide instructions for writing output data.

Parameter Type Default Description

Parameter	Type	Default	Description
`bufferSize`	`Number`	`8192`	Size of the buffer writer, in bytes. The value must be greater than 8.
`deferred`	`Boolean`	`false`	Generates the output as a data stream when set to `true`, and defers the script’s execution until the generated content is consumed. Valid values are `true` or `false`.
`encoding`	`String`	`'UTF-8'`	Encoding that the writer uses for output. Defaults to "UTF-8".
`skipNullOn`	`String`	`null`	Skips `null` values. By default, DataWeave does not skip. `arrays` Ignore and omit `null` values inside arrays from the JSON output, for example, with `output application/x-ndjson skipNullOn="arrays"`. `objects` Ignore key-value pairs that have `null` as the value, for example, with `output application/x-ndjson skipNullOn="objects"`. `everywhere` Apply `skipNullOn` to arrays and objects, for example, `output application/x-ndjson skipNullOn="everywhere"`. Valid values are `arrays` or `objects` or `everywhere`.
`writeAttributes`	`Boolean`	`false`	Converts attributes of a key into child key-value pairs of that key. The attribute key name starts with `@`. Valid values are `true` or `false`.

bufferSize

Number

8192

Size of the buffer writer, in bytes. The value must be greater than 8.

deferred

Boolean

false

Generates the output as a data stream when set to true, and defers the script’s execution until the generated content is consumed.

Valid values are true or false.

encoding

String

'UTF-8'

Encoding that the writer uses for output. Defaults to "UTF-8".

skipNullOn

String

null

Skips null values. By default, DataWeave does not skip.

arrays
Ignore and omit null values inside arrays from the JSON output, for example, with output application/x-ndjson skipNullOn="arrays".
objects
Ignore key-value pairs that have null as the value, for example, with output application/x-ndjson skipNullOn="objects".
everywhere
Apply skipNullOn to arrays and objects, for example, output application/x-ndjson skipNullOn="everywhere".

Valid values are arrays or objects or everywhere.

writeAttributes

Boolean

false

Converts attributes of a key into child key-value pairs of that key. The attribute key name starts with @.

Valid values are true or false.

Supported MIME Types

This format supports the following MIME types.

MIME Type

MIME Type
`application/x-ndjson`
`application/x-ldjson`

application/x-ndjson

application/x-ldjson