Flat File Format

MIME type: application/flatfile

ID: flatfile

The Flat File format supports multiple types of fixed width records within a single message. The schema structure allows you to define how different record types are distinguished, and how the records are logically grouped.

Flat File in DataWeave supports files of up to 15 MB, and the memory requirement is roughly 40 to 1. For example, a 1-MB file requires up to 40 MB of memory to process, so it’s important to consider this memory requirement in conjunction with your TPS needs for large flat files. This is not an exact figure; the value might vary according to the complexity of the mapping instructions.

Example: Specify a Flat File Schema

This example shows a DataWeave script that outputs data in the Flat File format. Notice that it uses the schemaPath and structureIndent writer properties.

%dw 2.0
output application/flatfile schemaPath="src/main/resources/test-data/QBReqRsp.esl", structureIdent="QBResponse"
---
payload

Configuration Properties

DataWeave supports the following configuration properties for this format.

Reader Properties

DataWeave accepts properties that provide instructions for reading input data in this format.

Parameter Type Default Description

Parameter	Type	Default	Description
`allowLenientWithBinaryNotEndElement`	`Boolean`	`false`	When the schema contains elements of type Binary or Packed, the `lenient` option does not allow short records, regardless of the last element’s format type. When you set this property to `true`, the validation applies only to cases in which the record ends with type Binary or Packed.
`enforceRequires`	`Boolean`	`false`	Error if required value missing. Valid values are `true` or `false`.
`missingValues`	`String`	`nulls` for copybook schema, `spaces` otherwise	Fill character used to represent missing values. To activate a non-default setting, set the `useMissCharAsDefaultForFill` property to `true`, and use one of the following values to `missingValues`: `none`: Treat all data as actual values `spaces`: Interpret a field consisting of only spaces as a missing value `zeroes`: Interpret numeric fields consisting of only '0' characters and character fields consisting of only spaces as missing values `nulls`: Interpret a field consisting only of 0 bytes as a missing value
`notTruncateDependingOnSubjectNotPresent`	`Boolean`	`false`	Fills the entire group when the DEPENDING ON subject is not present. Introduced in DataWeave 2.3 with the August 2021 release of Mule 4.3.0-20210622.
`recordParsing`	`String`	`strict`	Expected separation between lines/records: `strict`: Line break expected at exact end of each record. `lenient`: Line break is used, but records can be shorter or longer than the schema specifies. Do not use `lenient` if your payload lacks line breaks. The other options to `recordParsing` support records that lack line breaks. `noTerminator`: Records follow one another with no separation. This option is preferred for fixed-length records that lack a line break. `singleRecord`: Entire input is a single record. Note that schemas with type `Binary` or `Packed` don’t allow for line break detection, so setting `recordParsing` to `lenient` only allows long records to be handled, not short ones. These schemas also currently only work with certain single-byte character encodings (so not with UTF-8 or any multibyte format).
`schemaPath` (Required)	`String`	`null`	Schema definition. Location in your local disk of the schema file used to parse your input.
`segmentIdent`	`String`	`null`	Segment identifier in the schema for fixed width or copybook schemas (only needed when parsing a single segment/record definition and if the schema includes multiple segment definitions).
`structureIdent`	`String`	`null`	Structure identifier in schema for flatfile schemas (only needed when parsing a structure definition, and if the schema includes multiple structure definitions)
`truncateDependingOn`	`Boolean`	`false`	Truncate COBOL copybook DEPENDING ON values to the length used. Valid values are `true` or `false`.
`useMissCharAsDefaultForFill`	`Boolean`	`false`	By default, the flat file reader and writer use spaces for missing characters and ignore the setting of the `missingValues` property. When you set this property to `true`, the reader honors the setting of the `missingValues` property. Introduced in DataWeave 2.3 (2.3.0-20210823) for the September 2021 release of Mule 4.3.0-20210823. Valid values are `true` or `false`.
`zonedDecimalStrict`	`Boolean`	`false`	Use the `strict` ASCII form of sign encoding for COBOL copybook zoned decimal values. Valid values are `true` or `false`.

allowLenientWithBinaryNotEndElement

Boolean

false

When the schema contains elements of type Binary or Packed, the lenient option does not allow short records, regardless of the last element’s format type. When you set this property to true, the validation applies only to cases in which the record ends with type Binary or Packed.

enforceRequires

Boolean

false

Error if required value missing. Valid values are true or false.

missingValues

String

nulls for copybook schema, spaces otherwise

Fill character used to represent missing values. To activate a non-default setting, set the useMissCharAsDefaultForFill property to true, and use one of the following values to missingValues:

none: Treat all data as actual values
spaces: Interpret a field consisting of only spaces as a missing value
zeroes: Interpret numeric fields consisting of only '0' characters and character fields consisting of only spaces as missing values
nulls: Interpret a field consisting only of 0 bytes as a missing value

notTruncateDependingOnSubjectNotPresent

Boolean

false

Fills the entire group when the DEPENDING ON subject is not present. Introduced in DataWeave 2.3 with the August 2021 release of Mule 4.3.0-20210622.

recordParsing

String

strict

Expected separation between lines/records:

strict: Line break expected at exact end of each record.
lenient: Line break is used, but records can be shorter or longer than the schema specifies. Do not use lenient if your payload lacks line breaks. The other options to recordParsing support records that lack line breaks.
noTerminator: Records follow one another with no separation. This option is preferred for fixed-length records that lack a line break.
singleRecord: Entire input is a single record.

Note that schemas with type Binary or Packed don’t allow for line break detection, so setting recordParsing to lenient only allows long records to be handled, not short ones. These schemas also currently only work with certain single-byte character encodings (so not with UTF-8 or any multibyte format).

schemaPath (Required)

String

null

Schema definition. Location in your local disk of the schema file used to parse your input.

segmentIdent

String

null

Segment identifier in the schema for fixed width or copybook schemas (only needed when parsing a single segment/record definition and if the schema includes multiple segment definitions).

structureIdent

String

null

Structure identifier in schema for flatfile schemas (only needed when parsing a structure definition, and if the schema includes multiple structure definitions)

truncateDependingOn

Boolean

false

Truncate COBOL copybook DEPENDING ON values to the length used. Valid values are true or false.

useMissCharAsDefaultForFill

Boolean

false

By default, the flat file reader and writer use spaces for missing characters and ignore the setting of the missingValues property. When you set this property to true, the reader honors the setting of the missingValues property. Introduced in DataWeave 2.3 (2.3.0-20210823) for the September 2021 release of Mule 4.3.0-20210823. Valid values are true or false.

zonedDecimalStrict

Boolean

false

Use the strict ASCII form of sign encoding for COBOL copybook zoned decimal values. Valid values are true or false.

Writer Properties

DataWeave accepts properties that provide instructions for writing output data in this format.

Parameter Type Default Description

Parameter	Type	Default	Description
`bufferSize`	`Number`	`8192`	Size of the buffer writer.
`deferred`	`Boolean`	`false`	When set to `true`, DataWeave generates the output as a data stream, and the script’s execution is deferred until it is consumed. Valid values are `true` or `false`.
`encoding`	`String`	`null`	Encoding to be used by this writer, such as `UTF-8`.
`enforceRequires`	`Boolean`	`false`	Error if a required value is missing. Valid values are `true` or `false`.
`missingValues`	`String`	`NULLS` for copybook schema, `SPACES` otherwise	Fill character used to represent missing values. To activate a non-default setting, use `useMissCharAsDefaultForFill`, and use one of the following values to `missingValues`: `NONE`: Write nothing for missing values. `SPACES`: Fill the field with spaces. `ZEROES`: Fill numeric fields with `0` characters and character fields with space characters. `NULLS`: Fill the field with 0 bytes.
`notTruncateDependingOnSubjectNotPresent`	`Boolean`	`false`	Fills the entire group when the DEPENDING ON subject is not present. Introduced in DataWeave 2.3 with the August 2021 release of Mule 4.3.0-20210622.
`recordTerminator`	`String`	System property `line.separator`	Record separator line break. Valid values: `lf` `cr` `crlf` `none` Note that in Mule versions 4.0.4 and later, this is only used as a separator when there are multiple records. Values translate directly to character codes. (`none` leaves no termination on each record).
`retainEmptyStringFieldsOnParsing`	`Boolean`	`false`	When set to `true`, this property makes the reader retain fields that lack values and set their values to an empty string. The default behavior of the reader is to remove those fields. Introduced in the January 2023 version of DataWeave 2.4 for Mule version 4.4.0-20230102.
`schemaPath` (Required)	`String`	`null`	Schema definition. Path where the schema file to be used is located.
`segmentIdent`	`String`	`null`	Segment identifier in the schema for fixed width or copybook schemas (only needed when writing a single segment/record definition, and if the schema includes multiple segment definitions).
`structureIdent`	`String`	`null`	Structure identifier in schema for flatfile schemas (only needed when writing a structure definition and if the schema includes multiple structure definitions)
`trimValues`	`Boolean`	`false`	Trim string values longer than the field length by truncating trailing characters. Valid values are `true` or `false`.
`trimValues`	`Boolean`	`false`	Indicates whether trim values are longer than the field width. Valid Options are `true` or `false`.
`truncateDependingOn`	`Boolean`	`false`	Truncate COBOL copybook DEPENDING ON values to the length used. Valid values are `true` or `false`.
`useMissCharAsDefaultForFill`	`Boolean`	`false`	By default, the flat file reader and writer use spaces for missing characters and ignore the setting of the `missingValues` property. When you set this property to `true`, the writer honors the setting of the `missingValues` property. Introduced in DataWeave 2.3 (2.3.0-20210823) for the September 2021 release of Mule 4.3.0-20210823.
`zonedDecimalStrict`	`Boolean`	`false`	Use the `strict` ASCII form of sign encoding for COBOL copybook zoned decimal values. Valid values are `true` or `false`.

bufferSize

Number

8192

Size of the buffer writer.

deferred

Boolean

false

When set to true, DataWeave generates the output as a data stream, and the script’s execution is deferred until it is consumed. Valid values are true or false.

encoding

String

null

Encoding to be used by this writer, such as UTF-8.

enforceRequires

Boolean

false

Error if a required value is missing. Valid values are true or false.

missingValues

String

NULLS for copybook schema, SPACES otherwise

Fill character used to represent missing values. To activate a non-default setting, use useMissCharAsDefaultForFill, and use one of the following values to missingValues:

NONE: Write nothing for missing values.
SPACES: Fill the field with spaces.
ZEROES: Fill numeric fields with 0 characters and character fields with space characters.
NULLS: Fill the field with 0 bytes.

notTruncateDependingOnSubjectNotPresent

Boolean

false

Fills the entire group when the DEPENDING ON subject is not present. Introduced in DataWeave 2.3 with the August 2021 release of Mule 4.3.0-20210622.

recordTerminator

String

System property line.separator

Record separator line break. Valid values:

lf
cr
crlf
none

Note that in Mule versions 4.0.4 and later, this is only used as a separator when there are multiple records. Values translate directly to character codes. (none leaves no termination on each record).

retainEmptyStringFieldsOnParsing

Boolean

false

When set to true, this property makes the reader retain fields that lack values and set their values to an empty string. The default behavior of the reader is to remove those fields. Introduced in the January 2023 version of DataWeave 2.4 for Mule version 4.4.0-20230102.

schemaPath (Required)

String

null

Schema definition. Path where the schema file to be used is located.

segmentIdent

String

null

Segment identifier in the schema for fixed width or copybook schemas (only needed when writing a single segment/record definition, and if the schema includes multiple segment definitions).

structureIdent

String

null

Structure identifier in schema for flatfile schemas (only needed when writing a structure definition and if the schema includes multiple structure definitions)

trimValues

Boolean

false

Trim string values longer than the field length by truncating trailing characters. Valid values are true or false.

trimValues

Boolean

false

Indicates whether trim values are longer than the field width. Valid Options are true or false.

truncateDependingOn

Boolean

false

Truncate COBOL copybook DEPENDING ON values to the length used. Valid values are true or false.

useMissCharAsDefaultForFill

Boolean

false

By default, the flat file reader and writer use spaces for missing characters and ignore the setting of the missingValues property. When you set this property to true, the writer honors the setting of the missingValues property. Introduced in DataWeave 2.3 (2.3.0-20210823) for the September 2021 release of Mule 4.3.0-20210823.

zonedDecimalStrict

Boolean

false

Use the strict ASCII form of sign encoding for COBOL copybook zoned decimal values. Valid values are true or false.

Supported MIME Types

This format supports the following MIME types.

MIME Type

MIME Type
`*/flatfile`

*/flatfile