Avro Format

MIME type: application/avro

ID: avro

Avro is a binary data format that uses a schema to structure its data. DataWeave relies on the schema to parse the data. Avro data structures are mapped to DataWeave data structures.

Java Value Mapping

The following table shows how Avro types map to DataWeave types.

Avro Type DataWeave Type

Avro Type	DataWeave Type
`long`	`Number`
`int`	`Number`
`double`	`Number`
`boolean`	`Boolean`
`string`	`String`
`fixed`	`String`
`bytes`	`Binary`
`enum`	`String`
`map`	`Object`
`array`	`Array`
`null`	`Null`

long

Number

int

Number

double

Number

boolean

Boolean

string

String

fixed

String

bytes

Binary

enum

String

map

Object

array

Array

null

Null

Example: Use an Avro Schema

The following example shows how to specify a schema that the writer uses to output an Avro data structure.

Input

An Avro schema looks something like this.

schema.json:

{
    "type": "record",
    "name": "userInfo",
    "namespace": "my.example",
    "fields": [
        {
            "name": "username",
            "type": "string",
            "default": "NONE"
        },
        {
            "name": "age",
            "type": "int",
            "default": -1
        },
        {
            "name": "phone",
            "type": "string",
            "default": "NONE"
        },
        {
            "name": "housenum",
            "type": "string",
            "default": "NONE"
        }
    ]
}

Source

The schemaUrl property in the header of this DataWeave script passes a schema (schema.json) to the DataWeave writer. The writer uses the schema to structure content from the body of the script and output the results in Avro format.

%dw 2.0
output application/avro schemaUrl="classpath://schema.json"
---
[{
    username: "Mariano",
    age: 35,
    phone: "213",
    housenum: "123"
},
{
    username: "Leandro",
    age: 29,
    phone: "213",
    housenum: "123"
},
{
    username: "Christian",
    age: 25,
    phone: "213",
    housenum: "123"
}]

Configuration Properties

DataWeave supports the following configuration properties for this format.

Reader Properties

This format accepts properties that provide instructions for reading input data.

Parameter Type Default Description

Parameter	Type	Default	Description
`schemaUrl` (Required)	`String`	`''`	The URL for the Avro schema. Valid URL schemes are `classpath://`, `file://`, or `http://`. For the reader, this property is optional but defaults to the schema embedded in the input Avro file. The reader requires an embedded schema. For the writer, DataWeave requires a schema value.

schemaUrl (Required)

String

''

The URL for the Avro schema. Valid URL schemes are classpath://, file://, or http://. For the reader, this property is optional but defaults to the schema embedded in the input Avro file. The reader requires an embedded schema. For the writer, DataWeave requires a schema value.

Writer Properties

This format accepts properties that provide instructions for writing output data.

Parameter Type Default Description

Parameter	Type	Default	Description
`bufferSize`	`Number`	`8192`	Size of the buffer writer, in bytes. The value must be greater than `8`.
`deferred`	`Boolean`	`false`	Generates the output as a data stream when set to `true`, and defers the script’s execution until the generated content is consumed. Valid values are `true` or `false`.
`schemaUrl` (Required)	`String`	`''`	The URL for the Avro schema. Valid URL schemes are `classpath://`, `file://`, or `http://`. For the reader, this property is optional but defaults to the schema embedded in the input Avro file. The reader requires an embedded schema. For the writer, DataWeave requires a schema value.

bufferSize

Number

8192

Size of the buffer writer, in bytes. The value must be greater than 8.

deferred

Boolean

false

Generates the output as a data stream when set to true, and defers the script’s execution until the generated content is consumed.

Valid values are true or false.

schemaUrl (Required)

String

''

Supported MIME Types

This format supports the following MIME types.

MIME Type

MIME Type
`application/avro`

application/avro