Avro Format

MIME type: application/avro

ID: avro

Avro is a binary data format that uses a schema to structure its data. DataWeave relies on the schema to parse the data. Avro data structures are mapped to DataWeave data structures.

Java Value Mapping

The following table shows how Avro types map to DataWeave types.

Avro Type DataWeave Type

Avro Type	DataWeave Type
`long`	`Number`
`int`	`Number`
`double`	`Number`
`boolean`	`Boolean`
`string`	`String`
`fixed`	`String`
`bytes`	`Binary`
`enum`	`String`
`map`	`Object`
`array`	`Array`
`null`	`Null`

long

Number

int

Number

double

Number

boolean

Boolean

string

String

fixed

String

bytes

Binary

enum

String

map

Object

array

Array

null

Null

Example: Use an Avro Schema

The following example shows how to specify a schema that the writer uses to output an Avro data structure.

Input

An Avro schema looks something like this.

schema.json:

{
    "type": "record",
    "name": "userInfo",
    "namespace": "my.example",
    "fields": [
        {
            "name": "username",
            "type": "string",
            "default": "NONE"
        },
        {
            "name": "age",
            "type": "int",
            "default": -1
        },
        {
            "name": "phone",
            "type": "string",
            "default": "NONE"
        },
        {
            "name": "housenum",
            "type": "string",
            "default": "NONE"
        }
    ]
}

Source

The schemaUrl property in the header of this DataWeave script passes a schema (schema.json) to the DataWeave writer. The writer uses the schema to structure content from the body of the script and output the results in Avro format.

%dw 2.0
output application/avro schemaUrl="classpath://schema.json"
---
[{
    username: "Mariano",
    age: 35,
    phone: "213",
    housenum: "123"
},
{
    username: "Leandro",
    age: 29,
    phone: "213",
    housenum: "123"
},
{
    username: "Christian",
    age: 25,
    phone: "213",
    housenum: "123"
}]

Configuration Properties

DataWeave supports the following configuration properties for Avro.

Reader Properties (for Avro)

The Avro format accepts properties that provide instructions for reading input data.

Parameter Type Default Description

Parameter	Type	Default	Description
`schemaUrl` (Required)	`String`	`null`	The URL for the Avro schema. Valid values are `classpath://`, `file://`, or `http://`.

schemaUrl (Required)

String

null

The URL for the Avro schema. Valid values are classpath://, file://, or http://.

Writer Properties (for Avro)

The Avro format accepts properties that provide instructions for writing output data.

Parameter Type Default Description

Parameter	Type	Default	Description
`bufferSize`	`Number`	`8192`	Size of the buffer writer.
`deferred`	`Boolean`	`false`	When set to `true`, DataWeave generates the output as a data stream, and the script’s execution is deferred until it is consumed. Valid values are `true` or `false`.
`schemaUrl` (Required)	`String`	`null`	The URL for Avro schema. Valid values are `classpath://`, `file://`, or `http://`.

bufferSize

Number

8192

Size of the buffer writer.

deferred

Boolean

false

When set to true, DataWeave generates the output as a data stream, and the script’s execution is deferred until it is consumed. Valid values are true or false.

schemaUrl (Required)

String

null

The URL for Avro schema. Valid values are classpath://, file://, or http://.

Supported MIME Types

The Avro format supports the following MIME types.

MIME Type

MIME Type
`application/avro`

application/avro

Supported Top-Level Schema Types

DataWeave supports the following Avro types as the top-level element in the schema.

Avro Types

Avro Types
`record`

record