Nav

Apache Kafka Module Documentation Reference

Configurations


Consumer configuration

Parameters

Name Type Description Default Value Required

Name

String

The name for this configuration. Connectors reference the configuration with this name.

x

Connection

The connection types that can be provided to this configuration.

x

Expiration Policy

Configures the minimum amount of time that a dynamic configuration instance can remain idle before the runtime considers it eligible for expiration. This does not mean that the platform will expire the instance at the exact moment that it becomes eligible. The runtime will actually purge the instances when it sees it fit.

Connection Types

Kafka Basic Consumer Connection
Parameters
Name Type Description Default Value Required

Consumer Partitions

Number

The number of partitions to be used for the consumer

1

Group Id

String

A unique string that identifies the consumer group this consumer belongs to.

x

Bootstrap Servers

String

Comma-separated host-port pairs used for establishing the initial connection to the Kafka cluster. This is the same as the "bootstrap.servers" value you must provide to Kafka clients (producer/consumer). If this property is provided with producer/consumer properties files, this value is ignored and the one from the properties file is used.

x

Additional Properties

Object

Additional properties as key→value that you need for your connection.

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment will fail if the test doesn’t pass after exhausting the associated reconnection strategy

Kafka Kerberos Consumer Connection
Parameters
Name Type Description Default Value Required

Consumer Partitions

Number

The number of partitions to be used for the consumer

1

Group Id

String

A unique string that identifies the consumer group this consumer belongs to.

x

Bootstrap Servers

String

Comma-separated host-port pairs used for establishing the initial connection to the Kafka cluster. This is the same as the "bootstrap.servers" value you must provide to Kafka clients (producer/consumer). If this property is provided with producer/consumer properties files, this value is ignored and the one from the properties file is used.

x

Additional Properties

Object

Additional properties as key→value that you need for your connection.

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment will fail if the test doesn’t pass after exhausting the associated reconnection strategy

Principal

String

Kerberos principal

x

Keytab

String

Path to keytab file associated with "principal".

x

Service name

String

The Kerberos principal name that Kafka Broker runs as.

x

Additional JAAS Properties

Object

Additional parperties as key→value that you need to set on "sasl.jaas.config" and that you usually include in JAAS configuration file.

Kafka Kerbero SSL Consumer Connection
Parameters
Name Type Description Default Value Required

Consumer Partitions

Number

The number of partitions to be used for the consumer

1

Group Id

String

A unique string that identifies the consumer group this consumer belongs to.

x

Bootstrap Servers

String

Comma-separated host-port pairs used for establishing the initial connection to the Kafka cluster. This is the same as the "bootstrap.servers" value you must provide to Kafka clients (producer/consumer). If this property is provided with producer/consumer properties files, this value is ignored and the one from the properties file is used.

x

Additional Properties

Object

Additional properties as key→value that you need for your connection.

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment will fail if the test doesn’t pass after exhausting the associated reconnection strategy

Key Store Type

String

The file format of the key store file. This is optional for client.

JKS

Key Store Password

String

The store password for the key store file. This is optional for client and only needed if "keyStoreLocation" is configured.

Key Store Location

String

The location of the key store file. This is optional for client and can be used for two-way authentication for client.

Trust Store Type

String

The file format of the trust store file.

JKS

Trust Store Password

String

The password for the trust store file. If a password is not set access to the truststore is still available, but integrity checking is disabled.

x

Trust Store Location

String

The location of the trust store file.

x

Principal

String

Kerberos principal

x

Keytab

String

Path to keytab file associated with "principal".

x

Service name

String

The Kerberos principal name that Kafka Broker runs as.

x

Additional JAAS Properties

Object

Additional parperties as key→value that you need to set on "sasl.jaas.config" and that you usually include in JAAS configuration file.

Kafka SSL Consumer Connection
Parameters
Name Type Description Default Value Required

Consumer Partitions

Number

The number of partitions to be used for the consumer

1

Group Id

String

A unique string that identifies the consumer group this consumer belongs to.

x

Bootstrap Servers

String

Comma-separated host-port pairs used for establishing the initial connection to the Kafka cluster. This is the same as the "bootstrap.servers" value you must provide to Kafka clients (producer/consumer). If this property is provided with producer/consumer properties files, this value is ignored and the one from the properties file is used.

x

Additional Properties

Object

Additional properties as key→value that you need for your connection.

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment will fail if the test doesn’t pass after exhausting the associated reconnection strategy

Key Store Type

String

The file format of the key store file. This is optional for client.

JKS

Key Store Password

String

The store password for the key store file. This is optional for client and only needed if "keyStoreLocation" is configured.

Key Store Location

String

The location of the key store file. This is optional for client and can be used for two-way authentication for client.

Trust Store Type

String

The file format of the trust store file.

JKS

Trust Store Password

String

The password for the trust store file. If a password is not set access to the truststore is still available, but integrity checking is disabled.

x

Trust Store Location

String

The location of the trust store file.

x

Associated Sources


Producer configuration

Parameters

Name Type Description Default Value Required

Name

String

The name for this configuration. Connectors reference the configuration with this name.

x

Connection

The connection types that can be provided to this configuration.

x

Expiration Policy

Configures the minimum amount of time that a dynamic configuration instance can remain idle before the runtime considers it eligible for expiration. This does not mean that the platform will expire the instance at the exact moment that it becomes eligible. The runtime will actually purge the instances when it sees it fit.

Connection Types

Kafka Basic Producer Connection
Parameters
Name Type Description Default Value Required

Bootstrap Servers

String

Comma-separated host-port pairs used for establishing the initial connection to the Kafka cluster. This is the same as the "bootstrap.servers" value you must provide to Kafka clients (producer/consumer). If this property is provided with producer/consumer properties files, this value is ignored and the one from the properties file is used.

x

Additional Properties

Object

Additional properties as key→value that you need for your connection.

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment will fail if the test doesn’t pass after exhausting the associated reconnection strategy

Kafka Kerberos Producer Connection
Parameters
Name Type Description Default Value Required

Bootstrap Servers

String

Comma-separated host-port pairs used for establishing the initial connection to the Kafka cluster. This is the same as the "bootstrap.servers" value you must provide to Kafka clients (producer/consumer). If this property is provided with producer/consumer properties files, this value is ignored and the one from the properties file is used.

x

Additional Properties

Object

Additional properties as key→value that you need for your connection.

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment will fail if the test doesn’t pass after exhausting the associated reconnection strategy

Principal

String

Kerberos principal

x

Keytab

String

Path to keytab file associated with "principal".

x

Service name

String

The Kerberos principal name that Kafka Broker runs as.

x

Additional JAAS Properties

Object

Additional parperties as key→value that you need to set on "sasl.jaas.config" and that you usually include in JAAS configuration file.

Kafka Kerberos SSL Producer Connection
Parameters
Name Type Description Default Value Required

Bootstrap Servers

String

Comma-separated host-port pairs used for establishing the initial connection to the Kafka cluster. This is the same as the "bootstrap.servers" value you must provide to Kafka clients (producer/consumer). If this property is provided with producer/consumer properties files, this value is ignored and the one from the properties file is used.

x

Additional Properties

Object

Additional properties as key→value that you need for your connection.

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment will fail if the test doesn’t pass after exhausting the associated reconnection strategy

Key Store Type

String

The file format of the key store file. This is optional for client.

JKS

Key Store Password

String

The store password for the key store file. This is optional for client and only needed if "keyStoreLocation" is configured.

Key Store Location

String

The location of the key store file. This is optional for client and can be used for two-way authentication for client.

Trust Store Type

String

The file format of the trust store file.

JKS

Trust Store Password

String

The password for the trust store file. If a password is not set access to the truststore is still available, but integrity checking is disabled.

x

Trust Store Location

String

The location of the trust store file.

x

Principal

String

Kerberos principal

x

Keytab

String

Path to keytab file associated with "principal".

x

Service name

String

The Kerberos principal name that Kafka Broker runs as.

x

Additional JAAS Properties

Object

Additional parperties as key→value that you need to set on "sasl.jaas.config" and that you usually include in JAAS configuration file.

Kafka SSL Producer Connection
Parameters
Name Type Description Default Value Required

Bootstrap Servers

String

Comma-separated host-port pairs used for establishing the initial connection to the Kafka cluster. This is the same as the "bootstrap.servers" value you must provide to Kafka clients (producer/consumer). If this property is provided with producer/consumer properties files, this value is ignored and the one from the properties file is used.

x

Additional Properties

Object

Additional properties as key→value that you need for your connection.

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment will fail if the test doesn’t pass after exhausting the associated reconnection strategy

Key Store Type

String

The file format of the key store file. This is optional for client.

JKS

Key Store Password

String

The store password for the key store file. This is optional for client and only needed if "keyStoreLocation" is configured.

Key Store Location

String

The location of the key store file. This is optional for client and can be used for two-way authentication for client.

Trust Store Type

String

The file format of the trust store file.

JKS

Trust Store Password

String

The password for the trust store file. If a password is not set access to the truststore is still available, but integrity checking is disabled.

x

Trust Store Location

String

The location of the trust store file.

x

Associated Operations

Producer Operation

<kafka:producer>

Operation that facilitates Kafka messages sending into the given topic.

Parameters

Name Type Description Default Value Required

Configuration

String

The name of the configuration to use.

x

Topic

String

Topic to send the message to

x

Key

String

Key belonging to the message that is going to be sent

x

Message

String

Message to be sent

#[payload]

Reconnection Strategy

A retry strategy in case of connectivity errors

For Configurations

Throws

  • KAFKA:CONNECTIVITY

  • KAFKA:CONNECTIVITY

  • KAFKA:UNKNOWN

  • KAFKA:RETRY_EXHAUSTED

Sources

Consumer Operationn

<kafka:consumer>

Operation that facilitates Kafka message consumption from a given topic.

Parameters

Name Type Description Default Value Required

Configuration

String

The name of the configuration to use.

x

Topic

String

Name of Kafka topic to consume messages from.

x

Partition Offsets

Array of Offset

Map representing partitions offsets configuration. It has to be in the following format [{"partition_number": "partition_offset"}] (e.g: [{"0":"2"}, [{"1":"5"}]]).

Redelivery Policy

Defines a policy for processing the redelivery of the same message

Streaming Strategy

Configure if repeatable streams should be used and their behavior

Reconnection Strategy

A retry strategy in case of connectivity errors

Output

Type

String

Attributes Type

Any

For Configurations

Types

Reconnection

Field Type Description Default Value Required

Fails Deployment

Boolean

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment will fail if the test doesn’t pass after exhausting the associated reconnection strategy

Reconnection Strategy

The reconnection strategy to use

Reconnect

Field Type Description Default Value Required

Frequency

Number

How often (in ms) to reconnect

Count

Number

How many reconnection attempts to make

Reconnect Forever

Field Type Description Default Value Required

Frequency

Number

How often (in ms) to reconnect

Expiration Policy

Field Type Description Default Value Required

Max Idle Time

Number

A scalar time value for the maximum amount of time a dynamic configuration instance should be allowed to be idle before it’s considered eligible for expiration

Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

A time unit that qualifies the maxIdleTime attribute

Offset

Field Type Description Default Value Required

Partition Number

String

Partition Offset

String

Redelivery Policy

Field Type Description Default Value Required

Max Redelivery Count

Number

The maximum number of times a message can be redelivered and processed unsuccessfully before triggering process-failed-message

Use Secure Hash

Boolean

Whether to use a secure hash algorithm to identify a redelivered message

Message Digest Algorithm

String

The secure hashing algorithm to use. If not set, the default is SHA-256.

Id Expression

String

Defines one or more expressions to use to determine when a message has been redelivered. This property may only be set if useSecureHash is false.

Object Store

The object store where the redelivery counter for each message is going to be stored.

Repeatable In Memory Stream

Field Type Description Default Value Required

Initial Buffer Size

Number

This is the amount of memory that will be allocated in order to consume the stream and provide random access to it. If the stream contains more data than can be fit into this buffer, then it will be expanded by according to the bufferSizeIncrement attribute, with an upper limit of maxInMemorySize.

Buffer Size Increment

Number

This is by how much will be buffer size by expanded if it exceeds its initial size. Setting a value of zero or lower will mean that the buffer should not expand, meaning that a STREAM_MAXIMUM_SIZE_EXCEEDED error will be raised when the buffer gets full.

Max Buffer Size

Number

This is the maximum amount of memory that will be used. If more than that is used then a STREAM_MAXIMUM_SIZE_EXCEEDED error will be raised. A value lower or equal to zero means no limit.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit in which all these attributes are expressed

Repeatable File Store Stream

Field Type Description Default Value Required

Max In Memory Size

Number

Defines the maximum memory that the stream should use to keep data in memory. If more than that is consumed then it will start to buffer the content on disk.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit in which maxInMemorySize is expressed