Contact Us 1-800-596-4880
  1. Homepage
  2. Amazon Kinesis Connector (1.0)

  3. Amazon Kinesis Connector Reference

All Connectors

Amazon Kinesis Data Streams Connector 1.0 Reference - Mule 4

Amazon Kinesis Data Streams Connector doesn’t support Windows because it uses the Kinesis Producer Library (KPL), which deprecated Windows support starting in version 0.14.0.

Configurations

Config

The configuration for the connector. You can use this configuration for both the producer and consumer.

Parameters

Name Type Description Default Value Required

Name

String

The name for this configuration. Connectors reference the configuration with this name.

x

Connection

Kinesis Connection

The connection types to provide to this configuration.

x

Aggregation Enabled

Boolean

When set to true, this option packs multiple user data records into a single Amazon Kinesis data record.

true

Response Timeout

Number

The maximum total time that can elapse between the start of an HTTP request and when the responses are received. If this time is exceeded, the request times out.

30

Request Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

A time unit that qualifies the Response Timeout value

SECONDS

Record Max Buffered Time

Number

Maximum amount of time that an Amazon Kinesis data record spends buffering before it is sent. Records can be sent earlier than this value, depending on other buffering limits.

3

Record Max Buffered Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

A time unit that qualifies the Record Max Buffered Time value

SECONDS

Aggregation Max Count

Number

Maximum number of items to pack into an aggregated Amazon Kinesis data record.

4294967295

Aggregation Max Size

Number

Maximum data size to pack into an aggregated Amazon Kinesis data record.

51200

Aggregation Max Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

A data unit that qualifies the Aggregation Max Size value

BYTE

Collection Max Count

Number

Maximum number of items to pack into a batch request.

500

Collection Max Size

Number

Maximum amount of data to send with a Put Record operation.

5242880

Collection Max Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

A data unit that qualifies the Collection Max Size value

BYTE

Connection Timeout

Number

The amount of time to wait when initially establishing the TCP connection between the connector and Amazon Kinesis server before throwing an exception if the connection fails

30

Connection Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

The time unit for the Connection Timeout value

SECONDS

Max Connections

Number

Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput.

24

Min Connections

Number

Minimum number of connections to keep open to the backend. In most cases, there is no need to increase value.

1

Rate Limit

Number

Limits the number of data records per second and number of bytes per second sent to a shard. This limit can reduce bandwidth and CPU cycle wastage from sending requests that will fail from throttling. The default value, 150%, enables a single producer instance to completely saturate the allowance for a shard. To reduce throttling errors rather than completely saturate the shard, consider reducing this setting.

150%

Record TTL

Number

Time limit for data records to be put. Records that are not successfully put within this limit fail. When setting this value, take into consideration the fact that data records still incur network latency after they leave the Amazon Kinesis Producer Library (KPL).

30

Record TTL Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

The time unit for the Record TTL value.

SECONDS

Thread Pool Size

Number

Maximum number of threads with which to configure the native process thread pool. The default value, 0, enables the KPL process to choose the size of the thread pool.

0

Connection Types

Kinesis Connection

Connects to AWS Kinesis through the use of a user-specified access and secret key and, optionally, a role. These credentials securely sign requests sent to AWS services.

Parameters
Name Type Description Default Value Required

Proxy Configuration

Proxy Configuration

Proxy connection settings for outbound connections. This setting applies to the Listener source and Checkpoint operation. It does not affect the Put Record operation.

TLS Configuration

TLS

Protocol to use for communication. When using the HTTPS protocol, the HTTP communication is secured using TLS or SSL. If HTTPS is configured as the protocol, then, at a minimum, you must configure the keystore in the tls:context child element of the listener-config.

This setting applies to the Listener source and Checkpoint operation. It does not affect the Put Record operation.

HTTP

Access Key

String

Access key provided by Amazon.

x

Secret Key

String

Secret key provided by Amazon.

x

Region Endpoint

String

Topic region endpoint

us-east-1

Role

Role

Role configuration

Connection Timeout

Number

The amount of time to wait when initially establishing the TCP connection between the connector and Amazon Kinesis server before throwing an exception if the connection fails

30

Connection Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

The time unit for the Connection Timeout field.

SECONDS

Response Timeout

Number

The maximum time to wait between the time an HTTP request is sent and when a response is received

30

Response Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Response Timeout value.

SECONDS

Reconnection

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy.

Supported Operations

Associated Sources

Operations

Checkpoint

<kinesis:checkpoint>

The operation checkpoint position of a listener into the Amazon DynamoDB table. Upon failover, the connector starts fetching records that are located after the check pointed position. The Checkpoint operation does not modify the current position of the listener.

Parameters

Name Type Description Default Value Required

Configuration

String

The name of the configuration to use.

x

Stream Name

String

The stream name.

x

Application Name

String

Name of the DynamoDB table that holds data about the current stream positions.

x

Shard ID

String

The shard identifier.

Sequence Number

String

A sequence number at which to checkpoint in the shard.

Subsequence Number

Number

A subsequence number at which to checkpoint within the shard.

Config Ref

ConfigurationProvider

The name of the configuration to use to execute this component

x

For Configurations

Throws

  • KINESIS:ILLEGAL_STATE

  • KINESIS:INVALID_ARGUMENT

  • KINESIS:PROVISIONED_THROUGHPUT_EXCEEDED

  • KINESIS:TIMEOUT

  • KINESIS:UNAUTHORIZED

Put Record

<kinesis:put-record>

Puts a data record into an Amazon Kinesis data stream.

Parameters

Name Type Description Default Value Required

Configuration

String

The name of the configuration to use.

x

Stream Name

String

The stream name

x

Partition Key

String

The partition key. If not provided, the connector generates a UUID for this value.

Explicit Hash Key

String

Overrides the Amazon Kinesis Data Streams explicitHashKey value, which is normally computed with using MD5 function of the data record partition key.

Data

Any

The content to put in the Amazon Kinesis data stream. This content can be up to 1 MB in size.

#[payload]

Config Ref

ConfigurationProvider

The name of the configuration to use to execute this component

x

Streaming Strategy

Disables the repeatable stream functionality and uses non-repeatable streams to have lower performance overhead, memory use, and costs

Target Variable

String

Name of the variable in which to store the operation’s output

Target Value

String

Expression that evaluates the operation’s output. The expression outcome is stored in the target variable.

#[payload]

Reconnection Strategy

A retry strategy in case of connectivity errors

Output

Type

Any

For Configurations

Throws

  • KINESIS:CONNECTIVITY

  • KINESIS:INVALID_ARGUMENT

  • KINESIS:PROCESSING

  • KINESIS:RETRY_EXHAUSTED

  • KINESIS:TIMEOUT

  • KINESIS:UNAUTHORIZED

Sources

Listener

<kinesis:listener>

Uses the Amazon Kinesis Client Library (KCL) to listen from a specified point in a stream.

Due to varying batch sizes, checkpoints can occur in the incorrect sequence, potentially leading to transaction loss in certain cases. Set maxConcurrency=1 to ensure transactions are processed sequentially.

Parameters

Name Type Description Default Value Required

Configuration

String

The name of the configuration to use.

x

Output Mime Type

String

The MIME type of the payload that this operation outputs.

Output Encoding

String

The encoding of the payload that this operation outputs.

Config Ref

ConfigurationProvider

The name of the configuration to use to execute this component

x

Primary Node Only

Boolean

Whether this listener executes on the primary node only when running in a cluster

On Capacity Overload

Enumeration, one of:

  • WAIT

  • DROP

Strategy that Mule applies when the flow receives more messages than it can handle

WAIT

Redelivery Policy

Redelivery Policy

Policy for processing the redelivery of a message

Stream Name

String

The stream name.

x

Application Name

String

Name of the DynamoDB table that holds data about current stream positions. If the table doesn’t exist, then the connector creates a new one at the initial position.

x

Shard Configs

Array of Shard Config

The shard configuration. If specified, the listener listens only from selected shards.

Max Batch Size

Number

Maximum number of data records a batch can carry. A batch is the number of records that the listener receives from the stream.

10000

Max Leases For Worker

Number

Number of shards to which the listener can subscribe in parallel.

2137483647

Shard Prioritization

Enumeration, one of:

  • PARENTS_FIRST_SHARD_PRIORITIZATION

  • NO_OP_SHARD_PRIORITIZATION

Logic used to prioritize or filter the shards before their execution.

PARENTS_FIRST_SHARD_PRIORITIZATION Prioritizes parent shards first and limits the number of shards that are available for initialization based on their depth (Max Depth field).

NO_OP_SHARD_PRIORITIZATION Returns the original list of shards without any modifications.

NO_OP_SHARD_PRIORITIZATION

Max Depth

Number

Used by the PARENTS_FIRST_SHARD_PRIORITIZATION shard prioritization option. Any shard that is deeper than this value is excluded from processing.

Read Capacity Units

Number

The maximum number of strongly consistent reads consumed per second before Amazon DynamoDB returns a ThrottlingException.

10

Write Capacity Units

Number

The maximum number of writes consumed per second before Amazon DynamoDB returns a ThrottlingException.

10

Failover Time

Number

Duration of a lease for a worker. This parameter also determines the frequency with which a worker looks for new leases to work on. The frequency is roughly twice the specified value.

10

Failover Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Failover Time field.

SECONDS

Checkpoint On Complete

Boolean

If true, the listener checkpoints the current reading position to Amazon DynamoDB after data records are processed by the owning flow, without taking into account whether the processing finished successfully.

true

Absolute Position

Enumeration, one of:

  • LATEST

  • FIRST

Whether the listener listens from the latest data record or the oldest data record

Timestamp

DateTime

Date and time from which the listener listens

Reconnection Strategy

A retry strategy in case of connectivity errors

Output

Type

Array of binary messages with [Record Attributes]

For Configurations

Types

Proxy Configuration

Field Type Description Default Value Required

Host

String

Host where the proxy requests are sent.

x

Port

Number

Port where the proxy requests are sent.

x

Username

String

The username to authenticate against the proxy.

Password

String

The password to authenticate against the proxy.

Non Proxy Hosts

Array of String

A list of hosts against which the proxy should not be used.

Ntlm Domain

String

The domain to authenticate against the proxy.

TLS

Field Type Description Default Value Required

Enabled Protocols

String

A comma-separated list of protocols enabled for this context.

Enabled Cipher Suites

String

A comma-separated list of cipher suites enabled for this context.

Trust Store

Trust Store

Key Store

Key Store

Revocation Check

Trust Store

Field Type Description Default Value Required

Path

String

The location (which is resolved relative to the current classpath and file system, if possible) of the truststore.

Password

String

The password used to protect the truststore.

Type

String

The type of store used.

Algorithm

String

The algorithm used by the truststore.

Insecure

Boolean

If set to true, no certificate validations are performed, which makes connections vulnerable to attacks. Use at your own risk.

Key Store

Field Type Description Default Value Required

Path

String

The location (which is resolved relative to the current classpath and file system, if possible) of the keystore.

Type

String

The type of store used.

Alias

String

When the keystore contains many private keys, this attribute indicates the alias of the key that should be used. If not defined, the first key in the file is used by default.

Key Password

String

The password used to protect the private key.

Password

String

The password used to protect the keystore.

Algorithm

String

The algorithm used by the keystore.

Standard Revocation Check

Field Type Description Default Value Required

Only End Entities

Boolean

Verifies the last element of the certificate chain only.

Prefer Crls

Boolean

Tries CRL instead of OCSP first.

No Fallback

Boolean

Does not use the secondary checking method.

Soft Fail

Boolean

Avoids a certification failure when the revocation server cannot be reached or is busy.

Custom Ocsp Responder

Field Type Description Default Value Required

Url

String

The URL of the OCSP responder.

Cert Alias

String

Alias of the signing certificate for the OCSP response (must be in the truststore), if present.

Crl File

Field Type Description Default Value Required

Path

String

The path to the CRL file.

Role

Field Type Description Default Value Required

Arn

String

The Amazon Resource Name (ARN) of the role to assume.

x

External Id

String

A unique identifier that might be required when you assume a role in another account. If the administrator of the account to which the role belongs provides an external ID, then provide that value in this field.

Duration

Number

The duration of the role session.

3600

Duration Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Duration field

SECONDS

Referred Policy Arns

Array of String

The Amazon Resource Names (ARNs) of the IAM-managed policies to use as managed session policies. The policies must exist in the same account as the role.

Tags

Object

A list of session tags that you want to pass. Each session tag consists of a key name and an associated value.

Reconnection

Field Type Description Default Value Required

Fails Deployment

Boolean

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy.

Reconnection Strategy

The reconnection strategy to use.

Reconnect

Field Type Description Default Value Required

Frequency

Number

How often in milliseconds to reconnect

Count

Number

How many reconnection attempts to make.

Reconnect Forever

Field Type Description Default Value Required

Frequency

Number

How often in milliseconds to reconnect

Record Attributes

Field Type Description Default Value Required

Aggregated

Boolean

Approximate Arrival Timestamp

String

Encryption Type

String

Partition Key

String

Partition Key

String

Sequence Number

String

Shard Id

String

Sub Sequence Number

Number

Redelivery Policy

Field Type Description Default Value Required

Max Redelivery Count

Number

The maximum number of times a message can be redelivered and processed unsuccessfully before triggering a process-failed message

Message Digest Algorithm

String

The secure hashing algorithm to use.

SHA-256

Message Identifier

Redelivery Policy Message Identifier

The strategy used to identify the messages.

Object Store

ObjectStore

The object store that stores the redelivery counter for each message.

Redelivery Policy Message Identifier

Field Type Description Default Value Required

Use Secure Hash

Boolean

Whether to use a secure hash algorithm to identify a redelivered message.

Id Expression

String

One or more expressions to use to determine when a message was redelivered. This property can be set only if the value of the Use Secure Hash field is false.

Shard Config

Field Type Description Default Value Required

Shard Id

String

The shard identifier.

x

Shard Initial Position

Shard Initial Position

The shard initial position.

x

Shard Initial Position

Field Type Description Default Value Required

At Sequence Number

String

The listener starts listening at this position in the data stream.

After Sequence Number

String

The listener starts listening after this position in the data stream.

Absolute Position

Enumeration, one of:

  • LATEST

  • FIRST

The listener starts listening from the latest or oldest record, depending on the value.

Timestamp

DateTime

Repeatable In Memory Stream

Field Type Description Default Value Required

Initial Buffer Size

Number

The amount of memory to allocate to consume the stream and provide random access to it. If the stream contains more data than can fit into this buffer, then the buffer expands according to the Buffer Size Increment value, with an upper limit of the Max In Memory Size value.

Buffer Size Increment

Number

This is by how much the buffer size expands if it exceeds its initial size. Setting a value of zero or lower means that the buffer should not expand, meaning that a STREAM_MAXIMUM_SIZE_EXCEEDED error is raised when the buffer is full.

Max Buffer Size

Number

The maximum amount of memory to use. If more memory is used, then a STREAM_MAXIMUM_SIZE_EXCEEDED error is raised. A value less than or equal to zero means no limit.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit in which the fields in this table are expressed

Repeatable File Store Stream

Field Type Description Default Value Required

In Memory Size

Number

Maximum memory that the stream should use to keep data in memory. If more than that is consumed, content on the disk is buffered.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit in which the In Memory Size value is expressed

See Also

Back to top
View on GitHub