Amazon Kinesis Data Streams Connector 1.0 Reference - Mule 4
Amazon Kinesis Data Streams Connector doesn’t support Windows because it uses the Kinesis Producer Library (KPL), which deprecated Windows support starting in version 0.14.0. |
Configurations
Config
The configuration for the connector. You can use this configuration for both the producer and consumer.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Name |
String |
The name for this configuration. Connectors reference the configuration with this name. |
x |
|
Connection |
The connection types to provide to this configuration. |
x |
||
Aggregation Enabled |
Boolean |
When set to |
true |
|
Response Timeout |
Number |
The maximum total time that can elapse between the start of an HTTP request and when the responses are received. If this time is exceeded, the request times out. |
30 |
|
Request Timeout Unit |
Enumeration, one of:
|
A time unit that qualifies the Response Timeout value |
SECONDS |
|
Record Max Buffered Time |
Number |
Maximum amount of time that an Amazon Kinesis data record spends buffering before it is sent. Records can be sent earlier than this value, depending on other buffering limits. |
3 |
|
Record Max Buffered Time Unit |
Enumeration, one of:
|
A time unit that qualifies the Record Max Buffered Time value |
SECONDS |
|
Aggregation Max Count |
Number |
Maximum number of items to pack into an aggregated Amazon Kinesis data record. |
4294967295 |
|
Aggregation Max Size |
Number |
Maximum data size to pack into an aggregated Amazon Kinesis data record. |
51200 |
|
Aggregation Max Size Unit |
Enumeration, one of:
|
A data unit that qualifies the Aggregation Max Size value |
BYTE |
|
Collection Max Count |
Number |
Maximum number of items to pack into a batch request. |
500 |
|
Collection Max Size |
Number |
Maximum amount of data to send with a Put Record operation. |
5242880 |
|
Collection Max Size Unit |
Enumeration, one of:
|
A data unit that qualifies the Collection Max Size value |
BYTE |
|
Connection Timeout |
Number |
The amount of time to wait when initially establishing the TCP connection between the connector and Amazon Kinesis server before throwing an exception if the connection fails |
30 |
|
Connection Timeout Unit |
Enumeration, one of:
|
The time unit for the Connection Timeout value |
SECONDS |
|
Max Connections |
Number |
Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput. |
24 |
|
Min Connections |
Number |
Minimum number of connections to keep open to the backend. In most cases, there is no need to increase value. |
1 |
|
Rate Limit |
Number |
Limits the number of data records per second and number of bytes per second sent to a shard. This limit can reduce bandwidth and CPU cycle wastage from sending requests that will fail from throttling. The default value, 150%, enables a single producer instance to completely saturate the allowance for a shard. To reduce throttling errors rather than completely saturate the shard, consider reducing this setting. |
150% |
|
Record TTL |
Number |
Time limit for data records to be put. Records that are not successfully put within this limit fail. When setting this value, take into consideration the fact that data records still incur network latency after they leave the Amazon Kinesis Producer Library (KPL). |
30 |
|
Record TTL Unit |
Enumeration, one of:
|
The time unit for the Record TTL value. |
SECONDS |
|
Thread Pool Size |
Number |
Maximum number of threads with which to configure the native process thread pool. The default value, 0, enables the KPL process to choose the size of the thread pool. |
0 |
Connection Types
Kinesis Connection
Connects to AWS Kinesis through the use of a user-specified access and secret key and, optionally, a role. These credentials securely sign requests sent to AWS services.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Proxy Configuration |
Proxy connection settings for outbound connections. This setting applies to the Listener source and Checkpoint operation. It does not affect the Put Record operation. |
|||
TLS Configuration |
Protocol to use for communication. When using the HTTPS protocol, the HTTP communication is secured using TLS or SSL. If HTTPS is configured as the protocol, then, at a minimum, you must configure the keystore in the |
HTTP |
||
Access Key |
String |
Access key provided by Amazon. |
x |
|
Secret Key |
String |
Secret key provided by Amazon. |
x |
|
Region Endpoint |
String |
Topic region endpoint |
us-east-1 |
|
Role |
Role configuration |
|||
Connection Timeout |
Number |
The amount of time to wait when initially establishing the TCP connection between the connector and Amazon Kinesis server before throwing an exception if the connection fails |
30 |
|
Connection Timeout Time Unit |
Enumeration, one of:
|
The time unit for the Connection Timeout field. |
SECONDS |
|
Response Timeout |
Number |
The maximum time to wait between the time an HTTP request is sent and when a response is received |
30 |
|
Response Timeout Unit |
Enumeration, one of:
|
Time unit for the Response Timeout value. |
SECONDS |
|
Reconnection |
When the application is deployed, a connectivity test is performed on all connectors. If set to |
Checkpoint
<kinesis:checkpoint>
The operation checkpoint position of a listener into the Amazon DynamoDB table. Upon failover, the connector starts fetching records that are located after the check pointed position. The Checkpoint operation does not modify the current position of the listener.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Stream Name |
String |
The stream name. |
x |
|
Application Name |
String |
Name of the DynamoDB table that holds data about the current stream positions. |
x |
|
Shard ID |
String |
The shard identifier. |
||
Sequence Number |
String |
A sequence number at which to checkpoint in the shard. |
||
Subsequence Number |
Number |
A subsequence number at which to checkpoint within the shard. |
||
Config Ref |
ConfigurationProvider |
The name of the configuration to use to execute this component |
x |
Put Record
<kinesis:put-record>
Puts a data record into an Amazon Kinesis data stream.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Stream Name |
String |
The stream name |
x |
|
Partition Key |
String |
The partition key. If not provided, the connector generates a UUID for this value. |
||
Explicit Hash Key |
String |
Overrides the Amazon Kinesis Data Streams explicitHashKey value, which is normally computed with using MD5 function of the data record partition key. |
||
Data |
Any |
The content to put in the Amazon Kinesis data stream. This content can be up to 1 MB in size. |
#[payload] |
|
Config Ref |
ConfigurationProvider |
The name of the configuration to use to execute this component |
x |
|
Streaming Strategy |
|
Disables the repeatable stream functionality and uses non-repeatable streams to have lower performance overhead, memory use, and costs |
||
Target Variable |
String |
Name of the variable in which to store the operation’s output |
||
Target Value |
String |
Expression that evaluates the operation’s output. The expression outcome is stored in the target variable. |
#[payload] |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
Listener
<kinesis:listener>
Uses the Amazon Kinesis Client Library (KCL) to listen from a specified point in a stream.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Output Mime Type |
String |
The MIME type of the payload that this operation outputs. |
||
Output Encoding |
String |
The encoding of the payload that this operation outputs. |
||
Config Ref |
ConfigurationProvider |
The name of the configuration to use to execute this component |
x |
|
Primary Node Only |
Boolean |
Whether this listener executes on the primary node only when running in a cluster |
||
On Capacity Overload |
Enumeration, one of:
|
Strategy that Mule applies when the flow receives more messages than it can handle |
WAIT |
|
Redelivery Policy |
Policy for processing the redelivery of a message |
|||
Stream Name |
String |
The stream name. |
x |
|
Application Name |
String |
Name of the DynamoDB table that holds data about current stream positions. If the table doesn’t exist, then the connector creates a new one at the initial position. |
x |
|
Shard Configs |
Array of Shard Config |
The shard configuration. If specified, the listener listens only from selected shards. |
||
Max Batch Size |
Number |
Maximum number of data records a batch can carry. A batch is the number of records that the listener receives from the stream. |
10000 |
|
Max Leases For Worker |
Number |
Number of shards to which the listener can subscribe in parallel. |
2137483647 |
|
Shard Prioritization |
Enumeration, one of:
|
Logic used to prioritize or filter the shards before their execution.
NO_OP_SHARD_PRIORITIZATION Returns the original list of shards without any modifications. |
NO_OP_SHARD_PRIORITIZATION |
|
Max Depth |
Number |
Used by the PARENTS_FIRST_SHARD_PRIORITIZATION shard prioritization option. Any shard that is deeper than this value is excluded from processing. |
||
Read Capacity Units |
Number |
The maximum number of strongly consistent reads consumed per second before Amazon DynamoDB returns a ThrottlingException. |
10 |
|
Write Capacity Units |
Number |
The maximum number of writes consumed per second before Amazon DynamoDB returns a ThrottlingException. |
10 |
|
Failover Time |
Number |
Duration of a lease for a worker. This parameter also determines the frequency with which a worker looks for new leases to work on. The frequency is roughly twice the specified value. |
10 |
|
Failover Time Unit |
Enumeration, one of:
|
Time unit for the Failover Time field. |
SECONDS |
|
Checkpoint On Complete |
Boolean |
If |
true |
|
Absolute Position |
Enumeration, one of:
|
Whether the listener listens from the latest data record or the oldest data record |
||
Timestamp |
DateTime |
Date and time from which the listener listens |
||
Reconnection Strategy |
A retry strategy in case of connectivity errors |
Output
Type |
Array of binary messages with [Record Attributes] |
Types
Proxy Configuration
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Host |
String |
Host where the proxy requests are sent. |
x |
|
Port |
Number |
Port where the proxy requests are sent. |
x |
|
Username |
String |
The username to authenticate against the proxy. |
||
Password |
String |
The password to authenticate against the proxy. |
||
Non Proxy Hosts |
Array of String |
A list of hosts against which the proxy should not be used. |
||
Ntlm Domain |
String |
The domain to authenticate against the proxy. |
TLS
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Enabled Protocols |
String |
A comma-separated list of protocols enabled for this context. |
||
Enabled Cipher Suites |
String |
A comma-separated list of cipher suites enabled for this context. |
||
Trust Store |
||||
Key Store |
||||
Revocation Check |
Trust Store
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Path |
String |
The location (which is resolved relative to the current classpath and file system, if possible) of the truststore. |
||
Password |
String |
The password used to protect the truststore. |
||
Type |
String |
The type of store used. |
||
Algorithm |
String |
The algorithm used by the truststore. |
||
Insecure |
Boolean |
If set to |
Key Store
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Path |
String |
The location (which is resolved relative to the current classpath and file system, if possible) of the keystore. |
||
Type |
String |
The type of store used. |
||
Alias |
String |
When the keystore contains many private keys, this attribute indicates the alias of the key that should be used. If not defined, the first key in the file is used by default. |
||
Key Password |
String |
The password used to protect the private key. |
||
Password |
String |
The password used to protect the keystore. |
||
Algorithm |
String |
The algorithm used by the keystore. |
Standard Revocation Check
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Only End Entities |
Boolean |
Verifies the last element of the certificate chain only. |
||
Prefer Crls |
Boolean |
Tries CRL instead of OCSP first. |
||
No Fallback |
Boolean |
Does not use the secondary checking method. |
||
Soft Fail |
Boolean |
Avoids a certification failure when the revocation server cannot be reached or is busy. |
Custom Ocsp Responder
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Url |
String |
The URL of the OCSP responder. |
||
Cert Alias |
String |
Alias of the signing certificate for the OCSP response (must be in the truststore), if present. |
Role
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Arn |
String |
The Amazon Resource Name (ARN) of the role to assume. |
x |
|
External Id |
String |
A unique identifier that might be required when you assume a role in another account. If the administrator of the account to which the role belongs provides an external ID, then provide that value in this field. |
||
Duration |
Number |
The duration of the role session. |
3600 |
|
Duration Time Unit |
Enumeration, one of:
|
Time unit for the Duration field |
SECONDS |
|
Referred Policy Arns |
Array of String |
The Amazon Resource Names (ARNs) of the IAM-managed policies to use as managed session policies. The policies must exist in the same account as the role. |
||
Tags |
Object |
A list of session tags that you want to pass. Each session tag consists of a key name and an associated value. |
Reconnection
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Fails Deployment |
Boolean |
When the application is deployed, a connectivity test is performed on all connectors. If set to |
||
Reconnection Strategy |
The reconnection strategy to use. |
Reconnect
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Frequency |
Number |
How often in milliseconds to reconnect |
||
Count |
Number |
How many reconnection attempts to make. |
Reconnect Forever
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Frequency |
Number |
How often in milliseconds to reconnect |
Record Attributes
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Aggregated |
Boolean |
|||
Approximate Arrival Timestamp |
String |
|||
Encryption Type |
String |
|||
Partition Key |
String |
|||
Partition Key |
String |
|||
Sequence Number |
String |
|||
Shard Id |
String |
|||
Sub Sequence Number |
Number |
Redelivery Policy
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Max Redelivery Count |
Number |
The maximum number of times a message can be redelivered and processed unsuccessfully before triggering a process-failed message |
||
Message Digest Algorithm |
String |
The secure hashing algorithm to use. |
SHA-256 |
|
Message Identifier |
The strategy used to identify the messages. |
|||
Object Store |
ObjectStore |
The object store that stores the redelivery counter for each message. |
Redelivery Policy Message Identifier
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Use Secure Hash |
Boolean |
Whether to use a secure hash algorithm to identify a redelivered message. |
||
Id Expression |
String |
One or more expressions to use to determine when a message was redelivered. This property can be set only if the value of the Use Secure Hash field is |
Shard Config
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Shard Id |
String |
The shard identifier. |
x |
|
Shard Initial Position |
The shard initial position. |
x |
Shard Initial Position
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
At Sequence Number |
String |
The listener starts listening at this position in the data stream. |
||
After Sequence Number |
String |
The listener starts listening after this position in the data stream. |
||
Absolute Position |
Enumeration, one of:
|
The listener starts listening from the latest or oldest record, depending on the value. |
||
Timestamp |
DateTime |
Repeatable In Memory Stream
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Initial Buffer Size |
Number |
The amount of memory to allocate to consume the stream and provide random access to it. If the stream contains more data than can fit into this buffer, then the buffer expands according to the Buffer Size Increment value, with an upper limit of the Max In Memory Size value. |
||
Buffer Size Increment |
Number |
This is by how much the buffer size expands if it exceeds its initial size. Setting a value of zero or lower means that the buffer should not expand, meaning that a |
||
Max Buffer Size |
Number |
The maximum amount of memory to use. If more memory is used, then a |
||
Buffer Unit |
Enumeration, one of:
|
The unit in which the fields in this table are expressed |
Repeatable File Store Stream
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
In Memory Size |
Number |
Maximum memory that the stream should use to keep data in memory. If more than that is consumed, content on the disk is buffered. |
||
Buffer Unit |
Enumeration, one of:
|
The unit in which the In Memory Size value is expressed |