Apache Kafka Connector 4.10 - Mule 4
Anypoint Connector for Apache Kafka (Apache Kafka Connector) enables you to interact with the Apache Kafka messaging system and achieve seamless integration between your Mule app and a Kafka cluster, using Mule runtime engine (Mule).
For compatibility information, see the Apache Kafka Connector Release Notes.
Before You Begin
Before creating an app, you must:
-
Have access to the Apache Kafka target resource and Anypoint Platform
-
Understand how to create a Mule app using Anypoint Studio
-
Have access to Apache Kafka to get values for the fields that appear in Studio
Common Use Cases for the Connector
Common use cases for Apache Kafka Connector include:
-
Log Aggregation
Leverage the low-latency processing from Apache Kafka to collect logs from multiple services and make them available in a standard format to multiple consumers. For example, you can publish a message to Apache Kafka and then retrieve it.
-
Analysis and Metrics
Optimize your advertising budget by integrating Apache Kafka and your big data analytics solution to analyze end user activity, such as page views, clicks, shares, and so on, to serve relevant ads. For example, a department store can use a website activity tracker to improve the online shopping experience. The gathered data is sent to multiple departments for various computations. Each department reviews the received information to stay informed about what the customer is looking for and then provides recommendations accordingly.
-
Notifications and Alerts
Notify customers about various financial events, such as the recent amount of a transaction or more complex events like future investment suggestions based on integrations with credit agencies, location, and so on.
-
Time-Sensitive Applications
Build integration applications that are time-sensitive. For example, you can build an application that ensures that patients in a hospital receive the care they need in a timely manner with a hospital server that processes emergency requests for patient admissions and orders the requests based on priority determined by specified criteria. In this scenario, an application processes Apache Kafka messages in the order that they are received, relying on the order and idempotency of the messages sent through the queue. For example, a newsroom can use the Apache Kafka system to deliver the latest news. Retrieve the latest news by building an integration application by reading from the end of the Apache Kafka queue first.
For examples of these use cases, refer to Apache Kafka Connector Examples.
Publish-to Use Cases
With Kafka Connector, applications can publish binary messages to the Kafka server. Some examples of use cases include:
-
Publishing binary data to the Kafka server and the underlying topic.
-
Setting the batch size of the data to publish to Kafka when working with large volumes of data.
-
To optimize for performance, companies can batch messages to send. Kafka stores the message until the batch size is reached and completes the send at that time.
-
-
Configuring the message size to send to focus on optimal performance.
-
Kafka is not intended to process and handle large messages. Messages are typically in the 1 MB range and users set the size to avoid performance degradation. In the case of an error, the message is rejected.
-
-
Blocking further commits on a configurable level of acknowledgment for the messages published to the Kafka server to ensure successful data transmission.
-
Acknowledgments enable users to ensure that the message is sent prior to publishing more messages. This is specifically useful in time-ordered or event-ordered messages, such as logs or patient drug administration orders. Ensuring that messages are sent in order helps the downstream application process the messages in the right order.
-
-
Configuring how data is serialized during a publish.
-
Securing messages sent during Mutual TLS.
The following lists the acceptance criteria for publishing:
-
Publish a message to a Kafka topic.
-
Configure the batch size and size of the message.
-
Obtain an acknowledgment once publishing is complete at the configured level.
-
Send the message securely via Mutual TLS.
-
Configure the serialization type.
Subscribe-from Use Cases
With Kafka Connector, applications can obtain messages from Kafka for further consumption downstream. Some examples of use cases include:
Reading from Kafka:
-
Subscribing to the Kafka server and the underlying group to read and consume messages for downstream processing.
-
Subscribing to a specific Kafka partition to process messages.
-
Users with a high availability microservice don’t want to rely on Kafka’s rebalancing in the event of a failure.
-
If users have a local state associated with their partition, they want to process only that partition.
-
-
Configuring the offset value to a set value or to the beginning or end of the topic.
-
Users have a local store and want to use that information to start processing their records.
-
Users have time-sensitive records and want to skip to the end if the application must be up-to-date or skip to the beginning if it is a new application processing the records.
-
-
Reading the offset value from an alternate storage system on restart.
-
Users have a local store with the offset information and want to read the value from their local storage.
-
-
Setting the interval for which to block the receipt of messages for varying downstream processing intervals.
Processing the messages:
-
Ensuring there is no data loss when the downstream application crashes by manually configuring commits to have more control.
-
Storing a commit value in a local store or in the Kafka storage and blocking until the store is successful.
-
Storing an offset asynchronously in a local store or in the Kafka storage.
-
Using Kafka’s in-built capability of load-balancing with multiple applications reading from the same group ID.
-
Configuring the number of records to process in a single call.
The following lists the acceptance criteria for subscribing:
-
Read a message from Kafka from a specific partition or topic.
-
Read a message from Kafka after obtaining the offset from a local store.
-
Read a message from the beginning of the queue, the end of the queue, and a pre-specified offset.
-
Ensure a high level of performance when there is one consumer per thread. Multiple consumers exist in a single group.
Audience
-
Starting user
To create a Mule app, see Apache Kafka Studio Configuration.
-
Power user
Read XML and Maven Support and Examples.
Next
After you complete the prerequisites, you are ready to create an app with Anypoint Studio.