Reliability Patterns
A reliability pattern is a design that results in reliable messaging for an application, even if the application receives messages from a nontransactional connector. A reliability pattern couples a reliable acquisition flow with an application logic flow, as shown in the following diagram:
The reliable acquisition flow (the left side of the diagram) delivers a message reliably from a Mule event source that does not implement transactions to an outbound operation of a connector that implements transactions. The operation can be of any type of the transactional endpoints, such as VM or JMS. If the reliable acquisition flow cannot deliver the message, it ensures that the message isn’t lost.
-
For socket-based connections like HTTP, this means returning an "unsuccessful request" response to the client so that the client can retry the request.
-
For resource-based connections like File or FTP, it means not deleting the file, so that it can be reprocessed.
The application logic flow (the right side of the diagram) delivers the message from the event source that uses a transactional connector to the business logic for the application.
Reliable Messaging
High-reliability applications must have zero tolerance for message loss. This means that both the underlying Mule and its individual connections need to be reliable.
If your application uses a transactional connection such as JMS, VM, or DB, reliable messaging is ensured by the built-in support for transactions in the connector. This means, for example, that you can configure a transaction on a JMS listener that makes sure messages are removed only from the JMS server when the transaction is committed. Doing this ensures that if an error occurs while processing the message, it is still available for reprocessing. In other words, the transactional support in these connectors ensures that messages are delivered reliably from a source to an operation, or between processors within a flow.
If you want to move messages between different connectors that support transactions, use XA transactions to ensure that both connectors' transactions are committed as one atomic unit.
See Transaction Management for more information about XA and other types of transactions.
If you have a web application that uses a nontransactional connector, such as HTTP, follow a reliability pattern to ensure reliable messaging for your application.
Comparing Endpoints in Reliability Patterns
You can couple the reliable acquisition flow with any type of transactional endpoint. You don’t have to use a VM endpoint in a reliability pattern. The following diagram illustrates a reliability pattern where the reliable acquisition flow delivers a message to a JMS endpoint:
The following table lists the advantages and disadvantages of using VM, JMS, and JDBC endpoints in a reliable acquisition flow.
| Endpoint | Advantage | Disadvantage |
|---|---|---|
File-based VM (standalone Mule) |
|
|
JDBC |
|
|
JMS (enabled for persistence) |
|
|
| The comparison doesn’t include in-memory persistence setups for VM and JMS because they can lose messages and are not considered reliable. |
Implementing a Reliability Pattern
Below is a code example that shows what a complete reliability pattern would look like, including the reliable acquisition flow and the application logic flow. In this example, the reliable acquisition flow part of the reliability pattern is HTTP-to-VM.
Example: HTTP to VM
<http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" >
<http:listener-connection host="0.0.0.0" port="8081" />
</http:listener-config>
<vm:config name="VM_Config" doc:name="VM Config" >
<vm:queues >
<vm:queue queueName="toTransactionalVM" queueType="PERSISTENT"/>
</vm:queues>
</vm:config>
<flow name="reliable-data-acquisition">
<http:listener config-ref="HTTP_Listener_config" path="transactionalEndpoint"/>
<vm:publish config-ref="VM_Config" queueName="toTransactionalVM" sendCorrelationId="ALWAYS"/> (1)
</flow>
<!-- This is the application logic flow in the reliability pattern.
It is a wrapper around the sub-flow "business-logic-processing". -->
<flow name="main-flow">
<vm:listener doc:name="Listener" config-ref="VM_Config" queueName="toTransactionalVM"
transactionalAction="ALWAYS_BEGIN"/> (2)
<flow-ref name="business-logic-processing"/>
</flow>
<!-- In this sub-flow, the application starts processing the message. -->
<sub-flow name="business-logic-processing">
<logger level="INFO" doc:name="Logger" />
<!--
This is where the actual business-logic is performed.
-->
</sub-flow>| 1 | The message is written to the VM queue. It is now available for processing by the main flow. |
| 2 | The message is read from the VM queue transactionally. This ensures that if an error occurs, the reading is rolled back and the message is reprocessed. |
To test this XML example in Studio, add the VM connector dependency to your project’s pom.xml file.
| This example shows VM file persistency, which does not work on clusters. |
Implementing a Reliable Acquisition Flow
The example provided in Implementing a Reliability Pattern showed a reliable acquisition flow for an HTTP listener to a VM publish operation. This scenario focuses on a reliable acquisition flow that has a nontransactional event source: FTP-to-VM (Or SFTP, FTPS).
In this case, a resource-based connection is used. When using resource-based connections, ensure to read the resource only once. This is done by setting watermarkEnabled to true.
Do not set the auto-delete parameter to true because doing so deletes the file as soon as the flow finishes, even if the processing is done by another flow. The same applies to moveToDirectory.
Example: FTP to VM
The following code implements a reliable acquisition flow from an FTP listener (On New or Updated File) to a VM queue by a publish operation:
<ftp:config name="FTP_Config">
<ftp:connection host="localhost" username="max" password="mulesoft"/>
</ftp:config>
<flow name="reliable-data-acquisition">
<ftp:listener config-ref="FTP_Config" watermarkEnabled="true" directory="someDirectory/in"> (1)
<scheduling-strategy >
<fixed-frequency />
</scheduling-strategy>
</ftp:listener>
<vm:publish config-ref="VM_Config" queueName="toTransactionalVM"/>
</flow>
<flow name="doc-examplesFlow">
<vm:listener queueName="toTransactionalVM" config-ref="VM_Config" transactionalAction="ALWAYS_BEGIN"/>
<set-variable value="#[attributes.filename]" variableName="filepath"/>
<!-- File content is already present in payload -->
<flow-ref name="business-logic-processing"/>
<ftp:delete path="#[vars.filepath]"/> (2)
</flow>| 1 | To perform a transformation before publishing to VM, add a max redelivery count. |
| 2 | If the file is not intended to be deleted, do not use the ftp:delete operation. The same applies if the intended operation is ftp:move. |
A similar implementation is done for any other of the resourced-based connectors.
General Considerations
When you implement a reliability pattern, consider the following points:
-
When the connector (event source) allows it, always use a transaction.
-
When you want to enlist multiple managed resources within the same transaction, use an XA transaction to bridge event sources.
-
The reliability of JMS is tied to the MQ implementation and how it is configured. Most MQ implementations allow you to configure whether messages are stored only in memory, or persisted. You can achieve reliability only if you configure the MQ server to persistently store messages before sending them forward. Otherwise, you risk losing messages in case of an MQ server crash.
-
Reliability has performance implications.
-
If the outbound operation in the reliable acquisition flow is not transactional (for example, a flow from file-to-FTP), perform that operation inside a Try Scope. With this practice, you can confirm if the operation completes successfully or, in case it fails, log the error message.