<AsyncLogger name=“org.mule.extensions.jms” level=“DEBUG”/>
<AsyncLogger name=“org.mule.jms.commons” level=“DEBUG”/>
Troubleshooting JMS Connector - Mule 4
To troubleshoot Anypoint Connector for JMS (Java Message Service Connector), become familiar with the information about enabling verbose logging, troubleshooting acknowledgment issues, troubleshooting JMS stopped consuming messages, troubleshooting redeployment errors when using binding mode, and interpreting commonly thrown messages.
Enable Verbose Logging
Sometimes you are not sure why the application is failing and you require a better understanding of what is going on in the app’s interaction with the JMS server.
To gather detailed logs:
-
Use the configuration file to increase the logging level on the server side.
-
Enable verbose logging for JMS Connector:
-
Access Anypoint Studio and navigate to the Package Explorer view.
-
Open your application’s project name.
-
Open the
src/main/resources
path folder. -
Open the
log4j2.xml
file inside the folder. -
If the following line is already in the
log4j2.xml
file, uncomment it to enable it; otherwise, add it:
-
Troubleshoot Acknowledgment Issues
There are several ways an application setup can be inconsistent in terms of message acknowledgment. You can configure the session acknowledgment for an application as:
-
IMMEDIATE
-
AUTO
-
MANUAL
-
DUPS_OK
When you set the session acknowledgment to IMMEDIATE, the message gets acknowledged as soon as it’s received. Do not mix IMMEDIATE acknowledgment with Ack or Recovery session operations.
When you set the session acknowledgment to AUTO, the On New Message source takes control of the acknowledgment when the event finishes, or the session recovers when an error is thrown while processing the event. Adding a manual acknowledgment or session recover in an application with AUTO acknowledgment leads to duplicate operations on a channel.
When you set the session acknowledgment to either MANUAL or DUPS_OK, check that every path of the flow execution gets a message acknowledgment or a session recover. Otherwise, a message can leak and lead to an unresponsive consumer when the max depth of the unit of work is reached.
To resolve this issue:
Take a detailed look at the flow design and be sure that every message is acknowledged or the session recovers.
Troubleshoot JMS Stopped Consuming Messages
Usually when JMS stopped consuming messages is related to a large number of unacknowledged messages. When your Mule app that has an On New Message source or Consume operation with manual acknowledgement of messages, and not every path of the message has the proper acknowledgement or session recovery configuration, the Mule app can lead to an unresponsive consumer.
To resolve this error:
Check the depth of unacknowledged messages on the broker side and review the application logic.
Troubleshoot Redeployment Errors When Using Binding Mode
When using binding mode for connections, the driver loads a JNI resource. Because the JNI resource can be loaded only once, when the application is redeployed, an error is thrown.
To resolve this error:
Add the driver dependency at the domain level, which enables application redeployment.
Troubleshoot HA Environments and Durable Subscriptions
The configuration of HA (High Availability) systems consists of two components:
-
Primary node
Is the main node that provides continuous service. -
Secondary node
Is a clone of the primary node and provides the same service.
Both nodes are identical and are ready to receive requests so the end-user is not aware of the system crash and can continue to use the system transparently.
However, when you work with durable subscriptions, only one node initializes a connection while the other node fails to do it. In these scenarios, the nodes are identical, but both of them cannot be ready at the same time. That’s because one node has a connection while the other one doesn’t. The current design of Mule running in HA clusters aims that all nodes initialize connections to ensure that they are capable of facing a failure of the primary node.
Because of these reasons, JMS Connector does not support durable ActiveMQ subscriptions in HA cluster environments.
To resolve this scenario:
Implement a fallback solution by managing the life cycle of containers to detect when the primary node fails and starts a new node.
Troubleshoot Prefetch Issues
A known issue is that a JMS driver or broker prefetch configuration combined with MANUAL ACK and concurrent consumers or consuming more than one message per flow lead to an unexpected ordering of the consumed messages.
JMS brokers implement prefetch policies for efficient usage of network resources using a push model to dispatch messages to consumers, ensuring that a consumer always has a local buffer of messages ready to process. When the JMS driver or broker has a prefetch size configuration, each consumer holds a buffer of messages ready to consume. The number of messages in the buffer depends on the broker implementation and custom parameterization.
The JMS Connector Consume operation consumes one message at a time using a per operation consumer, which means that each operation holds a buffer of messages, but the operation consumes only the first message in the buffer by design. Which results in an unexpected behavior in the order in which messages are consumed. Consider the following scenarios:
Concurrent Flows
Conditions
-
The same flow runs concurrently in two different threads of execution (thead A and thread B).
-
The JMS driver or broker prefetch size is configured to 10.
-
There are 100 messages in the queue.
Flow iteration 1:
-
Thread A: consumer A is created, and there are 10 messages in its local buffer that are ready to process and be pushed by the broker (messages 1 to 10).
-
Thread B: consumer B is created, and there are 10 messages in its local buffer that are ready to process that were pushed by the broker (messages 11 to 20 because the first 10 messages were pushed to consumer A).
-
Thread A: consumer A acknowledged message 1, and messages 2 to 10 were returned to the broker.
-
Thread B: consumer B acknowledged message 11, and messages 12 to 20 were returned to the broker.
-
Messages 1 and 11 were consumed.
Flow iteration 2:
-
Thread A: consumer A is created, and there are 10 messages in its local buffer that are ready to process and be pushed by the broker (messages 2,3,4,5,6,7,8,9,10,12).
-
Thread B: consumer B is created, and there are 10 messages in its local buffer that are ready to process that were pushed by the broker (messages 13 to 22).
-
Thread A: consumer A acknowledged message 2 and messages 3,4,5,6,7,8,9,10,12 were returned to the broker.
-
Thread B: consumer B acknowledged message 13 and messages 14 to 20 were returned to the broker.
-
Messages 2 and 13 were consumed.
Flow iteration 3:
-
Thread A: consumer A is created, and there are 10 messages in its local buffer that are ready to process and be pushed by the broker (messages 3,4,5,6,7,8,9,10,12,14).
-
Thread B: consumer B is created, and there are 10 messages in its local buffer that are ready to process that were pushed by the broker (messages 15 to 24).
-
Thread A: consumer A acknowledged message 3 and messages 4,5,6,7,8,9,10,12,14 were returned to the broker.
-
Thread B: consumer B acknowledged message 15 and messages 16 to 24 were returned to the broker.
-
Messages 3 and 15 were consumed.
Workaround:
-
Set the Max Concurrency field value of the flow to
1
to force the executions to be serialized, and the order of the consumed messages will be as expected. -
Another option is to disable prefetch on the broker. Check your third-party JMS vendor provider documentation to disable prefetch.
Multiple Consumes on the Same Flow
Conditions
-
Messages are consumed in a foreach block from 1 to 4.
-
Messages consumed in the first point are added to a collection of messages that are later processed and acknowledged.
-
The JMS driver/broker prefetch size is configured to 2.
-
There are 4 messages in the queue.
-
The JMS Consume operation Ack mode field is configured to
MANUAL
, and messages are acknowledged in a foreach block using the collection created in the first point.
Flow iteration 1:
-
Foreach iteration1 1:
-
Consumer 1 is created, and there are 2 messages in its local buffer that are ready to process and be pushed by the broker (messages 1 and 2).
-
Message 1 is added to the collection.
-
-
Foreach iteration 2:
-
Consumer 2 is created, and there are 2 messages in its local buffer that are ready to process and be pushed by the broker (messages 3 and 4).
-
Message 3 is added to the collection.
-
-
Foreach iteration 3:
-
No available messages in the queue.
-
-
Foreach iteration 4:
-
No available messages in the queue.
-
Messages on the collection created by the foreach are acknowledged (1,3) and the rest of the messages are returned to the queue (2,4).
Flow iteration 2:
-
Foreach iteration 1:
-
Consumer 1 is created, and there are 2 messages in its local buffer that are ready to process and be pushed by the broker (messages 2 and 4).
-
Message 2 is added to the collection.
-
-
Foreach iteration 2:
-
No available messages in the queue.
-
-
Foreach iteration 3:
-
No available messages in the queue.
-
-
Foreach iteration 4:
-
No available messages in the queue.
-
Messages on the collection created by the foreach are acknowledged (2) and the rest of the messages are returned to the queue (4).
Flow iteration 3:
-
Foreach iteration 1:
-
Consumer 1 is created, and there is 1 message in its local buffer that is ready to process and be pushed by the broker (message 4).
-
Message 4 is added to the collection.
-
-
Foreach iteration 2:
-
No available messages in the queue.
-
-
Foreach iteration 3:
-
No available messages in the queue.
-
-
Foreach iteration 4:
-
No available messages in the queue.
-
Messages on the collection created by the foreach are acknowledged (4)
Summarizing the flow iterations, the 3 iterations of the flow consume 4 messages in the queue and the order is not honored:
-
Iteration 1: [1,3]
-
Iteration 2: [2]
-
Iteration 3: [4]
Workaround:
-
Disable prefetch on the broker, and check your third-party JMS vendor provider documentation to disable prefetch.
-
If you cannot disable prefetch on the broker, in the JMS Consume operation set the Ack mode field to
IMMEDIATE
to avoid holding the messages in the local buffer until the message is acknowledged. -
Depending on the use case, use a JMS On New Message listener to process messages when they arrive instead of polling for messages using the JMS Consume operation. If you need to process messages in chunks, use the Aggregators module Size based aggregator scope to create a collection of messages for later processing.
Understand Common Throws
Here is a list of common throw messages and how to interpret them.
-
JMS:ACK
There was an error in the context of the acknowledgment of an inflight message.
-
JMS:CONNECTIVITY
The connection is no longer valid. It is disposed of, but you can apply a reconnection policy if defined in the application setup.
-
JMS:CONSUMING
An error occurred while consuming messages from WMQ. This error might relate to the absence of a pending message.
-
JMS:DESTINATION_NOT_FOUND
JMS Connector is not able to find or create the destination.
-
JMS:RETRY_EXHAUSTED
The maximum number of retries for the operation is reached.
-
JMS:SECURITY
The thrown exception in due to of security-related errors.
-
JMS:TIMEOUT
The timeout for the requested operation is exceeded.
-
JMS:ILLEGAL_BODY
The message body is invalid or cannot be converted to a supported type.
-
JMS:PUBLISHING
There is an error while publishing to the target queue or topic.