Posts Tagged ‘MQ’

Scenario

A Spring Boot application using ActiveMQ with @JmsListener suddenly stops receiving messages after running for a while. No errors in logs, and the queue keeps growing, but the consumers seem idle.

Setup

• ActiveMQConnectionFactory was used.

• The queue (myQueue) was filling up.

• Restarting the app temporarily fixed the issue.

Investigation

1. Checked ActiveMQ Monitoring (Web Console)

   • Messages were enqueued but not dequeued.

   • Consumers were still active, but not processing.

2. Thread Dump Analysis

   • Found that listener threads were stuck in a waiting state.

   • The problem only occurred under high load.

3. Checked JMS Acknowledgment Mode

   • Default AUTO_ACKNOWLEDGE was used.

   • Suspected an issue with message acknowledgment.

4. Enabled Debug Logging

   • Added:

     logging.level.org.springframework.jms=DEBUG

   • Found repeated logs like:

     JmsListenerEndpointContainer#0-1 received message, but no further processing

   • This hinted at connection issues.

5. Tested with a Different Message Broker

   • Using Artemis JMS instead of ActiveMQ resolved the issue.

   • Indicated that it was broker-specific.

Root Cause

ActiveMQ’s TCP connection was silently dropped, but the JMS client did not detect it.

• When the connection is lost, DefaultMessageListenerContainer doesn’t always recover properly.

• ActiveMQ does not always notify clients of broken connections.

• No exceptions were thrown because the connection was technically “alive” but non-functional.

Fix

1. Enabled keepAlive in ActiveMQ connection

   ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory(); factory.setUseKeepAlive(true); factory.setOptimizeAcknowledge(true); return factory;

2. Forced Reconnection with Exception Listener

   • Implemented:

     factory.setExceptionListener(exception -> { log.error("JMS Exception occurred, reconnecting...", exception); restartJmsListener(); });

   • This ensured that if a connection was dropped, the listener restarted.

3. Switched to DefaultJmsListenerContainerFactory with DMLC

   • SimpleMessageListenerContainer was less reliable in handling reconnections.

   • New Configuration:

     @Bean public DefaultJmsListenerContainerFactory jmsListenerContainerFactory( ConnectionFactory connectionFactory) { DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory(); factory.setConnectionFactory(connectionFactory); factory.setSessionTransacted(true); factory.setErrorHandler(t -> log.error("JMS Listener error", t)); return factory; }

Final Outcome

✅ After applying these fixes, the issue never reoccurred.
🚀 The app remained stable even under high load.

Key Takeaways

• Silent disconnections in ActiveMQ can cause message listeners to hang.

• Enable keepAlive and optimizeAcknowledge for reliable connections.

• Use DefaultJmsListenerContainerFactory with DMLC instead of SMLC.

• Implement an ExceptionListener to restart the JMS connection if necessary.

Case

In a Mule ESB workflow, the endpoint is a <jms:outbound-endpoint>, pointing to a JMS queue hosted on MQ Series and accessed through WebLogic 10.3.3.

I get the following stracktrace

Exception stack is:
1. MQJE001: Completion Code 2, Reason 2027 (com.ibm.mq.MQException)
com.ibm.mq.MQQueue:1624 (null)
2. MQJMS2007: failed to send message to MQ queue(JMS Code: MQJMS2007) (javax.jms.JMSException)
com.ibm.mq.jms.services.ConfigEnvironment:622 (http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/jms/JMSException.html)
3. Failed to create and dispatch response event over Jms destination "queue://MQSERVER/AMQ.4C8A5E112285475605?persistence=1". Failed to route event via endpoint: null. Message payload is of type: JMSObjectMessage (org.mule.api.transport.DispatchException)
org.mule.transport.jms.JmsReplyToHandler:154 (http://www.mulesource.org/docs/site/current2/apidocs/org/mule/api/transport/DispatchException.html)

Fix

On Mule config file, explicitly set the attribute disableTemporaryReplyToDestinations at true in the JMS outbound tag:

<jms:outbound-endpoint
 queue="jonathan.lalou.jms.queue"
 connector-ref="jmsConnector"
 transformer-refs="foo" disableTemporaryReplyToDestinations="true"/>

Abstract

You have an application deployed on WebLogic 10 (used version for this tutorial: 10.3.3). You have to use an external provider for JMS, in our case MQ Series / WebSphere MQ.
The client side is a Mule ESB launched in standalone.

Prerequisites

You have:

• a running WebLogic 10 with an admin instance and an another instance, in our case: Muletier.
• a file file.bindings, used for MQ.

JARs installation

• Stop all your WebLogic 10 running instances.
• Get the JARs from MQ Series folders:
  • providerutil.jar
  • fscontext.jar
  • dhbcore.jar
  • connector.jar
  • commonservices.jar
  • com.ibm.mqjms.jar
  • com.ibm.mq.jar
• Copy them in your domain additional libraries folder (usually: user_projects/domains/jonathanApplication/lib/)
• Start WebLogic 10 admin. A block like this should appear:

  <Oct 15, 2010 12:09:21 PM CEST> <Notice> <WebLogicServer> <BEA-000395> <Following extensions directory contents added to the end of the classpath:
  C:\win32app\bea\user_projects\domains\jonathanApplication\lib\com.ibm.mq.jar;C:\win32app\bea\user_projects\domains\jonathanApplication\lib\com.ibm.mqjms.jar;C:\win32app\bea\user_projects\domains\jonathanApplication\lib\commonservices.jar;C:\win32app\bea\user_projects\domains\jonathanApplication\lib\connector.jar;C:\win32app\bea\user_projects\domains\jonathanApplication\lib\dhbcore.jar;C:\win32app\bea\user_projects\domains\jonathanApplication\lib\fscontext.jar;C:\win32app\bea\
  user_projects\domains\jonathanApplication\lib\providerutil.jar>

Config

• Get file.bindings, copy it into user_projects/domains/jonathanApplication/config/jms, rename it as .bindings (without any prefix)
• Launch the console, login
• JMS > JMS Modules > Create JMS System Module > Name: JmsMqModule. Leave other fields empty. > Next > target server MuleTier > Finish
• Select JmsMqModule > New > Foreign Server > Name: MQForeignServer > keep check MuleTier > Finish
  • Select MQForeignServer >
    • JNDI Initial Context Factory: replace weblogic.jndi.WLInitialContextFactory with: com.sun.jndi.fscontext.RefFSContextFactory
    • JNDI Connection URL: set the URI of the folder containing the .bindings file, eg: file://c/win32app/bea/user_projects/domains/jonathanApplication/config/jms
  • Tab Connection Factories > New >
    • Name: MQForeignConnectionFactory
    • Local JNDI Name: the JNDI name on WebLogic side, eg: jonathanApplication/jms/connectionFactory/local (convention I could observe: separator on WebLogic: slash '/' ; unlike clients for which the separator in a dot '.')
    • Remote JNDI Name: the JNDI name on MQ side, eg: JONATHAN_APPLICATION.QCF
    • OK
  • Tab Destinations > New >
    • Queue of requests:
      • Name: JONATHAN.APPLICATION.REQUEST
      • Local JNDI Name: JONATHAN.APPLICATION.REQUEST
      • Remote JNDI Name: JONATHAN.APPLICATION.REQUEST
    • Queue of response:
      • Name: JONATHAN.APPLICATION.REPONSE
      • Local JNDI Name: JONATHAN.APPLICATION.REPONSE
      • Remote JNDI Name: JONATHAN.APPLICATION.REPONSE
    • NB: usually, MQ data are upper-cased and Java’s JNDI names are low-cased typed ; anyway (because of Windows not matching case?) here we use uppercase in for both names.

Mule

This part of the tutorial deals with a case of Mule ESB being your client application (sending and/or receiving JMS messages).

• Get the archive wlfullclient.jar (56MB). Alternatively, you can generate it yourself: go to the server/lib directory of your WebLogic installation (usually: C:\win32app\bea\wlserver_10.3\server\lib, and run: java -jar wljarbuilder.jar
• Copy the archive into $MULE_HOME/lib/user
• Copy the seven jars above (providerutil.jar, fscontext.jar, dhbcore.jar, connector.jar, commonservices.jar, com.ibm.mqjms.jar, com.ibm.mq.jar) into the same folder: $MULE_HOME/lib/user
• You can launch the mule. The config file is similar to any other configuration using standard JMS.