April 2017
« Sep    
Your widget title

Posts Tagged ‘Mule’

PostHeaderIcon Thread leaks in Mule ESB 2.2.1


The application I work on packages Mule ESB 2.2.1 in a WAR and deploys it under a WebLogic 10.3 server. My team mates and I noticed that, on multiple deploy/undeploy cycles, the PermGen size dramatically decreased. The cause of this was the number of threads, which hardly decreased on undeployment phases, unlike the expected behaviour.
Indeed, Mule is seldom deployed as a WebApp. Rather, it is designed to be run as a standalone application, within a Tanuki wrapper. When the JVM is killed, all the threads are killed, too, and therefore no thread survives ; hence, the memory is freed and there is no reason to fear a thread leak.

Moreover, when the application is redeployed, new threads -with the same names as the “old” threads- are created. The risk is that, for any reason, a thread-name-based communication between threads may fail, because the communication pipe may be read by the wrong thread.

In my case: on WebLogic startup, there are 31 threads ; when the application is deployed, there are 150 ; when the application works (receives and handles messages), the number of threads climbs to 800 ; when the application is undeployed, only 12 threads are killed, the other remaining alive.

The question is: how to kill Mule-created threads, in order to avoid a Thread leak?

WebLogic Threads

I performed a thread dump at WebLogic startup. Here are WebLogic threads, created before any deployment occurs:

Attach Listener
ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'
ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'
ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'
JMX server connection timeout 42
RMI Scheduler(0)
RMI TCP Accept-0
RMI TCP Connection(1)-
RMI TCP Connection(2)-
Reference Handler
Signal Dispatcher
VDE Transaction Processor Thread
[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'
[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'
[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'
[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'
[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'
[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'

Dispose Disposables, Stop Stoppables…

The application being deployed in a WAR, I created a servlet implementing ServletContextListener. In the method contextDestroyed(), I destroy Mule objects (Disposable, Stoppable, Model, Service, etc.) one per one.

        final Collection<Model> allModels;
        try {
            allModels = MuleServer.getMuleContext().getRegistry().lookupObjects(Model.class);
            if (LOGGER.isDebugEnabled()) {
                LOGGER.debug("Disposing models " + allModels.size());
            for (Model model : allModels) {
        } catch (Exception e) {


    private void stopStoppables() {
        final Collection<Stoppable> allStoppables;
        try {
            allStoppables = MuleServer.getMuleContext().getRegistry().lookupObjects(Stoppable.class);
            if (LOGGER.isDebugEnabled()) {
                LOGGER.debug("Stopping stoppables " + allStoppables.size());
            for (Stoppable stoppable : allStoppables) {
        } catch (MuleException e) {

This first step is needed because default mechanism is flawed: Mule re-creates objects that were destroyed.

Kill Threads

The general idea to kill Mule threads is the following: perform a Unix-style “diff” between WebLogic native threads, and the threads still alive once all Mule objects have been stopped and disposed.

On Application Startup

In the ServletContextListener, I add a field that will be set in a method called in the constructor:

    private List<String> threadsAtStartup;
     * This method retrieves the Threads present at startup: mainly speaking, they are Threads related to WebLogic.
    private void retrieveThreadsOnStartup() {
        final Thread[] threads;
        final ThreadGroup threadGroup;
        threadGroup = Thread.currentThread().getThreadGroup();
        try {
            threads = retrieveCurrentActiveThreads(threadGroup);
        } catch (NoSuchFieldException e) {
            LOGGER.error("Could not retrieve initial Threads list. The application may be unstable on shutting down ", e);
            threadsAtStartup = new ArrayList<String>();
        } catch (IllegalAccessException e) {
            LOGGER.error("Could not retrieve initial Threads list. The application may be unstable on shutting down ", e);
            threadsAtStartup = new ArrayList<String>();

        threadsAtStartup = new ArrayList<String>(threads.length);
        for (int i = 0; i < threads.length; i++) {
            final Thread thread;
            try {
                thread = threads[i];
                if (null != thread) {
                    if (LOGGER.isDebugEnabled()) {
                        LOGGER.debug("This Thread was available at startup: " + thread.getName());
            } catch (RuntimeException e) {
                LOGGER.error("An error occured on initial Thread statement: ", e);
     * Hack to retrieve the field ThreadGroup.threads, which is package-protected and therefore not accessible 
     * @param threadGroup
     * @return
     * @throws NoSuchFieldException
     * @throws IllegalAccessException
    private Thread[] retrieveCurrentActiveThreads(ThreadGroup threadGroup) throws NoSuchFieldException, IllegalAccessException {
        final Thread[] threads;
        final Field privateThreadsField;
        privateThreadsField = ThreadGroup.class.getDeclaredField("threads");

        threads = (Thread[]) privateThreadsField.get(threadGroup);
        return threads;

On application shutdown

In the method ServletContextListener.contextDestroyed(), let’s call this method:

     * Cleanses the Threads on shutdown: theorically, when the WebApp is undeployed, should remain only the threads
     * that were present before the WAR was deployed. Unfornately, Mule leaves alive many threads on shutdown, reducing
     * PermGen size and recreating new threads with the same names as the old ones, inducing a kind of instability.
    private void cleanseThreadsOnShutdown() {
        final Thread[] threads;
        final ThreadGroup threadGroup;
        final String currentThreadName;

        currentThreadName = Thread.currentThread().getName();

        if (LOGGER.isDebugEnabled()) {
            LOGGER.debug("On shutdown, currentThreadName is: " + currentThreadName);

        threadGroup = Thread.currentThread().getThreadGroup();
        try {
            threads = retrieveCurrentActiveThreads(threadGroup);
        } catch (NoSuchFieldException e) {
            LOGGER.error("An error occured on Threads cleaning at shutdown", e);
        } catch (IllegalAccessException e) {
            LOGGER.error("An error occured on Threads cleaning at shutdown", e);

        for (Thread thread : threads) {
            final String threadName = thread.getName();
            final Boolean shouldThisThreadBeKilled;

            shouldThisThreadBeKilled = isThisThreadToBeKilled(currentThreadName, threadName);
            if (LOGGER.isDebugEnabled()) {
      "should the thread named " + threadName + " be killed? " + shouldThisThreadBeKilled);
            if (shouldThisThreadBeKilled) {
                thread = null;


     * Says whether a thread is to be killed<br/>
     * Rules:
     * <ul><li>a Thread must NOT be killed if:</li>
     * <ol>
     * <li>it was among the threads available at startup</li>
     * <li>it is a Thread belonging to WebLogic (normally, WebLogic threads are among the list in the previous case</li>
     * <li>it is the current Thread (simple protection against unlikely situation)</li>
     * </ol>
     * <li>a Thread must be killed: in all other cases</li>
     * </ul>
     * @param currentThreadName
     * @param threadName
     * @return
    private Boolean isThisThreadToBeKilled(String currentThreadName, String threadName) {
        final Boolean toBeKilled;
        toBeKilled = !threadsAtStartup.contains(threadName)
                &amp;&amp; !StringUtils.contains(threadName, "weblogic")
                &amp;&amp; !threadName.equalsIgnoreCase(currentThreadName);
        return toBeKilled;


My application uses an EhCache. Its threads names usually end with “.data”. They are not killed by the previous actions. To get rid of them, the most elegant way is to add this block in the web.xml:


cf EhCache documentation

With all these operations, almost all threads are killed. But Java VisualVM still displays 34, vs. 31 at startup.

Tough Threads

A thread dump confirms that, at this point, 3 rebellious threads still refuse to be kill:


Let’s examine them:

  • MuleServer.1: This thread is an instance of the inner class MuleServer.ShutdownThread. Indeed, this is the first thread created by Mule, and therefore appears among the threads available at startup, before the ServletContextListener is called… I did not succeed in killing it, even why trying to kill it namely, which makes sense: killing the father thread looks like suiciding the ServletContextListener.
  • SocketTimeoutMonitor-Monitor.1: This thread is created by Mule’s TcpConnector and its daughter classes: HttpConnector, SslConnector, etc. Again, I could not kill them.


We have seen Mule suffers of major thread leaks when deployed as a WAR. Anyway, most of these leaks may be sealed.
I assume MuleSoft was aware of this issue: in the version 3 of Mule, the deployment of webapps was refactored.

PostHeaderIcon Tutorial: Re-package Mule ESB as a standalone client


You have to deliver Mule 2.2.1 as a standalone application, or, more accurately, as a simple archive ready-to-use by someone else (customer, co-team worker, etc.).

In this tutorial, we assume that:

  • you have to include external jars, eg. MQ and WebLogic jars
  • you have written your XML configuration file for Mule, of which all properties are externalized in an external property file. We don’t mind the actual workflow, we assume you’re skilled enough with Mule 😉



Prior to building standalone:

  • get Mule ESB 2.2.1 standalone archive, available on MuleSoft website
  • get the JARs needed by MQ
    • providerutil.jar
    • fscontext.jar
    • dhbcore.jar
    • connector.jar
    • commonservices.jar
  • get WebLogic’s wlfullclient.jar
  • install the zip and the jars on your local repository:
    mvn install:install-file -DgroupId=org.mulesource -DartifactId=mule-esb -Dversion=2.2.1 -Dpackaging=zip
    mvn install:install-file -Dfile=wlfullclient.jar  -DgroupId=weblogic -DartifactId=wlfullclient -Dversion=10.3 -Dpackaging=jar -DgeneratePom=true
    mvn install:install-file -Dfile=fscontext.jar  -DgroupId=fscontext -DartifactId=fscontext -Dversion=1.2 -Dpackaging=jar -DgeneratePom=true
    mvn install:install-file -Dfile=providerutil.jar  -DgroupId=fscontext -DartifactId=providerutil -Dversion=1.2 -Dpackaging=jar -DgeneratePom=true
    mvn install:install-file -DgroupId=mq -Dversion= -Dpackaging=jar
    mvn install:install-file -DgroupId=mq -Dversion= -Dpackaging=jar
    mvn install:install-file -DgroupId=mq -DartifactId=dhbcore -Dversion= -Dpackaging=jar -Dfile=dhbcore.jar
    mvn install:install-file -DgroupId=mq -DartifactId=commonservices -Dversion= -Dpackaging=jar -Dfile=commonservices.jar
    mvn install:install-file -DgroupId=connector -DartifactId=connector -Dversion=1.0 -Dpackaging=jar -Dfile=connector.jar

Files to be edited

  • Create a mule-jonathan.xml file in src/main/resources/ folder.
  • Externalize all properties in file in src/main/resources/ folder. As you may anticipate it, you will have add this property file in Mule classpath
  • To perform that:
    • Copy the wrapper.conf of Mule standalone archive as src/main/resources/wrapper.conf
    • After the line:*.jar

      , add the line:
  • in src/main/resources/, create a file start-mule-jonathan.bat, with the content:
    set MULE_HOME=%CD%
    cd %MULE_HOME%\bin
    mule.bat -config mule-jonathan.xml


Here is the pom.xml of our project:

<?xml version="1.0"?>
<project xmlns="" xmlns:xsi=""  xsi:schemaLocation="">

Maven Assembly

We will use Maven Assembly: this plugin allows unpack archives, copy files, insert files, delete folders, etc.

Here is the assembly.xml file that should be located in src/main/assembly/ folder of your project. The code is commented so that you understand what we do.

<assembly xmlns=""           xmlns:xsi=""           xsi:schemaLocation="                     ">
          <!-- excluse original wrapper.conf, to include our tuned wrapper.conf-->
          <!--remove the these folders, useless in a standalone client-->
        <!--include the property file -->
        <!-- include Mule XML config file-->
        <!-- modified wrapper.conf to stake in account the etc/ folder, containing the property file-->

Build process

To build go to the folder yourproject/jonathan, then launch a mvn clean install. A complete installation package is output on target folder:

The archive is built thanks to Maven Assembly plugin.



Copy or move the archive to any folder of your choice. Then unzip it.

(optionnal) Checks


Here is a tree of the installation, with some important file that must appear:

¦   +---mule-jonathan.xml
¦   +---wrapper.conf
¦   +---boot
¦   ¦   +---exec
¦   +---endorsed
¦   +---mule
¦   +---opt
¦   +---user
¦       +------commonservices-
¦       +------connector-1.0.jar
¦       +------dhbcore-
¦       +------fscontext-1.2.jar
¦       +------providerutil-1.2.jar
¦       +------wlfullclient-10.3.jar
¦       +------connector-1.0.jar


Check the files listed above in the tree appear. Besides, check the conf/wrapper.conf file contains the line


Edit etc/ file and set the right properties.


Execute start-mule-jonathan.bat to launch Mule on Windows. On first attempt, Mule will display the user licence and ask you your confirmation you accept the terms of the agreement.

PostHeaderIcon Mule / MQJE001 / MQJMS2007


In a Mule ESB workflow, the endpoint is a <jms:outbound-endpoint>, pointing to a JMS queue hosted on MQ Series and accessed through WebLogic 10.3.3.

I get the following stracktrace

Exception stack is:
1. MQJE001: Completion Code 2, Reason 2027 ( (null)
2. MQJMS2007: failed to send message to MQ queue(JMS Code: MQJMS2007) (javax.jms.JMSException) (
3. Failed to create and dispatch response event over Jms destination "queue://MQSERVER/AMQ.4C8A5E112285475605?persistence=1". Failed to route event via endpoint: null. Message payload is of type: JMSObjectMessage (org.mule.api.transport.DispatchException)
org.mule.transport.jms.JmsReplyToHandler:154 (


On Mule config file, explicitly set the attribute disableTemporaryReplyToDestinations at true in the JMS outbound tag:

 transformer-refs="foo" disableTemporaryReplyToDestinations="true"/>

PostHeaderIcon Deploy a webservice under Mule ESB using CXF

This short tutorial is aimed at showing the main steps allowing to deploy a WebService, using CXF framework, under a Mule ESB instance.

Java code

Declare an interface:

  public interface BasicExampleServices {
       @WebResult(name = "myReturnedInteger")
        Integer getInteger(@WebParam(name="myInteger") Integer myInteger);

Implement this interface:

@WebService(endpointInterface = "", serviceName = "basicExampleServices")
public class WSBasicExampleServices implements BasicExampleServices {

     public Integer getInteger(Integer param) {
          return 12345;

XML files

Create a Spring config file ws-basicExample.xml:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns=""
 xmlns:xsi="" xmlns:aop=""

      <bean id="basicExampleService" scope="singleton"/>

Create a Mule configuration file ws-basicExample-config.xml:

<?xml version="1.0" encoding="UTF-8" ?>
<mule xmlns=""

      <spring:import resource="ws-basicExample.xml"/>

 <model name="wsBasicExampleModel">
      <service name="wsBasicExampleService">
              <cxf:inbound-endpoint address="http://localhost:8282/services/basicExampleServices"/>
              <spring-object bean="basicExampleService"/>


  • Run the Mule, pointing your config file.
  • In your favorite webbrowser, open the URL:
  • The webservice contract is expected to be displayed.

  • You can also execute a runtime test:
    public class WSBasicExampleServicesRuntimeTest {
     private BasicExampleServices basicExampleServices;
       public void setup() {
         JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();
         factory.getInInterceptors().add(new LoggingInInterceptor());
         factory.getOutInterceptors().add(new LoggingOutInterceptor());
         basicExampleServices = (BasicExampleServices) factory.create();
       public void testGetInteger() {
         final Integer expectedAnswer = 12345;
         final Integer actualAnswer;
         final Integer param = 912354;
         actualAnswer = basicExampleServices.getInteger(param);
         assertEquals(expectedAnswer, actualAnswer);

PostHeaderIcon Cannot run this command because Java services are not enabled. A user with System Administrator (SA) role must reconfigure the system to enable Java.


An object is marshallized and sent on TibcoRV 8.1. An instance of Mule ESB 2.2.1 listens to TibcoRV, reads the message, unmarshalls the object and builds an SQL insert query.

The query is very simple:

insert into myTable(dateColumn, stringColumn) values(...)

It is executed in a Jdbc connector declared on Mule configuration file:

    <jdbc:connector name="jdbcConnector" dataSource-ref="mySybaseDB" pollingFrequency="1000">
        <jdbc:query key="writeTest"
                    value="INSERT INTO myTable(dateColumn, stringColumn)

The DB is Sybase. The first column is of type datetime, the second one is varchar. The values are retrieved from a java.util.Date and a java.lang.String.


When the query is run, I get the error:

java.sql.SQLException: Cannot run this command because Java services are not enabled. A user with System Administrator (SA) role must reconfigure the system to enable Java.

(you may get a com.sybase.jdbc2.jdbc.SybSQLException instead of java.sql.SQLException)

Explanation – Fix

The error is related only to Sybase, and not to TibcoRV and Mule: by default, Sybase cannot manage Java Dates (java.util.Date). You have to start explicitly the Java services. To perform that:

  • login with an username owing the rights “sa_role
  • run the SQL query:
    sp_configure 'enable java', 1
  • the restart the Sybase server

Now it should work. A similar error may occur with JBoss.