May 2017
« Sep    
Your widget title

Posts Tagged ‘Jonathan Lalou’

PostHeaderIcon Thread leaks in Mule ESB 2.2.1


The application I work on packages Mule ESB 2.2.1 in a WAR and deploys it under a WebLogic 10.3 server. My team mates and I noticed that, on multiple deploy/undeploy cycles, the PermGen size dramatically decreased. The cause of this was the number of threads, which hardly decreased on undeployment phases, unlike the expected behaviour.
Indeed, Mule is seldom deployed as a WebApp. Rather, it is designed to be run as a standalone application, within a Tanuki wrapper. When the JVM is killed, all the threads are killed, too, and therefore no thread survives ; hence, the memory is freed and there is no reason to fear a thread leak.

Moreover, when the application is redeployed, new threads -with the same names as the “old” threads- are created. The risk is that, for any reason, a thread-name-based communication between threads may fail, because the communication pipe may be read by the wrong thread.

In my case: on WebLogic startup, there are 31 threads ; when the application is deployed, there are 150 ; when the application works (receives and handles messages), the number of threads climbs to 800 ; when the application is undeployed, only 12 threads are killed, the other remaining alive.

The question is: how to kill Mule-created threads, in order to avoid a Thread leak?

WebLogic Threads

I performed a thread dump at WebLogic startup. Here are WebLogic threads, created before any deployment occurs:

Attach Listener
ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'
ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'
ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'
JMX server connection timeout 42
RMI Scheduler(0)
RMI TCP Accept-0
RMI TCP Connection(1)-
RMI TCP Connection(2)-
Reference Handler
Signal Dispatcher
VDE Transaction Processor Thread
[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'
[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'
[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'
[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'
[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'
[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'

Dispose Disposables, Stop Stoppables…

The application being deployed in a WAR, I created a servlet implementing ServletContextListener. In the method contextDestroyed(), I destroy Mule objects (Disposable, Stoppable, Model, Service, etc.) one per one.

        final Collection<Model> allModels;
        try {
            allModels = MuleServer.getMuleContext().getRegistry().lookupObjects(Model.class);
            if (LOGGER.isDebugEnabled()) {
                LOGGER.debug("Disposing models " + allModels.size());
            for (Model model : allModels) {
        } catch (Exception e) {


    private void stopStoppables() {
        final Collection<Stoppable> allStoppables;
        try {
            allStoppables = MuleServer.getMuleContext().getRegistry().lookupObjects(Stoppable.class);
            if (LOGGER.isDebugEnabled()) {
                LOGGER.debug("Stopping stoppables " + allStoppables.size());
            for (Stoppable stoppable : allStoppables) {
        } catch (MuleException e) {

This first step is needed because default mechanism is flawed: Mule re-creates objects that were destroyed.

Kill Threads

The general idea to kill Mule threads is the following: perform a Unix-style “diff” between WebLogic native threads, and the threads still alive once all Mule objects have been stopped and disposed.

On Application Startup

In the ServletContextListener, I add a field that will be set in a method called in the constructor:

    private List<String> threadsAtStartup;
     * This method retrieves the Threads present at startup: mainly speaking, they are Threads related to WebLogic.
    private void retrieveThreadsOnStartup() {
        final Thread[] threads;
        final ThreadGroup threadGroup;
        threadGroup = Thread.currentThread().getThreadGroup();
        try {
            threads = retrieveCurrentActiveThreads(threadGroup);
        } catch (NoSuchFieldException e) {
            LOGGER.error("Could not retrieve initial Threads list. The application may be unstable on shutting down ", e);
            threadsAtStartup = new ArrayList<String>();
        } catch (IllegalAccessException e) {
            LOGGER.error("Could not retrieve initial Threads list. The application may be unstable on shutting down ", e);
            threadsAtStartup = new ArrayList<String>();

        threadsAtStartup = new ArrayList<String>(threads.length);
        for (int i = 0; i < threads.length; i++) {
            final Thread thread;
            try {
                thread = threads[i];
                if (null != thread) {
                    if (LOGGER.isDebugEnabled()) {
                        LOGGER.debug("This Thread was available at startup: " + thread.getName());
            } catch (RuntimeException e) {
                LOGGER.error("An error occured on initial Thread statement: ", e);
     * Hack to retrieve the field ThreadGroup.threads, which is package-protected and therefore not accessible 
     * @param threadGroup
     * @return
     * @throws NoSuchFieldException
     * @throws IllegalAccessException
    private Thread[] retrieveCurrentActiveThreads(ThreadGroup threadGroup) throws NoSuchFieldException, IllegalAccessException {
        final Thread[] threads;
        final Field privateThreadsField;
        privateThreadsField = ThreadGroup.class.getDeclaredField("threads");

        threads = (Thread[]) privateThreadsField.get(threadGroup);
        return threads;

On application shutdown

In the method ServletContextListener.contextDestroyed(), let’s call this method:

     * Cleanses the Threads on shutdown: theorically, when the WebApp is undeployed, should remain only the threads
     * that were present before the WAR was deployed. Unfornately, Mule leaves alive many threads on shutdown, reducing
     * PermGen size and recreating new threads with the same names as the old ones, inducing a kind of instability.
    private void cleanseThreadsOnShutdown() {
        final Thread[] threads;
        final ThreadGroup threadGroup;
        final String currentThreadName;

        currentThreadName = Thread.currentThread().getName();

        if (LOGGER.isDebugEnabled()) {
            LOGGER.debug("On shutdown, currentThreadName is: " + currentThreadName);

        threadGroup = Thread.currentThread().getThreadGroup();
        try {
            threads = retrieveCurrentActiveThreads(threadGroup);
        } catch (NoSuchFieldException e) {
            LOGGER.error("An error occured on Threads cleaning at shutdown", e);
        } catch (IllegalAccessException e) {
            LOGGER.error("An error occured on Threads cleaning at shutdown", e);

        for (Thread thread : threads) {
            final String threadName = thread.getName();
            final Boolean shouldThisThreadBeKilled;

            shouldThisThreadBeKilled = isThisThreadToBeKilled(currentThreadName, threadName);
            if (LOGGER.isDebugEnabled()) {
      "should the thread named " + threadName + " be killed? " + shouldThisThreadBeKilled);
            if (shouldThisThreadBeKilled) {
                thread = null;


     * Says whether a thread is to be killed<br/>
     * Rules:
     * <ul><li>a Thread must NOT be killed if:</li>
     * <ol>
     * <li>it was among the threads available at startup</li>
     * <li>it is a Thread belonging to WebLogic (normally, WebLogic threads are among the list in the previous case</li>
     * <li>it is the current Thread (simple protection against unlikely situation)</li>
     * </ol>
     * <li>a Thread must be killed: in all other cases</li>
     * </ul>
     * @param currentThreadName
     * @param threadName
     * @return
    private Boolean isThisThreadToBeKilled(String currentThreadName, String threadName) {
        final Boolean toBeKilled;
        toBeKilled = !threadsAtStartup.contains(threadName)
                &amp;&amp; !StringUtils.contains(threadName, "weblogic")
                &amp;&amp; !threadName.equalsIgnoreCase(currentThreadName);
        return toBeKilled;


My application uses an EhCache. Its threads names usually end with “.data”. They are not killed by the previous actions. To get rid of them, the most elegant way is to add this block in the web.xml:


cf EhCache documentation

With all these operations, almost all threads are killed. But Java VisualVM still displays 34, vs. 31 at startup.

Tough Threads

A thread dump confirms that, at this point, 3 rebellious threads still refuse to be kill:


Let’s examine them:

  • MuleServer.1: This thread is an instance of the inner class MuleServer.ShutdownThread. Indeed, this is the first thread created by Mule, and therefore appears among the threads available at startup, before the ServletContextListener is called… I did not succeed in killing it, even why trying to kill it namely, which makes sense: killing the father thread looks like suiciding the ServletContextListener.
  • SocketTimeoutMonitor-Monitor.1: This thread is created by Mule’s TcpConnector and its daughter classes: HttpConnector, SslConnector, etc. Again, I could not kill them.


We have seen Mule suffers of major thread leaks when deployed as a WAR. Anyway, most of these leaks may be sealed.
I assume MuleSoft was aware of this issue: in the version 3 of Mule, the deployment of webapps was refactored.

PostHeaderIcon Start Mule ESB as an NT service under a Windows server


You would like to start a Mule ESB instance as an NT service, under a Windows server. You would also like to give a specific name and description


  • add an environment variable MULE_HOME, with value the path of Mule install, for instance: C:\jonathan\mule-standalone-3.0.1
  • edit the file %MULE_HOME%\conf\wrapper.conf
  • replace the following properties default values:
  • (you can also set other properties related to NT service configuration)
  • launch the command:
  • %MULE_HOME%\bin\mule.bat install -config %MULE_HOME%\bin\mule-conf.xml
  • then you can see in the administration services that the service has started
  • to remove the service, only launch the following command
  • %MULE_HOME%\bin\mule.bat remove

Known issue:

On certain installations (among them the servers on which CygWin is installed), a conflict may happen between the files %MULE_HOME%\bin\mule (standard launcher for Unix and Linux) and %MULE_HOME%\bin\mule.bat (standard launcher for Windows). In this case, rename %MULE_HOME%\bin\mule as %MULE_HOME%\bin\mule.OLD

PostHeaderIcon Tutorial: Re-package Mule ESB as a standalone client


You have to deliver Mule 2.2.1 as a standalone application, or, more accurately, as a simple archive ready-to-use by someone else (customer, co-team worker, etc.).

In this tutorial, we assume that:

  • you have to include external jars, eg. MQ and WebLogic jars
  • you have written your XML configuration file for Mule, of which all properties are externalized in an external property file. We don’t mind the actual workflow, we assume you’re skilled enough with Mule 😉



Prior to building standalone:

  • get Mule ESB 2.2.1 standalone archive, available on MuleSoft website
  • get the JARs needed by MQ
    • providerutil.jar
    • fscontext.jar
    • dhbcore.jar
    • connector.jar
    • commonservices.jar
  • get WebLogic’s wlfullclient.jar
  • install the zip and the jars on your local repository:
    mvn install:install-file -DgroupId=org.mulesource -DartifactId=mule-esb -Dversion=2.2.1 -Dpackaging=zip
    mvn install:install-file -Dfile=wlfullclient.jar  -DgroupId=weblogic -DartifactId=wlfullclient -Dversion=10.3 -Dpackaging=jar -DgeneratePom=true
    mvn install:install-file -Dfile=fscontext.jar  -DgroupId=fscontext -DartifactId=fscontext -Dversion=1.2 -Dpackaging=jar -DgeneratePom=true
    mvn install:install-file -Dfile=providerutil.jar  -DgroupId=fscontext -DartifactId=providerutil -Dversion=1.2 -Dpackaging=jar -DgeneratePom=true
    mvn install:install-file -DgroupId=mq -Dversion= -Dpackaging=jar
    mvn install:install-file -DgroupId=mq -Dversion= -Dpackaging=jar
    mvn install:install-file -DgroupId=mq -DartifactId=dhbcore -Dversion= -Dpackaging=jar -Dfile=dhbcore.jar
    mvn install:install-file -DgroupId=mq -DartifactId=commonservices -Dversion= -Dpackaging=jar -Dfile=commonservices.jar
    mvn install:install-file -DgroupId=connector -DartifactId=connector -Dversion=1.0 -Dpackaging=jar -Dfile=connector.jar

Files to be edited

  • Create a mule-jonathan.xml file in src/main/resources/ folder.
  • Externalize all properties in file in src/main/resources/ folder. As you may anticipate it, you will have add this property file in Mule classpath
  • To perform that:
    • Copy the wrapper.conf of Mule standalone archive as src/main/resources/wrapper.conf
    • After the line:*.jar

      , add the line:
  • in src/main/resources/, create a file start-mule-jonathan.bat, with the content:
    set MULE_HOME=%CD%
    cd %MULE_HOME%\bin
    mule.bat -config mule-jonathan.xml


Here is the pom.xml of our project:

<?xml version="1.0"?>
<project xmlns="" xmlns:xsi=""  xsi:schemaLocation="">

Maven Assembly

We will use Maven Assembly: this plugin allows unpack archives, copy files, insert files, delete folders, etc.

Here is the assembly.xml file that should be located in src/main/assembly/ folder of your project. The code is commented so that you understand what we do.

<assembly xmlns=""           xmlns:xsi=""           xsi:schemaLocation="                     ">
          <!-- excluse original wrapper.conf, to include our tuned wrapper.conf-->
          <!--remove the these folders, useless in a standalone client-->
        <!--include the property file -->
        <!-- include Mule XML config file-->
        <!-- modified wrapper.conf to stake in account the etc/ folder, containing the property file-->

Build process

To build go to the folder yourproject/jonathan, then launch a mvn clean install. A complete installation package is output on target folder:

The archive is built thanks to Maven Assembly plugin.



Copy or move the archive to any folder of your choice. Then unzip it.

(optionnal) Checks


Here is a tree of the installation, with some important file that must appear:

¦   +---mule-jonathan.xml
¦   +---wrapper.conf
¦   +---boot
¦   ¦   +---exec
¦   +---endorsed
¦   +---mule
¦   +---opt
¦   +---user
¦       +------commonservices-
¦       +------connector-1.0.jar
¦       +------dhbcore-
¦       +------fscontext-1.2.jar
¦       +------providerutil-1.2.jar
¦       +------wlfullclient-10.3.jar
¦       +------connector-1.0.jar


Check the files listed above in the tree appear. Besides, check the conf/wrapper.conf file contains the line


Edit etc/ file and set the right properties.


Execute start-mule-jonathan.bat to launch Mule on Windows. On first attempt, Mule will display the user licence and ask you your confirmation you accept the terms of the agreement.

PostHeaderIcon Tutorial: from an application, make a clustered application, within WebLogic 10


You have a non-clustered installation, on the host with DNS name jonathanDevDesktop, with an admin (port: 7001), a muletier (port: 7003) and a webtier (port: 7005) instances.
You need set your muletier as a clustered installation, with two nodes, on the same server. The second node will dedeployed on port 7007.

We assume you have a configured JMS Modules (in our case: JmsMqModule, even though the bridge between WebLogic and MQ has no impact here).



  • Copy $DOMAINS\jonathanApplication\start-muletier-server.bat" as $DOMAINS\jonathanApplication\start-muletier-server-2.bat"
  • Edit it:
    • Possibly, modify the debug port (usually: 5006)
    • Replace the line
      call "%DOMAIN_HOME%\bin\startManagedWebLogic.cmd" muletier t3://jonathanDevDesktop:7001


      call "%DOMAIN_HOME%\bin\startManagedWebLogic.cmd" muletier2 t3://jonathanDevDesktop:7001

Second Node Creation

  • Following points are not required.
    • Copy the folder %DOMAIN_HOME%\servers\muletier as %DOMAIN_HOME%\servers\muletier2
    • Delete the folders %DOMAIN_HOME%\servers\muletier2\cache and %DOMAIN_HOME%\servers\muletier2\logs
  • Stop the server muletier
  • On WebLogic console:
    • Servers > New > Server Name: muletier2, Server Listen Port: 7007 > Check Yes, create a new cluster for this server. > Next
    • Name: jonathanApplication.cluster.muletier > Messaging Mode: Multicast, Multicast Address:, Multicast Port:5777
    • Clusters > jonathanApplication.cluster.muletier > Configuration > Servers > Select a server: muletier
    • Clusters > jonathanApplication.cluster.muletier > Configuration > Servers > Select a server: muletier2
  • Start the instances of muletier and muletier2 in MS-DOS consoles.
  • On the WebLogic console:
    • Deployments > jonathanApplication-web (the mule instance) > Targets > check “jonathanApplication.cluster.muletier” and “All servers in the cluster” > Save
  • On the muletier2 DOS console, you can see the application is deployed.

JMS Configuration

The deployment of JMS on clustered environment is a little tricky.

  • On WebLogic console: JMS Modules > JmsMqModule > Targets > check “jonathanApplication.cluster.muletier” and “All servers in the cluster
  • Even though it is not required, restart your muletiers. Then you can send messages either on port 7003 or 7007, they will be popped and handled the same way.

PostHeaderIcon Tutorial: an Event Bus Handler for GWT / GXT



Let’s consider a application, JonathanGwtApplication, divided in three main panels

  • a panel to select animal name name
  • a panel to display, expand and collapse trees of the animal ancestors
  • a panel of information to display many indicators (colors, ages, etc.).

An issue we encounter is: how to make the different panels communicate? In more technical terms, how to fire events from a panel to another one?

A first solution would be to declare each panel as listener to the other panels. Indeed, this principle may go further, and declare each component as listener to a list of other components…
Main drawbacks:

  • the code becomes hard to read
  • adding or removing a component requires to modify many parts of the code
  • we don’t follow GWT 2’s “philosophy”, which is to use Handlers rather than Listeners.

Hence, these reasons incited us to provide a global EventBusHandler.

The EventBusHandler concept

The EventBusHandler is a global bus which is aware of all events that should be shared between different panels, and fires them to the right components.
The EventBusHandler is a field of JonathanGwtApplicationContext.


  • generic interface for a event. Abstract method:
    EventTypeEnum getEventEnum();
  • lalou.jonathan.application.web.gwt.animal.handler.EventHandler: generic interface for a component able to handle an event. Abstract method:
    void handleEvent(HandledEvent handledEvent);
  • lalou.jonathan.application.web.gwt.animal.handler.EventHandlerBus: the actual bus. As a concrete class, it has two methods:
    	 * Fires an event to all components declared as listening to this event
    	 * event type.
    	 * @param baseEvent
    	public void fireEvent(HandledEvent baseEvent) {
                   // ...
    	 * Adds an listener/handler for the event type given as parameter
    	 * @param eventTypeEnum
    	 * @param eventHandler
    	 * @return The List of handlers for the key given as parameter. This list
    	 *         contains the eventHandler that was given as second parameter
    	public List<EventHandler> put(EventTypeEnum eventTypeEnum,
    			EventHandler eventHandler) {
                  // ...

How to use the bus?

  1. Define an event: in JonathanGwtApplication, an event is decribed by two elements:
    • a functionnal entity: eg: “animal”, “food”, “tree node”. The functionnal entity must be isomorph to a technical DTO, eg: AnimalDTO for an entity Animal.(in the scope of this turoriel we assume to have DTOs, even though the entities may ne sufficient)
    • a technical description of the event: “selection changed”, “is expanded”
  2. Add an entry in the enum EventTypeEnum. Eg: “ANIMAL_SELECTION_CHANGED
  3. in, create an event, implementing HandledEvent and its method getEventEnum(). The match between EventTypeEnum and DTO is achieved here. Eg:
    public class AnimalSelectionChangedEvent extends
    		SelectionChangedEvent<AnimalDTO> implements HandledEvent {
    	public AnimalSelectionChangedEvent(
    			SelectionProvider<AnimalDTO> provider,
    			List<AnimalDTO> selection) {
    		super(provider, selection);
    	public EventTypeEnum getEventEnum() {
    		return EventTypeEnum.ANIMAL_SELECTION_CHANGED;
  • When an event that should interest other component is fired, simply call the bus. The bus will identify the event type and dispatch it to the relevant handlers. eg:
    animalComboBox.addSelectionChangedListener(new SelectionChangedListener<AnimalDTO>() {
    			public void selectionChanged(SelectionChangedEvent<AnimalDTO> se) {
    				final AnimalDTO selectedAnimalVersion;
    				selectedAnimalVersion= se.getSelectedItem();
    						final AnimalSelectionChangedEvent baseEvent = new AnimalSelectionChangedEvent(
    								se.getSelectionProvider(), se.getSelection());
  • Handlers:
    • easy case: the component handles only one type of event: this handler must implement the right interface (eg: AnimalSelectionChangedEventHandler) and its method, eg:
      protected void handleAnimalSelectionChangedEvent(HandledEvent handledEvent) {
    • frequent case: the component handles two or more event types. No matter, make the component implement all the needed interfaces (eg: AnimalSelectionChangedEventHandler, FoodSelectionChangedEventHandler). Provide a unique entry point for the method to implement, which is common to both interfaces. Retrieve the event type, and handle it with ad hoc methods. Eg:
      public void handleEvent(HandledEvent handledEvent) {
      		final EventTypeEnum eventTypeEnum;
      		eventTypeEnum = handledEvent.getEventEnum();
      		switch (eventTypeEnum) {
      	protected void handleAnimalSelectionChangedEvent(HandledEvent handledEvent) {
      		// do something
      	protected void handleFoodSelectionChangedEvent(HandledEvent handledEvent) {
      		// do something else