Jonathan Lalou's blog

Posts Tagged ‘RMI’

(long tweet) JOnAS / no security manager: RMI class loader disabled

On server side:

an EJB2 packaged in a JAR within an EAR, deployed on JOnAS 5.

On client side:

java.lang.ClassNotFoundException: org.ow2.jonas_gen.com.clam.indice.api.interfaces.JOnASHelloWorldService150707405Home_Stub (no security manager: RMI class loader disabled)]

Explanation:

This means there is some kind of issue with generated Stubs on client side. Should check whether the Stubs depended on are available in classpath.

Posted in en-US | Tags: EJB, JOnAS, long tweet, RMI | No Comments »

Sniffing RMI Traffic… Rather log it!

Author: Jonathan Lalou

Suspecting a thread leak, there is some traffic I’d like to track on my JOnAS server: most of all, the calling IPs, with the methods and parameters sent. Actually, I lack some tools, so I tried to snif the network traffic.
Two softwares may make the job:

I discarded these solutions for several reasons. First, there are some issues with Windows Seven compatibility. Moreover, traffic on RMI protocol is SSL-encrypted… therefore not easy to read.

At least, I withdrew from this idea to snif, and I decided to intercept calls thanks to loggers.
In order to enable RMI logs, add the following properties to your JVM (to JAVA_OPTS parameters):

on client side: -Dsun.rmi.client.logCalls=true
on server side: -Djava.rmi.server.logCalls=true

More details and options are available in Oracle’s documentation: RMI Implementation Logging.

By default, the logs look like this:

FINEST: RMI TCP Connection(5)-10.76.35.25: [10.76.35.25: sun.rmi.transport.DGCImpl[0:0:0, 2]: java.rmi.dgc.Lease dirty(java.rmi.server.ObjID[], long, java.rmi.dgc.Lease)]
2012-05-03 15:35:51,053 : Log$LoggerLog.log : RMI TCP Connection(5)-10.76.35.25: [10.76.35.25: sun.rmi.transport.DGCImpl[0:0:0, 2]: java.rmi.dgc.Lease dirty(java.rmi.server.ObjID[], long, java.rmi.dgc.Lease)]

(Don’t be fool: this is Java logging, but not Log4J!)

By the way, a very interesting result I got is the following: monitoring and profile tools such as JVisualVM or JProfiler “ping” the RMI server, disturbing the measurements. Let’s consider that as “the Heisenberg uncertainty principle” applied to softwares 😉

Posted in General | Tags: JProfiler, JVisualVM, logging, RMI | No Comments »

java.net.ConnectException: (…) Bootstrap to (…) failed. It is likely that the remote side declared peer gone on this JVM

Author: Jonathan Lalou

Case and Topology

RMI services are deployed on UAT, exposed via a F5, at the following address: t3://my-f5-frontal.my.domain.extension:7090
The actual servers are my-first-node.my.domain.extension and my-second-node.my.domain.extension.

The client application is deployed in a remote location, on a QA server.
The ports are open between QA and UAT, and we can ping and use telnet with no issue on QA.

Anyway, when I launch the client application from QA, I get the following error:

2011-10-31 06:41:03,277 INFO  support.DefaultListableBeanFactory  - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@79e304: defining beans [jonathanServiceClient]; root of factory hierarchy
Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'smartServiceClient' defined in class path resource [com/lalou/jonathan/rmi-client-spring.xml]: Invocation of init method failed; nested exception is org.springframework.remoting.RemoteLookupFailureException: JNDI lookup for RMI service [rmiServices] failed; nested exception is javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3://my-f5-frontal.my.domain.extension:7090: Bootstrap to my-f5-frontal.my.domain.extension/111.222.012.123:7090 failed. It is likely that the remote side declared peer gone on this JVM]

Explanation and Fix

Owing to my understanding, here is the point: when the client tries to connect to F5, it presents itself with its name, and the F5 returns the name of the actual server. If both client and server are not on the same domain (“domain” as network domain, no link with Weblogic domain), then the DNS resolution may fail.

To fix this issue, you have to follow one or both of the following points: everything depends on your local topology.

WebLogic: “Listen Address”

Modify the “Listen Address” in WebLogic administration console, from home: Servers > MyFirstNode/MySecondNode > Configuration > General > Listen Address > update it

By “update” the “Listen Address”, I mean providing the complete name of the machines, including the domain extension.
eg: my-first-node.my.domain.extension and my-second-node.my.domain.extension, rather than my-first-node and my-second-node (or, even worse, localhost).
You can also provide an IP, cf. WebLogic documentation on Oracle’s website.

Of course, you can decide to set it directly in WebLogic’s config.xml.

Caution! This option may also be set via the command line running WebLogic, using the flag -Dweblogic.ListenAddress=... Therefore, take care to be consistent between the content of console/config.xml and the command line option.

Hosts

On client side, check the content of hosts file. Usually, you can found it at /etc/hosts (or C:\WINDOWS\system32\drivers\etc\hosts on Windows XP).
Assuming your machine is myClientMachine with an IP 123.123.123.123 and a domain extension remote.domain, then your hosts file should look like:

127.0.0.1       localhost
123.123.123.123  myClientMachine

Update it to:

127.0.0.1       localhost
123.123.123.123  myClientMachine myClientMachine.remote.domain

Posted in en-US | Tags: F5, It is likely that the remote side declared peer gone on this JVM, RMI, Spring, t3, Weblogic, Weblogic 10 | 1 Comment »

RMI / Spring / Cannot narrow remote object ClusterableRemoteRef

Author: Jonathan Lalou

Case

Under WebLogic, I deploy RMI services:

    <bean id="fakeRmiServer" class="org.springframework.remoting.rmi.JndiRmiServiceExporter">
        <property name="service" ref="fakeInterfaceImpl" />
        <property name="serviceInterface" value="lalou.jonathan.FakeInterface" />
        <property name="jndiName" value="fakeRmiServer" />
    </bean>
    <bean id="fakeInterfaceImpl" class="lalou.jonathan.FakeInterfaceImpl" />

The WAR is hosted by a WebLogic 10.3.3 server (this point does not matter).

I retrieve the RMI services on client side, with this Spring config:

    <bean id="fakeServiceClient" class="org.springframework.remoting.rmi.JndiRmiProxyFactoryBean">
        <property name="jndiName" value="fakeRmiServer"/>
        <property name="jndiEnvironment">
            <props>
                <prop key="java.naming.factory.initial">weblogic.jndi.WLInitialContextFactory</prop>
                <prop key="java.naming.provider.url">t3://localhost:7003</prop>
            </props>
        </property>
        <property name="serviceInterface" value="lalou.jonathan.FakeInterface"/>
    </bean>

On launching the client side, I get this error:

java.lang.ClassCastException: Cannot narrow remote object ClusterableRemoteRef(-2462835584319760815S:123.123.123.123:[7003,7003,-1,-1,-1,-1,-1]:JonathanApplication:JonathanAdminServer [-2462835584319760815S:123.123.123.123:[7003,7003,-1,-1,-1,-1,-1]:JonathanApplication:JonathanAdminServer/375])/375 to lalou.jonathan.FakeInterface

Complete Stacktrace

java.lang.ClassCastException: Cannot narrow remote object ClusterableRemoteRef(-2462835584319760815S:123.123.123.123:[7003,7003,-1,-1,-1,-1,-1]:JonathanApplication:JonathanAdminServer [-2462835584319760815S:123.123.123.123:[7003,7003,-1,-1,-1,-1,-1]:JonathanApplication:JonathanAdminServer/375])/375 to lalou.jonathan.FakeInterface

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'fakeServiceClient' defined in class path resource [lalou/jonathan/java/webservices/jonathan-services.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [lalou.jonathan.java.webservices.FakeServiceWSServer]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'fakeServiceClient' defined in class path resource [lalou/jonathan/java/webservices/jonathan-rmi-client-spring.xml]: Invocation of init method failed; nested exception is org.springframework.remoting.RemoteLookupFailureException: Could not narrow RMI stub to service interface [lalou.jonathan.FakeInterface]; nested exception is java.lang.ClassCastException: Cannot narrow remote object ClusterableRemoteRef(-2462835584319760815S:123.123.123.123:[7003,7003,-1,-1,-1,-1,-1]:JonathanApplication:JonathanAdminServer [-2462835584319760815S:123.123.123.123:[7003,7003,-1,-1,-1,-1,-1]:JonathanApplication:JonathanAdminServer/375])/375 to lalou.jonathan.FakeInterface
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:883)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:839)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:440)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:221)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:429)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:729)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:381)
        at org.apache.cxf.transport.servlet.CXFServlet.loadAdditionalConfig(CXFServlet.java:171)
        at org.apache.cxf.transport.servlet.CXFServlet.updateContext(CXFServlet.java:139)
        at org.apache.cxf.transport.servlet.CXFServlet.loadSpringBus(CXFServlet.java:101)
        at org.apache.cxf.transport.servlet.CXFServlet.loadBus(CXFServlet.java:70)
        at org.apache.cxf.transport.servlet.AbstractCXFServlet.init(AbstractCXFServlet.java:78)
        at (...)
        at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
        at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:736)
        at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
        at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
        at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
        at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
        at org.mortbay.jetty.plugin.Jetty6PluginWebAppContext.doStart(Jetty6PluginWebAppContext.java:115)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
        at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
        at org.mortbay.jetty.Server.doStart(Server.java:224)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.plugin.Jetty6PluginServer.start(Jetty6PluginServer.java:132)
        at org.mortbay.jetty.plugin.AbstractJettyMojo.startJetty(AbstractJettyMojo.java:454)
        at org.mortbay.jetty.plugin.AbstractJettyMojo.execute(AbstractJettyMojo.java:396)
        at org.mortbay.jetty.plugin.Jetty6RunWarExploded.execute(Jetty6RunWarExploded.java:170)
        at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:490)
        at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:694)
        at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandaloneGoal(DefaultLifecycleExecutor.java:569)
        at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:539)
        at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:387)
        at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:348)
        at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:180)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:362)
        at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:60)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
        at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
        at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
        at org.codehaus.classworlds.Launcher.main(Launcher.java:375)

Fix

Indeed, this issue appeared with Spring 2.5.4, and was fixed with 2.5.6 release. I let you guess which version I was using…
(cf. Spring 2.5.6 Changelog: * JndiRmiClientInterceptor skips narrowing for RmiInvocationHandler stubs (fixing a regression in 2.5.4)
Therefore, to fix this issue you have to upgrade your Spring version to 2.5.6.

Posted in en-US | Tags: RMI, Spring, Weblogic | No Comments »

April 2024
S	M	T	W	T	F	S
« Sep
	1	2	3	4	5	6
7	8	9	10	11	12	13
14	15	16	17	18	19	20
21	22	23	24	25	26	27
28	29	30