JMX Lesson

JMX is an interesting technology which some of use to view vm metrics and to perform centralization of system attributes. In the last few days a new project was started to introduce JMX to a core set of components within my current project.

In this project a problem arose where a jconsole running could not connect to a JMX server running on a flavor of linux. I was able to resolve the issue and wanted to post the solution here so I won't forget and hopefully help someone else who may find themselves in the same spot.

The symptom of the problem is a JMX server running on windows and a jconsole running on windows works without any issue. Once you deploy the JMX server on linux and run a jconsole on windows you get the following error:

As you can see the problem seems a bit frustrating because there are no real reasons giving for the problem. So my first step was creating a logging.properties which I pulled the sample from Daniel Fuchs (Daniel has lots of good JMX information on his blog). My logging.properties looks like:

handlers= java.util.logging.ConsoleHandler.level=DEBUG
java.util.logging.FileHandler.pattern = java%u.log
java.util.logging.FileHandler.limit = 50000
java.util.logging.FileHandler.count = 1
java.util.logging.FileHandler.formatter = java.util.logging.XMLFormatter

java.util.logging.ConsoleHandler.level = FINEST
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter

// Use FINER or FINEST for javax.management.remote.level - FINEST is
// very verbose... //
javax.management.level=FINEST
javax.management.remote.level=FINER

Once I created the logging.properties I started the jconsole using the commandline:

jconsole -J-Djava.util.logging.config.file=c:\logging.properties (make sure you change the path to match your location). With logging on I noticed the stack trace with the following error:

FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi: ///jndi/rmi://19.16.14.15:9090/jmxrmi] failed to connect: java.rmi.ConnectEx ception: Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused: connect

Please notice the error contains a loopback address for the host. If your asking yourself how can this be since I explicitly asked to use 19.16.14.15, well the issue lies with how the jmx server looks up its own ip address. If your linux machine is not setup to use the proper named look up you end up with the /etc/host entry which by defualt is 127.0.0.1.

So, now that I have determined I need to get the stub to return to the client with the proper address I found a great forum post on the SUN Forum. The post had the same issue and provides a linux method to resolve the issue. Basically start the jmx server with the following jvm start up param:

-Djava.rmi.server.hostname=`ifconfig eth0.1:1|grep "inet addr:" | sed "s/inet addr://" | awk '{print $1}'`

what this does is make sure the JMX server knows what address to register with so when a request for the stub is made it uses the right IP address and not 127.0.0.1. Please run the command on your system before start up to make sure you get your IP address. You may have to leave the .1:1 off the eth0.1:1 if you are not using more than one IP.

Once I started the jvm with the above command I was able to connect from windows to the linux box.