Inspired by Charles Nutter's talk on the JIT's optimizations I mentioned a couple of days ago, I designed a little benchmark to learn more about the JIT optimizations. Being curious if the Java performance improved in the last years, I ran the benchmark with every major JDK version since JDK 1.2.
Before continuing I should emphasize that the my benchmark[1] is a microbenchmark. It's more complicated than the average microbenchmark[2], but basically it is just a little bubblesort algorithm. I suppose my benchmark is a good benchmark when it comes to areas like number crunching, but I don't know if you can apply it's result to everyday desktop applications. I suppose you can do so, but only to a limited extend.
32 bit Java
That said, I observed a steady performance improvement with every new version of the JDK:
The y-axis is the time elapsed during each loop of the benchmark in milliseconds. The x-axis depicts the number of repetitions of the benchmark. The primal intention of the benchmark was to verify if the JIT optimizes the program while it runs. It does indeed, as you can see clearly in the diagram. It did so since the introduction in JDK 1.2. Unfortunately, the times measured are imprecise a the left hand side, because the JDKs prior to 1.5 didn't offer the time resolution needed to measure the performance of a single loop of my benchmark.
Nonetheless you can observe both the effect of JIT compilation as well as the performance gains of the newer JDK versions. Java 7 passes my benchmark approximately three times faster than Java 1.2 does.
64 bit Java
Things became a little confusing when I looked at the 64 bit version of Java 7:
On the long run the 64 bit version of Java 7 is much faster than its 32 bit counterpart. It passes each reperition of the benchmark in 46 microseconds, while the 32 bit version take 135 microseconds and Java 1.2 takes 409 microseconds. However, 64 bit Java seems to be a slow starter: it needs 20 repetitions to overtake even Java 1.2.
When I run the benchmark using the JVM flags -XX:+PrintCompilation and -XX:+PrintInlining, I noticed that the 64 bit JIT seems to optimize the code more aggressively that its 32 bit companion. What puzzles me is that almost every web page I looked at when I prepared this article claims that 64 bit Java is slower because of memory access bandwidth limitations.
Nonetheless, 64 bit Java also runs Barak Cohen's raytracer faster than its 32 bit counterpart, at least on my PC.[3] However, the effect is far less impressive: 24 seconds versus 39 seconds. Complex images benefit more from the 64 bit version than simple images that need less number crunching.
However, when I measured the time Liferay needs to start up at my office PC, I noticed no difference at all.
My numbers seem to indicate that the 64 bit version of JDK 7 is optimized for large-scale number crunching or server tasks, while the 32 bit version is better suited for smaller and more diverse tasks that are typical to desktop applications.
Conclusion
Over the years there have been considerable improvements of the JVM. Usually one doesn't observe this because the difference between two consecutive versions are small, but they sum up over the years.
My numbers are a little confusing when we look at 64 bit Java. I can't tell you whether it pays to switch to 64 bit or to stick to 32 bit. I observed a factor four penalty on the short run and a factor three gain on the long run. So I guess it depends on you application, and probably also on your OS and maybe even on your CPU. I'd rather you ran your own tests :) .