Benchmarking the CLR vrs the JVM

I've found that there are lots of flame wars regarding JVM/CLR performance, but very little solid data, so I thought I'd try running a benchmark of my own.

I decided the first benchmark would be sorting using a bubble sort. The bubble sort is highly inefficient, so it is a good demonstration of memory management and integer arithmetic.

I also wanted to see how well the JIT's would handle method by method optimization, so I wrote a version that used methods for each runtime, as well as a version that was entirely inline.

My theory was that the inline version would prove to be slower than the method invoking versions due to JIT optimizations when calling methods. In reality, it was even more apparent. However, when I did disable the JIT optimizations on the CLR by using a debug build, the inline CLR version was much faster than the method version.



This data was generated on my 2.4ghz laptop running Windows XP. I used the .NET 2.0 CLR, and JRE 1.6.0.5 (Which just happened to be the one that came with Eclipse). The Y axis is milliseconds.

The benchmark is performing a bubblesort on a 10,000 integer set, 100 times.

It shows that the CLR is a slight winner, at least in doing these operations. The big surprise was the speed difference the JVM JIT makes when optimizing methods.

Even the unoptimized CLR version was only slightly slower than the JVM.

The surprising lesson learned here is that premature optimization by inlining methods and unrolling loops may actually result in slower performance on the JVM.

If anyone wants to run these benchmarks on their machine, the full source is here.