The Pet Store.... Again

I've been writing Java code for over 6 years now, specifically in the back-end server space. (I was using the NetDynamics product as far back as late 1996, when JDK 1.0 as the norm.) I've been writing about Java since 1998, and teaching about Java since 1999. I've also been writing .NET code for over 2 years now, specifically in the back-end server space. I've been writing about .NET since 2001, and teaching about .NET for about the same amount of time. Living in both worlds gives me a really interesting perspective, both politically and technologically; I'm a member of the JSR-175 Expert Group, to bring metadata annotations into Java, and I'm writing a book on Rotor, the Shared Source .NET runtime implementation. I love Java, I love .NET. I want both to thrive, because the innovations produced as a result of competition between the two benefit us all.

Normally, I really really try to avoid the whole benchmarking thing--it's a black hole that ultimately does nobody any good. People ask me in my classes (J2EE or .NET, doesn't matter) which technology is faster, Java or .NET. I tell them that I've never run a benchmark, because I believe the variables between the two camps are too many and too diverse to accurately benchmark, and that ultimately it doesn't matter because the choice of which technology to use is usually made on a golf course or over a fancy dinner, not in a benchmarking lab. DevelopMentor, the company I teach for, went through this "should we do a benchmark to compare the two" discussion about two years ago, and we all almost unanimously voted against it, for many of the same reasons.

When Microsoft released their version of the Pet Store a year or so ago, I was moderately amused--they'd rewritten the sample to use what they believe to be .NET best practices (which at this point is realistically anybody's guess), and behold! Massive performance improvements. The Java community as a whole ridiculed the effort, and several J2EE app server vendors responded by writing their own versions of the Pet Store demo, and behold! Massive performance improvements. Nothing new here.

But now, The Middleware Company, a reputable Java and J2EE training and consulting company (and, I must, in the interests of full disclosure, point out, a rival to DevelopMentor) has taken the plunge and attempted to produce a benchmark that seriously compares J2EE and .NET on the subject of the Pet Store once again. They published their results last week, and, to put it politely, all Hell broke loose.

First and foremost, I want to point out that I deeply respect the pure cojones of The Middleware Company (TMC) in producing this--they knew, when they saw the results, what the reaction within the industry would be, and they published the results anyway. I have to believe that this was in the honest interests of trying to toe the careful line of honest reporting--either that, or TMC has decided to force themselves into bankruptcy. It simply makes no sense for them to do anything otherwise (unless they've suddenly decided to become a .NET training company, which I doubt). So I think it's important to give TMC some props on that score. I believe they did try to make it as fair of a benchmark as they could, and they did achieve a serious performance improvement (17 times better than Sun's original implementation, according to the report (p. 7). Whether they did everything they possibly could have done is another issue, addressed later.

In the meantime, however, it seems as if everybody within the industry, particularly the Java guys, have been leaping to their feet to decry the results of the benchmarks. In basketball, we'd be seeing a bench-clearing brawl right about now. More vitriolic poison has been slung between posters on message forums over this benchmark than in the last U.S. Presidential election--it's amazing.

As someone who lives and breathes in both camps, I feel compelled to offer my own spin on things, to anybody who's still listening. In particular, I want to address some comments made by Dion Almaer, in his OReilly weblog, and Rickard Oberg, in his online website review of the benchmark. I've never met either personally, but I have exchanged email with Rickard (a number of years ago). I deeply respect both.

Comments

• Dion points out that "What was the point of this report/benchmark? It seems unclear to me. Was it just about performance? Was it about ease of development (LOC)? It blurs." Absolutely. The LOC measurement in this benchmark definitely dilutes the results of a performance-oriented benchmark, and vice versa. More importantly, I'd like to see a benchmark that drills down even further--how would you optimize the J2EE/.NET application to optimize for performance as opposed to scalability? Or even the numbers when "optimizing" for design purity (which Dion touches on later)? The benchmark doesn't state clearly what it's trying to measure, and as such, leaves us with a sense of confusion regarding what we're trying to measure here.
• Dion writes that "M$ could even tweak their .NET core! Do you think you can download that .NET runtime right now?" Dion, I hate to say this, but that's a spurious argument and a foolish one to rely on. Do you honestly believe that Microsoft wrote a specialized version of the runtime just for this benchmark? I can honestly say that I've met a number of the guys on the CLR team in Redmond, and let me reassure you, they have MUCH bigger things to deal with than this benchmark. More importantly, I challenge anybody out there to download the Rotor source and point out what optimizations could be made to the .NET runtime (or to the JVM, for that matter) that would seriously impact the performance of this benchmark. This is zealotry talking, and we don't need that here.
• Dion says that the report "does not fully disclose who did the report, who was involved in the benchmarking, why certain "rules" were chosen, and who paid for it." This is a fair criticism; in addition, both Dion and Rickard point out that the way this thing played out, it *really* looks like it was skewed from the beginning to favor the Microsoft guys. I would think that, given the leak of the email from Microsoft showign that they knew about the benchmark results before it was posted, TMC at least owes an apology the community at large, *ASSUMING* they presented it to Microsoft before the results. In fairness to TMC, however, we must consider the possibility that they never actually presented the results to Microsoft--just as Rickard got the internal Microsoft email, it's fair to consider the possibility that somebody within TMC leaked the benchmark results to Microsoft, meaning TMC wasn't giving Microsoft a heads-up after all. Check out the exact verbase of the email: "The week of Monday Oct 21, a draft of the Middleware report was distributed to analsysts during private 1:1 analyst pre-brief meetings with...." NOWHERE does it say that TMC gave the report to Microsoft, or was even present at these meetings.
• A number of people have criticized that the benchmark wasn't run on Solaris or Linux. The benchmark report states, quite clearly, that "Both application servers were tested on both RedHat Linux 7.2 and on Windows 2000 Advanced Server (SP2)", and that TMC "used the operating system that provided the best performance with that application server for the final, published test runs." Should the tests run on Solaris, as a comparison? Absolutely. Will it be hard to find "comparable" hardware to run the tests on? Not at all--just buy whatever equivalent hardware I can get for the same price as the Compaq hardware used to run the tests, at list prices.
• Dion writes that certain optimizations weren't part of the app, and lists Local Interfaces and other JVMs as examples of what wasn't used. Again, however, TMC states that these were considered, and that Local Interfaces turned out to be no faster than remote ones, since any sane App Server would optimize the Remote Interface under the hood anyway, and that JRockit was, in fact, used, or at least attempted to: AppServer B couldn't support anything beyond JDK 1.3. This in turn makes me question whether AppServer B should even have been used, but that's another story for another day. (See the FAQ.) Honestly, the fact that AppServer B doesn't support 1.4 tells me that the major App Server vendors aren't all that concerned with the performance of their products, either. (And what the heck are they doing internally that prevents the app from running on 1.4, anyway?)
• A number of people have criticized the J2EE PetStore implementation for being unoptimized. First of all, let's not forget that this served as Sun's "Blueprints" document for a number of years, offering advice on how to build scalable systems using EJB. If it's not optimized for performance, then Sun's been leading all of us down a dark and unoptimized path for four years now. Shame on them.

  However, I am really getting sick and tired of Java developers trying to justify EJB as being the way to scale up and perform well, then claiming that the PetStore, which uses EJB, isn't optimized to support scalability and performance. Folks, it's high time we took the blinders off, stopped drinking the Kool-Ade, and realize that EJB doesn't magically give you performance and scalability--it's your application implementation and design that gives you that.

  So, then, would somebody *PLEASE* step up, write the PetStore the way the community thinks it should be written (using stored procs, no EJB, whatever), so we can kill that original implementation and be done with it? If it's not written the way it should be, then Sun should remove all traces of the Pet Store from their site and release one that does. If they don't, then they're implicitly still backing the idea that the Pet Store is the blueprint to building EJB applications, and they deserve what they get when they get crucified in benchmarks.

• Rickard writes that the .NET implementation is "seriously flawed", in that the .NET implementation mixes SQL logic into the middleware componentry to do the actual data-access layer. "Look at, for example, Product.cs which contains both object definition of the Product class as well as the SQL/code to access it as static methods. That is not very "object oriented", now is it?" Frankly, as I've been trying to say for years, EJB and other distributed technologies aren't about building distributed objects, but about building distributed components, also known as services. So the argument that this approach isn't very object-oriented is, to me, a red herring.

  I think it's also worth pointing out that having the SQL in the Product class really isn't much different from having the SQL hard-coded in the ProductEntity bean (if it's a BMP bean) or in the deployment descriptor (if it's a CMP bean, thinking specifically of finders and/or selectors). I will point out that layering business rules in a SessionBean and data-access code in an EntityBean yields better segregation of code, at the expense of violating good O-O encapsulation between the two objects--so which do we want? Segregation of business rules from data access, or good encapsulation boundaries?

• Much of the rest of Rickard's assessment of the benchmark, particularly his dissection of the J2EE implementation, sheds some serious question on how the J2EE implementation was optimized. Rickard comments that Ed Roman has been calling various people, "trying to convince them that [Rickard's] review is basically written because [Rickard] is out to 'get' [TMC]." If this is true, then I'm seriously disappointed in TMC's response. I grant you, TMC is definitely in the hot seat, but I would hope that they would welcome an opportunity to retest the benchmark with some of the optimizations that the Java community is calling for. What I would *really* love to see, as I said before, though, is somebody to write a version of the Pet Store that we can all agree is the "right" one to use, and put this question of Pet Store's implementation to bed forever.

Summary

1. Java developers, .NET is a serious contender and is here to stay. Simply calling Microsoft names and writing the derogatory "M$" everywhere doesn't erase the fact that Microsoft has produced a viable alternative to Java, particularly on the Win32 platform. Stop being silly and closing your eyes to this. Instead, do what Microsoft does so well, and steal from them: steal ideas, steal concepts, steal designs, then twist them around to make your Java apps faster. .NET doesn't provide any kind of support for entity beans? Ask yourself why that is. .NET doesn't provide any kind of support for object pooling? Ask yourself why that is. I'm not saying that everything Microsoft does is perfect--Windows DNA is a prime example of that--but don't lull yourself into complacency by simply chanting "Microsoft sucks" to yourself and ignoring what's going on over there. There's a lot of bright people there in Redmond, and they've been thinking about this problem a long time.
2. .NET developers, don't be lulled into complacency, either. The various Java vendors are *not* going to take this benchmark laying down, and you can bet that the Java guys are going to come after it with both barrels blazing. In particular, there is a tremendous amount of good ideas that the J2EE world embraces that are somewhat awkward to do in .NET; one such case in point is the whole Model-View-Controller paradigm for building webapps. Pick up a Servlets/JSP book, and figure out how to bring that over to ASP.NET. Don't start thinking that you can just dribble SQL all over your .aspx pages and expect that the .NET runtime will somehow make your application "just run faster"--it isn't going to work.
3. Java developers, this benchmark is essentially an indictment of EJB, not of Java. Demand to see a benchmark that doesn't use EJB in the middle, using stored procedures against the Oracle database, built in an intelligent fashion that doesn't include dead code or redundant transactional boundaries. Think you need EJB to work against simultaneous distributed databases? Think again--most servlet containers (the principal one I'm thinking of is Tomcat) support JTA, which means you can do distributed transactions against multiple databases directly from the servlet container. Think this means you're losing the benefits of connection pooling? Not at all--the servlet container naturally acts as a connection-pooling funnel, particularly when coupled with connection-pooling JDBC drivers. (The same, remarkably enough, can be said for .NET and COM+.)
4. .NET developers, this benchmark is essentially an indictment of EJB, not of Java. Demand to see a benchmark that doesn't use EJB in the middle, so you can get a fair and unbiased comparison of how the Java bits work against the .NET bits. Be prepared to admit that there's really not a whole lot of difference between the two once we get down to that level.
5. Everybody, step back for a second and realize that at the end of the day, IT DOESN'T MATTER. This benchmark only proves that .NET turned out to be faster for this application in this environment for this particular test. Is your application precisely similar, in both architecture and implementation, to the Pet Store? It's a hideous leap of logic to assume that "because the .NET implementation of the Pet Store runs faster in the lab, my online human resources system will run faster in our data center"; once the code diverges at all from the benchmark, entirely new variables come into question that can seriously affect the results.

To me, the verdict is clear: run your own benchmarks, form your own conclusions. It would be nice if we could find somebody who's completely neutral to run a benchmark, but what company exists today, working in both spaces (so as to provide equal degrees of expertise) that doesn't have ties to Microsoft or Sun/IBM/Oracle/etc? Let's forget the benchmarks, analyze the code to find out why they each performed the way they did, and apply the lessons to our own systems.

Ted Neward is an independent software development architect and mentor in the Sacramento, California area.

oreillynet.com Copyright © 2006 O'Reilly Media, Inc.