advertisement

Weblog:   The Python comunity has too many deceptive XML benchmarks
Subject:   deceptive benchmarks
Date:   2005-01-24 08:34:35
From:   faassen
The benchmark in this article is indeed rather deceptive, as the effbot said.


You definitely want to use time.time() or something like that *inside* the program to avoid measuring the Python startup and shutdown time, which is hardly relevant, unless you build web applications using CGI scripts. :)


That said, it is good idea to communicate our benchmarking strategy. What I did when I was curious what Fredrik used was just mail him and ask him. I've been using the same strategy as a result. I get numbers slightly different from his, though not drastically. It's likely due to platform/compiler differences (I'm on Linux, he's on Windows). See my weblog for some numbers:


http://faassen.n--tree.net/blog
Full Threads Oldest First

Showing messages 1 through 3 of 3.

  • deceptive benchmarks
    2005-01-24 12:09:20  effbot [Reply | View]

    I posted a rather long description of my test approach, and the issues involved in benchmarking extremely fast Python extensions that consume large amounts of memory, to the xml-sig in the middle of January. Unlike Uche, I know what I'm measuring.

    (fwiw, the virtual debunking team currently suspects that Uche has done most or all of the following mistakes: included Python startup and shutdown times in his figures, included module load times in his figures (cET 0.9 can parse OT.XML nine times in the time it takes Python to load Amara's bindtools component), sent output to a terminal instead of a file or /dev/null, used non-idiomatic solutions for the SAX and cET samples (for cET, Uche's code is 40% slower than the most obvious solution), and, quite possibly, used an unreleased version of the underlying cDomlette library, which is reportedly 3-4 times faster than the current release. And yes, the pystone figures don't seem to match his hardware description, either. This article should be archived in the "the whole bloody breakfast on my face" category, and replaced with an apology.)
    • deceptive benchmarks
      2005-01-24 14:51:55  faassen [Reply | View]

      It's indeed a surprisingly slow pystone rating for a dell inspiron 8600, 1700 megahertz. I'm on one now (with only 512 megabytes of RAM) and I get the following:

      Pystone(1.1) time for 50000 passes = 1.3
      This machine benchmarks at 38461.5 pystones/second

      perhaps some CPU scaling was going on on Uche's so it wasn't running at the full 1700 mhz?
      • deceptive benchmarks
        2005-01-24 15:08:22  effbot [Reply | View]

        38461 pystones/s matches the observation made here (37594 pystones/s). weird.

        but of course, running the Amara tests at a higher clockspeed, and the pystone/sax/cet tests at a lower speed, might also explain the 3X slowdown.
Showing messages 1 through 3 of 3.