advertisement

Weblog:   The Python comunity has too many deceptive XML benchmarks
Subject:   deceptive benchmarks
Date:   2005-01-24 12:09:20
From:   effbot
Response to: deceptive benchmarks

I posted a rather long description of my test approach, and the issues involved in benchmarking extremely fast Python extensions that consume large amounts of memory, to the xml-sig in the middle of January. Unlike Uche, I know what I'm measuring.


(fwiw, the virtual debunking team currently suspects that Uche has done most or all of the following mistakes: included Python startup and shutdown times in his figures, included module load times in his figures (cET 0.9 can parse OT.XML nine times in the time it takes Python to load Amara's bindtools component), sent output to a terminal instead of a file or /dev/null, used non-idiomatic solutions for the SAX and cET samples (for cET, Uche's code is 40% slower than the most obvious solution), and, quite possibly, used an unreleased version of the underlying cDomlette library, which is reportedly 3-4 times faster than the current release. And yes, the pystone figures don't seem to match his hardware description, either. This article should be archived in the "the whole bloody breakfast on my face" category, and replaced with an apology.)

Full Threads Oldest First

Showing messages 1 through 2 of 2.

  • deceptive benchmarks
    2005-01-24 14:51:55  faassen [Reply | View]

    It's indeed a surprisingly slow pystone rating for a dell inspiron 8600, 1700 megahertz. I'm on one now (with only 512 megabytes of RAM) and I get the following:

    Pystone(1.1) time for 50000 passes = 1.3
    This machine benchmarks at 38461.5 pystones/second

    perhaps some CPU scaling was going on on Uche's so it wasn't running at the full 1700 mhz?
    • deceptive benchmarks
      2005-01-24 15:08:22  effbot [Reply | View]

      38461 pystones/s matches the observation made here (37594 pystones/s). weird.

      but of course, running the Amara tests at a higher clockspeed, and the pystone/sax/cet tests at a lower speed, might also explain the 3X slowdown.

Showing messages 1 through 2 of 2.