My first patch to Perl 5 was a quick and dirty tiny feature enhancement. It also broke a couple of tests. That small act of public humiliation reinforced what I already knew was a good practice; automated testing is an important part of creating software that works.
I spent a couple of years chasing two goals. First, to create tools of such quality and ease of use that there’s no reason not to write good tests for Perl code. Second, to add tests so that we could immediately identify regressions and track them down to specific checkins.
Today, the core Perl 5 test suite (as of the most recent snapshot leading to Perl 5.8.9) has 121873 assertions. It could use more, but those tests cover the language and core libraries. Modern CPAN distributions are incomplete without tests written in the modern style, and test coverage and quality are topics of wider understanding and discussion.
I can’t imagine relying on a piece of software that I can’t verify with automated tests.
Analyzing the JRuby test suite, a recent weblog post from Christian Neukirchen stunned me. The most complete test suite for Ruby is the JRuby test suite and it has only 2747 assertions.
I know Ruby’s simpler than Perl, but it’s not that much simpler.
Comparing Test Assertion Counts
How many assertions does the core Ruby test suite have? If you build Ruby 1.8.6 and run
make test, 867 assertions run. (To be fair, there is a
make test-all target which runs library tests too.)
Perhaps there’s a better comparison. Perl 5’s test suite runs the standard library tests for
make test. The language tests alone have 64793 assertions. That’s almost 75 times as many tests as Ruby 1.8.6.
By way of comparison, Perl 1, released in 1987, had 328 assertions. Parrot has about 7200 (see Parrot smoke reports. Perl 6 has around 18700 (see Pugs smoke reports. Neither Pugs nor Parrot is complete. Both numbers include some library tests.
I didn’t find an easy way to count the number of assertions in the Python test suite, so I used a very quick and dirty
grep in the Lib/ subdirectory of Python 2.5.1:
$ grep 'assert' $( find . -name '*.py' ) | wc -l 11437
That’s 13.2 times as many assertions as Ruby. However, the Python numbers do include tests for the standard library.
Now I’ve only shown raw numbers here, and I hope I’ve documented well where I’ve made assumptions. If I’ve missed test files that I should count, I’m happy to count again. I’ve said nothing about the relative quality of any implementation or language. I’ve only given you the numbers as I found them in a couple of hours of research on x86 GNU/Linux. (Different platforms may have different numbers of assertions; Perl 5’s 64-bit tests ran zero assertions for me.) I don’t mean to assert a complete correspondence between the number of assertions and the quality of an implementation. That’s ridiculous, and it’s easy to let such numbers mislead you.
I also don’t know how many tests it takes to have enough tests for a language or a language feature. Still, I find it difficult to believe that Ruby has adequate test coverage with 867 or even 2747 assertions–Perl 5 has several times that many tests just of the
unpack operators. (I counted four assertions for Ruby’s