Just before I started my job at the BBC, one of our developers committed code which reduced our test suite run time from an hour and twenty minutes down to twenty-two minutes. One of my first tasks was to improve that. However, improving performance begs the old question of “cpu or developer performance?” Both are equally important, but I’ll just talk about making the tests run faster. Right now, it looks like we’re on track to get our test suite to run in under ten minutes. Here’s how we did this.

Don’t Recreate the Database

The biggest win, of course, was to reduce the 80 minute runtime to 22 minutes. The problem was that many tests would set up and tear down the entire test database. While this is, in theory, a clean solution, the trade off is that it’s very slow. The solution was to divide tables into ’static’ tables whose data does not change or changes infrequently (ISO country codes, for example) and ‘dynamic’ tables which change constantly (customer orders). Each test program which uses the database creates a database handle and when the handle goes out of scope it does the following:

1. Disable foreign keys
2. Truncate all dynamic tables
3. Enable foreign keys
4. Scream if any static tables have changed

That was the bulk of the savings and for many slow test suites, this could be enough. However, we still had a 22 minute test run and we were targeting 10 minutes. With five developers, every time they each run a test suite, that’s an extra hour of waiting for information.

Don’t Duplicate Tests

The next step was examining our tests. I noticed that we had many tests in the form of test/yaml pairs and they looked like this:

t/unit/customer.t
t/unit/customer.yml
t/unit/order.t
t/unit/order.yml

Each of the .t files would look like this:

use Our::Test::API;
Our::Test::API->run;

Or this:

use Our::Test::System;
Our::Test::System->run;

Each of those classes (we had several, but different names from what’s here) inherited from Our::Test and used the name of the test program to determine the name of the YAML file. I deleted all of them and made one test program. I turned Our::Test into a factory which returned the correct class for the YAML being used.

use Our::Test;

foreach my $yml (get_yaml_tests()) {
    Our::Test->new($yml)->run;
}

The test suite now ran in 16 minutes.

How did that save six minutes? Well, instead of invoking the perl interpreter for a bunch of test files and reloading all of the related classes every time, perl was invoked once and each class was loaded only once. This is similar to how Perl’s Test::Class can dramatically improve a test suite’s performance.

When factoring out duplicate code like this, you’ll also often discover that you’re factorinng out duplicate tests. Running the same test more than once is often meaningless.

Don’t Reload Anything

Now that the obvious performance issues were out of the way, I need to figure out how to get squeeze out an extra six minutes of performance. I had thought a lot about using Test::Class because it’s an excellent tool, but this meant rewriting a bunch of working tests and our performance wasn’t bad enough that I could really justify that. So I started writing Test::Aggregate. This doe something very similar to what the YAML tests did but in a more generic manner.

The idea is to slowly move your tests, one by one (this eases debugging) into a separate test directory and write a driver test like the following:

use Test::Aggregate;
my $tests = Test::Aggregate->new( {
    dirs          => 'aggtests',
    set_filenames => 1,
} );
$tests->run;

What this does, effectively, is concatenate all of the tests together into one program and run them all at once. Of course, there’s a lot of magic under the hood to make this work, but just moving a few test programs over to this gained us an extra two minutes. One experimental branch had the test suite completing in under nine minutes (with a couple of strange failures). Not bad for being at 80 minutes a couple of months ago, eh?

Sometimes It’s OK to Fail

For diehard testers like myself, that sounds like heresy, but it’s true for a very limited set of circumstances. For example, we have one very slow test which loads every module and verifies that it uses strictures. While this is an important thing to do, disabling strictures does not automatically mean your software is broken. These tests are very slow and we’re likely to move them into a separate directory which only gets run during integration.

We also have tests which verify that our documentation is complete and properly formatted. If these tests fail, it doesn’t mean that the software is broken. These tests can also be moved into a separate ‘integration’ directory. While developers should still run all tests before checking in their code, it’s more important that they run those tests which really check that the software is doing what it should.

Using this and previous techniques, it’s possible we’ll have our core test suite running in under seven minutes.

Why Is This Important?

Why is a fast running test suite so important? At one company I worked with, the test suite took an hour and a half to run and developers would often check in code when they verified that their tests passed. Running the entire test suite took so long that they skipped it. More than once bugs got into production because no one was running the entire suite!

The faster your test suite runs, the more likely developers will run it. Proper test management is key to developing good software and this is one of the many areas developers should focus on.

In a later post, I hope to document ways that developers can develop tests easier (it will be far more Perl-specific, though). Just as a faster test suite is more likely to be run, easy-to-write tests are more likely to get written.