TAP::Parser is the intended replacement for the venerable Test::Harness module. The intent is to clean up the code in such a way that writing custom test harnesses and supporting new TAP features is possible. (I’ve hacked on Test::Harness:: Straps; it wasn’t the easiest programming task I’ve ever tackled.)

I added TODO tests to Parrot’s test tools a while ago, to make it easier to distinguish expected test failures from accidental failures. Unfortunately, Test::Harness displays very little information about TODO tests that passed. In TAP terms, these are bonus tests. The programmer expected them to fail, but they actually passed, so they need further investigation.

It would be nice to collect information on skipped, TODO, and bonus tests in the normal test run. Though I could write a harness via Test::Harness::Straps, I decided to try TAP::Parser instead. Here’s what I discovered.

Installation

The installation process was painless, through the CPAN shell. There was one prompt; it asked whether to install the runtests program. This is the equivalent of prove in Test::Harness. The tests ran quickly and without error.

Investigation

My next work was to skim the documentation. The starting point is TAP::Parser. Create a new TAP::Parser object and optionally pass in arguments:

  use TAP::Parser;
  my $parser = TAP::Parser->new({ tap => $tap_stream });

For my initial tests, I wrote a very bare-bones TAP stream with passing and failing and TODO and bonus tests and stored it in $tap_stream. The documentation originally led me to believe that passing the stream as source would suffice. However, TAP::Parser complained that that file did not exist. I switched to tap and things just worked.

Given a source of TAP, TAP::Parser uses an iterator-style interface to provide access to test results:

  while ( my $result = $parser->next() )
  {
      # do something...
  }

$result is a subclass of TAP::Parser::Result. To find only TODO tests, I stacked some method calls:

      next unless $result->is_test();
      next unless $result->has_todo();

Some $result objects represent diagnostic messages and most represent normal passing tests. Given a $result that is a TODO test, it’s easy to see if it’s a bonus test and, if so, get the explanation:

      next unless $result->is_actual_ok();
      my $explanation = $result->explanation();

After processing all of the tests (or all of the TAP), there are summary methods available on the parser. The todo() and todo_passed() methods on $parser return the number of TODO and bonus tests in scalar context, and a list of the test numbers in list context:

  my @todo  = $parser->todo();
  my @bonus = $parser->todo_passed();

Harnessing the Information

With all that knowledge, I turned my attention to TAP::Harness. This code is the wrapper around TAP::Parser and performs analysis and reporting. It’s easy to begin:

  my $harness = TAP::Harness->new();
  $harness->runtests( @test_filenames );

The documentation led me to believe that calling aggregate() on $harness after this point would return a TAP::Parser:: Aggregator object. It didn’t, so I asked the developers; TAP::Harness 0.52 and later return the aggregator from the runtests() call.

The aggregator contains results of all of the tests run in the process. runtests() actually runs the tests and displays brief results. The parsers() method gives access to all of the aggregated parsers:

  my @parsers = $aggregate->parsers();

Originally I thought that I could use a while loop and call the next() method on each, but reading that documentation more closely revealed that a TAP stream is actually a stream. Once you’ve processed a line in the stream, it’s gone.

After I discovered that, I rethought my approach. The aggregator has summary methods that allow you to find all test files with TODO, skipped, and failed tests. Again, the documentation mislead me a little bit. It refers to descriptions, which is an identifier related to a TAP source. In a TAP stream, individual tests have optional descriptions. For my purposes, descriptions are the names of tests files.

  my @todo    = $aggregate->todo();
  my @skipped = $aggregate->skipped();
  my @failed  = $aggregate->failed();

Passing the filenames to the aggregator’s parsers() method returns a list of parsers for the test files. From there, I can call the todo() and skipped() and todo_passed() methods on each parser and report the appropriate test numbers.

The only remaining problem was that the summary from TAP::Parser was more detailed than I wanted in some ways and in others.

A Final Solution

The documentation for TAP::Parser suggests that it’s much more subclassable than is Test::Harness. I decided to test this by subclassing it and overriding the summary() method.

Here’s the final code:

package Parrot::TAP::Harness;

use base 'TAP::Harness';

sub summary { my ($self, $args) = @_; my $aggregate =
$args->{aggregate};

    my %seen;
    my @descriptions;

    my $total  = $aggregate->total();
    my $passed = $aggregate->passed();
    $self->output( "\nTest summary\n------------\n" );

    for my $desc ( map { $aggregate->$_() } qw( skipped todo failed ) )
    {
        push @descriptions, $desc unless $seen{$desc}++;
    }

    for my $description ( @descriptions )
    {
        my ($parser)    = $aggregate->parsers( $description );

        my @fail        = $parser->failed();
        my @todo        = $parser->todo();
        my @skip        = $parser->skipped();
        my @todo_passed = $parser->todo_passed();
        my $seen_header = 0;

        $self->summarize( \@fail,        ' Fail', $description, \$seen_header );
        $self->summarize( \@todo,        ' TODO', $description, \$seen_header );
        $self->summarize( \@skip,        ' Skip', $description, \$seen_header );
        $self->summarize( \@todo_passed, 'Bonus', $description, \$seen_header );
    }
}

sub summarize
{
    my ($self, $test_numbers, $label, $description, $seen_header) = @_;
    return unless @$test_numbers;

    my $pad_len     = length( $description ) + 3;
    my $padding     = " " x $pad_len;

    $self->output( "$description:\n" ) unless $$seen_header++;
    $self->output( "  $label: ", join( "\n$padding",
        $self->balanced_range( 80 - $pad_len, @$test_numbers )), "\n" );
}

The only unclear code is the use of $seen_header in the summarize() method. There may be a cleaner way to print data only for the relevant test files, but this worked for me so far. I don’t think too much of the padding and alignment code, but it was a quick proof of concept and did the job.

Conclusion

I’m impressed with how much cleaner TAP::Parser is than Test::Harness. The last time I used Test::Harness::Straps, I had to know way too much about its internals. Barring a few confusing parts of the documentation, I’m reasonably satisfied with how easy it was to write this code.

The documentation mentioned callbacks on certain types of test results, but it looked somewhat inapplicable to this situation. Perhaps it was the right approach, but I’m not sure.

I dislike somewhat overriding summary() completely, but it’s a large method in the harness as it is. I don’t know how feasable it is to extract several smaller methods for easier overriding, but that may have made my life simpler.

Though it looks like TAP::Parser will eventually become the new version of Test::Harness, its biggest benefit right now is for people who want to write their own test harnesses, for reasons similar to mine. If you’re merely a user of Test::Harness, this distribution won’t do much for you at the moment. Within a year, I expect that you may use tools built on it, however.