This post introduces developer tests that constrain exceptions. Our platform, as usual, is Ruby, yet these topics apply to any system. We will extend assert_raise() for more control over program faults.

The best developers write tests to keep their projects on track. Tests should cover every aspect of a program, and should take special care with program details that are sticky, hard, and mysterious. Exception handling is a murky topic, because when a program fails its control flow might not be obvious. Many a program has failed in the field because nobody in the lab tested all its error paths. Our test cases must ensure that faults make our programs degrade gracefully, not derail.

Ruby programs throw their errors with raise object, and catch them with rescue type => variable. Consider a program with a risky method that might raise an error. A real test would use a “Mock Object“, to temporarily replace that method with a method that always faults. Mocks are a different topic, so we will just use a fanatical method, perpetually intent on destroying our program:

  def fanatic()
    raise 'bad news'
  end

You can neutralize it by guarding its statement with rescue:

  fanatic() rescue puts($!)

The $! is a link to the raised object, and puts prints out its bad news.

Now we write a test case to demonstrate our method’s fanaticism:

  def test_fanatic()
    fanatic()
    flunk('fanatic failed its mission')
  rescue RuntimeError
    # nothing
  end

flunk() is the assertion method that fails a test with a message. If control-flow reaches the flunk() line, the Ruby Unit Testing Framework will raise its own error type, Test::Unit::AssertionFailedError. The test runner catches that and converts it into a “Red Bar” - an error report.

The rescue RuntimeError statement won’t catch any other error type, so only the type we expect will allow the test case to pass.

assert_raise

Test::Unit wraps that test up in a convenient assertion, so this works the same:

  def test_assert_fanatic_raises()
    assert_raise RuntimeError do
      fanatic()
    end
  end

When a program faults, the best way to handle the situation is roll all program state back to its condition when the user started the last input. Good developer tests will help us write the correct manipulations.

Then the program should report its error message to the user. These error messages are high-level features; they might use text substitutions, and even localizations. We should test these manipulations directly, but ultimately we will need to see if the exceptions propagate them correctly.

This test detects our fanatic’s error message:

  def test_assert_fanatic_raises_good_message()
    exception = assert_raise(RuntimeError){  fanatic()  }
    assert_match /bad news/, exception.message
  end

assert_raise explicitly returns the exception object it caught, so we can then use assert_match to spot-check the message’s important details.

Exception messages seem important enough to deserve an assertion of their own.

assert_raise_message

Test::Unit does not contain this assertion, so we must split that test case into a simpler case and a reusable assertion:

  def test_assert_fanatic_raises_message()
    assert_raise_message RuntimeError, /bad news/ do
      fanatic()
    end
  end

  def assert_raise_message(types, matcher, message = nil, &block)
    args = [types].flatten + [message]
    exception = assert_raise(*args, &block)
    assert_match matcher, exception.message, message
  end

The curious operators around *args are Ruby’s way of manipulating a method’s arguments as an array. assert_raise_message takes either a single exception type, or an array of types inside [] operators. The [].flatten trick turns the single type into an array, to pass into assert_raise. That assertion can take more than one exception because sometimes the same fault might raise more than one exception type. For example, the timeout{} method will raise Timeout::Error if statements inside its {} block run for too long. But if those statements call database methods with ActiveRecord, and if these methods happen to rescue that error, they will decorate it with ActiveRecord::StatementInvalid and re-raise it. So assert_raise should take both error types.

  assert_raise Timeout::Error, ActiveRecord::StatementInvalid, ...

And assert_raise_message requires [], to distinguish its matching pattern:

  assert_raise_message [Timeout::Error, ActiveRecord::StatementInvalid], //, ...

Messages

A developer test should store information, and save it for after we forget about the current feature. A test could fail while we are working on other features, so every test assertion should accept an error message argument. Composing a descriptive error message now could save lots of time debugging later, so these error messages are like exception messages. Their user is you, the developer.

So lets use our new assertion to test that our new assertion raises its error message. This is bad logic, but its heart is in the right parsec:

    def test_assert_raise_message_raises_message
      assert_raise_message Test::Unit::AssertionFailedError, /ringer/ do
        assert_raise_message RuntimeError, /no raise me/, 'ringer' do
          fanatic()
        end
      end
    end

That test case would be typical diabolical cleverness … if its logic actually worked. But because it uses assert_raise_message to test itself, a single bug could create a symetric error in the test, and nothing would catch that.

Use assertions like assert_raise to ensure your programs unwind their stacks correctly, and break down any partly-constructed objects safely, at exception time. Your customers should not even notice them.