ONDotNet.com    
 Published on ONDotNet.com (http://www.ondotnet.com/)
 See this if you're having trouble printing code examples


Unit Testing in .NET Projects

by Jay Flowers
07/18/2005

If you didn't know it already, now is a pretty exciting time for unit testing in .NET. Tremendous progress is being made on several fronts: IDE integration, process integration, and new test fixtures. This article will cover unit testing in Visual Studio 2005, including VSTS unit testing, NUnit and MBUnit--the Superman of unit testing.

NUnit

NUnit is the unit testing framework that has the majority of the market share. It was one of the first unit testing frameworks for the .NET platform. It utilizes attributes to identify what a test is. The TestFixture attribute is used to identify a class that will expose test methods. The Test attribute is used to identify a method that will exercise a test subject. Let's get down to business and look at some code.

First we need something to test:


public class Subject { 
  public Int32 Add(Int32 x, Int32 y)
  { 
    return x  + y; 
  } 
}

That Subject class has one method: Add. We will test the Subject class by exercising the Add method with different arguments.


[TestFixture]
public class tSubject
{
  [Test]
  public void tAdd()
  {
    Int32 Sum;
    Subject Subject = new Subject();
    Sum = Subject.Add(1,2);
    Assert.AreEqual(3, Sum);
  }
}

The class tSubject is decorated with the attribute TestFixture, and the method tAdd is decorated with the attribute Test. You can compile this and run it in the NUnit GUI application. It will produce a successful test run.

That is the basics of what NUnit offers. There are attributes to help with setting up and tearing down your test environment: SetUp, SetUpFixture, TearDown, and TearDownFixture. SetUpFixture is run once at the beginning when the fixture is first created; similarly, TearDownFixture is run once after all tests have completed. SetUp and TearDown are run before and after each test.

NUnit tests can be run several different ways: from the GUI application, from the console's application, and from a NAnt task. NUnit has been integrated into Cruise Control .NET as well. In the last product review, you will see how it has been integrated into the VS.NET IDE as well.

NUnit GUI Application
Figure 1. NUnit GUI Application

Visual Studio Hacks

Related Reading

Visual Studio Hacks
Tips & Tools for Turbocharging the IDE
By James Avery

MbUnit

This is where we see some of the most innovative work in unit testing, regardless of language. There are 11 types of test fixtures, nine types of tests, and a myriad of other attributes to be applied. What one can do with MbUnit boggles the mind. So let's start with something reasonable; let's build on an existing example.


[RowTest()]
[Row(1,2,3)]
[Row(2,3,5)]
[Row(3,4,8)]
[Row(4,5,9)]
public void tAdd(Int32 x, Int32 y, Int32 expectedSum)
{
  Int32 Sum;
  Sum = this.Subject.Add(x,y);
  Assert.AreEqual(expectedSum, Sum);
}

The attribute Test has been replaced with RowTest. The attribute Row decorates providing data to be passed as parameters to the test. Notice the signature to the test: (Int32 x, Int32 y, Int32 expectedSum). MbUnit's test runner will run the test for as many Row decorators have been applied; four, in this case. And yes, the test runner will count them as separate tests. The results of running these tests would be four tests run, three passed, one failed. The RowTest leads into the XMLDataProvider attribute. It is similar to the functionality of a FitNesse fixture. The general idea is to provide an XML file and XPath query. The test runner will loop through the nodes, passing the node or (if desired) deserializing the node and passing it as a parameter to the test. Again, the test runner will count each run of the test as a test (i.e., three nodes would be counted as three tests). Jumping up in complexity, MbUnit provides advanced fixtures like the attribute TestSuite, which can be combined with other features like reflection to produce a test like this:


[TestSuite]
public TestSuite TestGenerator(){
  this.LoadAssemblies(); 
  TestSuite suite = new TestSuite("ExceptionSuite"); 
  foreach (Assembly Asm in AppDomain.CurrentDomain.GetAssemblies()) 
  { 
    if (!(Asm.FullName.StartsWith("MbUnit"))) 
    { 
      foreach (Type Tp in Asm.GetExportedTypes()) 
      { 
        if (Tp.IsSubclassOf(typeof(Exception))) 
        { 
          suite.Add(Tp.FullName, new TestDelegate(GenericTest), Tp);
        } 
      } 
    } 
  } 
  return suite;
}

The method decorated with the attribute TestSuite must return a TestSuite. To add to the TestSuite, a name for the test and an instance of the delegate TestDelegate must be provided. In the example, we will be testing the serialization and deserialization of exceptions. You will be surprised at how many types of exceptions are not able to be serialized or deserialized. The method LoadAssemblies reads a config file and loads all assemblies specified. A test is added to the TestSuite for each type found that is derived from Exception.


public void GenericTest(Type exceptionType){
  if (exceptionType == typeof(System.Data.SqlTypes.SqlTypeException))
  { 
    Assert.Ignore("The type {0} does not implement ISerializable correctly.", 
                  exceptionType.FullName);
  } 
  else if (exceptionType == typeof(System.Data.SqlTypes.SqlNullValueException))
  { 
    Assert.Ignore("The type {0} does not implement ISerializable correctly.", 
                  exceptionType.FullName);
  } 
  else if (exceptionType == typeof(System.Data.SqlTypes.SqlTruncateException))
  { 
    Assert.Ignore("The type {0} does not implement ISerializable correctly.", 
                  exceptionType.FullName);
  }

The method GenericTest was provided as the delegate for each test. Notice the parameter exceptionType; it was specified when the test was added to the TestSuite. The next interesting thing is MbUnit's ability to dynamically ignore a unit test. Here we are ignoring the SqlTypes exceptions because they do not implement ISerializable correctly.


Exception ThrownException = null; 
Stream Stream = null; 
Exception Clone = null; 

try 
{ 
  throw ((Exception)(this.ClassFactory.CreateInstanceOf(exceptionType))); 
} 
catch (Exception ex) 
{ 
  if (ex.GetType() == exceptionType) 
  { 
    ThrownException = ex; 
  } 
  else 
  { 
    Assert.Ignore(string.Format("Not able to create {0} {1}", 
                                exceptionType.FullName, ex.ToString())); 
  } 
}

Next, a class factory wil be used to create an instance of the exception and throw it. Again, the Assert.Ignore is used when the class factory has an error (remember, we are testing the exception's ability to be serialized and deserialized).


try 
{ 
  try 
  { 
    Stream = this.Serializer.SerializeToStream(ThrownException); 
  } 
  catch (SerializationException SerEx) 
  { 
    Assert.Fail(string.Format("Not able to serialize {0} {1}", 
                              exceptionType.FullName, SerEx.Message)); 
  } 
  catch (Exception OtherEx) 
  { 
    if (!(OtherEx.InnerException is SerializationException)) 
    { 
      Assert.Fail(string.Format("Problem with {0} {1}", 
                                exceptionType.FullName, OtherEx.ToString()));
    } 
    else 
    { 
      Assert.Fail(string.Format("Not able to serialize {0} {1}", 
                                exceptionType.FullName, OtherEx.ToString()));
    } 
  }

Now that we have a thrown exception, let's get to the test. First we will try to serialize it, and because we are going to want to deserialize it in just a second, we will hold on to the stream. At that point, because we are actually testing to see if there is a problem, we fail the test with Assert.Fail.


    try 
    { 
      Clone = ((Exception)(this.Serializer.DeserializeToObject(Stream)));
    } 
    catch (SerializationException SerEx)
    { 
      Assert.Fail(string.Format("Not able to deserialize {0} {1}",
                                exceptionType.FullName, SerEx.Message));
    } 
    catch (Exception OtherEx) 
    { 
      if (!(OtherEx.InnerException is SerializationException))
      { 
        Assert.Fail(string.Format("Problem with {0} {1}",
                                  exceptionType.FullName, OtherEx.ToString()));
      } 
      else 
      { 
        Assert.Fail(string.Format("Not able to deserialize {0} {1}",
                                  exceptionType.FullName, OtherEx.ToString()));
      } 
    } 
  } 
  finally
  { 
    if (Clone != null)
    { 
      Assert.AreEqual(ThrownException.StackTrace, Clone.StackTrace);
    } 
    if (Stream != null)
    { 
      Stream.Close();
    } 
  }
}   

Now we will try and deserialize the stream back into the exception, creating a clone. If this were not an example, I would need to be more thorough, but here I have only verified the StackTrace. I want to point out that the class used to test exception serialization is 170 lines, and that 161 tests are generated from loading 18 assemblies. Now that is cool. By the way, 24 exceptions failed; interestingly, 16 of them are from WSE. There are many more cool, time-saving features in MbUnit. The last one I will mention is the ability to take an existing NUnit test and simply change the reference and using statement to MbUnit and voila, your conversion is done. MbUnit can be run from the GUI application, from a console application, or from a NAnt task. It has been integrated into Cruise Control .NET. And yes, you will see how it has been integrated into the VS.NET IDE.

MbUnit GUI Application
Figure 2. MbUnit GUI application

VSTS

Visual Studio Team System (VSTS) is a beta-level commercial SCM and testing platform from Microsoft, with unit testing as one of its testing types. I will be covering only the unit testing features here, but it's worth noting that VSTS supports other testing, such as functional and load testing. This article is based on Beta 2 of VSTS; as this is a beta product, some features could change for the final release.

Test Class

The class used to demo the VSTS tests is a class with simple addition and subtraction methods.


public class Simple
{
  public int Add(int intNum1, int intNum2)
  {
    return intNum1 + intNum2;
  }

  public int Subtract(int intNum1, int intNum2)
  {
      return intNum1 - intNum2;
  }
}

Assert Tests

VSTS supports the Assert range of tests like MbUnit and NUnit, with commonly found tasks such as Assert.AreEqual, etc. VSTS has also added other tasks to its Assert test, including tasks such as GetHashCodeTests, Inconclusive, IsInstanceOf, and IsNotIstanceOf. The following is an example of an Assert test, here showing the AreEqual task.


[TestClass]
public class SimpleTest
{

  VSTSTest.Simple objSimple;

  [TestInitialize()]
  public void Initialize()
  {
      objSimple = new VSTSTest.Simple();
  }

  [TestMethod]
  public void TestAdd()
  {
    Assert.AreEqual(2, objSimple.Add(1, 1));      
  }

  [TestMethod]
  public void TestSubtract()
  {
    Assert.AreEqual(2, objSimple.Subtract(3,1));
  }
}

Note how VSTS lays out its unit tests. The TestClass attribute marks a class as a unit test class. The TestInitalize attributed is fired up each and every time by a test runner (e.g., the IDE) to allow a test class to set up any objects or values that the class might need. While not shown here, it is worth considering the TestCleanup attribute, which allows a class to clean up when the tests are complete. Finally, note that each test method in our class is marked with the TestMethod attribute.

Data-Driven Test Fixture

It's often vital to drive different sets of data through your unit tests; by running a broad spectrum of data through your tests, you can build a bigger picture of inputs and expected outputs. VSTS supports running tests through a database table of your choice; for example, a sample table may be as follows.


CREATE TABLE [UnitTest] (
  [ID] [int] IDENTITY (1, 1) NOT NULL ,
  [TestVal1] [int] NULL ,
  [TestVal2] [int] NULL ,
  [Result] [int] NULL 
) ON [PRIMARY]
GO

We can then alter our unit test as follows:


[TestClass]
public class SimpleDBTest
{
  VSTSTest.Simple objSimple;

  private TestContext testContextInstance;

  public TestContext TestContext
  {
      get
      {
          return testContextInstance;
      }
      set
      {
        testContextInstance = value;
      }
  }
  
  [TestInitialize()]
  public void Initialize()
  {
    objSimple = new VSTSTest.Simple();
  }

  [TestMethod, DataSource("System.Data.SqlClient", 
                          "Data Source=LOCALHOST;
                          Initial Catalog=TestDB;
                          Integrated Security=True", 
                          "UnitTest", 
                          DataAccessMethod.Sequential)]
  public void TestAddDB()
  {
    Assert.AreEqual((int)TestContext.DataRow["Result"], 
                    objSimple.Add((int)TestContext.DataRow["TestVal1"], 
                    (int)TestContext.DataRow["TestVal2"]));
  }
}

This is almost like the previous example; however, to enable data-driven testing, we extend the TestMethod attribute with the additional DataSource value. Here we add a database connection string and table name (in this case, UnitTest), and then reference the data using the TestContect.DataRow.

VSTS and the VS IDE

VSTS enjoys a close relationship with the VS IDE, which you would expect from a Microsoft tool. The IDE allows you to use a wizard to generate the unit tests from the code you are testing. This supports testing both public and private methods; however, opinion about testing private methods remains divided. VSTS includes TestManager, which is a GUI to allow you to select tests to run and to see the results of those tests. TestManager allows you to run all of your unit tests at once, or you can select which tests you want to run. You can also add additional information to a test, such as adding the DataSource attribute that we featured in the example. Finally, VSTS supports debugging unit tests in the VS IDE, so that it's possible to set breakpoints and start debug run via your unit test.

MSTest

VSTS supports running its test types (including unit tests) via a command-line tool called MSTest. MSTest does require VSTS to be installed to work (client side tools) and is not available individually for set up on a build server, etc.

TeamBuild

VSTS unit tests can be run in a build cycle using an extended version of MSBuild called TeamBuild.

Other Notes

You need to be a MSDN subscriber to download CTP releases; beta 2 can be downloaded from MSDN or you can receive it in the mail for a small charge.

TestDriven.NET

This product is not a unit testing framework, but a Visual Studio add-in for integrating existing unit testing frameworks into the IDE. It offers test execution from the IDE, both in and out of the debugger. It currently supports all versions of VS.NET and the unit testing frameworks NUnit, MbUnit, and MS Team System. Test execution can begin from several actions. Right-clicking (in a test method, in a class test fixture, in a namespace, on the solution, on a project, or on a class) will expose the Run Test(s) context menu item. This context menu item will begin executing the test(s). There is a feature called Ad-Hoc testing that relates to Run Test(s) context menu. If you right-click in a method that is not part of a test fixture class and run it as a test, the method will in effect become sub-main. If there is a return value from the method, it will be displayed in the output pane. Regular tests results are printed to the output pane as well. If there is an exception and the line numbers are available, they are active and can be double-clicked to navigate to that line. Task list entries are created for failing lists as well.

Conclusion

I don't see a benefit to pitting these products against each other. They each serve different needs. NUnit provides the basics needed for unit testing in a clean and elegant fashion. It has been integrated into many other products and adopted as standard by the .NET community. MbUnit is an advanced unit testing framework. It has been integrated into almost as many products as NUnit. MbUnit's motivation is to provide a more robust framework, taking on more of the common unit testing tasks so that you don't have to. VSTS has an entirely different goal. This product was born from a desire to integrate into the process of creating software. Not only does it provide a basic unit testing framework, but it integrates into the IDE. Its integration into the IDE is for the purpose of injecting itself before a developer can submit to source control. It can impose itself on a developer and force unit testing into the process. TestDriven.NET has its own motivations. It lowers the effort required to run tests. If I don't have to leave the IDE to run a test, if I can begin debugging a test with a single click, I am more likely to write more tests. It makes it easier for me to do the right thing. That is the criteria by which I would urge you to judge which of these products is right for you and or your team. Which one will make it easiest for you to do the right thing?

Resources

Jay Flowers is a .NET developer working as a design lead at Northrop Grumman.

Andrew Stopford hails from Manchester, UK and is a .NET Developer with the Mobile & Wireless Group.


Return to OnDotNet.com

Copyright © 2009 O'Reilly Media, Inc.