May 12, 2010

Structuring tests using contexts and scenarios

One of the greatest challenges of writing good tests is to keep the tests short and readable. This can be achieved by composition - here is the way I use to structure AAA-style tests:

[Test]
public void ProductValidator_IllegalProductCode_Fails()
{
    // Arrange
    var context = new TestContextBuilder<ProductValidatorContext>()
        .WithScenario(AnonymousProduct)
        .WithScenario(ProductHasIllegalProductCode)
        .BuildContext();
    
    ProductValidator sut = context.CreateSubjectUnderTest();

    // Act
    bool res = sut.Validate();

    // Assert
    Assert.IsFalse(res);
}

The goal is to provide more readable tests and to encapsulate (and reuse) test setup code. As a bonus it will be easy to move between tests and these context/scenario-style tests and using a framework like StoryQ .

Using AAA syntax, the Arrange part of each test uses a specific test context, and aditionally applies one or more scenarios to this context. In the Act part the Subject Under Test is exercised in the context, and in the Assert part the assertions are made, typically on the Subject Under Test or on the context.

The test context:

The test context is the context needed to run a test on the subject under test. In other words, it is the fixed part that stays the same for each test in your test class. You want to keep the context at a minimal level, meaning that the context establishes just enough stuff to be able to create the subject under test. Also the context exposes the things you might want to modify from test to test or make assertions against. We often place the test context class as a private, nested class inside the test class itself.

public class ProductValidatorContext : TestContextBase
{
    public Product TheProduct { get; set; }

    public ProductValidator CreateSubjectUnderTest()
    {
        return new ProductValidator(TheProduct);
    }
}

The scenarios:

Each specific test will use a context builder to build the context. The context builder initializes the context and allows you specify one or more scenarios to apply to the context. Scenarios can be expressed using simple lambda expressíons, methods or classes:

Using lambda:
var context = new TestContextBuilder<ProductValidatorContext>()
    .WithScenario(AnonymousProduct)
    .WithScenario(x => x.TheProduct.Code = "Illegal product code")
    .BuildContext();

Using a method:
var context = new TestContextBuilder<ProductValidatorContext>()
    .WithScenario(AnonymousProduct)
    .WithScenario(ProductHasIllegalProductCode)
    .BuildContext();

...

private void ProductHasIllegalProductCode(ProductValidatorContext context)
{
    context.TheProduct.Code = "Some illegal product code";
}

Using a class:
var context = new TestContextBuilder<ProductValidatorContext>()
    .WithScenario(AnonymousProduct)
    .WithScenari<ProductHasIllegalProductCode>()
    .BuildContext();

...

private class ProductHasIllegalProductCode : TestScenarioBase<ProductValidatorContext>
{
    public override void Apply(ProductValidatorContext context)
    {
        context.TheProduct.Code = "Some illegal product code";
    }
}

Scenario classes can expose properties that are specific to the scenario in case you need to assert against these:

Assert.That(context.Scenario<SomeProductInEditing>.TheProduct.Name, Is.Not.Null);

So typically the context establishes mocks or test instances of all the dependencies of the subject under test. The scenario(s) set the behavior of these mocks or builds the test instances. Here I typically use frameworks like Moq and Autofixture.

I have put the context builder as well as base classes for the contexts and scenarios on Code Gallery.

March 19, 2010

Best practices for Assert statements in unit tests

Some may find this trivial, but I find myself going over this in almost every code review: You’ve got to pay attention to your unit test assert statements:

Don’t hide the comparison in your assert
Consider this assert statement:

Assert.IsTrue(myObj.Name == "Expected name");

Why is this bad? Take a look at the test runner output when the test fails:

NUnit.Framework.AssertionException:   Expected: True
But was:  False

The comparison is hidden, so the outcome of the test is always a non-informative true/false. Change the statement to

Assert.AreEqual("Expected name", myObj.Name);

This will give the following output when the test fails:

NUnit.Framework.AssertionException:   Expected string length 13 but was 12. Strings differ at index 0.
Expected: "Expected name"
But was:  "Another name"
-----------^

Much more informative, right?

I personally like the alternate fluent assert constructs you can do with NUnit, because it makes you write the assert statement right almost without thinking about it:

Assert.That(myObj.Name, Is.EqualTo("Expected name"));

The assert message should add information or be omitted

How about:

Assert.AreEqual(myObj.Name, "Peter", "Expected name to be Peter.");

The assert message is a waste of typing efforts, because the output of the test runner will tell you anyway. Use the assertion messages only to provide extra information, or omit them:

myObj.DoSomething();
Assert.That(myObj.HasError, Is.True, "Expected HasError to be set because DoSomething produces an error.");

Multiple assert statements

When completing a complex arrange part of a test, it is tempting to make a lot of assert statements (after all this trouble, it’s time to assert to heck out of it, right?) Well not really…

The assert statement is the test. When you look at a test that has multiple assert statements, the test will most likely violate the SRP principle (yes, it also applies to tests), meaning that the test is testing to many things at once. Why is this bad? Because the test will most likely be big and difficult to understand. What will you name the test if it tests 5 different things? Have you seen each of the assert statements fail? If you haven’t, how do you know that the test is working?

Resolution is simply to split it up. You can extract any complex arrange/setup part to a private method and reuse this from your split up tests.

The only case where I see myself diverting from this practice is when doing progressive assertions against the same object. Again it is about making the test informative. Here my expected outcome is that a person with the expected name is added to Persons:

Assert.That(myObj.Persons[0].Name, Is.EqualTo("Expected name"));

Will potentially throw a NullReferenceException or an IndexOutOfRangeException, which doesn’t give a clear clue to what the problem is. Here I would use:

Assert.That(myObj.Persons, Is.Not.Null);

Assert.That(myObj.Persons, Is.EqualTo(1));

Assert.That(myObj.Persons[0].Is.EqualTo("Expected name"));

This will be more informative in case the Persons list is not instantiated or populated.

Conclusion

The general issue here is, that when you are constructing your test, you know exactly what is going on, at that precise moment. It is very easy to be blinded by focusing only on making this test go green. But try to look ahead – in 3 months, this test may fail. Imagine your trusted coworker having to find out what went wrong – fast. That’s why your tests should be easy to understand and informative when failing. And that, my friend, is achieved by paying attention to the assert statement.

March 11, 2010

Code Coverage analysis for .NET projects in TeamCity 5

I personally don't use test code coverage as a goal in itself, but it is a great tool to find the blind spots - finding classes and methods that don't have coverage, but really should.
Speaking in numbers, I don't set a certain code coverage percentage as a strict required minimum, instead I look for classes or namespaces that falls below a certain treshold, like 60%.

My favorite build server of choice now makes it easier to keep track of the code test coverage. Team City Version 5.0+ has a nice little new part (Build Configuration –> Runner) :


Out of the box TeamCity supports NCover (commercial and community versions) and PartCover.  Let’s look at NCover – there is a free (however old) community edition. NCover provides much more progressed commercial versions if you want to do advanced coverage analysis and Visual Studio integration, but let’s start small with the community edition.

On the build server:
Download and install NCover Community 1.5.8 (you need to register, which is free).
Download and install NCover Explorer 1.4.0.7 (this is just a zip, unzip it to an “Explorer” sub folder to the NCover install location).
Now you just need to set it up from the TeamCity Build Configuration –> Runner page:


A few things to note:
Don’t count on the autodetection of the NCover install files, especially if you are on a 64-bit server. Put those paths in manually.
Enter the assemblies to analyze – without extension, and one on each line.

Now you are ready to go. Run a build and check out the code coverage report:


You can choose between different reports to view assemblies, namespaces, methods and classes to find potential blind spots. I find this a valuable extension to your build server that it is hard to be without, once you have started using it.

February 23, 2010

Folder structure for a .NET application

I usually stick to the same base folder structure for my .NET applications. This one works well for both my open source projects and enterprise projects at work:

  • Branch name (usually ‘trunk’)
  • This root folder contains the solution file.
    • Application
    All the projects of the application goes here. Each project in it’s own subfolder.
    • Build
    I keep all build related stuff here, like:
    • The signing key for the application
    • A C# project holding the main MSBuild script file and any custom MSBuild tasks
    • Common
    Usually not much here, except always a SolutionInfo.cs file.

    • Documentation
    All developer-related documents of the project are kept here. This way a developer can easily get the latest version of a document without leaving Visual Studio.
    If the project contains auto-generated documentation (for instance using Sandcastle), these generator-projects are kept here as well.

    • Installers
    All installer projects are kept here (Setup projects, WIX projects). Each project in it’s own subfolder.

    • Lib
    This folder contains all the external assemblies used by the project.
    All Visual Studio projects reference the needed dll’s directly from this folder.

    • Tests
    Contains all test projects, usually divided in unit tests, integration tests and test helpers. Each project in it’s own subfolder.


    Note that the above is the physical folder structure. I always make sure that the version control has the excact same folder structure. Never try to build this folder structure from within Visual Studio. Instead build it by hand and add it to source control by hand.

    In the main Visual Studio solution file however, I alter the structure slightly by putting the contents of the Application folder directly in the solution root, and mimic the rest of the folders using solution folders:
    • Solution root
      • All projects of the Application folder
      • Build
      • Common
      • Documentation
      • Installers
      • Lib
      • Tests

    Now, with everything being nicely structured, it’s time to do some coding.

    February 12, 2010

    Creating a .NET project’s development environment

    This week a co-worker and I set up the envrionment for a new development project. It's starting to look like a pattern. Here is our list:

    The development server

    We use virtual servers running on some huge hardware farm somewhere. This is nice. We order a new Windows Server instance from our IT services, and within 24 hours it is up and running.

    • IIS - used for staging our builds (if web)
    • Visual Studio Database Edition GDR2
    • SQL Server with:
      • CI database, used as the database for CI build integration tests
      • Staging Database (typically backing a nightly staging build, so the PM can track to progress)
      • Pre-Production Database – should always reflect the scheme of the current in-production build. We use this for schema comparisons.
    • Instal Visual SVN (server)
    • Install Team City 5.0  with:
      • CI Build
      • Staging Build (usually nightly)
      • Release Build (for the releases)

    All the installations are basically "next, next, finish" installations, using default settings.

    This setup means that every project has it's own Subversion server and build server running on the dev server. This works well for the small to medium sized projects we are having, and we have had no performance issues for our typical 1-10 team member projects.

    The development environment

    To keep the development environment consistent for all team members we prepare a complete development environment in a virtual machine, ready for distribution. The install list we're currently using is:

    After setting up the development environment, we run a sysprep and close it down, compress and distribute. The way sysprep works is that when the developer starts up this virtual machine for the first time, she is presented with a setup wizard resembling the last install steps in a clean Windows install. She can then give the virtual machine a name, set administrator password, etc. The virtual machine will now be unique.

    I intentionally left our the version numbers for Windows Server, IIS and SQL Server. We used 2003/6.0/2005 for the latest projects, and are hoping that the next customer's hosting provider gives green light for 2008 versions.

    Now we can start coding…


    January 25, 2010

    Speeding up your Build & Run (F5)

    Have you ever been in the situation where you need to run your application, check something, edit some code, build, run again, repeatately? And each time you are pressing F5 run run, it’s just taking too long to build.
    There are many ways to improve the build time (including buying cool and speedy hardware like an SSD drive), but this one is free:
    In Visual Studio, select Tools –> Options --> Projects and Solutions –> Build and Run

    Check the “Only build startup project and dependencies on Run” option.
    I can’t believe that this is not a default setting.

    August 21, 2009

    New release of FormStateKeeper

    I just made a new release of the FormStateKeeper - earlier known as FormStateSaver, but hey, a naming change is also a way of binging in something new ;-). Most important news for this version is support for ASP.NET MVC. As the source code has grown from a small hack to a nice little project, I also moved it from MSDN Code Gallery to Codeplex.

    For those who don’t know FormStateKeeper: It is a small HttpModule that will assist in the scenario where the user looses all the contents of the web page form fields when redirected to the login page. This will typically happen using forms authentication if an authentication timeout happens before the user submits the form. FormStateKeeper prevents this by intercepting the http post the user makes just before being redirected to the login page. It then stores the posted field values, and recreates them just before the user is being redirected to the original page.

    FormStateKeeper has a small footprint of one .dll file an one line in the web.config file.