What’s the problem with TDD?

Have you ever worked on a system where the tests slow you down?

Do the tests break even with the smallest change?

Are the tests brittle?

Is it really hard to write a test? Lots of setup, mocks and expectations?

I’ve worked on these and I’m pretty sure I’m not alone.

So what’s the problem with TDD?

Feedback loops

To understand the problem we have to understand feedback loops.

The use of feedback loops is crucial in engineering; the components in a system provide feedback that reinforces or reduces change.

There are two types of feedback loops; positive and negative.

Negative feedback

Negative feedback loops move systems towards stability.

Negative feedback loops act as a damper to, or prohibits, change.

An example of a negative feedback loop in engineering is the centrifugal governor. This limits the speed of the engine by limiting fuel supply.

Positive feedback

Positive feedback loops moves systems towards instability.

Positive feedback loops reinforce change. A change reinforces itself and creates more of the same change.

An oscillator is an example of a positive feedback loop.

Tests as a feedback loop

In much the same way, unit tests are an example of a negative feedback loop.  When one breaks we stop and fix it. It is enforcing stability.

However, tests are also providing feedback when you are writing them. If it’s hard to write a test, your design is likely sub-optimal for the change you are trying to make.

For example, if there are lots of dependencies to mock and expectations to set up, we may have missed a needed abstraction that combines them.

The longer we put up with tests that are getting harder to write, the more our software degrades.

Designing feedback loops

We have to be careful how we design our feedback loops. If our tests are too fine grained our feedback loop will be too tight, limiting the changes we can make to our system. This will reduce our ability to change our software.

We need the feedback to ensure we haven’t broken anything while at the same time we want to be able to change the implementation details of our software.

Conclusion

Our tests are feedback loops, telling us, not only when changes are wrong, but also when our changes cannot easily be supported in the design.

If we ignore our tests it will become harder to work within our system over time.

We need to be careful how fine grained we make our tests; too fine and we won’t be able to change our software. Not fine enough and we will inadvertently break some behaviour of our software.

Testing emails

We’ve recently added email notifications to our application.  I really wanted to avoid configuring email servers or setting up email accounts to test our software. Luckily there are a number of tools to help with this for development, and testing.

Development

We’re using .Net so we can configure our application to send all SMTP to a file directory. This can be done in the application configuration file with the below snippet;

<system.net>
    <mailSettings>
        <smtp deliveryMethod="SpecifiedPickupDirectory">
            <specifiedPickupDirectory pickupDirectoryLocation="c:\temp\mailbox\"/>
        </smtp>
   </mailSettings>
</system.net>

Our acceptance tests relied on this mechanism to redirect SMTP traffic to the file system. We then had an interface in our automation framework to parse these files so we could easily check the contents.

Component Testing

For manual testing we wanted to avoid having to change the application configuration so looked for a simple way of setting up SMTP.

We decided to use an SMTP fake. There are a few open source project for this.  Two examples are SMTP4Dev and PaperCut.

I use SMTP4Dev simply because I’ve used it before but will try papercut when I get some time.

System Testing

This is the only time we needed to have a real SMTP server configured.  For this we have a Dogfood environment that our operations department takes care of. Taking this approach completely removed the need to become experts in configuring email servers to test sending emails.

Mistake proofing

In previous posts I described how I test serialization of objects.

However, I still forget to do this. So, I’ve written tests to remind me. This is an idea from Lean engineering called poka yoke (see http://en.wikipedia.org/wiki/Poka-yoke).

The concept is simple – make it impossible or hard for someone to make mistakes that go unnoticed.

I use the convention of putting all the classes that need to be serialized in a contracts namespace. This makes it simple to write a test to check all these classes are defined with the DataContract attribute.

        [Test]
        public void All_classes_in_contract_namespace_should_implement_datacontract()
        {
            IList allClassesInContractsNamespace = ReflectionHelper.GetAllTypesWhere(x=>x.Namespace.Contains("Contracts") && x.IsClass).ToList();

            allClassesInContractsNamespace.Count.ShouldBeGreaterThan(0);

            foreach (object obj in allClassesInContractsNamespace)
            {
                var dataContract = obj.GetType().GetCustomAttributes(typeof(DataContractAttribute), false).OfType().FirstOrDefault();
                dataContract.ShouldNotBe(null);
            }
        }

The ReflectionHelper extension method encapsulates how I’m using Autofac to get a list of types. This makes it simpler to re-use in my tests as well as change the method of reflecting over types in the future;

public static IEnumerable GetAllTypesWhere(Func predicate)
        {
            var builder = new ContainerBuilder();
            builder.RegisterAssemblyTypes(typeof(Invoice).Assembly)
                   .Where(predicate.Invoke)
                   .As();

            IContainer container = builder.Build();
            var controllers = container.Resolve<IEnumerable>();

            return controllers;
        }

This technique can be extended to any frequent mistake.

Conclusion

We have a lot of tools to help us not make mistakes; modern development tools help a lot. Third party tools exist such as re-sharper, style-cop, and FxCop. Our build systems run our unit tests and these static analysis tools.

We can take this further by writing tests that check for our most common mistakes.

Testing serialization part II

My previous post on testing serialization was in-complete.  Since then my tests have evolved enough to warrant another post (at least I think so).

Here’s the excerpt from my previous post;

[Test]
public void Should_Serialise_across_service_boundary()
{
    var s = new DataContractSerializer(typeof(BankAccount));
    var account = new BankAccount() { Owner = new Person() };
    s.WriteObject(new MemoryStream(), account);
}

So, what are the shortcomings I’d like to address?

  • The test doesn’t assert anything. As long as the serializer doesn’t throw an exception, the test passes
  • It isn’t particuarly readable and we’ll probably end up with a lot of code duplication

However, before addressing the above issues I’d like to explain why we became interested in testing serialisation and how we failed to solve this previously.

Motivation

Unlike my normal tests this is not a TDD practise.  TDD is about driving your design through tests.  This is not my motivation for doing this type of test.

These tests are supporting continuous integration by providing early feedback.

On the project I was designing at the time we’d spent some time tracking down the cause of defects that turned out to be serialization issues.

First attempt

To address this we starting creating a comprehensive set of integration and acceptance tests. However, these took too long to run and we realised things would only get worse as our system grew.

Also, detecting these problems in integration tests increases the feedback loop. This is bad. It would be much better to find these problems sooner – preferably in unit tests.

Unit testing serialization

So let’s get back to the tests.

Take the following classes;

public class BankAccount
{
    public string AccountNumber { get; set; }
    public string SortCode { get; set; }
    public Person Owner { get; set;}

public class Person
{
    public string Name { get; set; }
    public DateTime DateOfBirth { get; set; }
}

BankAccount and Person are simple classes, so I’m confident this will serialize correctly.

Now, how about after refactoring this to use interfaces;

public class BankAccount
{
    public string AccountNumber { get; set; }
    public string SortCode { get; set; }
    public IParty Owner { get; set; }
}

public class Person : IParty
{
    public string Name { get; set; }
    public DateTime DateOfBirth { get; protected set; }
}

public interface IParty
{
    string Name { get; set; }
}

This will now fail as the serializer has no knowledge of the interface and the Person’s DateOfBirth property has a protected setter.

So here’s my test;

[Test]
 public void Should_Serialise_across_service_boundary()
 {
     GivenBankAccount();
     GivenAccountHasBeenSerialised();
     WhenDeserializingFromStream();

     ThenAllPropertiesShouldBeSet();
  }

I like to refactor my test code into methods to reduce noise and increase the intent on the code.  this makes it easier to understand them later. The Givens sets up the scenario I am testing;

private void GivenBankAccount()
{
    account = Builder<BancAccount>.CreateNew().Build();
    person = Builder<Person>.CreateNew().Build();
    account.Owner = person;
}
private void GivenAccountHasBeenSerialised()
{
    storeStream = new MemoryStream();
    serializer = new DataContractSerializer(typeof(BankAccount));
    serializer.WriteObject(storeStream, account);
}

The When..() method deserializes the stream back into a BankAccount instance;

private void WhenDeserializingFromStream()
{
    storeStream.Position = 0;
    savedAccountDetails = this.serializer.ReadObject(storeStream) as BankAccount;
}

And finally, we test the deserialized object’s properties are the same as the original;

private void ThenAllPropertiesShouldBeSet()
{
    savedAccountDetails.ShouldNotBe(null); 
    savedAccountDetails.SortCode.ShouldBe(account.SortCode); 
    savedAccountDetails.AccountNumber.ShouldBe(account.AccountNumber); 
    savedAccountDetails.Owner.Name.ShouldBe(person.Name
    ((Person)savedAccountDetails.Owner).DateOfBirth.ShouldBe(person.DateOfBirth);

}

This test will pick up both problems; the DateOfBirth property won’t be set correctly and the serializer will prompt to add the [KnownType(typeof(Person))] attribute to the BankAccount class;

To aid re-use and reduce code clutter I’ve created some extension methods to serialize and de-serialize objects;

public static class SerialisationExtensions
{
    public static Stream Serialize(this T target)
    {
        Stream storeStream = new MemoryStream();
        var serializer = new DataContractSerializer(typeof(T));
        serializer.WriteObject(storeStream, target);
        return storeStream;
    }

    public static T DeSerialize<T>(this Stream fromStream) where T: class
    {
        fromStream.Position = 0;
        var serializer = new DataContractSerializer(typeof(T));
        return serializer.ReadObject(fromStream) as T;
    }
}

This reduces code duplication in my tests. The below methods;

private void GivenAccountHasBeenSerialised()
{
    storeStream = new MemoryStream();
    serializer = new DataContractSerializer(typeof(BankAccount));
    serializer.WriteObject(storeStream, account);
} 

private void WhenDeserializingFromStream()
{
    storeStream.Position = 0;
    savedAccountDetails = this.serializer.ReadObject(storeStream) as BankAccount;
}

are replaced with these;

private void GivenAccountHasBeenSerialised()
{
   storeStream = account.Serialize();
}

private void WhenDeserializingFromStream()
{
    savedAccountDetails = storeStream.DeSerialize<BancAccount>();
}

This makes these methods redundant and we could move this code back into the test.

What we learnt

Creating a comprehensive suite of integration and acceptance tests did prevent our team deploying broken software. However, these were slow running tests and slowed us down.

Finding a way of moving these tests from integration to unit tests enabled us to be faster by reducing our feedback loop.

I’ve still got some more work to do on testing serialization but this method is easier to understand and more reliable than testing that attributes exist on your classes.

Testing serialization

A common problem I see is broken integration tests because objects can’t be serialised across service boundaries. It’s easy to forget to decorate a class or property with the DataContract attribute.

Catching this during integration tests can lead to large slow running test suites and wasted time troubleshooting, as it is rarely obvious what has the issue is.

One goal of continuous integration is to catch errors as early in the process as possible and it turns out this can be caught during unit tests.

It’s simple to add unit tests to ensure a class or property is decorated with the correct attribute;

[Test]     
public void DataMembersSetAsExpected()   
  {            
       var type = typeof(BankAccount);           
       Assert.That(type.IsDefined(
            typeof(System.Runtime.Serialization.DataContractAttribute), true));
       CheckForDataMemberAttribute(type, "SortCode"); 
  }
  private static void CheckForDataMemberAttribute(Type type, string property)
      {
          var idProperty = type.GetProperty(property);
          Assert.That(idProperty.IsDefined(
             typeof (System.Runtime.Serialization.DataMemberAttribute), true));
        }
    }

The main problem with this implementation is developers need to remember to update the test when a new property is added or removed.

It would be simple to refactor this to check all properties on a class using reflection.

However, this isn’t ideal as it will lead to false negatives as properties that won’t be serialised are still checked.

The approach I’ve adopted is to use the DataContractSerialiser to test if an object can serialised;

    [Test]
    public void Should_Serialise_across_service_boundary()
    {
        var s = new DataContractSerializer(typeof(BankAccount));
        var account = new BankAccount() { Owner = new Person() };
        s.WriteObject(new MemoryStream(), account);
    }

This has the advantage that it will only test properties that will be serialised and developers don’t have to remember to change the test when adding or removing properties.

Conclusion

Testing objects can be serialised in integration tests results in slow test suites and fails to catch the defect as early as possible.

One approach to unit testing serialisation is to verify classes have been decorated with the DataContract and DataMember attributes.

However, this leads to false positives when developers forget to refactor tests when adding properties.

The approach I am currently using it to use the DataContractSerialiser class to ensure a class instance can be successfully serialised.

running tests all the time

I’ve been using a great TDD tool called ncrunch (http://www.ncrunch.net/)

It’s a Visual Studio add-in that automatically runs tests in the background. It analyses your code changes and runs the affected tests.