What’s the problem with TDD?

Have you ever worked on a system where the tests slow you down?

Do the tests break even with the smallest change?

Are the tests brittle?

Is it really hard to write a test? Lots of setup, mocks and expectations?

I’ve worked on these and I’m pretty sure I’m not alone.

So what’s the problem with TDD?

Feedback loops

To understand the problem we have to understand feedback loops.

The use of feedback loops is crucial in engineering; the components in a system provide feedback that reinforces or reduces change.

There are two types of feedback loops; positive and negative.

Negative feedback

Negative feedback loops move systems towards stability.

Negative feedback loops act as a damper to, or prohibits, change.

An example of a negative feedback loop in engineering is the centrifugal governor. This limits the speed of the engine by limiting fuel supply.

Positive feedback

Positive feedback loops moves systems towards instability.

Positive feedback loops reinforce change. A change reinforces itself and creates more of the same change.

An oscillator is an example of a positive feedback loop.

Tests as a feedback loop

In much the same way, unit tests are an example of a negative feedback loop.  When one breaks we stop and fix it. It is enforcing stability.

However, tests are also providing feedback when you are writing them. If it’s hard to write a test, your design is likely sub-optimal for the change you are trying to make.

For example, if there are lots of dependencies to mock and expectations to set up, we may have missed a needed abstraction that combines them.

The longer we put up with tests that are getting harder to write, the more our software degrades.

Designing feedback loops

We have to be careful how we design our feedback loops. If our tests are too fine grained our feedback loop will be too tight, limiting the changes we can make to our system. This will reduce our ability to change our software.

We need the feedback to ensure we haven’t broken anything while at the same time we want to be able to change the implementation details of our software.

Conclusion

Our tests are feedback loops, telling us, not only when changes are wrong, but also when our changes cannot easily be supported in the design.

If we ignore our tests it will become harder to work within our system over time.

We need to be careful how fine grained we make our tests; too fine and we won’t be able to change our software. Not fine enough and we will inadvertently break some behaviour of our software.

Mocking dangers

Mocking dependencies allow us to create tests that are fast by removing our dependencies on external systems.

However, it’s easy to overuse and create brittle tests that become a maintenance burden.

For example, imagine we are creating a To-do list application. We might come up with the below simple design;

todo_design

Given this design we might end up with tests similar to the ones below;

public class TodoSpecs
{
    public class TodoSpecs
    [Test]
    public void Service_invokes_repository()
    {
        var repository = Substitute.For<IRepository>();
        var service = new ToDoService(repository, Substitute.For<IMapper<ToDoTask, TaskResource>>());
        service.GetAll();
        repository.Received(1).GetAll();
     }

     public void Service_should_return_resources()
     {
         var repository = Substitute.For<IRepository>();
         var toDoTask = new ToDoTask() { Note = "new todo task" };
         repository.GetAll().Returns(new[] { toDoTask });
         var mapper = Substitute.For<IMapper<ToDoTask, TaskResource>>();
         mapper.Map(Arg.Any()).Returns(new TaskResource() { Note = "new todo task" });
      
         var service = new ToDoService(repository, mapper);
         var todos = service.GetAll();
      
         <mapper.Received(1).Map(Arg.Is(toDoTask));
         todos.Count().ShouldBe(1);
      }

      public void mapper_should_map_task_to_resource()
      {
          var mapper = new TaskResourceMapper();
          var task = new ToDoTask() { Note = "new todo task" };
          var resource = mapper.Map(task);
          resource.Note.ShouldBe(task.Note);
      }
}

These tests are trying to prove the service gets all tasks from the repository, converts them to a resource and returns them. At first glance this seems fine. The first test mocks the repository dependency and tests we call the getall() method. This is reasonable as the real repository will be going to a database of some kind.  If we didn’t mock this we would have a slow integration test.

Brittle Tests

The problem is in the second test. The first smell that something is wrong it that the test is hard to understand. We have to mock two things – the repository and the mapper.

This means that any change to the mapper requires us to change our tests in two places; the tests that cover the mapper and the tests that create mock mappers.

Another problem is that to prove we return the correct result to the client, we are testing the collaboration between the mapper and the service. If we remove the mapper we have to change all our tests.

Testing from the outside

Now, imagine if we just tested this code from the service and treated the mapper as an internal object. We’d probably have something like this;


[TestFixture]
public class TodoSpecs2
{
    [TestFixture]
    [Test]
    public void Service_invokes_repository()
    {
        var repository = Substitute.For<IRepository<ToDoTask>>();
    
        var service = new ToDoService(repository);
        service.GetAll();
    
        repository.Received(1).GetAll();
    }

    public void Service_should_return_resources()
    {
        var repository = Substitute.For<IRepository<ToDoTask>>();
        var toDoTasks = Builder<ToDoTask>.CreateListOfSize(1).Build();    
        repository.GetAll().Returns(toDoTasks);

        var service = new ToDoService(repository);
        var todos = service.GetAll().ToList();

        todos.Count().ShouldBe(1);
        todos[0].Note.ShouldBe(toDoTasks[0].Note);
    }
}

We have removed the mapper from the services constructor and are not testing the mapper anymore. We are still using the mapper in the service but it is being created in the service.

As a result, the test is much more readable and nothing is lost – the mapper is still covered by this test. However, rather than testing it directly, requiring us to create unneeded dependencies and brittle tests, it is being tested as a side-effect of this test.

That’s completely valid – the mapper is cheap to create and consume and it only collaborates privately with the service. We could refactor the code to remove the mapper and our test would still be be valid.

Conclusion

Overusing mocking creates brittle tests that will inhibit flexibility and decrease productivity. We should only mock objects that are expensive to create or use.

In the example, the mapper was cheap to create and use and did not need to be an external dependency. Treating it as private to the service resulted in more flexible and easier to understand tests.

Faking third party web services

I’ve worked on a number of projects that relied on third party web services or API’s.  Most of these had a test service but didn’t provide a way to return a predetermined response.

This left two choices when testing our software; call the test service or create a fake service of our own

Third party test services

Usually the purpose of a test service is to ensure that your production systems will work with their production systems.  So it is used for testing the integration of two systems.

This usually means the only difference is the data is not real or is a sub-set of the live data.  Using it still requires calls across service boundaries (HTTP), expensive computation and storage (like databases).

This creates a number of issues when executing an automated test suite for your software;

  • Calls across services boundaries are expensive, resulting in slow tests and slow feedback
  • Tests will be brittle because the data can change, leading to different results over time
  • You may not have access even to the test service in the environment you are running your tests

Faking services

The solution is invariably to create a fake service that can be configured to return a specific response.

I’ve done this a few times and without exception the solution has become a maintenance burden. Some  of the problems I’ve experienced are below;

  • Fake is not re-usable across projects
  • Duplication of fakes across teams
  • Different fakes for different third party services

This leads to a lot of duplication of effort and lost time.

Proxy servers

After realising this could be solved with a proxy server I created Boomerang to make this easier.  This enables developers to set predetermined responses for HTTP requests.

For example, I can specify a json object should be returned from a GET for a specific relative address;

var dictionary = new Dictionary<string, string>() { { "content-type", "application/json" } };
Boomerang.Server(5100).Get("/api/products").Returns(products.SerialiseToJsonString(), 200, dictionary);

This creates a proxy server on port 5100 and specifies the response to a HTTP GET request to the relative uri /api/products should be a list of Products formatted as json.

I created an extension method to help with the json serialisation;

public static string SerialiseToJsonString(this object target)
        {
            string str;

            var dataContractJsonSerializer = new DataContractJsonSerializer(target.GetType());

            using (var mem = new MemoryStream())
            {
                dataContractJsonSerializer.WriteObject(mem, target);
                mem.Flush();
                mem.Position = 0;
                str = Encoding.Default.GetString(mem.ToArray());
                Console.WriteLine(str);
            }

            return str;
        }

Conclusion

GET, POST, PUT and delete are supported and the base address is ignored.  If Boomerang receives a request it hasn’t got a response for it will return an error response.

I hope this makes testing with third party services a bit easier!

Mistake proofing

In previous posts I described how I test serialization of objects.

However, I still forget to do this. So, I’ve written tests to remind me. This is an idea from Lean engineering called poka yoke (see http://en.wikipedia.org/wiki/Poka-yoke).

The concept is simple – make it impossible or hard for someone to make mistakes that go unnoticed.

I use the convention of putting all the classes that need to be serialized in a contracts namespace. This makes it simple to write a test to check all these classes are defined with the DataContract attribute.

        [Test]
        public void All_classes_in_contract_namespace_should_implement_datacontract()
        {
            IList allClassesInContractsNamespace = ReflectionHelper.GetAllTypesWhere(x=>x.Namespace.Contains("Contracts") && x.IsClass).ToList();

            allClassesInContractsNamespace.Count.ShouldBeGreaterThan(0);

            foreach (object obj in allClassesInContractsNamespace)
            {
                var dataContract = obj.GetType().GetCustomAttributes(typeof(DataContractAttribute), false).OfType().FirstOrDefault();
                dataContract.ShouldNotBe(null);
            }
        }

The ReflectionHelper extension method encapsulates how I’m using Autofac to get a list of types. This makes it simpler to re-use in my tests as well as change the method of reflecting over types in the future;

public static IEnumerable GetAllTypesWhere(Func predicate)
        {
            var builder = new ContainerBuilder();
            builder.RegisterAssemblyTypes(typeof(Invoice).Assembly)
                   .Where(predicate.Invoke)
                   .As();

            IContainer container = builder.Build();
            var controllers = container.Resolve<IEnumerable>();

            return controllers;
        }

This technique can be extended to any frequent mistake.

Conclusion

We have a lot of tools to help us not make mistakes; modern development tools help a lot. Third party tools exist such as re-sharper, style-cop, and FxCop. Our build systems run our unit tests and these static analysis tools.

We can take this further by writing tests that check for our most common mistakes.

Testing serialization part II

My previous post on testing serialization was in-complete.  Since then my tests have evolved enough to warrant another post (at least I think so).

Here’s the excerpt from my previous post;

[Test]
public void Should_Serialise_across_service_boundary()
{
    var s = new DataContractSerializer(typeof(BankAccount));
    var account = new BankAccount() { Owner = new Person() };
    s.WriteObject(new MemoryStream(), account);
}

So, what are the shortcomings I’d like to address?

  • The test doesn’t assert anything. As long as the serializer doesn’t throw an exception, the test passes
  • It isn’t particuarly readable and we’ll probably end up with a lot of code duplication

However, before addressing the above issues I’d like to explain why we became interested in testing serialisation and how we failed to solve this previously.

Motivation

Unlike my normal tests this is not a TDD practise.  TDD is about driving your design through tests.  This is not my motivation for doing this type of test.

These tests are supporting continuous integration by providing early feedback.

On the project I was designing at the time we’d spent some time tracking down the cause of defects that turned out to be serialization issues.

First attempt

To address this we starting creating a comprehensive set of integration and acceptance tests. However, these took too long to run and we realised things would only get worse as our system grew.

Also, detecting these problems in integration tests increases the feedback loop. This is bad. It would be much better to find these problems sooner – preferably in unit tests.

Unit testing serialization

So let’s get back to the tests.

Take the following classes;

public class BankAccount
{
    public string AccountNumber { get; set; }
    public string SortCode { get; set; }
    public Person Owner { get; set;}

public class Person
{
    public string Name { get; set; }
    public DateTime DateOfBirth { get; set; }
}

BankAccount and Person are simple classes, so I’m confident this will serialize correctly.

Now, how about after refactoring this to use interfaces;

public class BankAccount
{
    public string AccountNumber { get; set; }
    public string SortCode { get; set; }
    public IParty Owner { get; set; }
}

public class Person : IParty
{
    public string Name { get; set; }
    public DateTime DateOfBirth { get; protected set; }
}

public interface IParty
{
    string Name { get; set; }
}

This will now fail as the serializer has no knowledge of the interface and the Person’s DateOfBirth property has a protected setter.

So here’s my test;

[Test]
 public void Should_Serialise_across_service_boundary()
 {
     GivenBankAccount();
     GivenAccountHasBeenSerialised();
     WhenDeserializingFromStream();

     ThenAllPropertiesShouldBeSet();
  }

I like to refactor my test code into methods to reduce noise and increase the intent on the code.  this makes it easier to understand them later. The Givens sets up the scenario I am testing;

private void GivenBankAccount()
{
    account = Builder<BancAccount>.CreateNew().Build();
    person = Builder<Person>.CreateNew().Build();
    account.Owner = person;
}
private void GivenAccountHasBeenSerialised()
{
    storeStream = new MemoryStream();
    serializer = new DataContractSerializer(typeof(BankAccount));
    serializer.WriteObject(storeStream, account);
}

The When..() method deserializes the stream back into a BankAccount instance;

private void WhenDeserializingFromStream()
{
    storeStream.Position = 0;
    savedAccountDetails = this.serializer.ReadObject(storeStream) as BankAccount;
}

And finally, we test the deserialized object’s properties are the same as the original;

private void ThenAllPropertiesShouldBeSet()
{
    savedAccountDetails.ShouldNotBe(null); 
    savedAccountDetails.SortCode.ShouldBe(account.SortCode); 
    savedAccountDetails.AccountNumber.ShouldBe(account.AccountNumber); 
    savedAccountDetails.Owner.Name.ShouldBe(person.Name
    ((Person)savedAccountDetails.Owner).DateOfBirth.ShouldBe(person.DateOfBirth);

}

This test will pick up both problems; the DateOfBirth property won’t be set correctly and the serializer will prompt to add the [KnownType(typeof(Person))] attribute to the BankAccount class;

To aid re-use and reduce code clutter I’ve created some extension methods to serialize and de-serialize objects;

public static class SerialisationExtensions
{
    public static Stream Serialize(this T target)
    {
        Stream storeStream = new MemoryStream();
        var serializer = new DataContractSerializer(typeof(T));
        serializer.WriteObject(storeStream, target);
        return storeStream;
    }

    public static T DeSerialize<T>(this Stream fromStream) where T: class
    {
        fromStream.Position = 0;
        var serializer = new DataContractSerializer(typeof(T));
        return serializer.ReadObject(fromStream) as T;
    }
}

This reduces code duplication in my tests. The below methods;

private void GivenAccountHasBeenSerialised()
{
    storeStream = new MemoryStream();
    serializer = new DataContractSerializer(typeof(BankAccount));
    serializer.WriteObject(storeStream, account);
} 

private void WhenDeserializingFromStream()
{
    storeStream.Position = 0;
    savedAccountDetails = this.serializer.ReadObject(storeStream) as BankAccount;
}

are replaced with these;

private void GivenAccountHasBeenSerialised()
{
   storeStream = account.Serialize();
}

private void WhenDeserializingFromStream()
{
    savedAccountDetails = storeStream.DeSerialize<BancAccount>();
}

This makes these methods redundant and we could move this code back into the test.

What we learnt

Creating a comprehensive suite of integration and acceptance tests did prevent our team deploying broken software. However, these were slow running tests and slowed us down.

Finding a way of moving these tests from integration to unit tests enabled us to be faster by reducing our feedback loop.

I’ve still got some more work to do on testing serialization but this method is easier to understand and more reliable than testing that attributes exist on your classes.

Testing serialization

A common problem I see is broken integration tests because objects can’t be serialised across service boundaries. It’s easy to forget to decorate a class or property with the DataContract attribute.

Catching this during integration tests can lead to large slow running test suites and wasted time troubleshooting, as it is rarely obvious what has the issue is.

One goal of continuous integration is to catch errors as early in the process as possible and it turns out this can be caught during unit tests.

It’s simple to add unit tests to ensure a class or property is decorated with the correct attribute;

[Test]     
public void DataMembersSetAsExpected()   
  {            
       var type = typeof(BankAccount);           
       Assert.That(type.IsDefined(
            typeof(System.Runtime.Serialization.DataContractAttribute), true));
       CheckForDataMemberAttribute(type, "SortCode"); 
  }
  private static void CheckForDataMemberAttribute(Type type, string property)
      {
          var idProperty = type.GetProperty(property);
          Assert.That(idProperty.IsDefined(
             typeof (System.Runtime.Serialization.DataMemberAttribute), true));
        }
    }

The main problem with this implementation is developers need to remember to update the test when a new property is added or removed.

It would be simple to refactor this to check all properties on a class using reflection.

However, this isn’t ideal as it will lead to false negatives as properties that won’t be serialised are still checked.

The approach I’ve adopted is to use the DataContractSerialiser to test if an object can serialised;

    [Test]
    public void Should_Serialise_across_service_boundary()
    {
        var s = new DataContractSerializer(typeof(BankAccount));
        var account = new BankAccount() { Owner = new Person() };
        s.WriteObject(new MemoryStream(), account);
    }

This has the advantage that it will only test properties that will be serialised and developers don’t have to remember to change the test when adding or removing properties.

Conclusion

Testing objects can be serialised in integration tests results in slow test suites and fails to catch the defect as early as possible.

One approach to unit testing serialisation is to verify classes have been decorated with the DataContract and DataMember attributes.

However, this leads to false positives when developers forget to refactor tests when adding properties.

The approach I am currently using it to use the DataContractSerialiser class to ensure a class instance can be successfully serialised.

running tests all the time

I’ve been using a great TDD tool called ncrunch (http://www.ncrunch.net/)

It’s a Visual Studio add-in that automatically runs tests in the background. It analyses your code changes and runs the affected tests.