This week, I have had a bit of an epiphany with regard to how we should approach automated testing our projects.
Misgivings about TDD style testing
The value of TDD style testing comes as an aid to code design rather than testing. If you are exploring designs and then writing tests afterwards, where is the net benefit? They do give you confidence that the changes you make are not breaking existing functionality, such as inadvertently inverting some boolean logic, that might otherwise have slipped through.
The value of unit tests utilising mocks is dubious. They are often painful to set-up, inevitably difficult to read, and a pain to maintain. Unless you refactor your code in a very strict way, your tests will need be rewritten as the API they call completely changes.
Is there really much inherent benefit in isolating everything but one object and mocking interactions with other objects? Again, in my opinion, not unless you are driving the design of your code through mocking.
How can you know that you have written the correct tests? It is “easy” to get 100% code coverage, but not so easy to see that you have covered your code with relevant and correct assertions.
I am simply too lazy, not smart/disciplined enough to write unit tests that strictly satisfy the definitions of a unit.
What do I want from automated testing?
A safety net that will protect our code from changes or additions unintentionally affecting algorithms
Mainstream tooling and language support (e.g. NUnit, mocking frameworks, integration with Visual Studio and Continuous Integration software)
A focus on the desired behaviour of the software rather than infrastructure and internals. This forces developers and domain experts to think about what the specifications (specs) of the software should be immediately.
Tests/Specs to be as frictionless as possible to write. Tests that are difficult to write, in my experience don’t get written. As much as possible, avoid complex data setup, external services to encourage people to write tests.
Ability to generate test fixtures using a Domain Specific Language any domain expert/developer can understand
Use as many of our concrete implementations of objects where possible, but retain the option to mock our externals interfaces such as database, file system, web services. Much behaviour or functionality in applications is expressed via the interaction of multiple moving parts. Our tests should reflect this.
Ability to generate reports on the number of specs implemented and passing/failing in an readable and portable manner
The result is a series of BDD styles specs using a .Net project called SpecFlow, which are written in “english” using the gherkin DSL which comes from the Ruby Project Cucumber.
I’m not going to describe the rationale behind BDD as others have done so before, better than I can, here and here
SpecFlow is currently at version 1.2.0 and has versions for Visual Studio 2008 and 2010. The reason for the different versions is that there are specific Item Templates within Visual Studio to aid the creation of SpecFlow Features and Steps.
Steve Sanderson has an excellent post which covers SpecFlow with ASP.net MVC which was my inspiration for exploring this framework in the first place. We have a Windows Forms client, so our Specs cover the layers immediately below the UI (presenter and downwards).
References
SpecFlow Home Page
Cucumber Project
Steve Sanderson Article
Hi, I'm Simon Whittemore, a developer living in London.
I've been a Developer for about 14 years, and have been working with .NET and C# since version 1.0.
I have extensive experience developing maintainable, well architected Web and Desktop Applications in the Financial and IT Services industries, and have successfully integrated Open Source .NET applications and frameworks into processes and software.
Pingback: Tweets that mention Simon on Software » Blog Archive » Thoughts on TDD, Automated Testing and BDD with Specflow -- Topsy.com
Hi Simon,
thanks a lot for writing about SpecFlow and contributing patches!
I also want to point you to a TekPub scrrencast episode where Rob Conery explains SpecFlow. Its episode 15 of the series “Mastering ASP.NET MVC2”. The episode is freely available here:
http://tekpub.com/production/starter
I would also be very interested in your insights, findings and patterns how to drive/automate rich-client applications with SpecFlow. Looking forward to find more posts here 😉
Great to read such a pragmatic description of testing. (Also looking forward to the final episode on Tekpub in which SpecFlow is mentioned – as I bought the series :-))