JP's blog

Tech talk with a french twist.

OMG, Test Driven Development Actually Works!

You often hear people advocate that writing tests (unit, functional, etc.) when building software leads to less bugs in production. How can one verify this statement? Usually, those people are already using TDD or BDD and might be working for companies that have a strong testing culture from day one. Then, how can one measure the impact of NOT writing tests? How can we verify that practicing TDD is actually producing less bugs? Can we stop writing tests for a period of time and look at the consequences it has on the defect count? That doesn’t seem very realistic.

In this post, I am trying to answer the question by analyzing real numbers from my current job. No tests were written originally and once tests were included, I became a strong advocate of the TDD methodology.

Background

I’ve been a web developer for over 10 years and in 2009 I started to hear a lot about test driven development (TDD). It was then I decided to learn more about it. At that time I had been working for my current employer for about 2 years. Fast forward to today (2012). I’ve now been with my company for 5 years and have spent the last 3 years applying TDD principles.

This is very interesting because today I can look at our bug tracker and gather statistics about the number of defects found over time and see the impact TDD has had on our code base.

A quick word about the software stack: I build a UI based on PHP and Javascript. I am the lead UI engineer, which means if there is a UI bug, I am likely to be the one responsible. The UI interacts with backend C++ services which are leveraging PLSQL code running on Oracle.

The PHP code of the UI is the only area where TDD was applied. Our Javascript code does not have tests (yet). Sadly, no other engineers in the company are practicing TDD. All the other layers of the code base are not unit or functionally tested (the testing is handled by a dedicated QA team post development).

This actually makes it really easy to look at statistics. I can look over time at the number of defects in the UI vs the entire product. That should allow me to answer the initial question, which in context can be rephrased as: does TDD adoption lead to less defects in the UI that I am responsible for?

So does TDD actually work?

The following graph is generated from our bug tracker. It is showing the ratio of bugs in the UI compared to the whole product.

There are a few interesting things to notice:

  • First and foremost, we can answer the question: the amount of bugs found in the UI (in green above) has been reduced by about 50% (the UI accounted for roughly 35% of the bugs in the product and today it’s down to hovering around 15%).

  • Applying TDD on a legacy code base does not magically change things overnight. I started applying TDD in Q3 of 2009. You can see the number of bugs starting to decline around Q3 of 2010 with a bigger drop in Q3 of 2011. It took nearly 2 years to achieve the 50% drop.

  • The graph shows a couple of bumps (e.g. slight increase in bug count) in Q2 of 2010 and Q2 of 2011. Looking at our releases, this corresponds to a time where we added features based on 3rd party maps API that heavily relied on Javascript. As I mentioned previously, we did not have Javascript tests at the time. This also confirms that not writing tests leads to more bugs.

What’s next?

As I said earlier, I started applying TDD to the PHP side of my code. Looking at the bugs I get nowadays, I noticed they usually fall in one of two categories:

  • Bugs in our Javascript code (which again is not automatically tested)

  • Integration bugs (for instance where independent PHP components are passing unit tests, but are not working together properly)

This leads me to the next obvious step of my testing adventure: I need to add a Javascript test suite as well as a end to end integration testing suite.

This is what I’ve been working on over the summer. For Javascript, I’ve setup a test suite based on the Jasmine BDD framework. For end to end testing, I’ve setup Cucumber. In both cases, I can run the tests using real browsers with Selenium Web Driver or headless using the awesome Phantom.js.

We are about to start our next development cycle and I am looking forward to leveraging this new setup to build higher quality software. I am also looking forward to looking at this topic next year and seeing what impact these tools will have. It will hopefully help to reduce our bug count further.

Also, sharing those numbers with my teammates helped convince some engineers of the benefits of TDD. Engineers working on other services of our product have shown interest and are willing to investigate how they can apply TDD to their codebases. It’ll be interesting to see what kind of benefits they get out of the exercise.

Wrap up

What about you? How did TDD or BDD affect your code base? Do you have similar statistics to share? Feel free to comment below.

As usual, if you have feedback on this post, feel free to leave a note or to contact me via Twitter (@jphpsf).

Possibly related posts

Comments