You often hear people advocate that writing tests (unit, functional, etc.) when building software leads to less bugs in production. How can one verify this statement? Usually, those people are already using TDD or BDD and might be working for companies that have a strong testing culture from day one. Then, how can one measure the impact of NOT writing tests? How can we verify that practicing TDD is actually producing less bugs? Can we stop writing tests for a period of time and look at the consequences it has on the defect count? That doesn’t seem very realistic.
In this post, I am trying to answer the question by analyzing real numbers from my current job. No tests were written originally and once tests were included, I became a strong advocate of the TDD methodology.
I’ve been a web developer for over 10 years and in 2009 I started to hear a lot about test driven development (TDD). It was then I decided to learn more about it. At that time I had been working for my current employer for about 2 years. Fast forward to today (2012). I’ve now been with my company for 5 years and have spent the last 3 years applying TDD principles.
This is very interesting because today I can look at our bug tracker and gather statistics about the number of defects found over time and see the impact TDD has had on our code base.
This actually makes it really easy to look at statistics. I can look over time at the number of defects in the UI vs the entire product. That should allow me to answer the initial question, which in context can be rephrased as: does TDD adoption lead to less defects in the UI that I am responsible for?
So does TDD actually work?
The following graph is generated from our bug tracker. It is showing the ratio of bugs in the UI compared to the whole product.
There are a few interesting things to notice:
First and foremost, we can answer the question: the amount of bugs found in the UI (in green above) has been reduced by about 50% (the UI accounted for roughly 35% of the bugs in the product and today it’s down to hovering around 15%).
Applying TDD on a legacy code base does not magically change things overnight. I started applying TDD in Q3 of 2009. You can see the number of bugs starting to decline around Q3 of 2010 with a bigger drop in Q3 of 2011. It took nearly 2 years to achieve the 50% drop.
As I said earlier, I started applying TDD to the PHP side of my code. Looking at the bugs I get nowadays, I noticed they usually fall in one of two categories:
Integration bugs (for instance where independent PHP components are passing unit tests, but are not working together properly)
We are about to start our next development cycle and I am looking forward to leveraging this new setup to build higher quality software. I am also looking forward to looking at this topic next year and seeing what impact these tools will have. It will hopefully help to reduce our bug count further.
Also, sharing those numbers with my teammates helped convince some engineers of the benefits of TDD. Engineers working on other services of our product have shown interest and are willing to investigate how they can apply TDD to their codebases. It’ll be interesting to see what kind of benefits they get out of the exercise.
What about you? How did TDD or BDD affect your code base? Do you have similar statistics to share? Feel free to comment below.
As usual, if you have feedback on this post, feel free to leave a note or to contact me via Twitter (@jphpsf).