Thoughts on Software Testing

Less test coverage <==  ==> More test coverage
Y increases with cost
This is partly a response to the blog post by Derick Bailey linked below.  It is mostly an effort to help articulate my understanding of testing strategies that I am woefully unexperienced in.

http://www.lostechies.com/blogs/derickbailey/archive/2010/11/11/cost-vs-risk-in-testing.aspx?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+LosTechies+(LosTechies)


Each type of testing covers a different class of bugs, which combined only add to 100% of error cases coverable by testing.  Yes, there is some overlap, but that brings me to my next point.  


Different types of testing also have drastically different costs associated.  If we could afford true 100% end to end testing, maybe unit testing wouldn't be required.  In reality, we rarely ever do.  100% end to end testing is too expensive in terms of time and money.


We do unit testing and integration testing because each is much cheaper than end to end testing.  This is in no small part do to the fact that it is agile.  We get a cheaper bang for our buck, but we also get it during early development.  Sure end to end could handle a working system, you get no help from end to end testing when building a system bottom up or developing a new slice from the middle.  


So why not skip end to end all together?  Well we have some diminishing returns.  Your first end to end test is going to be more productive than your unit tests at some point.  It doesn't matter if all your code is unit tested if you have a show stopper that an end to end test would have caught.

Overlap certainly is waste, but is not alway economically avoidable waste.


Consider a car.  We can install a roll cage, air bags, and a seatbelt costing 1000, 100, and 10 dollars respectively.  A roll cage may reduce the chance for injury in certain crashes just as effectively as an air bag or seatbelt, but it may do nothing in other cases or it may be vital.  In reality, car manufacturers provide a combination of techniques which provide a sweet spot between economy and safety.  They focus on those efforts which give the greatest return, even if they necessarily create overlap in function.


Risk is not easily compartmented into each category of testing.  Sometimes you need both belt and suspenders.