Code Coverage Conundrum

I once received an email that said:

Please be prepared to explain how you will increase your unit test coverage to  target %

A week later on another project I saw something similar:

Code coverage is “green”, 100% test coverage.

So here is the conundrum, test automation is one of the best ways I know to design, build, and create a sustainable system.  So isn’t more better and wouldn’t 100% be nirvana? Nope.  Remember this is a conundrum (what a fun word to say, conundrum, conundrum, conundrum).  Here is the conundrum with code coverage:

Goodhart’s Law: Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.  – George Dinwiddie

Now George was stating the above in the context of velocity, but if you read his forward to the book, Software Development Metrics, you will see that he applies that statement more broadly to any metric.

In my experience, individuals who grasp at code coverage are under pressure from people who understand managing risk more than they understand managing software development.  Which presents two issues: 1) dealing with the reality these people and 2) how to measure the effectiveness of your unit testing.

First, let’s explore the space of an objective metric that allows everyone to sleep better at night.  Sorry there isn’t one–blame Goodhart.  So the cheap answer is that we right click our project in eclipse and generate test automation stubs which immediately turns our code coverage 100% green. #GoTeamGoodhart.  Really, isn’t there a better way?  Sorry, any way you slice or dice it, Goodhart’s law comes into play.

Now let’s move onto something a bit more useful, how do we measure the effectiveness of your unit testing?  Let’s try the 5 whys technique and see where it takes us.

  1. Why do I want 100% unit test coverage?  Because unit tests are good.  They help me with simple design, understand requirements, form a regression test suite, increase my mpg, give me confidence in my builds, etc.
  2. Why do I want these good things? I want to deploy early, frequently, and embrace change.  Those agile principles are hard to achieve without automated tests.
  3. Why can’t I achieve those principles without unit tests?  Well I can, but my chances of success, especially in the long term increase (almost in lockstep) with the effectiveness of my unit test program.
  4. Ok, maybe I asked the wrong question. Why do I want to deploy more frequently?  Good things happen when I deploy more frequently, most importantly my feedback loops shorten and I accelerate learning.
  5. Hmm, I think I’m getting closer (which good because I’m at my fifth why).  Why do unit tests accelerate learning?  Well they accelerate learning both as I build the functionality by understanding and thinking about the software’s behavior and once built by providing stable builds that end-users can give me focused feedback.

So now that I understand my goal: accelerate learning, how do I measure the impact of my unit testing approach both while I am building the software and then once built when I validate it? Oooh tough question.  Maybe if we look at what happens when my unit testing is weak we will find some clues.