Talk About Quality

Tom Harris

The Dangers of Counting

leave a comment »

On Slow Leadership today they’re talking about Occam’s Razor. Choosing the simplest explanation. And the dangers of setting numerical targets.

But metrics are so easy to collect these days. The temptation to count things is great.

Let’s say you are deploying a new process, or tracking progress on an existing one. What harm could there be in counting, looking at the numbers, and making decisions based on them? Isn’t that sound management?

The problem is assuming that by counting something you learn more about it.

Actually all you learn is how many of something there were.

You don’t know anything about those somethings, nor why the count came out as it did.

When you count deployment events, there is no positive integer count that tells you much by itself.

Even zero doesn’t tell you unequivocally what happened. I can think of at least 3 possibilities about the events:

a) Didn’t happen
b) Happened but insubordination or fear among the counters
c) Happened but misunderstanding about what to count

Things get worse if the numbers are positive. How do you know whether a number is large or small? More, or less?

For example, trying to measure software development, let’s say you counted 24 high-severity defects last month. Is that high or low? Is it more or less than the month before? Kind of depends, doesn’t it, on how much code you wrote, how much testing was done, how good the testing was, which defects were recorded, and so on.

Forget distributions — pie charts and the like. How do you know if people understood which category an event belonged in?

What does all this leave us with? No graphs? No counting? No decision-making based on numbers? Back to the Stone Age?

No. It just means something counter-intuitive in today’s modern world of computers: knowledge before counting.

First learn the qualitative details of your events. What is a satisfactory defect entry? A relevant test result? A productive code review? Make sure all participants can recognize them.

Then you can start counting.

1, 2, 3 …

P.S. Want extra credit? Read just the first page of What is Mathematics? by Courant and Robbins. The section is called “The Natural Numbers: Introduction”.

Written by Tom Harris

September 14, 2006 at 9:11 pm

Posted in Metrics

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s