Controlling what you can’t measure

I recently came across Tom DeMarco’s “Controlling Software Projects” for a second time, and I remembered my problem with it immediately: The very first line in the book states that “You can’t control what you can’t measure”, and the rest of the text builds upon that phrase to argue that we need metrics to rein the chaos of software development.

But the book falls apart because the statement is wrong, and you and me are living proof of that. We control things we don’t measure, all the time. You are able to control your body (your breathing, your hunger, your thoughts) without measurements, and you are an expert at it. In some cases measurements can help –for example, for controlling your weight, or your cholesterol. In many others, however, measurements would not even make sense (do you need to measure the length and number of your hairs to know when to get a haircut?)

What bothers me is that the phrase has picked up among software project managers, and is often used to justify absurd metrics and policies: Relying on measurements such as lines of code for programmer efficiency, or number of errors found for software quality, are excellent ways to lose control over a project. They are deficient and misleading proxies for the real constructs that one wants to study.

There is a kernel of truth to the phrase, however. The unknown is, by definition, uncontrollable, and sometimes numbers help us increase our knowledge. But I would much rather read a software development book that relaxes the statement to something like “You can’t control the unfamiliar”, and builds a thesis from there. It doesn’t have the same punch, but it has the benefit of being true.

Robert Glass, by the way, puts it better than me in his excellent book “Facts and Fallacies of Software Engineering” (the relevant passage is available here for free):

“The problem with the saying “you can’t manage what you can’t measure”—what makes it a fallacy—is that we manage things we can’t measure all the time. We manage cancer research. We manage software design. We manage all manner of things that are deeply intellectual, even creative, without any idea of what numbers we ought to have to guide us. Good knowledge worker managers tend to measure qualitatively, not quantitatively.”


About Jorge Aranda

I'm currently a Postdoctoral Fellow at the SEGAL and CHISEL labs in the Department of Computer Science of the University of Victoria.
This entry was posted in Books, Software development. Bookmark the permalink.

19 Responses to Controlling what you can’t measure

  1. Mike says:

    I prefer what Tom Gilb said, “Anything you need to quantify can be measured in some way that is superior to not measuring it at all. “

  2. Jorge says:

    Thanks for chipping in, Mike!

    Gilb’s phrase does not really contradict my argument. I’m not against measurements, I’m against believing that one *requires* measurements to achieve control.

    Having said that, I’d take Gilb’s phrase with a grain of salt. Sometimes getting proper and accurate measurements may cost more than the benefits we might obtain from having them.

  3. The idea that you need to measure everything to control it make sense when you include non-quantitative measurements. You don’t measure your hair but you look at a mirror and compare its lenght to you usual standard. That is a way of measure that relies on qualitative standards. You can measure the amount of food you have by the sensation of fullness and, with hope, happiness of your belly. Not all measures have to be in ones and zeroes, but DeMarco’s work miss this fact and falls in excess when measure the efficiency of a worker by the lines of code he writes….

  4. Jorge says:

    Right –if we expand the definition of measurement to include qualitative assessments, then I have no problem with the phrase. But the standard meaning of the word calls for numbers, and that’s the way the phrase is interpreted most of the time!

  5. Leopoldo Lopez says:

    I do not agree on your statement that “We control things we don’t measure”, since in all your examples there is some type of measurement. Not all the measurements have to be quantified some can be qualified (as good, normal, etc); these are also types of measurements.

    For example, not because you think you can not measure the number of times your hurt beats per minute doesn’t mean that your body (or your mind) isn’t doing some kind of measurement unconsciously. Have you ever had a heart attack? Do you know what you feel before your body knows that your heart is going to stop? This is because your unconscious self measured something in your body that’s not right, and this is a type of metrics an it has a specific way of evaluation and of showing a result as simple as healthy or sick or as complicated as any name of strange disease you can come up with..

    If you think thoroughly on every example you gave you can find a kind of metric that is helping control the process, and of course it is measurable.

    The main issue here is to find the appropriate metric and also to find the appropriate way to measure it, because if you do not do that you might reach wrong conclusions like thinking that you can control things that you can not measure.

  6. Jorge says:

    Thanks for your comments, Leopoldo. Again, if in the meaning of ‘measurement’ we include all sorts of qualitative assessments, then of course, “you can’t control what you can’t measure” —meaning, “you can’t control it if you don’t know what’s going on”.

    But really, the phrase is consistently used as an argument for quantitative measurements, and against qualitative and unconscious ones. This is what bugs me.

  7. Green Queen says:

    The concept of measurement is a left-brain activity and precludes all emotion and intuition. Human beings are vibrant souls who are designed to rely on emotions and intution for decision-making but when we have cut ourselves off from that internal wisdom, we fall back on measurements to tell us what we should know. For sure, we need both sides of the brain firing away to find solutions.

  8. Wolverine says:

    Apparently the “You can’t manage what you can’t measure” disciples also suffer from an inability to understand that “feedback” is different from quantitative measurement.

    Of course management depends upon a feedback loop. That is so stunningly self-evident it doesn’t merit comment. Otherwise, all managers could sit at home in a sound-proof room and issue outbound emails all day long and never check their inbox.

    However, quantifiable metrics and “dashboards” aren’t sufficent to drive good management. In fact in many instances such “hard data” fails the manager because: (a) we’re measuring the wrong thing; (b) the measurement is flawed or doesn’t include all the necessary factors; or (c) the interpretation of the measurement is wrong. Most importantly having quantifiable numbers can lull a manager into a false sense of security, deluding himself / herself that he has things under control, when in fact, all he has is a piece of paper which doesn’t describe emerging competitive threats, employee morale, or thousands of other factors which impact a business but are difficult to quantify with great granularity.

    “You can’t manage what you can’t measure” is a crutch for the insecure and the inept.

  9. Sam Cannon says:

    That’s why Maslow created B-language, so that we can assemble a discriminator circuit. We can begin to measure Beauty, Justice, Fairness, etc., at the group level as we develop cyberspaces reflecting our value systems. Maslow has a list that jives well with a Gardner view. But more important is to sniff out the nay-sayers from the yea-sayers. At the bottom of the nay-sayers ( life negative) are the mentors of the suicide bombers, who, if they had the courage to blow THEMSELVES up, would solve the problem.

  10. Sam Cannon says:

    You said, “Good knowledge worker managers tend to measure qualitatively, not quantitatively.”

    I beg to differ. They want ALL of the good guys.

  11. Pingback: More on measurement « Catenary

  12. alexjulien says:

    I have not read DeMarco’s book, but I’m certainly familiar with the quote. I agree with you that there are irrational ways of measuring. Lines of code per se can’t give an insight of the quality of the code, unless you happen to measure the same group of programmers over enough lines of code over enough time for the measurement to be meaningful. And really: who will do this? Who NEEDS this at all?

    This is what bugs me about methodologies like CMMI: you spend so much time measuring that eventually you don’t have enough of anything to be measured.

    But on the other hand, I do believe that if your definition of “measuring” is broad enough, and you never lose sight of common sense, yes: you can’t control what you can’t measure.

    Hair, to take your own example (I just had a haircut last week). I certainly didn´t grab a ruler, measured a statistically meaningful number of hairs and threw an average length of my hair to match against a table that stated that I was 35.43% above the acceptable length, and that it was time for a haircut.

    My measurement method is simpler, but it can be somehow described both qualitatively and a quantitatively:

    Quantitatively: it’s been more than 4 weeks since my last haircut. I know for experience that by week #6 my hair is unmanageable, so by week 4 or 5 I’m ready for scissors.

    Qualitative: by the third consecutive morning I tried to comb my hair and said to myself “I look like crap”, I knew a haircut was in my plans for the weekend.

    So yes: measure, just don’t overdo it. Measure until it’s meaningful, not more.

    BTW: ¡Viva México, C…!

    • Jorge says:

      Alex — thanks for the comment. I think we’re in agreement.

      I often speak against measurement and quantitative data not because I think it’s not valuable, but because it seems to me we’re biased to give an unwarranted weight to quantitative measurements over any other kind of evidence.

  13. Rob says:

    I totally disagree with the author’s premise and agree with the original thought that control comes through measurement and that qualitative measurement is still a measurement – albeit a less exact one. But in all the examples that you provided, a quantified example would probably have provided a better result than the qualitative measure. Your hair example is a good one. What if there were a way to easily measure the length, weight and density of your hair. Say for example you had an device that could give you a read out on those three measures. You could then use those measures to know precisely when you needed a hair cut and perhaps where exactly on your head the hair stylist should spend more time and attention. These quantifiable measures would be better than than qualitative ones – in every case. Before we had ways to look into arteries and see how much of one or more of them were blocked, the qualitative measure would be how “tired” am I feeling after some amount of physical exertion. But is that as good of a measure as knowing that the artery is 70% blocked? We result to qualitative measures where it is hard to get to the more precise and ultimately meaningful quantifiable measures. Sure, you can control things that do not have precise measures for – but the saying is real control (repeatable, precise control) comes when you have a strong measurement system in place. I don’t think you are advocating that we should give up on finding precise measurements for things – but your argument seems to lack the proper weighting – objective always trumps subjective unless you are talking about something that is purely subjective (aesthetics).

  14. zee says:

    quote: “You are able to control your body (your breathing, your hunger, your thoughts) without measurements, and you are an expert at it.”

    Very nice post. But the question arises are we really in control of these functions? My strong feeling is we all have some “unconscious competence” in these areas but we do not have absolute control or even understand what control might mean in these instances.

    Often times we discover what to do through a process of trial and error.

    Measurement defines or refines how we do things allowing us greater insight and control within limits.

    Certainty has limits.

    • Jorge Aranda says:

      When measurements provide us with better insights, then yes, they lead to greater control. But sometimes the drive for measurements overrides the behaviors it was intended to assess.

      To clarify: I don’t mean to say that measurements are always detrimental, merely that they are often unnecessary. For a large proportion of activities, you *can* control what you can’t measure.

  15. zee says:

    Yes, I agree partly. Although we control a large portion of our activities without measurement this sort of control falls under *unconscious-competence*. Such control is undefined. In such situations we find that the majority of our activities are basically ritualistic in nature and actually contribute little to the process of what it is we are doing. Meanwhile, a few activities, many of them subtle and unconscious, dominate the activity. Such ritualistic behavior may extend to include, as you state, the drive for measurement that overrides the need to assess behavior. Knowing and balancing what’s measurable with what’s not measurable maybe what the doctor ordered.

    But, ultimately, it comes down to choice and what feels good to a particular person.

  16. Pingback: Codermetrics? - It will never work in theory

  17. Pingback: DeMarco, Deming, and “Cannot control what you cannot measure” « Semantic Werks

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s