Standish, the CHAOS report, and science

Every software engineering researcher has heard of the 1994 CHAOS report from the Standish Group. It reported that the large majority of software projects fail, and that they incur in an average of 189% cost overrun. It’s a very popular report –the go-to reference for researchers that want to make the case that the software industry is in crisis and is therefore desperately in need of whatever tool or method we throw their way.

I suspect, however, that most software engineering researchers haven’t read it. If they had they wouldn’t be citing it: as a piece of science it is terrible. Its methods are not fully disclosed, and the bits that are disclosed are deeply flawed. Its results are completely unreliable. There are good critiques of the report out there, notably by Robert Glass and by Jørgensen and Moløkken, that point to the unreliability of the study and question the claim of a software crisis.

All this is old news, but yesterday I stumbled upon this gem: In the comments section of this InfoQ interview with Standish’s Jim Johnson, Magne Jorgensen brought up his critique of the CHAOS report and asked two very fair questions. Johnson’s reply is almost unbelievable (emphasis mine):

General answer to Magne Jørgensen, Robert Glass, et al. We are an advisory research firm much like a Gartner or Forrester. What sets us apart from them is that we use our primary research to form our opinions, whereas they use their individual consultants. Neither they nor we can afford to give our opinions away for free. We have facilities, utilities, and personnel and we must, the same as you, be able to pay our bills. Just because someone asks a question, does not mean we will respond with an answer. In fact, we most likely will not. It is not rebuff or slight, it is just our business model to survive. Our current standard answer to a CHAOS inquiry is, first: please purchase our new book, “My Life is Failure” in our online store. If that does not satisfy you, then you need to join CHAOS University. If you do not find your answer or answers there then you need to purchase our inquiry services. Then we will work to answer your questions. Besides being a client you can get CHAOS information by participating in the research forum. If you would like to participant in our research you canjoin the Standish User Research Forum. You need to be an organization that develops software for your own use to qualify. This year we are offering a free reviewers copy of a book we just completed titled “The Pubic Execution of Miss Scarlet” as the honorarium. It is a tale of a fictional project in peril, but uses much of our lessons learned. Every year since 1995 we hold a CHAOS University and discuss many of these issue. It is strange that both Jørgensen and Glass have never applied or professed interest in joining us. Some answers can be found if you join us at CHAOS University 2007 or one of the many outreach events. So you can contribute to the CHAOS research by providing funding or sweat, but short of that you will and must be ignored by design. We do provide much of our research to the legitimate press and I have personally written many articles on our CHAOS and other research. We have given many talks to public and private organizations. In June, I presented some of our research to an IEEE conference in Philadelphia and the Delivering Project Excellence conference in Phoenix. This month I am doing a keynote talk to the National Association for Justice Information Systems in Boca Raton, Florida. As to Jørgensen’s two questions:Question 2: How was (and is) the cost overrun in the report calculated? The simple answer to this is that we ask“what was the estimated cost verses the actual cost?” We use a number of different instruments to collect the data and derive our overrun assessment. While we really do have a rocket scientist on staff, this is basic math. On the first question and any other questions you will have to read “My Life is Failure” to find those answers.

Really? Purchase our book and our inquiry services? Join our “university”? It’s clear that the CHAOS report is a business, not science. We must stop treating it otherwise, and we must call out those who do not.