Standish, the CHAOS report, and science

Every software engineering researcher has heard of the 1994 CHAOS report from the Standish Group. It reported that the large majority of software projects fail, and that they incur in an average of 189% cost overrun. It’s a very popular report –the go-to reference for researchers that want to make the case that the software industry is in crisis and is therefore desperately in need of whatever tool or method we throw their way.

I suspect, however, that most software engineering researchers haven’t read it. If they had they wouldn’t be citing it: as a piece of science it is terrible. Its methods are not fully disclosed, and the bits that are disclosed are deeply flawed. Its results are completely unreliable. There are good critiques of the report out there, notably by Robert Glass and by Jørgensen and Moløkken, that point to the unreliability of the study and question the claim of a software crisis.

All this is old news, but yesterday I stumbled upon this gem: In the comments section of this InfoQ interview with Standish’s Jim Johnson, Magne Jorgensen brought up his critique of the CHAOS report and asked two very fair questions. Johnson’s reply is almost unbelievable (emphasis mine):

General answer to Magne Jørgensen, Robert Glass, et al. We are an advisory research firm much like a Gartner or Forrester. What sets us apart from them is that we use our primary research to form our opinions, whereas they use their individual consultants. Neither they nor we can afford to give our opinions away for free. We have facilities, utilities, and personnel and we must, the same as you, be able to pay our bills. Just because someone asks a question, does not mean we will respond with an answer. In fact, we most likely will not. It is not rebuff or slight, it is just our business model to survive. Our current standard answer to a CHAOS inquiry is, first: please purchase our new book, “My Life is Failure” in our online store. If that does not satisfy you, then you need to join CHAOS University. If you do not find your answer or answers there then you need to purchase our inquiry services. Then we will work to answer your questions. Besides being a client you can get CHAOS information by participating in the research forum. If you would like to participant in our research you canjoin the Standish User Research Forum. You need to be an organization that develops software for your own use to qualify. This year we are offering a free reviewers copy of a book we just completed titled “The Pubic Execution of Miss Scarlet” as the honorarium. It is a tale of a fictional project in peril, but uses much of our lessons learned. Every year since 1995 we hold a CHAOS University and discuss many of these issue. It is strange that both Jørgensen and Glass have never applied or professed interest in joining us. Some answers can be found if you join us at CHAOS University 2007 or one of the many outreach events. So you can contribute to the CHAOS research by providing funding or sweat, but short of that you will and must be ignored by design. We do provide much of our research to the legitimate press and I have personally written many articles on our CHAOS and other research. We have given many talks to public and private organizations. In June, I presented some of our research to an IEEE conference in Philadelphia and the Delivering Project Excellence conference in Phoenix. This month I am doing a keynote talk to the National Association for Justice Information Systems in Boca Raton, Florida. As to Jørgensen’s two questions:Question 2: How was (and is) the cost overrun in the report calculated? The simple answer to this is that we ask“what was the estimated cost verses the actual cost?” We use a number of different instruments to collect the data and derive our overrun assessment. While we really do have a rocket scientist on staff, this is basic math. On the first question and any other questions you will have to read “My Life is Failure” to find those answers.

Really? Purchase our book and our inquiry services? Join our “university”? It’s clear that the CHAOS report is a business, not science. We must stop treating it otherwise, and we must call out those who do not.

About Jorge Aranda

I'm currently a Postdoctoral Fellow at the SEGAL and CHISEL labs in the Department of Computer Science of the University of Victoria.
This entry was posted in Academia, Software development and tagged , , . Bookmark the permalink.

9 Responses to Standish, the CHAOS report, and science

  1. Lorin Hochstein says:

    Fortunately, we don’t need to depend on the Standish group for this type of data anymore. For example, check out the article by El Emam and Koru in the September/October 2008 issue of IEEE Software (doi:10.1109/MS.2008.107).

  2. Jorge says:

    Thanks for the pointer, Lorin!

  3. Lee Fischman says:

    Since the CHAOS results are so controversial, I’ve decided a grass roots effort might be interesting. I created a single-question survey here:

    http://www.surveymonkey.com/s.aspx?sm=oOq7Hzgz6BCYZfgZoNC72w_3d_3d

    And will be posting results here:

    http://swprojectsurvey.blogspot.com/

  4. Pingback: Endlich - In die Tonne mit dem Chaos Report der Standish Group - Daniel Liebhart

  5. Pingback: Against SEMAT « Catenary

  6. Pingback: Data and science in enterprise computing « Semantic Werks

  7. Michael Flum says:

    I didn’t really think this was a surprise to anyone. It’s not touted as science, it’s survey. Like any “Best Practices” study, it is entirely empirical, largely subjective, and clearly doesn’t have the controls of a formal scientific study. As my 10 year old would say, “Doy”. Does anyone use this proposing it is fact and certainty in all cases? That it must be followed and adhered to as gospel? Seriously? It did have the unique aspects of timeliness and industry feedback previously not obtained. Hardly conclusive, but usefully indicative of problems throughout the industry. Certainly a good justification for scientific research, wouldn’t you say? Instead of pointing out the obvious, use it for what it’s worth.

  8. Pingback: Most interesting links of June ’13 « The Holy Java

  9. Pingback: Best practices for software development | Dave Nicolette

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s