Well, my ICSE talk has come and gone, and I’m very happy with it and with the feedback I got. Now, with the talk behind, I feel it’s a good idea to give a brief summary of what the paper Gina and I wrote is all about. I should say though, if you’re interested in the topic, it’s much better if you read the paper than my summary. You can also download the slides I used during the talk, although they miss much of what I (and the paper) said.
To understand this work you may need a bit of context. In recent years there has been a lot of research that mines repositories to discover stuff about software projects. For instance, you can extract a list of people that changed the code of one file, then you can extract the number of emails that they’ve sent to each other during the same period, and through some math magic you can find out if these people coordinated properly (that is, emailed each other enough) to work on the code – because if they didn’t, they should’ve.
This is a simplification, of course, but that’s the main idea: mine software repositories to discover how teams (ought to) coordinate and communicate. A lot of this research is quite interesting and useful, and the people that do it are extremely smart. But I often find their conclusions questionable because they make one important, dangerous assumption: that the electronic repositories they mine provide an accurate, or at least appropriate, picture of what really happenned in the projects they come from. For example, can we really assume that if two developers didn’t send each other an email on a particular period then they were working uncoordinated? It’s an (until recently) untested assumption, but that doesn’t seem to give much pause to the researchers in the area.
This is the assumption Gina Venolia and I set out to test. In my talk I made an analogy between paleontologists and entomologists: when it comes to studying bugs, the first are stuck with fossils as the only evidence they have, while the second have relatively easy access to live specimens. They have a much easier time confirming or refuting simple hypotheses because of this. Paleontologists would love to have that kind of opportunity. And yet, it seemed to us, our community seems more comfortable handling fossils than venturing to study live specimens.
We wanted to find out the consequences of this focus on fossils (or electronic records) over the real deal. We investigated in-depth the stories of ten bugs at Microsoft – a friend rather accurately described the process as one of investigative journalism. We tried to find out all possible information about the bugs in question. At the same time, we kept track of what the story of each bug would be like if we didn’t have access to interview the people involved, or if we hadn’t spent time making sense of little oddities of each story. In effect, we compared our in-depth stories with the (rather plain) story you get if you’d only mined the same data. When we were almost done with our case study we conducted a survey, also at Microsoft, to validate its findings.
The differences between the two kinds of stories are huge. We spend a good deal of the paper describing them. In every case the electronic records are missing important information; for most they actually contain erroneous information. In the paper we make the case that a lot of the information that matters the most for each bug is the kind that is the hardest to document or to extract automatically: personal, political, social, tacit knowledge.
We also go to some depth in describing what the stories of bugs actually look like. We do not narrate the stories we found (each of them fills several pages); rather, we list and describe the most important patterns that we observed in them, as well as the goals that people appear to have when they are working on bug-fixing activities.
That’s what we did. After the talk, a friend told me that he thought our message was mainly that qualitative studies are better than quantitative studies. Though I do prefer qualitative studies I hope that’s not the message most people got: it’s not that qualitative studies are better, but that for questions of communication and coordination, many quantitative studies rely on data that are wrong or misleading in important ways, and the researchers that carry them out need to be extremely careful with their assumptions.
Photo credits: John Cancalosi (fossil), Andre Karwath (yellow-winged darted dragonfly)