In the past I’ve made it clear that for software research I personally prefer to do qualitative case studies over other empirical methods. I can see the value in data mining, experiments, and surveys, but I’ve argued that those methods have significant weaknesses that are often unacknowledged in our field. In fairness, however, qualitative work is no less susceptible to pitfalls, and I think failing to make the weather/climate distinction is probably one of the greatest.
To explain what I mean, think of our reaction to the weather. Lately it seems that whenever we get a cold snap or a freak weather event somewhere, some people will come out and say that this is clear evidence against global warming. The latest example is the recent snowstorm in Northeast US. Stephen Colbert summarizes the problem brilliantly. You can watch his video on the Huffington Post (I would link from it here but the Comedy Network won’t let me):
After showing clips of Fox News correspondents explaining that the weather is burying Al Gore’s “hysterical” theories, Colbert joined in on the ridiculous logic, deeming it “simple observational research: whatever just happened is the only thing that is happening.”
Using the same rationale as Fox News, Colbert couldn’t help but point out that, due to it being nighttime, the city was covered in darkness. “Based on this latest data, we can only assume that the sun has been destroyed.” Who’s to blame for this “forever-night”? Gore, of course.
Failing to discriminate between weather and climate is a novice mistake in understanding climate change, of course. But I wonder if those of us doing qualitative software research are sometimes caught off-guard by the equivalent mistake in our own observations. We join a team for a few hours, days, or weeks, and report back on the “weather” we observed as if it was the “climate” of the team. Without a historical record, it is quite difficult to determine whether what we are observing is representative of the climate under study, or rather, which elements of our observation are representative and which are freak weather events.
As an example, last Summer I was doing fieldwork on a software organization. The very first day of my observations I sat with a team that was going through a crisis: their demo wasn’t working as expected, their client was getting restless, the team had been growing significantly to meet their long list of urgent requirements, and many team members were rather frantic and tense. The second day of observations was similar, but milder. By the fifth day there was still some underlying tension; the crisis had dissolved, although some of its causes were still latent. Two weeks later the issue was mostly under control.
While I was conducting my observations a couple of people in the team told me, on several occasions, that this was not the way their team usually worked—in other words, I was observing a snowstorm. I made sure to take note of that, but I confess that on a more instinctual level I didn’t fully believe them. It was only later, when I repeatedly saw the team in a different and calmer context, when I interviewed people to get their impression on what is their usual climate, that I came to understand to what extent my initial perception had been misleading.
From a research perspective, this turned out to be a great event to witness: you rarely get the chance to watch an evolving snowstorm with your notepad ready. But it’s only a great event when I realized it was unique and treated it as such; I can think of several other qualitative studies, mine or from others, where the danger of extrapolating from unique events hasn’t been fully explored or acknowledged.
How can we address this problem? My preferred mechanism (and the one Robert Yin suggests repeatedly) is to collect and triangulate information from as many sources as possible. It also helps to be constantly skeptical of anything you hear; I’ve lost count of how many times I’ve been given conflicting accounts of the same events. And of course, you should know that the first few days of your observations are going to be rather unrepresentative simply by your presence; you’re altering the weather yourself.
Ultimately qualitative work is impossible to pull off spotlessly, and there’s always a nagging fear that you’re over- or under-representing some observations, that you’re not doing things right. Then again, no matter which empirical method you choose, you should have that fear, and you should act on it.