Then a Miracle Occurs?

I think one of the reasons that make many people uncomfortable with qualitative research lies in the difficulty of doing data analysis properly. You prepare a study design with carefully worded research questions, you have a bunch of interview guides, you do your observations, perform your interviews, take lots and lots of notes, and then you’re supposed to take all that and bring it down to a set of easily communicable findings and (if your data and epistemological persuasion allows you) generalizations. That data analysis step may feel phony, a bit like that old classical cartoon:

Then a miracle occurs

In his Case Study Research book, Yin acknowledges this is the step that is least developed, methodologically speaking. And though we may attack it with all kinds of coding and techniques and structure, nothing guarantees that we’ll actually extract the essence from our data. You actually need to think, to question yourself, and to try many things, see them fail, and try again. That’s what makes it hard.

I used to think this was a drawback of qualitative research when compared to its quantitative sibling and I was not alone in my belief. But I’ve come to realize that, deep down, quantitative research suffers from the same problem—it’s merely often ignored. In an experiment, for instance, you may observe an effect in your sample, and you may be able to generalize to your population with enough statistical significance. But this (in a field like mine) doesn’t get you very far. You then need to examine whether the effects you observed would hold in an uncontrolled environment, with lots of other confounding factors, in different contexts and for people and organizations with differing motivations. Your experimental data does not help you there—you again need to think, to question yourself, and so on, though if you have good numbers you may be giving short shrift to those concerns in a rushed “Threats to Validity” discussion near the end of your paper.

(Incidentally, Yin also talks about this in his book—he discusses the difference between generalizing to a population and generalizing to a theory, and how experiments need to—or should—do both, but case studies, by design, are usually only concerned with the latter.)

As I am currently at this stage with an empirical study, my mind goes back to this idea of the difficulty of doing this kind of research. I always feel like there’s a thread hanging up there somewhere—that if I jump high enough I can reach it. Or as if there’s a hidden melody that I can uncover if I listen intently. And I try and grope, and sometimes I find something and I feel exhilarated, and sometimes I find nothing and I want to just forget about it all. But if there were a straightforward list of steps to follow, this wouldn’t be research, and it wouldn’t be interesting.

About Jorge Aranda

I'm currently a Postdoctoral Fellow at the SEGAL and CHISEL labs in the Department of Computer Science of the University of Victoria.
This entry was posted in Academia. Bookmark the permalink.

4 Responses to Then a Miracle Occurs?

  1. Neil says:

    Great post Jorge.
    Replication is important too. That way I can see whether e.g. the effect you saw at Microsoft, and the generalizations you claim, are indeed true at other places.

    I don’t think a single study of either type (quan/qual) really lets one claim insight into Truth.

    • Jorge Aranda says:

      I agree, and the difficulty to replicate is one of the drawbacks of qualitative work. You can’t aim for a literal replication, as you are dealing with an uncontrolled environment (though you can aim for theoretical replication—“if the theory is true, it should hold under these other circumstances…”).

      • Neil says:

        Do you think your ICSE conclusions would hold at IBM, for example?

        One concern is that if you make your theory *too* general, you can always find support (or rejection!). It seems a bit of a black art to make specific claims from qualitative research.

        E.g., “bug databases don’t contain all information” seems tautologous. I’m trying to think of a more specific claim that could be tested (been a while since I read the paper). Maybe something about prioritization?

      • Jorge Aranda says:

        “Do you think your ICSE conclusions would hold at IBM, for example?”

        Yes, absolutely. The only context in which I think they might not work is in projects that self-impose communication on only one or two channels. Many open source projects fall in this category. At IBM, the Jazz project may fall in this category too; I’m not sure. I’m curious to explore this.

        “E.g., “bug databases don’t contain all information” seems tautologous. I’m trying to think of a more specific claim that could be tested (been a while since I read the paper). Maybe something about prioritization?”

        The main claims from our ICSE 2009 paper were:

        * Bug databases miss information that is essential to understanding the story of their bugs.

        * Bug databases often include misleading and incorrect information to understand the story of their bugs (though these errors are not necessarily harmful for the organization itself).

        * A number of coordination patterns (included as a table) occur regularly in bug fixing activities. (And we speculate that these patterns could be used as the basic vocabulary of bug story descriptions.)

        * Thinking of bug stories in terms of phases or stages may help the organization, but it doesn’t help us as much (as researchers or tool designers). Thinking in terms of goals instead (we included a table of our proposed goals) gives us good insights for research and tool design.

        Of these four, the first two were more spectacular and received almost all the attention, perhaps due to the rise of the mining repositories community. But I was hoping that the other two claims would be a greater long-term contribution.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s