I think one of the reasons that make many people uncomfortable with qualitative research lies in the difficulty of doing data analysis properly. You prepare a study design with carefully worded research questions, you have a bunch of interview guides, you do your observations, perform your interviews, take lots and lots of notes, and then you’re supposed to take all that and bring it down to a set of easily communicable findings and (if your data and epistemological persuasion allows you) generalizations. That data analysis step may feel phony, a bit like that old classical cartoon:
In his Case Study Research book, Yin acknowledges this is the step that is least developed, methodologically speaking. And though we may attack it with all kinds of coding and techniques and structure, nothing guarantees that we’ll actually extract the essence from our data. You actually need to think, to question yourself, and to try many things, see them fail, and try again. That’s what makes it hard.
I used to think this was a drawback of qualitative research when compared to its quantitative sibling and I was not alone in my belief. But I’ve come to realize that, deep down, quantitative research suffers from the same problem—it’s merely often ignored. In an experiment, for instance, you may observe an effect in your sample, and you may be able to generalize to your population with enough statistical significance. But this (in a field like mine) doesn’t get you very far. You then need to examine whether the effects you observed would hold in an uncontrolled environment, with lots of other confounding factors, in different contexts and for people and organizations with differing motivations. Your experimental data does not help you there—you again need to think, to question yourself, and so on, though if you have good numbers you may be giving short shrift to those concerns in a rushed “Threats to Validity” discussion near the end of your paper.
(Incidentally, Yin also talks about this in his book—he discusses the difference between generalizing to a population and generalizing to a theory, and how experiments need to—or should—do both, but case studies, by design, are usually only concerned with the latter.)
As I am currently at this stage with an empirical study, my mind goes back to this idea of the difficulty of doing this kind of research. I always feel like there’s a thread hanging up there somewhere—that if I jump high enough I can reach it. Or as if there’s a hidden melody that I can uncover if I listen intently. And I try and grope, and sometimes I find something and I feel exhilarated, and sometimes I find nothing and I want to just forget about it all. But if there were a straightforward list of steps to follow, this wouldn’t be research, and it wouldn’t be interesting.