Just a reflection spawned from a recent post by Greg Wilson and a later discussion at the lab (I had to leave early, so I don’t know how it ended):
In a research discipline with few or no mature constructs, such as ours, experimentation is little more than disguised, expensive, and poorly grounded argumentation. It is misleading: it proves nothing, it can be made to support almost anything, and it has an aura of “truth” disproportionate to the weakness of its real validity.
It is best to develop constructs and their underlying theories through argumentation enriched by qualitative observations, and to proceed to experimentation only when the constructs seem sound.
I realize that a great part of the appeal of experimentation is psychological. It is comforting to hear that someone ran an experiment and “demonstrated” this or that. But these demonstrations are flimsy, and will be superseded and forgotten once we develop a better understanding of the relevant constructs and theories.
4 responses so far ↓
Lorin // February 25, 2009 at 2:24 pm |
People who are influenced by the outcome of software engineering studies? I thought they were a myth.
Jorge // February 25, 2009 at 4:43 pm |
That’s what we’re trying to fix!
More on measurement « Catenary // February 26, 2009 at 11:46 am |
[...] About me Experimentation and argumentation [...]
Andy Ko and the semblance of objectivity in numbers « Catenary // May 31, 2009 at 10:01 am |
[...] I’ve written about this before. As a community, our excessive focus on quantitative methods is preventing us from developing the right constructs to study. It’s a shame that when researchers address this fundamental problem they’re shot down by their peers. [...]