A controlled experiment with three hundred paid subjects participating for a full day is flimsy evidence in the medical field, but for software engineering researchers it is huge. In the February 2007 issue of the Transactions on Software Engineering journal, Arisholm et al., from the Simula Research Lab in Norway, report on a pair programming study with 295 professional Java consultants . One third of the participants worked alone, two thirds worked in pairs on the same problems.
Pair programming –that is, working shoulder-by-shoulder with another developer in the same computer– is one of those practices that has a few vocal convinced believers and a large majority of outsiders remaining silently skeptic. The standard claims are that these pairs are as efficient as individuals, or more; that they produce code of higher quality since everything is being verified as it is written; and that knowledge about the project flows better in the team since at least two people know how every bit of the system works, leading to all sorts of qualitative improvements.
The study of Arisholm & Co. explores the first two of those claims. From the abstract:
“The results of this experiment do not support the hypotheses that pair programming in general reduces the time required to solve the tasks correctly or increases the proportion of correct solutions. On the other hand, there is a significant 84 percent increase in effort to perform the tasks correctly”
So no improvements in time or quality were observed. The results also indicate that pair programming produced higher-quality code in more complex problems, and that junior consultants working in pairs benefit mostly in terms of correctness, while more senior consultants benefit mostly in terms of effort.
Although it’s an unusually strong experiment, mainly due to its magnitude, it has two important biases. First, against pair programming: Most subjects had no pair programming experience (while every developer has solo programming experience), and they had to get to work with strangers, so the benefits of knowing your pair’s work style and habits did not have a chance to ramp up. Everyone that has seen team members develop trust and collaboration over time would agree that evaluating recently formed pairs is artificial.
Second bias, in favour of pair programming: The real comparison should not be against solo programmers, but against pairs of developers working on the same problems, as a team, but on separate computers. For instance, Joe may choose to work on problems 1, 2, and 3, while Jane works on the much more complex problem 4 all along. The pairs of solo programmers would need to be familiarized with each other as well, of course, so that their long-time collaboration has a chance to kick in.
Still, this is the heaviest data point in pair programming research to date, and it’s worth taking a look at it if the topic interests you.
 Erik Arisholm, Hans Gallis, Tore Dyba, and Dag I.K. Sjoberg. “Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise.” IEEE Transactions on Software Engineering, 33:2, Feb 2007.