Why do old organizations die? Their death runs counter to our intuition of the nature of organizations as rational entities: if an organization has established itself and secured economic stability, then, through an efficient and rational management of its resources, it will almost always have a serious advantage over its younger challengers—and once it prevails over them, it will incorporate the spoils of its victories into its structure, presenting an even greater (at some point insurmountable) challenge to its future competitors. And yet, though we see them try to do this, we also witness an inability to keep up with the times, to understand the new context in which they need to perform, and to snatch the opportunities that appear before them. I’m not keen on anthropomorphizing organizations, but it is almost as if they, like humans, became rather sclerotic as they aged, or as if they lost their senses; their hearing dull, their sight blurred, and their teeth on decay, suffering from countless illnesses, slouching towards the fall that will end their days.
An explanation that I like for this phenomenon is the concept of organizational inertia. Broadly, organizations need to be reliable to survive (you wouldn’t frequent a store if you didn’t know whether you’d find it open for business, what kinds of goods you can expect to find there, at roughly what prices, with what quality, etc.). Reliability requires stability, but stability’s evil twin is sclerosis, or inertia. And so successful organizations (stable organizations) will exhibit a high resistance to change their ways, and this resistance, which helps them when the going is good, will be their downfall in the end.
(Note that this runs counter to the popular view of change as something that organizations do, and should do, all the time, and of managers as rational players pulling the levers of the social machinery and being generally in control of the activities and resources of their organizations. Organizational inertia tells us, first, that change is usually not desirable—it’s risky and unreliable—, and second, that when change is desirable, it is unfeasible. A corollary of this is that, when organizations claim to be changing, or to have changed, it’s a fair bet that their changes are superficial and inconsequential, mere wishful thinking or PR efforts.)
There are exceptions to this, of course, and you may be thinking of mentioning one or two in the comments. Generally speaking, though, the theory seems to be statistically sound—the book Organizational Ecology, by Hannan and Freeman, presents the idea and several empirical studies that support it. Hannan and Freeman, in fact, take the argument further: using Stinchcombe’s concept of organizational forms (Stinchcombe argues not only that organizations preserve their form over time, but that their form is a product of their time), they explain that we can actually see cohorts of organizations with essentially the same forms rising and falling, being created and becoming extinct, as if they were biological species in an ecosystem. Organizational forms do evolve over time, but Hannan and Freeman argue that this evolution is Darwinian, not Lamarckian: because of its inertia, an organization is stuck with the form it was given in its early stages, which is very similar to that of the rest of its cohort, and which worked well in the organization’s youth.
Now, in the context of software development organizations, what forms do we see? Though there could be many low-level variations depending on the domains we are dealing with, the concept of organizational forms is intended to be broad and comprehensive, not concerned with minutia. With this in mind, I see at least two clear basic forms to date: on one hand, traditional software firms, on the other, open source organizations, which have a social structure and a set of incentives, behaviours, and values, clearly different from those of traditional firms. It would be quite difficult for a traditional organization to transition into an open source group, and vice versa; this goes in accordance to Hannan and Freeman’s argument.
I had a harder problem determining whether Agile organizations have a form that is truly different from that of traditional firms (a question I struggled with as I write a paper on this topic). There are some indications that they don’t: their hierarchical structure is very similar to that of traditional firms, as are many of their goals and the incentives of their employees. Additionally, I hear all the time about firms “becoming Agile,” “switching to Scrum,” “testing XP,” and so on. If the change to Agile is as common as all that, then either Hannan and Freeman are wrong, or Agile, deep down, is not that different from business as usual.
But I’ve studied some Agile firms, and I’ve been impressed by the striking contrast they present when compared to traditional organizations. They favour generalists over specialists, enforcing knowledge sharing to an extent that seems wasteful to traditional firms. They embrace team autonomy and self-management to an extent that seems irresponsible to their counterparts. They welcome uncertainty in requirements and deliverables to an extent that threatens the long-term viability of their products, according to others. They insist upon having the customers adapt to doing things their way, with a zeal that seems to border on arrogance to more accommodating organizations. And they demand a level of personal interaction, of co-location, and of up-close communication that many developers find frankly uncomfortable and counterproductive.
It is these deeper elements, and the values from which they spring, that I think are what lies at the heart of Agile development. This is the reason why it’s so hard for an organization to truly transform its modus operandi. “We’re becoming Agile” doesn’t usually mean “we’re fully switching towards a model of autonomous, self-managed generalists working in close quarters to satisfy ever-shifting requirements whether the customer (and the boss) likes it or not;” it means “we’re asking our developers to write unit tests before they code, we’re keeping a backlog, and we’re referring to our project manager as the Scrum Master.” No wonder that Agile, for many, is simply a buzzword, and that they end up disappointed after their trials.
When I talk about truly Agile firms to managers and developers in more traditional software organizations, they often reply that it all sounds nice, but that couldn’t work in their setting. They would say this despite my assurances that I’m talking about firms that are contextually very similar to theirs, working in projects that they themselves could be working on. This would make me think that they must not be getting it, but on closer examination, and bringing in the work of Hannan and Freeman, I think they might be right: Agile demands a radically different arrangement of work and of responsibilities, and for most firms already used to more traditional software development, that may simply not be in their DNA.
I don’t know what will be the outcome of this market competition between Agile and traditional software firms. For a long time there may be room for both organizational forms, or it may well be that Agile won’t be as effective as traditional development, on the whole. It doesn’t help the Agile case that so many organizations have co-opted the term as a tired buzzword to impress their clients (nowadays everyone is Agile, or “as Agile as they can be”). But I suspect that the software development field today is a fertile ground for the underlying proposals and values of the movement, and that we’ll see them bearing many more fruits, under this or another name, in years to come.