There is a call for action making the rounds among software researchers. You may have heard of it: it goes by the name of SEMAT (Software Engineering Method and Theory); Ivar Jacobson presents it as a “revolution” whose “goal is to re-found software engineering as a rigorous discipline.” You can read the call for action here. The list of adherents is rather impressive and includes several people I admire deeply.
But this is an entirely misguided movement. Let’s analyze the call for action, bit by bit. It begins:
Software engineering is gravely hampered today by immature practices. […]
We start with what seems to be a minor point, a mere choice of words, but the problems begin by setting this supposedly re-foundational effort firmly within the old paradigm that got us where we are now. Thus we do “software engineering” research, and we should do what other engineers and engineering researchers do. Never mind that people in the community (most prominently Tom DeMarco) have spoken against this paradigm, or that software development has little to do with most established engineering disciplines: the people behind SEMAT have chosen to ignore these criticisms and move on.
Note that the SEMAT call for action presents the current state of the software industry as “hampered” by “immature practices,” which may be true (we’ve only been developing software for a few decades; our practices are necessarily immature), but as an observation it is symptomatic of the software-crisis hypochondria that afflicts many software researchers. The call continues:
[…] Specific problems include:
- The prevalence of fads more typical of fashion industry than of an engineering discipline.
This is how the consequences of the engineering paradigm I mentioned above manifest themselves. It’s true that the software industry is notoriously susceptible to fads. But notice how the alternative to these fads, according to the signatories of SEMAT, is to behave like an “engineering discipline.” That is, they offer a dichotomy between the archetypical fad-based industry (whimsical, pretentious, superficial, frivolous) and the archetypical measurements-based industry (conservative, positivist, serious, professional).
In fact, some “fads” end up being pretty good ideas. Object-orientation is mainstream today; it was once a fad. Extreme programming has been shown to be more effective than its alternatives under many conditions; some people still label it as a fad. There is a danger of labelling any grassroots innovation, any experimentation by practitioners, as a “fad”, as not conforming to the seriousness of our engineering discipline —there is a danger of stifling progress by an appeal to an unsuitable paradigm.
(This criticism doesn’t even get into the problem that several of the fads that have been inflicted on the software industry were started or promoted by some of the signatories!)
Let’s continue with SEMAT’s list of problems:
- The lack of a sound, widely accepted theoretical basis.
- The huge number of methods and method variants, with differences little understood and artificially magnified.
I agree with the first of these points, and I have argued in the past for the need to develop better foundational theories in our field. The second point, however, makes me doubt whether the signatories realize the implications of the first. We have plenty of evidence that software development is not homogeneous. Developing an operating system is not the same as developing videogames; developing mission-critical control systems is not the same as developing mission-critical data warehousing scripts. Different settings demand different methods. This diversity must play an essential role in our theories —unless we want our theories to be disconnected from the evidence we have collected.
In fairness, what the SEMAT adherents probably mean is that we have far too many academic software processes. Whole PhDs have been built on the basis of designing supposedly new methods, practices, or tools (really just a variation of old ones, under a different name) with almost no empirical validation. If this is the case, if what SEMAT refers to is our pathological growth of academic software methods without empirical validation, then I would agree with this point.
We continue:
- The lack of credible experimental evaluation and validation.
- The split between industry practice and academic research.
In this case I agree with the second point, but I partially object to the first. Industrial practice and academic research are certainly disconnected, and we need many more researchers bridging this gap if we are to solve the problem.
In fact I, too, believe that we need credible experimental evaluation and validation. I only object to this point because of its emphasis on experimentation, not on empiricism in general. This is another example of locking ourselves in the engineering paradigm: we want controlled experiments and measurements, the rest of our empirical work is not worth mentioning. As a researcher that has applied both experimental and non-experimental empirical methods in software development, I have seen that controlled experiments tend to abstract away the important elements of the phenomena we study, whereas non-experimental methods, such as case studies, provide a necessary richness to our evidence. Software development is hard, in part, because of the numerous elements and interactions in the socio-technical systems in which it takes place. These elements are almost impossible to replicate in the lab. Controlled experiments only give us a narrow piece of the story.
The SEMAT call for action then switches gears and offers a proposal:
We support a process to refound software engineering based on a solid theory, proven principles and best practices that:
- Include a kernel of widely-agreed elements, extensible for specific uses
- Addresses both technology and people issues
- Are supported by industry, academia, researchers and users
- Support extension in the face of changing requirements and technology
SEMAT, then, proposes to re-found software theory by decree. We decide to pick up “a solid theory” (which one?) that includes “a kernel of widely-agreed elements.” (which ones?) I simply cannot think of any scientific field that has prospered under such an epistemological basis. If we have a kernel of widely-agreed elements, that in itself is our theory and there is no need to refound anything. But we do not have a theory precisely because we do not yet have a kernel of widely-agreed elements; we cannot create such a kernel out of thin air.
Furthermore, I cannot see how such a kernel can arise out of the combined work of the signatories. The list includes, for instance, Alistair Cockburn and Watts Humphrey. It includes Larry Constantine and David Harel. It includes Ken Schwaber and Barry Boehm. The list itself suggests that, more than a kernel of widely-agreed elements, what we already have is several competing conceptions of software development, several vaguely formulated theories, that need to be refined and tested against each other. This is, I believe, where we should spend our efforts.
The rest of the items in SEMAT’s proposal are mush. Of course our theories need to address technological and social issues. Of course they need wide support by several communities to be successful. Of course they must be flexible. But what should they consist of? What stake is SEMAT putting on the ground? Unfortunately, beyond a wish to be more like an engineering discipline, this proposal is completely vague, and therefore I cannot support it.
Catenary 
Well said.
I would add one more complication to illustrate the heterogeneity of software development. Not only are operating systems different from video games, video games are different from each other. And not only are two video game development processes usually incomparable, I’ve seen no evidence to support the assumption that the set of appropriate software development processes remains constant over time within a single code base.
Pingback: The Third Bit » Blog Archive » Nothing to See Here, Folks—Move Along, Move Along
Jorge – Thanks for writing this. I too got the SEMAT email in my inbox. My reaction is that if these people think there is something seriously wrong with the discipline, they need to roll their sleeves up, produce a precise description of the specific problems, come up with potential solutions, and then evaluate the solutions. Oh, I guess that’s research. They don’t seem to be aware there’s already a lively research community doing exactly this…
Steve
‘software-crisis hypochondria’ – beautiful.
It astounds me sometimes just how ignorant some of these people are of research. Sure, some (most?) research is irrelevant and unsupported, but there is plenty of good stuff being done.
Just the other day I heard Dave Thomas (prag Dave) talk about code reading as if he’d just discovered it. He was seemingly completely ignorant of the rich and relevant research being done in software maintenance and software comprehension.
I’m not sure whose fault this gap is.
I completely agree with you Jorge, thanks for expressing the issues so well.
I have also experienced some of the complexities of applying experiments in an SE context, and agree with you that in many situations case studies can be more effective, especially once you get into new tools or techniques that involve the social aspects of Software Engineering to any large degree.
And I agree that there is often a gap between academic work and practice; however, we as grad students know, we are constrained by practices, traditions and perceptions that sometimes conflict with a desire to create practically applicable research. It can be difficult, especially in Computer Science, to get a degree or to publish without creating your own new widget, process, framework, whatever; and without some sort of implementation, algorithm, or proof. So although we may be very aware that the world does not really need yet another development process/modeling notation/framework/extension on any of the previous, we are also interested in graduating without too much of a struggle.
What would really help an initiative such as SEMAT would be a change in what constitutes good research, especially for graduate students. We would be well-served by a greater emphasis and regard for good empirical evaluations as well as comparisons, applications, and refinements of existing techniques. But we seem to have a fascinating in research for all things new, instead of application, refinement and improvement of existing work.
Excellent analysis. You put into words what I feel in my gut. Thank you.
Thanks, everyone, for your comments so far.
Jennifer, you bring up a very good point –we get the kind of research we do in our field largely because of the way the system is set up. We want to get a degree and we realize that they give them away for new processes or frameworks or notations, so we go ahead and create our own.
Pingback: Fourth Medium Consulting Inc.
When I read the semat manifesto it gave me a sense of deja-vu. Ivar Jacobson managed to get some very bright people to compromise on RUP. If you go back to the work of Grady Booch, it was much better before he became involved with RUP. Semat seems to be a new attempt along the same line as RUP.
Thanks, Niklas for reminding me of what a mess the Rational Unified Compromise made of Booch’s clear ideas and more usable, readable symbology.
I’m hoping to have some time to fight the battle from within because fighting UML non-design from the outside didn’t work.
If you are interested in history and the difficulty of influencing UML from out side, you should look at the OPEN Process Framework’s book on the OPEN Modeling Language (UML), which explicitly sought to make the notation more intuitively easy to understand and remember. A classic example is UML aggregation which clearly came from relational database design. The arrow points in the wrong direction and the Rational Rose tool even had you draw the arc in the wrong direction for object orientation. UML goes from part to whole (as in relational database tables), whereas OML went from whole to part (as in OO). We also used a Plus sign (to represent a philips head screw holding the parts together for aggregation) and a U (to represent a bag holding elements for sets and the like that have no relationships among parts). We, however, did not have Rational’s marketing budget.
Sigh.
I was a fan of OPEN and also the graphical modeling approach (grounded in theory) taken in Solution Based Modeling.
http://jbrugge.com/blog/2007/05/10/ode-to-solution-based-modeling/
I was kicked off the UML beta list for pushing Booch to admit that they had NOT consulted any experts in graphical communication when they designed the UML graphical communication symbols.
You talk about NOT consulting any experts in graphical communication when designing the UML graphical communications symbols. If my memory serves me, you may be granting the Three Amigos too much credit wrt “designing” the UML symbols. If my memory serves me, it seems that UML notation was basically just a rehash of OMT notation (primarily from the relational database members of the OMT team) mixed with some use case notation from Ivar. By leaving architecture to Grady and use cases to Ivar, Jim was able to essentially take over UML notation, which became the visible face of RUP. A very smart move if you want to position yourself as the first among equals in the eyes of people new to RUP and UML. Similar, it seems, to what happened to the many coauthors of the OMT book, whereby OMT eventually began to be seen as the Rumbaugh method. Sadly in this world, popularity and credit often have more to do with marketing and politics than technical excellence.
Jorge,
I do not agree with your point of view. (it look like I’m the only one based on the comments here).
I think its safe to say that we’ve all seen many different methodologies, but you can surely see that they all look pretty similar behind the surface.
If we put those different methodologies next to each other, and look at them from a distance I think we should be able to see the similarities and also clearly see in which way they differ from each other.
Now that is exactly what the SEMAT initiative is all about. Try to find the common elements in all the different methodologies, and rephrase them in a neutral, method agnostic way.
This should end up as a sort of framework that then allows us to specify the details of HOW we do things as opposed to WHAT we do.
In fact the WHAT we do is pretty much the same for all methods, but it gets far too less attention.
If we read the manuals and books about methods they all focus on HOW we should do something, not WHAT this something is about.
If indeed the SEMAT turns out as I hope it will, I don’t think it will stifle new initiatives or methods, but rather encourage it.
Look at it this way. If you now have a good idea on how to do requirements, or testing, or information modeling, the only way to get your message across, and sell this way of doing things, is to bundle it in a complete method. So you’ll have to incorporate lots of things that aren’t really new, but that are needed to get from a problem to working software.
Now image we have the SEMAT framework, where each of those activities are described in theory, and each of those have several practices that can be applied.
Now the only thing you need to do to get your great idea incorporated in the process is just add another practice to that activity.
If your idea is about information modeling that is the only thing you need to focus on. You no longer have to write stuff about testing, requirements etc… because these activities have already been described by other people who are probably better in this specific area.
I just hope it the whole SEMAT intiative turns out how I sketched it above.
Geert,
I can see the hypothetical appeal of SEMAT, I just don’t see how can it be brought to fruition.
I’ve seen many methodologies, yes. But what I see is that some of them are pretty *different* beneath the surface, and that in fact they are at ideological opposites. What happens, for instance, if my idea is not about how to do requirements, but about how requirements may be the wrong approach to the problem of software development? Will SEMAT be method-agnostic enough to say that a requirements phase is optional? And if anything can be made optional, how is the essence of SEMAT different from what we already do in practice?
I’ve also seen enough software development domains to understand that if we want a set of truths that will work across the software industry, it will have to be very general – far too general to be of use.
I admire many of the people that signed the initiative; I just don’t see how they can make it work satisfactorily.
Jorge,
It seems like we’ll have to agree to disagree. Only time will tell if your worries are have been correct. The fact that so many of the brightest minds in the industry seem to think this is a viable initiative is one of the reasons why I’m willing to give it the benefit of the doubt.
On the subject of the fundamental *differences* in the different methodologies, could you illustrate that with an example?
To me it looks like methodologies and religions have a lot in common with regards to that aspect. They tend to very strongly emphasize that they are the only one that is correct, and all the others are completely wrong in all aspects.
For me as an agnostic (both on methodology religion) they still look pretty much the same, carrying the same values and goals, only sometimes by different means.
As for your example of the requirements, the question is whether you are really saying requirements is something you don’t do, or whether you are saying that you don’t formally write down the requirements.
In the latter case you are actually doing requirements, only the *way* you are doing them is different. In stead of writing them down in a formal way you keep the requirements in your head; which could very well be the best approach for a specific problem.
Finally I would like to emphasize that SEMAT is not searching for *the* final method that will replace all others.
To the contrary, it is searching for the theory that will provide a sound foundation to build all those different methods upon.
Geert,
Thanks for the follow up.
The analogy of methodologies and religions is good, to an extent. It’s true that all methodologies have somewhat similar goals. I don’t think it’s true that all methodologies hold the same values.
You asked me for an example of fundamental differences, and that’s what the requirements example was meant to be: not just whether you write down your requirements, but whether the word “requirements” even makes sense in your domain.
There are many people who argue that software development should not be approached as something that requires a methodology, that the engineering paradigm is wrong. SEMAT, by its emphasis on methodology, chooses to be against them.
Jorge,
You have brought up the practical side very well. I was just wondering why don’t we fight the battle from within SEMAT since it is still in the evolution stage? Here are some of my thoughts.
While building and maintaining software, we need to take care of two diagramatically opposite systems, namely the production system and the innovation system.
The production system needs to be predictable, repeatable, measurable, deterministic, hierarchial and low risk. On the other hand, the innovation systems are uncertain, exploratory, judgemental, ambiguous, cross-functional and high risk.
By whatever name we call it, software engineering MUST help us balance these two systems in the most approrpiate way for a given situation.
Prabhakar,
Thanks for your comments. I have two replies. First, I think it’s fine to make a “production” and “innovation” distinction, as long as by “production” we mean to compile code, burn CDs, etc. I think coding requires innovation, and that it is very much unlike industrial production.
Second, you ask why don’t we fight the battle from within SEMAT. I’d ask a similar question to them –why create a separate entity to address the questions that our research community is addressing anyway? My fear, and I hope I’m wrong, is that they choose to do so to steamroll a particular conception of software development without worrying about the peer review process and about demands for solid evidence.
Jorge,
I’m one of the Signatories of SEMAT, and I agree with some (NOT ALL) of your objections. As and see the SEMAT Initiative as Prabhakar suggests, I believe that inside SEMAT is the best place to address your concerns.
Another thing, one idea that I will push is the idea of situational method engineering. Systems, projects, organizations differ so much in so many ways that their is no one best way to engineer software. That was a failing of RUP, which I feel tries to be a unified process rather than a unifying framework. I recommend developing a set of reusable method components and create appropriate situation-specific methods by selecting, tailoring, and integrating the relevant method components. There are quite a few concerns that must be addressed to make situational method engineering work, but it does give you the advantages of standardization (metamodel and method components) with the advantages of flexibility (method component selection and tailoring).
Given the risks of trying to get big names in the SE community to agree reminds me of the following joke (which sadly is less politically correct now than when it was first formed):
What is the difference between a methodologist and a terrorist?
Answer: You can negotiate with a terrorist.
Donald,
Yes, I recognized your name in the SEMAT website – I’ve been lurking at the re-online mailing list and have read many of your messages there.
Thanks for the thoughts (and the joke). It seems we disagree, but I’m glad to hear that you’ll be pushing for situational method engineering —I would, too, if I were constrained to work on this kind of framework.
I applaud any effort to bring some level of rationality to the field of software development, but I do retain a sense of scepticism regarding its potential long-term effect. In many fields, “best practices” often become “writ in stone” and remain long after their applicability or usefulness has passed. That’s bad enough, but it can get worse; a variety of certification schemes often complicates issues when “gatekeepers” far removed from current proctices enter the picture.
On the other hand, I’m an inveterate optimist, so let’s get started!
Thank you for this, Jorge.
As someone who just signed up, I have to say that I’m heartened by SEMAT’s aim to include the “people” aspects – and by the combination of competence and good intentions.
Your posting has provoked some (mostly) useful dialogue along the lines of “Why we disagree about software development”.
Hi Jorge,
very interesting and controversial.
I also read Neil’s blog and would like to focus on the engineering-similarity issue:
think, it’s simply very much like physics and engineering. e.g. as well as there is statical mechanics and civil engineering there is/ should be software-science* and software-engineering.
Is this so far completely/ partially contradictive to your position? (see also ‘production’ and ‘innovation’ comment above)
*or what ever it should be called
ModelPractice,
I don’t mind making a distinction between “software science” and “software engineering” if by engineering we mean the application of science. Unfortunately that’s not what we usually mean in our field –“engineering” is a very loaded word.
Oh yes. The civil eng. departments at university also do research (science), but they don’t think of themselves as physicists and vice versa.
Pingback: Parijat’s Weblog
I beleive that the software community would do itself a big favor by dropping the phrase software “engineering” like a bad habit. I suggest that the specialilzation that engineers supply is applying formulae like F=ma, or PV=nrT, to create useful products. While there is great variation in the path from the formulae to those products, the formulae at the root of the path are not subjects of debate. I am unaware of any similar foundation underpinning software development and I wince/cringe each time I hear the phrase Software Engineering.
In this sense I strongly support Jorge’s recommendations to avoid using engineering disciplines as a role model.
Blake, your “engineers apply formulae” is probably a man-in-the-street impression but certainly not what engineering is about.
If you look at what is required for Chartered Engineer status, and particularly the continued professional development provisions, you will see that engineering is about a managed process applying technical knowledge.
http://www.engineersaustralia.org.au/
”
To maintain chartered status you must log a minimum of 150 hours over a three year period:
At least 50 hours must relate to your area(s) of practice
At least 10 hours must cover risk management
At least 15 hours must address business and management skills
The remainder must cover a range of activities relevant to your career & interests.”
The basic problem is that software is not an independent discipline. Software is simply an implementation choice that extends the solution space for an engineering problem to allow more complex algorithmic solutions than are practical by other means.
The foundational issue falls rather in the larger engineering domain to identify when such solutions are either required — e.g., due to complexity — or are desirable — e.g., cost (total lifecycle), flexibility, etc. It is the concept of elevating software to an independent discipline that has led to what people view as a software problem — the problem is not software, it is the choice to use software ubiquitously, expecting software to be able to address complexity that we have not yet learned to express well. Back to the old saw: “If you only have a hammer, you approach every problem as a nail.” ANY tool that we choose to apply to problems of the complexity that have been assigned software in the last 15-20 years would have difficulties addressing them all.
Find an alternate solution to deal with complexity … the human brain works pretty well … emulate it …
I would say that “The split between industry practice and academic research” touches a very significant issue. That is that:
a) academic research is not generally conducted with the variables and pressure that practicing in an industry applies.
b) developers fail to see themselves as users, both of the software which they are producing, and also the code which they are creating and will have to work with later.
Currently the “best practices” developers have been taught, and the process required in order to produce a marketplace product are very often divergent. Often this is a result of the layer which a programmer is building on top of, hence the need for programmers to see themselves as users of their code, and academics to live in the real world thereby enabling best practices to cover the creation of good layers on top of less-good ones.
I agree –one of the best explorations of this divergence that I’ve seen is Martin et al on “Good Organizational Reasons for Bad Software Testing”:
http://portal.acm.org/citation.cfm?id=1248890
@Si, firstly when you comment on the “best practices” developers have been taught – are you referring to them being taught in an academic institution or practices they are taught in the workplace or by courses outside academia?
Secondly, I disagree strongly with the word “required” – I’d phrase your point as “the process ‘used’ to produce a marketplace product” as I think there are a lot of suboptimal processes in use but not necessarily required.
I have worked in organisations that thrived despite woeful software practices and another that went under due to business reasons independent of the software engineering (on time, bug-free and on budget!).
Delivering a marketplace product is a wider scope than software engineering and includes a lot of business considerations. The role of Product Manager is one of the innovations bridging those worlds.
Pingback: Notional Slurry » links for 2010-02-25
Pingback: Software as an engineering discipline - Multi Shoring - A Perficient Blog
Jorge, thanks for writing this. I had a similar reaction reading about SEMAT.
In particular, I doubt that creating yet another method will solve any of the problems. Education might, standards and regulation certainly will – most if not all of the established engineering disciplines have come to where they are not last through regulation, which got started because too many people suffered from poor practices. Observing that software (and thus unfortunately bugs) are a crucial part of many widely used products these days, some of which have the potential to harm people (say cars, for example), I’d propose we’re not too far from regulation starting at large.
Apart from this rather dark (realistic?) view, here’s another thought: what SE could really profit from is to adopt the way in which other engineering disciplines have established *properties* for their products which allow very precise specification. IMHO SE is yet to establish such properties for software, i.e. a universally agreed standard to specify and thus measure the properties of a software product in a precise, unambiguous manner.
It seems to me that SEMAT is trying to find a GUT (Grand Unified Theory) in computer sciense, similar as physicists try do in physics. I do belive though, that software is much to diversified for that to be succesful. although I certainly understand, and sympathise, with the reasons to do so. SEMAT will eventually fail and fade away, probably forgotten by most people, and the endless debate on how to develop software will continue.
Pingback: Blog commentaries about SEMAT « SEMAT Iberia Chapter
Thanks very much for this critique! Although this is already an old topic, we continued the discussion within SEMAT. You can find our discussion paper here: http://karismolander.net/2013/04/forming-theories-of-practices-for-software-engineering/
Thanks, Kari! I’ll read your discussion paper.
Pingback: So: What does “#NoEstimates” even mean, Anyway? | cumulativehypotheses
Pingback: SEMAT – Software Engineering Method and Theory – InfoQ.com - Trending Topicss