Programming - A craft or an engineering science?
Manager: "I want some serious answers. How long will this program take to make?" And yet it's actually true. Before I justify this statement, let's step back a bit and take a wider look at what provoked this somewhat contentious idea. I was perusing the ACCU General mailing list, as one does, when someone asked whether people thought software engineering was a true branch of engineering. The question flowed out of a discussion on what the use of membership of the British Computer Society (BCS) was. There was quite a lot of discussion on the issue - hotly debated different views and the usual multiple branches off into different topics. One of the great things about ACCU General is that you never know who is going to hijack your thread and take it somewhere you never thought of. This got me thinking and doing a bit of research on the web. The first thing I had a look at was the latest 44 page magnum opus entitled 'Engineering Values in IT' from the BCS, Royal Academy of Engineering, and the Institution of Engineering and Technology [1]. Predictably, the report's conclusions fell into the "programming is now mature, therefore it should be a profession, and we should have its practitioners required to have professional chartered status" viewpoint. To quote from the document summary, "The concerns of this report relate to the area of IT dealing with the development of software-based IT systems. This area of IT has benefited greatly from the input of engineering methods and skills. It is the argument of this report that IT systems engineering has now reached the point of maturity where it is a codified discipline that must be recognised as a required capability for system developers. Like all branches of engineering, IT systems engineering has a basis in scientific theory." I think they should give a citation to the Wizard of Oz who first pointed out that you don't need a brain, you only need a diploma... [2] The problem is that, among other things, the idea that the 'maturity' of a discipline is a sufficient condition for declaring it a profession requiring the membership of a professional body to practice is unproven. The scientific basis of software engineering is also a matter of opinion. I posted to ACCU General, pointing out that the engineering disciplines are based on physical laws, and asking what physical laws programming was based on. No one came up with an answer. Neither did the BCS et al document. So what else is out there? Well, as it happens, the well respected Tom DeMarco, inventor of the term software engineering, has recently written a redux on this very issue, in a short and pithy piece in 'Viewpoints' entitled "Software Engineering: An idea whose time has come and gone?" [3]. It's only two pages, but it takes a sharp look at his 1982 book on software metrics [4], and comes to conclusion that he was wrong. Wrong because you can't measure everything, and wrong because you can control what you can't measure. Though, of course, it's more difficult! Interestingly, he points out that it makes excellent sense to engineer software, but that isn't what 'software engineering' has come to mean. He also looks at controlling software in terms of its Return on Investment (RoI). To use his example, consider two projects. Project A will eventually cost about a million dollars and produce value of around $1.1 million. Project B will eventually cost about one million dollars and will eventually produce value of around $50 million. Clearly the most important thing for project A is controlling the costs, or it will make a loss. The most important thing for project B is making sure that it realises its potential! Actually, the real question we need to ask is, "Why on earth are we wasting time on so many projects like project A that only produce marginal value?" Moving on, the next piece of interesting writing on the topic I found was a paper by Chuck Connell in Dr.Dobbs, entitled 'Software Engineering != Computer Science' [5]. I guess the title says it all. Connell draws a line. Below it are things like complexity, algorithms, cryptography, network analysis, and queuing theory. This he refers to as 'clean', meaning that they are formal mathematical entities which can be proven. Above the line, which he refers to as the 'bright line', are things like design patterns, architectural style, estimation, portability, testability, and maintainability (I would also add usability to his list). These parts he argues are inherently messy, and the reason is that they involve humans. He proposes a thesis - Connell's Thesis: Software engineering will never be a rigorous discipline with proven results, because it involves human activity. Hmm. I'm not convinced. Are regular engineers not human? Do humans not use the product of the engineering discipline? Connell is, in my view definitely on to something, but, I don't think his conclusions are correct. (Read his paper, it's only three pages, easy to read and well argued. You may feel I'm doing him an injustice.) He's clearly correct that software engineering falls into two categories though - a computing-science base and 'the rest'. The question is, though, 'why'? To answer that question, at least partially, it was back to searching the web and the discovery of an interesting article by Koni Buhrer [6]. Buhrer argues that the difference between a craft and engineering is that craft is based on trial and error, while engineering is based on a set of first principles (usually the laws of physics) from which all knowledge in the discipline can be derived. Note that this does not involve any concept of 'maturity' (however you want to define that). I would suggest, contrary to the views of the BCS, that maturity is probably necessary for a move from a craft to engineering, but it is far from a sufficient requirement. Buhrer takes a hard line on the issue of software as engineering. Not only does he believe that current software development is a craft and not engineering, he is of the view that software development will never be an engineering discipline! As he points out, although we have rules and methods for how we go about software development (think 'best practices') they are not rules derived from first principles, they are something that has come out of collective experimentation. Let me give you an example. If you give a qualified civil engineer fresh out of college a building to design, the result may well look like something that was ripped out of the Maginot line, but it won't fall down, and the engineer will be able to prove that it won't fall down from first, physical, principles. A more experienced engineer would probably design something more elegant, and probably more economic to build, but they too would be able to prove that it wouldn't fall down [7]. Contrast that with even an experienced programmer. How many people do you know who can prove, from some sort of first principles, not using tests, that a program of modest complexity will work as intended? The interesting thing about Buhrer's paper is that he has an explanation for why software development is so different and so much more difficult. If you look at engineering processes, there are basically two stages - design and manufacture. Buhrer's thesis is that software development is design, and that the manufacture is the running of the compiler and linker. I suspect that this idea was developed in an enterprise development environment, and would need some re-working for shrink wrap. Nonetheless, the idea gives some intriguing answers to a number of common problems in software development. Take for instance the difficulty in providing a detailed work breakdown and schedule for software development, in the way you can for, say, building a skyscraper. To quote Buhrer, "Because software implementation is the equivalent of skyscraper design (i.e., the creation of the blueprints for a skyscraper) and not skyscraper construction. Design work is naturally less amenable to planning than construction work -- which is true for skyscrapers as well as for software -- because the scope and complexity of the end product are discovered only in the course of the design work." It also explains the lack of economies of scale in software programs. I'm sure my readers will be only too well aware that if you increase the required size of a program by (say) ten times, it costs way more than ten times to produce. That's because, in engineering, the economies of scale come from the economies in the manufacturing cost, not the design stages, which will, probably, take proportionally longer! Furthermore, the construction is where most of the cost is in engineering - building the skyscraper costs far more than designing the skyscraper, and so the cost benefits of scale are realised in the building (manufacturing), not the design. It is interesting to speculate whether civil engineering would have emerged from craft if it had been possible to press a button on a blueprint and have the building appear within a few hours at no cost, and facilities to dispose of it at the press of a button! Without the need to have to prove the design before the bulk of the upfront costs were committed, would civil engineering have developed, or would the craft have just proceeded down the route of design, construct, test, demolish, tweak design, construct... Buhrer's paper goes further, and tries to provide, with much less success, the idea of some first principles. Unfortunately, the exposition is left to a part two of the article, which I have been unable to trace, so I guess it may never have been developed or published. OK. So why is all this important? It's important because the bulk of the scientific breakthroughs in the last 20 years have been enabled by the rise of cheap computing power. At the same time, software development appears to be little further advanced than that of a medieval craft guild state. That, rightly, concerns its practitioners [8]. At the same time as we are struggling to understand the basis of our discipline, powerful forces are moving to place limits on our activities. Governments are not happy with us. Spectacular failures of government contracts (mostly their own fault, but what government would admit that), combined with the fact that software development is a very portable skill, making it easy to pack and leave the country, is providing the impetus for registration and regulation of commercial programmers. It seems likely that my generation will be the last of the self-taught programmers. On the bright side, while commercial programming may become tied up in regulation, it will be difficult to completely stop 'amateur' programming. After all, as long as open source software exists, all you need is a general purpose computer, an editor and a compiler to write your own programs! And if you break the program, it's not as though you have to buy ones and zeros from the shop to fix it... I think we are on the cusp of something big which will determine the way in which our profession will develop for many years to come. Programming is now not something separate from society, if it ever was, it genuinely is becoming ubiquitous, and we can no longer shelter behind geekiness. We can either set our own course, which means we need genuine discussion, or we can abdicate and let others decide what we are to become in the future.
[1] http://www.raeng.org.uk/news/publications/list/reports/Engineering_values_in_IT.pdf
Back to the Phlogiston Blue top page If you have any questions or comments about the articles on my web site, click here to send me email. |