The quest for the holy grail of software

Hardware pieces have long been assembled into functional products on the assembly line. With software, we’re finally ready to create a similar revolution.

 

It seems no coincidence that the term “software engineering” first appeared in an actual engineering context. Margaret Hamilton used it to describe her work on the Apollo Guidance Computer for the NASA Apollo Mission.

Inside the Apollo Guidance Computer was a core rope memory weaved by professional weavers. The whole construction was brittle and hard-coded: any mistake would entail the collapse of the entire mechanism.

The software was brittle as well. When Hamilton used the word engineering, then, she did not refer to the ease with which the software was replicated but merely to the precision and diligence with which it was built. She was well aware, in other words, that there was no assembly line for software yet.

Perhaps, however, a veritable assembly line was exactly what the software industry needed.

The battle for a world of components

In 1968, with the Apollo mission still in progress, at NASA’s Software Engineering Conference in Garmisch, Germany, software engineer Kenneth Kolence spoke up.

»The basic problem is that certain classes of systems are placing demands on us which are beyond our capabilities and our theories and methods of design and production«.

»It is large systems that are encountering great difficulties. We should not expect the production of such systems to be easy«.

In 1968, building large, reliable software systems such as those required for the Apollo Mission in a controlled, cost-effective way was an unsolved challenge. Before a veritable assembly line for software – one capable of constructing such large systems – could become a reality, the right reusable parts had to be discovered. 

One person who was present at the conference dreamed of a world where such components would exist.

In his talk, “Mass Produced Software Components”, Malcolm Douglas McIlroy, then of Bell Labs, laid out a vision. A vision where the construction of large systems of software was handled with ease.  

The failure to turn McIlroy’s vision into reality

Ever since McIlroy laid out his vision of a world of reusable components, the software industry has struggled to create it.

Several factors spoke against McIlroy’s ideas being instantly implementable. To work, reusable software components had to be discovered by large software companies. The larger those companies grew, the quicker their internal feedback loops deteriorated. A company of 20 people knew each other’s tasks, making double-work unlikely. In a company of 20.000, significant investment had to be put into maintaining strong feedback loops, and significant resources had to be put into maintaining and developing a common resource of reusable components. If no common resource was maintained, reuse would be reduced to opportunistic, scavenging approaches. To put a final nail in the coffin, psychological impediments such as the “not invented here” and “uniqueness bias” syndromes enforced barriers to successful reuse at large scales. 

In 1999, in a 1999 issue of C++ Report Magazine, developer Douglas C. Schmidt declared McIlroy’s vision anything but dead – or at least not yet awakened. Yet, human history was full of examples where reuse had worked.

Most familiar to all, alphabets had allowed for the “infinite use of finite means”, as naturalist Wilhelm van Humboldt put it.  

Later, the hardware industry made reuse work. Why then, had the software industry consistently failed to replicate earlier successes?  

One provocative possibility was that no one had really tried. Just like the reusable parts of language used to consist of entire symbols of objects in the world, only to emerge as right-sized letters after thousands of years of trial and error, perhaps getting reuse to work would require a similar significant investment by one or more software companies. 

In the 2010s, Netcompany decided it was time to make this investment. 

At Netcompany, the process of discovery took place over millions of hours of project work. Netcompany Foundations, spanning error handling, logging, caching, Web API, relational databases, and transaction management were discovered to be reusable components across nearly all IT projects. Components in Netcompany Platforms were discovered to be reusable across specific domains. Certain IT projects were discovered to be so similar that entire reusable Standard Products could be developed. 

»The components were never born to be reusable, but they were discovered to be so«, Martin Ellebæk, Principal at Netcompany, says. 

We also found out that components occur in groups or families, some belonging to the basic foundation of all systems, others to process-oriented applications, some to real-time scenarios, and again others perfectly fitted to be re-used for communication between systems and users.

Using ready-made components from these families, rapid deployment was possible. 

»With our platforms, we could start a project almost instantly and make a PoC with a Platform by two people in a couple of weeks«, Ellebæk says.  

Since McIlroy laid out his vision of a world of components, his vision remained just that – a vision. It turned out, however, that his vision was not impossible, it was just hard. Just like the assembly line of the industrial revolution did not reduce product diversity but increased it, in 2024, we can build anything creative that serves a unique purpose, and we can do so with speed.