Nick Carr (the Australian XML éminence grise not the US journalist) asked me whether the software world was fragmenting. Here’s my answer with some diagrams: software seems to be organizing itself into three layers of support/runner/plug-ins where each of those layers in turn is organized as support/runner/plug-ins.
The early model of PCs and simple computers had a model something like this diagram:three layers: device drivers, operating system and applications.
Exactly which features go where has always been subject to experimentation: microkernels, hardware extaction layers, and so on showed that the kernel conceptually belonged in a low layer. Windows seemed to try hard to unlayer matters, to make everything part of the operating system from device drivers to Internet Explorer. But almost no-one was fooled; just because you eat the whole cake doesn’t mean it has no layers.
The three layer model is re-establishing itself: this time with the layers of operating system, platform and suite.
Each of those layers in turn organized as similar threes, with libraries down the bottom, a runner in the middle and plug-ins providing alternative faces at the top.
In the Operating System layer, operating systems are being squeezed at both ends: the device driver layer is becoming more important, both because of smarter devices and the increasing power of BIOS. The rise of virtualization technologies is splitting the operating system into a run-time executive able to run multiple virtual operating systems. VMWare’s ESX is an example of that kind of executive for servers, while Apple’s virtual OS 9 is an example of a virtual OS for desktop systems.
Next comes the Platform layer: Java and .NET notably, but also Perl, Python and other platforms. They have a set of libraries, a virtual machine or runtime, and different languages.
Rather than applications, I think it is better to consider the top as the Suite layer. Suites are where all the action is and have subsumed libraries and component architectures. For example, you could take Eclipse code as a library and build your own application, or you could use Eclipse as a Rich Client Platform and use its platform capability, but Eclipse’s big gain is the network effect of using it optioned-up in conjunction with other plug-ins. Other suites: Apache, Mozilla, Open Office, Microsoft Office, Visual Studio, NetBeans. (Ruby on) Rails is quite suite-like with its component architecture: certainly some of Oracle’s offerings are.
In the world of suites, software development becomes a matter of value-adding: of not needing to provide functionality that other plug-ins provide but also of needing to enhance them as well as being enhanced by them. If you cannot use the suite, you use the rich client platform; if you cannot use the client platform; you just use the library and framework. If you cannot use the framework, then you go to the non-suite world of SourceForge libraries.
Just as the Suite has its value because it provides a framework, client platform and one or more plug-in applications, so the Platform has value because it provides the libraries, the runtime and the language, and the Operating System has value because it provides the drivers, executive and a virtual operating system. Indeed, these built-in features are probably not well-layered at all, but integrated and required: but it seems a matter of time to me—over time code that starts of integrated will be refactored into this kind of layered pattern natrually, it seems to me.
Some of the untapped value to be gained at the moment is communication between different plug-ins of each layer: between virtual operating systems, between multiple languages (e.g. Java and JavaScript) on the same virtual machines, or between different application plug-in components.
In a way, the Marc Andreessen comment (actually Bob Metcalf’s?) about Windows reducing to a poorly (or partially) debugged set of device drivers for Netscape isn’t wrong: just that Windows is becoming a set of device drivers for virtual Windows (virtual Linux, etc). I think this is one reason why Microsoft doesn’t need to put out Vista to a deadline or in a hurry: the monolithic model is already superceded by events. This is not to say that Vista is dead in the water by any means: it is a cohesive bundle and a brand independent of any conceptual layering aspects. I was astounded to see that Microsoft itself distributes Red Hat Linux (as a virtual OS for its virtualization system)! Microsoft are repositioning themselves, to some extent, from being a pure software company to being a media company. And the need for cheaper, unbundled operating systems for poorer countries coupled with regulatory pressure should help make these layers more explicit over time.
The web fits into this model too. (It is tempting to see WS-* as the early attempts to make a fourth layer to bind together different three-layer stacks.) I haven’t thought them through completely, but I think Web Applications and Web Operating Systems fit (or will ultimately fit) into this pattern too.
The same 3-layer pattern can be discerned even inside plug-ins: application skinning and schemas are kinds of plug-in, I guess.
Anyway, after all that, the bottom line answer to Nick’s question is that the cracks we are seeing (i.e. all these different technologies that seem to challenge the old Windows/Win32 tight coupling) are not fragmentation but reflect natural layers.
Yep.
If you are predicting futures based on this model, you look for locales that control spending on the platforms. Those spaces transform the others.
A smart XML company looking for pure media plays looks for fat data transform pipelines with a reasonably frequent update rate. Then you factor in complexity of creating updates.
Some media models don't rely on XML or transformational economics. The porn model worked for those that understood it is an uncomplicated media with a limited range of expression. So in terms of cheaper faster better, one can only do cheaper faster. A good game costs what a major motion release costs and becomes stale in a single release cycle. Second Life? That is a very different model and interesting to compare to others.
For a media company, the cost of creating and owning the content BY MEDIA AND FORMAT TYPE is the limit on profitability. That is why open format standards win in the long run and proprietary standards win in the short run.
To a media company, platforms are stuff you need to host content, but otherwise, are just stuff. As a result, the period in which the technological aspects of the web fascinated the press and public is coming to a close.
len
As ever domain specifics will always lead the way...not VS200x but perhaps more like Open Lazlo or even (god forbid) license free FLEX beta3
Very salient essay, and I largely agree with the decompositions that you've laid out. I think that there are some aspects though that are creating alternative models that haven't completely settled out yet. Len's comments on transform pipelines brings up the whole notion of bindings within bindings within bindings, where each binding layer effectively creates a map between an underlying data abstraction at one end and an application representation on the other.
Case in point: a set of business rules and ACLs binds against schemas and templates to generate a core model mapping into components as an XHTML+XForms entity. This in turn reaches the client which instantiates the mappings into free-standing models, which in turn drive XForms content. Add in to that XBL bindings which turn custom intermediate markup into additional XForms content pulling from the data-model via a CSS binding layer, and tie that into posted XML that in turn gets passed into a second rules processing and validation engine to generate a data object which pushes back into the data store.
I see these playing right now primarily in the application space, but its easy to see this pushing further and further down the stack over time. They are (mostly) platform agnostic, they are increasingly underlaying low level services (such as file metadata and access) and they do not really differentiate strongly between local and remote resources.
I suspect that much of the fragmentation that is becoming apparent is thus also due to this recalibration around an XML paradigm.
-- Kurt
For different architectures (SOA, etc) I would model these as different arrangements of the 3-layersystems.
For pipelines, these are runners (middle layer) and the functions are plug-ins. So there could be pipelines at the Operating System level, at the Platform level, and at the Suite level. And because the plug-ins (and others) can be modeled (to varying degrees of acceptability) with the same 3-layer model, you could have pipelines inside plug-ins etc. too.
What about the brave new world of Web Operating Systems? Remotely hosted disk space fits into the model as a (virtual) device driver, and I bet at the other end there is enough complexity to model with this kind of three-layer model too. It would be interesting to see where or whether Google's MapReduce libaries fit in: suite, platform or OS.
I suppose my point is not that everything can be squeezed into this simplistic model (things have libraries, things have plugins, things have the bits that tie them together, big deal, ne?), but more that over time systems mature/evolve/grow into explicitly following this arrangement. Whether this means that systems should start off being designed using this pattern or not, I wouldn't like to say: however, presumably there can be a sweet spot between premature frameworkitis and inadequate top-level design.
[UPDATE] On the moving to make a distinct layer "Kernel and User Services", it seems that MS is indeed moving to a more modular construction, making a virtue of the necessity to have modularity that virtualization requires.
See MinWin at
http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9043359