Open Source: The Model for Collaboration in the Age of the Internet

This is the keynote presented by Tim O'Reilly at Computers, Freedom and Privacy, Toronto, Canada, April 6, 2000

• I'm going to talk about the rich environment of the Internet, why open source is central to that environment, and why we need to protect it.

Open Source is About Collaboration, Not Just about Software Licensing

• There are a number of myths about open source:
  • Linux is the first great open source success story.
  • Open source is primarily about software licensing.
  • Open source is about giving software away for free and allowing others to redistribute it.
These are important factors, but more importantly, OSS is about collaboration. Linux is a tremendous achievement, but the Internet and the WWW are the greatest OSS success stories to date.

• Much of the role of open source in the development of the Internet is well known: The most widely used TCP/IP protocol implementation was developed as part of Berkeley networking; Bind runs the DNS, without which none of the web sites we depend on would be reachable; sendmail is the heart of the Internet e-mail backbone; Apache is the dominant web server; Perl the dominant language for creating dynamic sites, etc. etc.

• Less often considered is the role of Usenet in mothering the Net we now know. Much of what drove public adoption of the Internet was in fact Usenet, that vast distributed bulletin board. You "signed up" for Usenet by finding a neighbor willing to give you a newsfeed. This was a true collaborative network, where mail and news were relayed from one cooperating site to another, often taking days to travel from one end of the net to another. Hub sites formed an ad-hoc backbone, but everything was voluntary.

  Rick Adams, who created Uunet, which was the first major commercial ISP, was the author of B News and the hostmaster of the world's largest Usenet hub. He realized that the voluntary Usenet was becoming unworkable, and that people would pay for reliable, well-connected access. UUnet started out as a nonprofit, and for several years, much more of its business was UUCP than TCP/IP. As the Internet caught on, UUNet and others liked it helped bring the Internet to the masses. But at the end of the day, the commercial Internet industry started out of a need to provide infrastructure for the completely collaborative UUCPnet and Usenet.

• The UUCPnet and Usenet were used for e-mail (the first killer app), but also for software distribution and collaborative tech support. The mechanisms that the early developers used to spread and support their work became the basis for a cultural phenomenon that reached far beyond the tech sector. The heart of that phenomenon was the use of wide-area networking technology to connect people around interests, rather than through geographical location or company affiliation. This was the beginning of a massive cultural shift that we're still seeing today.

• The Web was the next killer app. Tim's original web implementation was not just open source, it was public domain. NCSA's web server and Mosaic browser were not technically open source, but source was freely available. While the move of the NCSA team to Netscape moved key parts of the web infrastructure to the proprietary side, and the MS-NS battles made it appear that the Internet was primarily a proprietary creation, we all know better. Apache, the phoenix that grew from the NCSA server, has kept the open vision alive, keeping the standards honest, and not succumbing to proprietary embrace and extend strategies. Once Netscape was on the ropes, they returned to open standards with Mozilla.

• But even more significantly, the "View Source" menu item migrated from Tim's original browser, to Mosaic, and then on to Netscape Navigator and MSIE. Though no one thinks of HTML as an open source technology (because of the fixation on licensing), it's been absolutely key to the explosive spread of the Web. Barriers to entry for "amateurs" were low, because anyone could look "over the shoulder" of anyone else producing a web page. Dynamic content created with interpreted languages continued the trend toward transparency.

• A final note: While the open source community doesn't generally claim the IETF as its own, the Internet standards process has a great many similarities with an open source software project. The only substantial difference is that the IETF's output is a standards document rather than a code module. Anyone can participate, simply by joining a mailing list and having something to say, or by showing up to one of the three annual face-to-face meetings. Standards are decided on by participating individuals, irrespective of their company affiliations. (Though commercial participation is welcomed and encouraged, companies, like individuals, need to compete on the basis of their ideas and implementations, not their money or disproportional representation.) The IETF is where open source and open standards meet.

• I'd like to argue that open source is the "natural language" of a networked community, that the growth of the Internet and the growth of open source are interconnected by more than happenstance. As individuals found ways to communicate through highly leveraged network channels, they were able to share information at a new pace and a new level. Just as the spread of literacy in the late middle ages disenfranchised old power structures and led to the flowering of the renaissance, it's been the ability of individuals to share knowledge outside the normal channels that has led to our current explosion of innovation. Just as ease of travel helped new ideas to spread, wide area networking has allowed ideas to spread and take root in new ways. Open source is ultimately about communication.

• This is one reason behind one of O'Reilly's new open source ventures, the company collab.net , which we founded with Brian Behlendorf of the Apache project. Unlike many other OSS projects, Apache wasn't founded by a single visionary developer but by a group of users who'd been abandoned by their original "vendor" (NCSA) and who agreed to work together to maintain a tool they depended on. Apache gives us lessons about intentional wide-area collaborative software development that can be applied even by companies who haven't fully embraced open source licensing practices. For example, it is possible to apply open source collaborative principles inside a large company, even without the intention to release the resulting software to the outside world. More importantly, though, collab.net is teaching companies that it's not enough to slap an open source license on a piece of software; you need to build community and collaborative development processes around it as well.

• A side note: I like to say that software enables speech between humans and computers. It is also the best way to talk about certain aspects of computer science, just as equations are the best ways to talk about problems in physics. If you follow this line of reasoning, you realize that many of the arguments for free speech apply to open source as well. How else do you tell someone how to talk with their computer other than by sharing the code you used to do so? The benefits of open source are analogous to the benefits brought by the free flow of ideas through other forms of conversation. In the end, the conversation is more important than the actual words we use to talk to each other.

• In an exciting new development, most of you probably know by now that an appeals court agreed with this interpretation of software as speech in its recent decision in the Junger case. The treatment of software as speech is going to have enormous implications for areas such as software patents as well as for cryptography.

• If you believe me that open source is about Internet-enabled collaboration, rather than just about a particular style of software license, you'll open a much larger tent. You'll see the threads that tie together not just traditional open source projects, but also collaborative "computing grid" projects like SETIAtHome, user reviews on Amazon.com, technologies like collaborative filtering, new ideas about marketing such as those expressed in The Cluetrain Manifesto, weblogs, and the way that Internet message boards can now move the stock market. What started out as a software development methodology is increasingly becoming a facet of every field, as network enabled conversations become a principal carrier of new ideas.

Dangers Up Ahead

• When it comes time to chronicle the dangers facing the Internet and open source, myths also abound:
  • It's a battle between proprietary software and free software.
  • Microsoft is the great enemy.
  • Government regulation is going to spoil things.

• It's certainly true that companies have tried to "embrace, extend and extinguish" open standards. (And I should note that Microsoft is not the only one. In fact, Netscape started down this path with regards to the Web first; I gave my first talk warning about the dangers of Netscape's strategy to seize control of web APIs back in 1995.)

• It's also certainly true that the government could bollix things up pretty badly, but the greatest dangers are more insidious and more complex. To explain them, I want to take a brief discursion into the argument of a fabulous book, which all of you should read, if you haven't already, Lawrence Lessig's Code and Other Laws of Cyberspace.

The Architecture that Supports the Internet and Open Source

• Lessig argues quite compellingly that there is a complex interaction between architecture (code), social mores, the market, and government regulation in creating many of the features that we take for granted in our society. He makes the further case that as architecture (code) changes, we may need to make changes in regulation as well, in order to preserve the status quo rather than to change it.

• Lessig points out that many of the things we take for granted about the Net, which are perhaps best summed up by John Gilmore's famous dictum that the Net regards censorship as damage and routes around it, are becoming less true, due to changes in internet architecture brought about by e-commerce. The myth of the Net, Lessig says, is that it is unregulable. The myth of the Net is that it is a weapon of free speech, a weapon against aggression. The truth is that, as we are increasingly seeing, the web is also a tool for the invasion of privacy, for the tracking (and potential suppression) of free speech, for the diminishment and trivialization of human rights, on a previously undreamed scale.

• Lessig talks about the implications of changes in the Internet architecture for privacy, for free speech, and for the prospects of Internet regulation in his book. I won't go into details about how he characterizes the architecture of the Internet, but I will go on a Lessig-inspired meditation about the architecture that has thus far supported open source:

• Open source projects typically flourish in an environment consisting of small, cooperating programs with defined inputs and outputs, rather than monolithic integrated environments. Linux and the Internet are both ecologies of cooperating programs that have grown up in the wild have learned to live together. Linus points out in his essay in our book Open Sources that the architecture of Linux supports independent developers in a way that the architecture of Windows does not, even were Microsoft to offer Windows under an open source license.

• At bottom, open source depends on open standards. The ability of independently developed programs to cooperate via standard protocols and data exchange formats was one of the great insights that led to the development both of Unix and the Internet. One of the roles of open source software has in fact been to enforce open standards. It is questionable whether the web would still be open without Apache, or e-mail without Sendmail.

• The problem spaces being pursued were on the fringes, without great commercial attention. A program could be developed for a very narrow purpose, what has come to be called "a developer scratching his own itch". Often programs were developed for reasons that were indirect. For example, Sendmail was developed because Eric Allman thought it was easier to write a mail forwarding program than to give everyone in CS at Berkeley mail accounts on his Arpanet-connected machine. In this environment, your return on investment is the solution to your own problem; what you get by giving the software away is a dividend, as other people find bugs, enrich its functionality, or adapt it to other related problem spaces.

• Once the profit motive enters the picture, a developer is trying to scratch someone else's itch, which is a very different proposition. It requires different skills. One of the things that the open source community needs to do is to continue to foster work on the fringes, not just on the mainstream; where is the cool stuff that no one can find a real use for yet, but that breaks new ground? I like stuff like Kevin Lenzo's work with voice recognition, or with "answer bots" that participate in IRC or IM environments.

• One aspect of being on the fringes is that programs have time to take root, rather than being quickly discarded if they don't lead to immediate success. One of the myths of open source is that it's a rapid development methodology. There are contexts in which this is a true statement, but overall, the growth of open source is more akin to the development of a rich humus. Topsoil grows at a rate of an inch every 100 years. You can grow fabulous plants quickly in that soil, but the soil itself is a product of slow time. Look at most of the famous open source success stories. Many took 10, 15, or 20 years to take off. I have first-hand experience with my own company, which started out as a company providing documentation for many of these programs. Because the programs we wrote about were out of the mainstream, we had all the time in the world. We took six years (and six authors) before we produced our Sendmail book. Many of the books we produced focused on niches in which there was literally no competition.

  Now that open source and the Internet are front and center, we have to realize that the fundamental environment in which they evolved has changed dramatically. This is not to say that open source cannot compete and thrive in that environment, just that to deny that change is likely to lead to being blindsided. Most importantly, we need to realize that a project may be a huge success even though it doesn't give instant results. Open source projects often have unintended consequences. In fact, you might say that OS is the architecture of unintended consequences. Many of the greatest successes come not from the vision of the original designer but the uses to which newcomers put the original tool. Perl found its full flowering only when the Web came along. We're seeing the real fruit of Mozilla not in the next generation of Netscape's web browser, but in the ways that parts of Mozilla are showing up in other projects, from Zope as a web content management platform, to Eazel's next-generation Linux desktop.

• All of these metaphors have a lot of parallels in sustainable agriculture. I've already mentioned the time required to build up topsoil. What's more, we learn the importance of recycling, of putting nutrients back into the soil. A key part of open source is not just what big flowers you grow, but how much rots and is plowed back in to enrich the next generation. One of the big differences between open source and commercial software is the extent to which code is recycled -- and that doesn't just mean "code reuse" -- it means that ideas, freely shared, form the compost from which other ideas can grow. It means that failed projects are as important to the open source ecology as those that succeed.

Changes in the Architecture of Software Applications

• Moving away from the farming metaphor, there is a fundamental change in the architecture of the Internet that will have a profound effect on open source. This is probably the most important point I'm going to make, so pay attention!

• In the "old days," software needed to be distributed in order to be used. In the new world where the network is the computer, this is no longer true. The most powerful new applications are not traditional software applications, but web-hosted services: think MapQuest, Amazon, eBay, e*Trade, and you'll realize what I mean. These new-style applications, which I've elsewhere called "infoware" applications have profound implications for the open source software movement:

• Traditional open source licenses depend on software redistribution as the key to their requirement for source disclosure. What happens when software no longer needs to be distributed in order to be widely used? Oh GPL, where is thy sting?

• Rapid prototyping lowers the barrier to entry -- software becomes ever more of a conversation and less of a product. MP3.com says they try to introduce 20 new features every day. But on the other hand, this means that the traditional incentives to redistribute software don't exist. I've talked with developers at major web sites who've developed all kinds of cool internal tools for their sites, but they don't feel they have the resources to support those products, and they don't see the benefit of redistributing them. Since the web represents a key part of the future, it's essential to create an open source culture at leading web sites, even though they no longer need to redistribute software in order for it to be used.

• Returning to my point about open standards -- many of the new applications are being written at a new level above the old open standards stack. Where are the APIs that will allow an application to make use of Amazon's database, for example? These things will come, but they are likely to be proprietary unless we think through the benefits of standardization. XML offers promise, but because it's a commercial battleground, there's no assurance that the standards will be open, and that independent developers will be able to participate as they have been in the traditional architecture of Unix/Linux and the Internet.

• One of my biggest concerns about open source is that it has been slow to embrace these changes in the architecture of the Internet. Much of the attention of the Linux community, for example, has been focused on the recreation of a Macintosh- or Windows-style desktop environment, while Mozilla languishes. Microsoft is far ahead in the development of APIs that will allow developers to use other web sites as if they were software components. (For more information on this idea, see Jon Udell's keynote at the last Python Conference, The Next Generation Object Web.) Open Source is dominant on the server side of the web, but Microsoft now dominates the client side. There are good signs, though. The folks at Eazel grok the web at a fundamental level. Open source web developers are starting to grasp the new paradigm. But time is short.

Software Patents

• I haven't talked at all about software patents, which many of you know I've been spending some time on lately. I'll be glad to talk at length about patents during the question period, if we have time, but I do want to make a few short points.

• Given my concern for open source, you can quickly see why I believe that software patents are destructive to the Internet and open source culture. Where the essence of open source is collaboration, a heightened, speeded-up framework for sharing and communicating ideas, the essence of patents is to fence off ideas, and create monopolies on their use. In theory, patents result in the publication of ideas that would otherwise be kept secret, but in the world of software, the opposite is true. If you know of "prior art" you are supposed to disclose it in your patent application. If you don't know it, you can claim an idea as original. Therefore patent applicants go to great lengths not to know about prior art! Because so much prior art is not documented in the databases that are usually searched by patent examiners, this results in a huge transfer of rights from the collaborative community to unscrupulous companies that are patenting anything they can, in hopes of extracting royalties. At this point, even well-meaning companies realize that they need to file patents defensively, lest the tools of their trade be taken away from them.

• It's ironic that the patent examiners think that software patents will encourage innovation or protect the rights of inventors. In fact, patenting often sets in after the initial burst of creativity in a field has slowed down:

• When you're working on the fringes, without aim of commercial gain, you don't typically think to patent.
• In periods of rapid innovation, no one has time to patent. Faced with a choice between implementing a truly useful feature, and filing a patent on that feature, most developers would rather build software. Better to be imitated, improved upon, and then to leapfrog the competition, than to erect fences and hope they don't find their way around them.
• This situation is changing. The environment and the architecture we take for granted is seriously at risk.

A Few Big Picture Conclusions

• The environment in which open source and the Internet have flourished is changing and is at risk. We are moving from a situation in which developers scratch their own itches to one in which they try to please the market; we are moving from an architecture in which distributing code was the natural way to share ideas to one in which displaying the results of your work no longer requires distributing it; we are moving from a situation in which ideas were freely shared, and the best implementation won in the marketplace, to one in which patent licensees are laying claim to fundamental ideas (ideas not necessarily even their own).

• We need to take lessons from the environmental movement! We need a Sierra Club for the Internet. If we value the environment that brought us the Internet, we need to act to protect it.

• This starts with a realization that we could lose what we value. It's easy to think that it could never happen. My cousin, who is a fisheries statistician up here in Canada, tells the story of how fishermen will protest limits on their catch, saying "Why, we used to be able to take ten fish a day out of here." They don't realize that they get less now, because they took too many before. Problems take time to build, and by the time they are recognized, it's often too late.

• The next step is to build a bigger constituency. We need to tell the story of how we got the great benefits of the Internet from a particular style of cooperation. Open source and Internet-enabled cooperation should matter to people who will never write a line of software.

• We need to use our collaborative power to let those who threaten our environment know that it matters to us. This is the lesson of my protest over the Amazon patents. We can make a difference. I want to urge all of you to get involved in letting the world know that the culture of the Internet, the culture of open source, matters to all of you. Use it or lose it.

Tim O'Reilly is the founder and CEO of O’Reilly & Associates, Inc. He is also on the boards of ActiveState Tool Corp, Collab.Net, Invisible Worlds, and EPit.

Discuss this article in the O'Reilly Network General Forum.

Return to the Hub.

Copyright © 2009 O'Reilly Media, Inc.