The Architecture of Participation

		Print
		Email weblog link
		Discuss
		Blog this

Tim O'Reilly
Apr. 06, 2003 04:15 PM
Permalink

I had a great email conversation this morning with Adam Turoff, which he gave me permission to share here. Our first messages to each other are quoted completely; Adam's second message is quoted in bits, because it was interspersed with quoted bits of my response. I've done a minimal bit of formatting, and added some URLs to our exchange.

Adam started off:

I read Matthew Langham's post on oreillynet today, where he quotes Cocoon developer Stefan Mazzocchi's view on the role of code in open source software:
"Stefan Mazzocchi [...] often likes to make the sometimes drastic sounding statement that code is in fact uninmportant when it comes to success or failure of open source projects. He likes to state that "it's all about community".
I don't know which came first: open source (BSD) or software communities (DECUS, SHARE). Stefan's quote got me thinking about the key differences between communities focused around open vs. proprietary software. Now that I think about it, the proprietary communities are almost always in a reactive stance, while the open communities are almost always in an active stance.

The vendor led communities I've seen (PRIME users, Apple Developers, and various Microsoft user/developer communities) all focus on how to get you the individual to adapt to the latest software released by the vendor, or use obscure features properly to solve your problem. Even the organic communities I've seen (Local Mac User Groups, Campuswide Sun users/admins, etc.) focus on sharing secrets, or sharing software in a cargo cult like manner.

Without exception, the open source communities I've seen are all empowering. The basic premise is that there is a free market in place, and no one solution is forced upon anyone. Furthermore, there is a sense that it is not only possible but encouraged to create a custom solution to a problem and share your results.

Have you noticed anything similar? Do you think these factors have anything to do with the long-term viability and sustainability of open source software?

I replied:

I think that you've made a really good observation, Adam, one that is key to the failure of a lot of efforts by companies to manufacture developer communities. The question of whether these communities are "outside in" grassroots communities vs. corporate cheerleading is a good one for companies to ask themselves. Your image of corporate software community as "cargo cult" is a compelling way to drive home the difference between a captive community that waits for the next release, and one that's actively engaged in moving the product forward.

It leads back to what I've often said [see, for example, this blog entry from 2001, this posting to the Free Software Business mailing list or this editorial for Linux Magazine], that what really distinguishes open source is not just source, but an "architecture of participation" that includes low barriers to entry by newcomers, and some mechanism for isolating the cathedral from the bazaar. This architecture of participation allows for a real free market of ideas, in which anyone can put forward a proposed solution to a problem; it becomes adopted, if at all, by acclamation and the organic spread of its usefulness.

All of the most significant open source communities have some centralized "cathedral" elements -- look at the way Linus controls what goes into the Linux kernel, or the way Larry Wall controls what goes into the design of Perl. But the most successful open source communities surround that cathedral with a bazaar that is significantly open. In the case of Linux, this is the original Unix architecture, a set of "small pieces loosely joined" (to quote the title of David Weinberger's book about the architecture of the WWW). In the case of Perl, it was CPAN, as much as anything. By contrast, the Free Software Foundation was always fixated on control not just of the core but of the whole enchilada (cf. RMS's constant efforts to get Linux renamed GNU/Linux), and their community never took off in the same grassroots way. They leveraged the existing open source Unix community and architecture, which made their approach look more effective than it was, but I'm convinced that if they hadn't picked up on the existing Unix momentum, they would have been much less effective, since they have much the same approach as the old corporate communities, annointing winners and losers from the central core.

It's important to have an attitude that doesn't treat things that don't come from the center as second class citizens. Java has been a good example of how to go wrong there. While there's quite a bit of access to source, there isn't any ability to fork, and even people who build independent implementations of Java-based standards (think JBoss) are seen as enemies by the people in the core. There's a community process, but it focuses on premature standardization, rather than allowing users to self-select the offerings that provide real value. The process is also politicized, so that it's companies making decisions rather than individuals. Empowerment of individuals is a key part of what makes open source work, since in the end, innovations tend to come from small groups, not from large, structured efforts.

But it's not all about community. HTML is probably the single greatest testament to the power of source alone to jumpstart innovation. The "view source" menu item made it possible for anyone to see a neat feature on another web site, and immediately see how it was done. And a culture that encouraged leapfrogging (rather than blocking it via patents, copyrights, or standards committees.) In one sense, there was a "web developer community", but it was much more diffuse than most of the open source communities that people tend to celebrate, in which most of the key players actually know one another.

All of that being said, I don't think it's completely true that corporate sponsored communities are always one-way, and thus doomed to failure. A company like Sun or Macromedia or Microsoft can be (and often is) open to ideas from its developer community. Simply owning rights to the code doesn't make a community one way -- it's what you do with those rights. For example, I've heard tales of Linus' unwillingness to adopt necessary technologies (such as expanded threading support in the kernel) that match decisions by any corporate technology. But it is certainly true that since open source developers can be deserted more easily by their users through a fork, they are strongly incented to be more responsive to their users.

In the end, open source and the right to fork is a way of restoring competition to a software industry that has, for the most part, become anti-competitive through industry consolidation and the accretion of power to a few large players whose interest in maintaining the status quo becomes greater than their appetite for potentially disruptive innovation. A company that does want innovation needs to take risks. Like a surfer riding a big wave, they don't rely on containment or tight control of the environment to maintain their position, but rather, an exquisite balance and an ability to respond to rapidly changing conditions. This kind of responsiveness is hard for a large company to achieve, but not impossible, especially in the presence of the kind of competition that open source brings back to the market.

I'll also point out that if you have a large enough group of internal developers, open-source-like dynamics can occur within a proprietary software company. My favorite story in this regard is the birth of Microsoft's ASP.Net. As I understand it, Mark Anders and Scott Guthrie had always wanted to do a new version of ASP that wasn't backwards compatible, but hadn't been able to do so because of corporate strategy. But when they had a window after one project finished and before another began, they decided to do it just for the heck of it. Their project was picked up by other developers inside Microsoft, spreading just like any other open source project, until it came to the attention of senior management and was adopted as a strategic product.

Also consider the community that Amazon is building around their web services. They are definitely in charge, but they are also clear enough that they don't understand the potential of this new paradigm that they are actively engaging their users in co-creating the vision of the future. They put forth a starting point (Eric Raymond's plausible promise), watched what people did with it, and have tried to respond. What's interesting here is that Amazon isn't providing source; they are trying to open up their data to alternate interfaces. There's a lot to think about here. As we move into the era of dynamic, data-backed applications, and services built out from those applications, the traditional model of "open source" being defined by source availability seems even more limiting. The point that you and I are discussing, namely the "architecture of participation" seems even more important to understand.

My main point (and one that your posting seems to agree with) is that open source is not a litmus test, but a set of heuristics about how to encourage participation and innovation. We need to understand what works (and do more of it) and what does not (and do less of it).

I think some proprietary companies are doing some great experimentation, while some open source communities are convinced that they already know all the answers about how open source works. Complacency is a danger here. Companies like Microsoft, Sun, BEA, Macromedia, and IBM are all wrestling very hard with the issue of how to build effective developer communities, and all of them are making great strides in the right direction. So you definitely shouldn't compare today's corporate developer community with old style corporate communities like DECUS.

I hope it's clear that I couldn't agree more with your premise, that what makes open source go is empowerment. The question is how to maximize it.

Adam replied to my idea that "the architecture of participation" was a key concept with a little more detail on one particular proprietary developer community he'd had some experience with:

Yes. Very well put. Communities around a proprietary product lack that participatory aspect.

I briefly delved into Delphi many years ago. Attending a local Delphi users group had the feeling of a self-help group, where some people came across a key failing and traded coping strategies because there was no way to cause a change in Borland's product (at least for the product release in question). I've never sensed that feeling in a Perl Mongers meeting, because everything is up for discussion. Even if there's no Perl content in a Perl Mongers meeting, the sense is typically one of happiness and joy, not morbid resignation and praise/condemnation for the vendor.

He also responded to my idea that empowerment of individuals, not just corporations was key:

Certainly. (What's the definition of "Committee IQ"? It's the average IQ of the committee divided by the number of members in the committee. :-)

I've never thought of individual empowerment as centrally as you describe it here. Surely, individual empowerment is very important when looking at how open source gets created and used. The second order effect, that it almost naturally creates very strong self-sustaining communities is less obvious, at least to me.

Adam had some good thoughts about the continuum between a cargo cult community and one that is truly participatory. He doesn't know exactly where the dividing line is, but I agree very much with his idea that if you turn the participation dial only so far, you don't get community; if you turn it too far, you get anarchy.

I don't think it's important that a corporate sponsored community is strictly one way, as much as the dynamic is mostly one way. And the only way to measure this is through community participation on a shared resource: Perl, Java, .NET, WinXP, etc. While Sun, Macromedia and Microsoft do incorporate community feedback and contributions into their products, it seems to be the exception rather than the norm....

Similarly, the openness of an open source community is measured by its propensity to include individual contributions, not exclude them.

There's a continuum in play here. I don't know what the numbers are, but if you turn the knob to 3, there's a trickle of community feedback, and most individuals do not feel empowered, so the community doesn't gel. Turn the knob to 7, there's a feeling of community empowerment tempered with a benign dictatorship (or a benign sense of governance by a core team), so people feel empowered and a productive community forms.

Turn the knob to 10, and you get mass anarchy -- everyone is equal, so most of the activity is responding to flamewars on a mailing list. Turn the knob to zero, and you've created astroturf.

Adam also had a few choice comments about my citation of the Amazon web services community:

Yes, and that proves a few points.

First, there's Stefan's point that the source code is really irrelevant.

Second, there's your point about individual empowerment. Although Amazon has done little more than publish an API, they are making it possible for the individual to do a lot more than he could do if he had to screen scrape and worry about accepted use policies.

Third, there's the principle that healthy corporate sponsored community can form if they accept feedback and changes like an open source project would. Doubly so if they also provide a measure of leadership.

In closing, Adam wondered what made open source click when it did:

I do think there is value in understanding what works today, and what didn't work yesterday. I'm not accusing anyone today of acting like SHARE or DECUS did in the 1980s. Nevertheless, there were factors in play that inhibited something as powerful as open source from starting back then, even though all of the puzzle pieces were there. Perhaps it was sheerly a question of reaching a critical mass of alphageeks on a global network. Perhaps it was something more fundamental that Linus learned in the early 1990s.

I tend to think it was the critical mass that was brought about by the global network. It allowed for free association by developers on a massive scale. In the 80's, it was much harder to build a decentralized developer community. Only companies had the resources to build critical mass around a technology. But on the internet, freely redistributable software could find adherents worldwide, and those adherents could freely associate, work together, and build something that none of them could have done alone.

Still, physical proximity is useful to add to the mix once the community self-organizes on the net. I still remember the enormous buzz at the first Perl Conference I put together in 1997. All these people who'd worked together for years were meeting for the first time. "So you're Larry!" I heard more than one person say. The mind at the other end of the teletype suddenly given flesh and voice. And if they were meeting Larry Wall for the first time, how much more were they meeting each other. (That's why conferences have become so important a part of O'Reilly's business model. Open source communities can form in cyberspace, but getting together in the flesh can really help them to reach the next level of critical mass.)

Tim O'Reilly is the founder and CEO of O'Reilly Media, Inc., thought by many to be the best computer book publisher in the world, and an activist for open standards. O'Reilly Media also publishes online through the O'Reilly Network and hosts conferences on technology topics, including the O'Reilly Open Source Convention, the O'Reilly Emerging Technology Conference, and the Web 2.0 Conference. Tim's blog, the O'Reilly Radar "watches the alpha geeks" to determine emerging technology trends, and serves as a platform for advocacy about issues of importance to the technical community. For everything Tim, see tim.oreilly.com.

Return to weblogs.oreilly.com.