O'Reilly Network Weblogs: How Users Participate in Building Google

Published on The O'Reilly Network (http://www.oreillynet.com/)
http://www.oreillynet.com/pub/wlg/3204

How Users Participate in Building Google

by Tim O'Reilly
May. 18, 2003

I've always argued that sites like Amazon, Google, and EBay are the second generation Internet "killer apps." While the web browser, the first generation killer app, was really just a PC application that accessed the internet, these sites represent the leading edge of a new paradigm. Here are some of the ways they are different:

Most of them run on the LAMP (Linux-Apache-MySQL-PHP|Perl|Python) platform rather than on Windows. (EBay is a notable exception. Maps.Yahoo.com runs on BSD rather than Linux. But you get the idea.) That is, they are large, server-side applications with a massive database back-end that use a remote web browser on the PC for their user interface; most of the real work is done back at the server farm.
Of course, that far, they simply hark back to the timesharing era. An airline reservation system has a great deal in common with this model. But that's just the beginning.
They are content rich. Information is not just part of the results that are delivered to the user, but part of the interface. You navigate the information that is presented as part of your interaction with the software. I've argued that we ought to call this kind of software "infoware." And unlike an airline reservation system, this isn't just rows and columns of data, but is often unstructured or barely structured natural language text.
Maintaining that barely-structured content is part of why the "P" in LAMP is so important. Dynamic languages like PHP, Perl, and Python are especially good at handling text and related forms of content, and when that content is changing all the time, you need a lightweight programming methodology to keep up. I sometimes compare these apps to von Kempelen's Mechanical Turk, an eighteenth century hoax that purported to be a chess playing machine, but actually had a man hidden inside. These applications aren't a hoax, but they do have a programmer inside the application all the time. Take the programmer out, and before long the application stops working.
They are co-created by their users. (Well, not always. MapQuest and maps.yahoo.com haven't figured out yet that they need a way to let their users participate in improving the service; unless they do, I'll lay odds that whomever does so will be the long term winner in that market.) It's obvious that EBay depends on its users -- they provide both the reason to be there, and the mechanisms for evaluating what they bring to the table. At Amazon, it's a bit less clear. Most of the content comes from suppliers. But user-contributed reviews (as well as Amazon's own marketplace) clearly are a HUGE part of purchasers' decision-making process, and a key part of the interface. With Google, the involvement of users is even less obvious to those who don't know how the PageRank algorithm (and other key Google search algorithms) work.
Robert Scoble just told a great story that vividly explains how users help to build Google's product. He describes discovering a new Iranian restaurant in Redmond, WA, and linking to their website. He notes that the site doesn't now show up in google, but that, because of his weblog, it will now: "Oh, did you just realize that this weblog is nothing more than metadata for Google to use? Yeah, you'd be right. Google figured out how to get people like me to go around and look at websites and add meta data about those websites. How did Google do that? By giving us power. Think about it. That's how Google pays us back for the work we're doing to improve its index."
There's a dark side to this story. Scoble told it in the context of rumors that advertisers are lobbying Google to de-emphasize blogs in calculating its page ranks. I trust Google to do the right thing because of their relentless focus on the user. If they adjust the impact of blogs, it will be to get a truer result for users rather than for advertisers. But you can't underestimate the short-sightedness of many big players. Asking Google to take blogs out of the input is like asking EBay to stop taking product from small players and only take it from an approved vendor list, or asking Amazon to take reviews only from publishers and approved journalists! It's the essence of the new paradigm that users help to build the product.
(An aside: it's always been a key tenet of open source that you should treat your users as co-creators of the product. Sites like Google and Amazon aren't open source, but they clearly are fellow travellers when it comes to data. Not only do they have user participation in creating their database, but they are opening it for creative reuse via web services APIs. Open Source software proponents need to think about the significance of the internet paradigm shift, and what happens when the leading applications on Linux aren't open source (and we wouldn't get much benefit if they were) but instead relay on open data.)

Tim O'Reilly is the founder and CEO of O'Reilly Media, Inc., thought by many to be the best computer book publisher in the world. In addition to Foo Camps ("Friends of O'Reilly" Camps, which gave rise to the "un-conference" movement), O'Reilly Media also hosts conferences on technology topics, including the Web 2.0 Summit, the Web 2.0 Expo, the O'Reilly Open Source Convention, the Gov 2.0 Summit, and the Gov 2.0 Expo. Tim's blog, the O'Reilly Radar, "watches the alpha geeks" to determine emerging technology trends, and serves as a platform for advocacy about issues of importance to the technical community. Tim's long-term vision for his company is to change the world by spreading the knowledge of innovators. In addition to O'Reilly Media, Tim is a founder of Safari Books Online, a pioneering subscription service for accessing books online, and O'Reilly AlphaTech Ventures, an early-stage venture firm.