Web DevCenter    
 Published on Web DevCenter (http://www.oreillynet.com/javascript/)
 See this if you're having trouble printing code examples


Google Hacks

Why Try to Out-Google Google?

by Tara Calishain, coauthor of Google Hacks
05/16/2003

For a long time, the search engine wars were stagnant, but they appear to be heating up again. AskJeeves has launched a redesign, Overture bought AltaVista and the web-search portion of FAST, and MSN is starting to make rumblings about a search overhaul.

And what am I seeing every time I read one of these stories? "Challenge Google," "Rival Google," "Out-Google Google." And every time I hear about new initiatives like these my response is the same:

Why?

Why is every search engine out there so revved up to out-Google Google? The Google they aim at today won't be the Google they have to compete against tomorrow. The search tools they offer have a different set of strengths and weaknesses than Google's. Why compete on Google's terms?

When I think about this (and I find myself thinking about this a lot) it strikes me that the things that made Google successful initially were not technical. Yes, Google has great search technology, great algorithms, and so forth. But the average searcher does not care about these.

The average searcher is going to be attracted by a friendly interface and an easy-to-use site. Yes, the average searcher will be thrilled with Google's very relevant results. They may or may not tell their friends about that. It's more likely that they'll get a good laugh out of Google's PigeonRank joke and pass that on to their friends, who in turn can try Google for themselves.

I am very geeky. I do not like dwelling on "touchy-feely" aspects of business. But I am convinced that there are four very important things that have made Google successful, which have little to do with actual technology. Further, as long as we're talking about search engines wanting to "out-Google Google," I also wanted to take a look at ways in which Google could out-Google itself. It's got to evolve sometime, right?

Beyond Technology: What's Made Google Successful?

Related Article:

Eight Search Engine "C" Changes -- Tara Calishain, coauthor of Google Hacks, offers her first installment in a multi-part series on the latest developments for search engines and online data collections. In this article, Tara ponders the future direction search engines may take. How might pay-for-inclusion programs and other revenue generators, for example, change the way we search?

Yes, the search technology is excellent, and the "I'm Feeling Lucky" button is a stroke of genius. But that's not all there is to search engine success. If it were just the search syntax, AltaVista would still be number one. If it were just the sheer number of pages indexed, wouldn't AlltheWeb and Google be running neck and neck right now with market share? In addition to its technology, Google had several intangibles in its favor.

Good Timing

Google happened on the scene almost at the height of the portal craze. You remember the portal craze, right? That's when practically every search engine out there was offering weather, horoscopes, sports scores, and any other snippets of information that could be tied to a zip code. If you weren't actually focused on getting search results, I suppose these bits of information might have been of interest. But if you actually wanted to get your search and get on with your life, these drags on a page load could be distracting. And I won't even get into the banner ads.

Google's simple front page -- search box, two buttons, and logo -- were a big breath of fresh air. And not a banner ad to be found! It was almost too good to be true! When companies are thinking about out-Googling Google, do you think they're thinking about how to make the interface even faster-loading, even more streamlined, and even more friendly? Or do you think they're thinking about how to look exactly like Google?

A Sense of the Internet's Culture

It's been a source of much frustration to Google watchers everywhere that Google remains a private company. But it's been very good for Google, in my opinion. As a private company, Google can concentrate on its search engine and its associated properties. They don't have to publicly discuss the idea of profit and loss. To all external appearances, they can be just a happy-go-lucky company that really loves Internet searching.

Of course, there are grownups at Google whose job it is to make sure the search engine turns a profit. But since Google is a private company, it doesn't have to present that side of itself. They don't have to go on CNBC and explain their quarterly earnings. They can focus on what they do, and not how much they're earning or what their business model is.

On the other hand, if Google did not have a sense of the Internet's culture, this advantage wouldn't be an advantage. If Google didn't have its corporate culture to talk about, the former chef for the Grateful Dead, the odd logos on holidays, the April Fool's jokes, and so forth, it wouldn't have the mystique, the coolness, that it does. Google, as a whole, has a sense of the culture of the Internet, and its culture blends right in and looks really appealing. Being an east-coaster, I have never visited the Google office, but despite common sense, I imagine it as a 24-hour programming fest with buffet, foosball, and a piped-in soundtrack from Grand Theft Auto: Vice City.

Related Reading

Google Hacks
100 Industrial-Strength Tips & Tricks
By Tara Calishain, Rael Dornfest

I can't remember another search engine that has inspired such admiration on a cultural level, except perhaps AltaVista when it was still http://altavista.digital.com. That isn't to say that there weren't good search engines; there was a time when I adored Looksmart (I think it was around 1998), but I never thought, "Wow, Looksmart is cool. It must be really neat to work there." Most other major search engines and up-and-comers are publicly owned; what can they do to compete against the private- company-coolness of Google?

A Willingness to Share

It seems weird to include "A Willingness to Share" as one of Google's traits o' success, because they've been so aggressive about asking users not to scrape their search engine and not to access their index through automated processes.

But look at it another way: Google has been very good about releasing its new services in beta, or even the stuff that its programmers are just fooling around with (via Google Labs). Google's services tend to push toward the end of what's expected in a search engine. And finally, they released the Google API. No, it's not perfect, since it does access a very limited amount of the Google services. It's not finished, either, since it too is in beta. But it is available, and it's a tool that Google enthusiasts can use to build their own tool sets using Google's data, going in all different directions (witness the strange array of materials covered in Google Hacks).

Other search engines have not been as loud in their denunciation of scrapers, but at the same time, they've done little to make their indexes easily accessible to programmers and power users. It's like they developed the search engine for one audience -- a Web surfer -- and had no desire to go further, to see what services could be offered to a power user, a programmer, or a search enthusiast.

A Sense of Humor

If you've ever visited Google on April Fool's Day, you're aware of their sense of humor. There's also been the occasional odd logos that turn up, as well as some of the things in the Google store (a lava lamp? A bean bag chair?).

Now, of course, part of the advantage of the humor is that it fits in easily with the culture of the Internet, as has been mentioned before. But another advantage is that it gives new visitors something to pass along. It's not often that you run a routine search and get a set of search results that make you say, "Wow! I want to pass this set of search results on to all my friends!" On the other hand, when you come across a good online joke you tend to want to share it with other people. Google found a great way to spread memes and attract people to its site.

While other search engines have not been humorless, neither have they often been deliberately funny. I suspect that's more because it's hard to evolve a public sense of humor for a large organization -- things are developed and created by so many people that it's hard for them to maintain the unique voice that a sense of humor requires. AlltheWeb has probably come closest, with its skins contest and the way it presents its search engine.

Giving a large Internet property like one of the major search engines a humorous bent is also something that's very difficult to do -- when a company steps up and says, "From now on we're going to be ZANY and MADCAP and FILLED WITH SMILES!" it usually ends up embarrassing everybody. But if it evolves, and it's allowed to evolve, great.

So I've spent 1,400 words or so looking at the non-technical aspects of what Google did right. Some of the intangible reasons they got to the top. And now that they've got to the top, that's it, right? The Google experience is the pinnacle of everything and cannot be improved upon, right?

Shyeah. C'mon. This is the Internet.

How Google Could Out-Step Itself (Or How Other Search Engines Could Go Beyond What They're Offering Now)

On one hand, looking at the potential of Internet search is frustrating because of the limiting factors that aren't in your control. If only XML was widely adopted. If only everybody used title tags. If only domain names were more descriptive. Etcetera, etcetera. But on the other hand, other technology is developing that does make powerful and more extensive searching and crawling possible. Google could expand what they've got and become even cooler than they already are. How? Here are five ways, off the top of my head.

  1. RSS feeds of all its properties.

    The RSS format is a very handy way to read a lot of different Web sites without spending a lot of time waiting for pages to load. I'd sure love an RSS feed of Google News searches on the keyword of my choice.

    It's weird about RSS. Lesser-known search engines like Daypop are making great use of it. But none of the major general search engines are. Why? If the concern is losing ad revenue, why not include an ad in the RSS feed?

  2. A customizable "all-in-one" search.

    I know, I know; an all-in-one search is veering dangerously close to Portalville. But I think in this case, it's warranted. Google has so many properties that would provide complementary resources -- Froogle and Google Catalogs, for example, or Google's web search paired with results from Google News. I'd love an interface where I could say, "Give me the results from this query and use Google News, Google Web Search, and Google Blogs (that last one is only if the rumors floating around are true), and then present all the results on one page." Can't Google (or any other search engine, really) aggregate its own search sources without it being portalitis?

  3. Expand its API to other Google properties.

    Last summer will forever be burned into my mind as the summer that I ate, drank, slept, and breathed the Google API to write Google Hacks. Even now a small part of my brain patiently grinds away, coming up with neat things to do with Google and the Google API (and until this part of my brain gives up and goes away you can see its results at www.buzztoolbox.com/google/).

    But even though I rapidly discovered the tons of possibilities enabled by Google's API (and I'm very grateful to them for releasing it), I just as quickly found the limitations. No access to Google News or Google Images or most of the other collections. Only 1,000 queries per day, with only 10 results per query. Not all the special syntaxes (such as the phonebook: syntax) work. The API would be even more exciting if access to the other Google properties were available through it.

  4. Reach out to information-collection publishers.

    Google is often reluctant to discuss how the guts of its indexing technology work, and I can't say that I blame them. If too much were understood about how Google indexed and ranked its contents, people would spend too much time playing "Let's try to fool Google," instead of "Let's try to fill Google up with excellent content."

    The downside of this is that there's a group of content publishers who are caught in the middle. I'm referring to librarians and other information professionals who are often in charge of putting large collections of information online. Usually, those kinds of content publishers have far better things to do than spend extensive amounts of time trying to make sure their content gets indexed. This is a pity because it is exactly these kinds of information collections (extensive, unique, often not available anywhere else online) that are so valuable to search engines.

    It would be great if Google (or any other search engine) took some of its resources and made an effort to reach out to those groups (librarians, information professionals, government officials, and so forth) who are regularly publishing large information collections, and assist them in getting their content indexed as regularly and completely as possible. How should they be using title tags? Is their database-driven site restricting their chances of being indexed? How can they use Google tools to offer search on their own sites, perhaps with some specialty forms for sub-collection searching?

  5. Pay attention to successful uses of its API.

    I saved this one for last because I suspect that Google's already doing it, but it wouldn't hurt for it to be mentioned. It would be great if Google looked around at what folks are doing with the Google API and incorporated it into their offerings. No, I don't think they ought to have a "Goocookin'" section on their site, but what about Google Alert at www.googlealert.com? That site must have thousands of users. Isn't that an audience that Google wants to cultivate?

Like I said, I suspect Google is already doing this. I wonder if the other search engines are watching what's being created with the Google API and coming up with some ideas of their own?

Trying to "out-Google Google" is a bad idea in the fast-moving timestream of the Internet. By the time you think you've out-Googled Google, they've out-Googled themselves and you're still behind. But if search engines took at close look at the cultural factors that led to some of Google's success, and then considered how they could leapfrog what Google's doing now -- we'd have some search engine competition that I'd really look forward to!

Tara Calishain is the creator of the site, ResearchBuzz. She is an expert on Internet search engines and how they can be used effectively in business situations.


O'Reilly & Associates recently released (February 2003) Google Hacks.


Return to the Web Development DevCenter.

Copyright © 2009 O'Reilly Media, Inc.