If you search Google for…
- document
- net
- ruby
- python
…then, as a reader of O’Reilly weblogs, you’re probably going to see the kinds of results that you expected to find. But as the majority of people in the world aren’t web developers, are these results what most people would expect? If I asked someone in the street to tell me the most relevant thing they could about ‘ruby’ or ‘python’, it almost certainly wouldn’t be to do with the programming language.
From a quick glance at job vacancies on the Guardian website (one of the major newspapers in the UK), only about 1.8% of jobs have ‘development’ or ‘internet’ in them (limiting the search to just IT and Telecoms). So, in theory, the Google results would only be relevant for about 2% of the general population.
Which isn’t exactly the whole story, of course. Most people searching for ‘python’ on the web today are almost certainly searching for the programming language (i.e. the web population is different to the general ‘offline’ population). So is Google correct? Or is this a chicken and the egg situation, and - like the Nintendo Wii - the web will primarily be targeted at the hardcore until someone dares to design a system that targets the mainstream ‘non user’ over the tried and tested current user base? What other side effects could be caused by ‘web developers’ (or techies) being the people who put the most content on the web, and hence disproportionately telling Google what is ‘important’?
That's the whole point of PageRank: them that has gets.
Any appeal to the "wisdom of crowds" should be met by 'which crowd'? Call it long tail if you like, but really, this is just a power law with a feedback loop. The first statistically significant external class to use control sets the selectors for the loop. Think of it as organic adaptive programming of the web's controls (which by the way, makes the Sun execs' opinions about there only being a need for five web computers more than a little dumb).
That you get those results is unsurprising. The interesting question is how many queries are required to break the lock on the terms or if the emergence of a lock means that it is time to throw away those terms as names for the other semantics and invent new ones. It's a fun dilemma to contemplate.
try this: python snake ruby jewel
Rather than narrowing or expanding a search using the typical set of Boolean operators, maybe Google could copy ask.com and output not only the search results but also a list of related searches. On the other hand, I imagine the implementation would require a somewhat intelligent clustering algorithm of the search results.
Bryan> try this: python snake ruby jewel
Of the 10 hits on the first page, 5 of them are still for the programming languages.
Its an interesting point. Len's comment makes me wonder if there's going to be a market in the future for niche search engines. Or are search engines going to start using more of the informatioon they know about you to better page rank. Probably the latter.
It would make a lot of sense. If Google were to cross reference web searches with the a cross reference of a search of the users googlemail account I wonder how much better the search results would be. And of course it they were measurably better it would provide a further way to entice users to use there services.
Google tends to sway towards tech sites. I think that is just the nature of the internet...the plurality of information is probably tech related. Even searching for "vitamin" brings you to the ThinkVitamin webmagazine site.
why: Snake handlers and gem traders don't get as many links as tech sites. And they don't write as many pages, and they don't optimize the page layout correctly like the geeks do.
Google could factor in your search history. If you then quit your job as a programmer and become a snake handler it may take a while to rebuild your relationship with google. SERPS would no longer be universal for everyone; sectors of the population would see different results. downside: People would settle further into their own self re-inforcing reality tunnels.
Many new start ups are working hard trying to beat google in order to be the next google. But the fact that we have not seen any yet is an indication of how hard the problem is. Personally, google seems to be just good enough.
Google requires patience and skill plus some understanding of how it works to create a search for a particular task. The d'oh of searching is that the majority of searching is context-situated. That can be a person's history, yes, but that is problematic for privacy and intention. Are you thrilled to give up that much personal information to a system that creates tensor locks around its users thus holding them into those contexts? Do you like to be able to declare your own contexts and choose your situations? Do you still think that everyday is a new day?
Feedback is a means of narrowing or increasing selectors. Controls emerge naturally from that interaction and you have to be the chooser of your choices or the system will do it for you. Classification creates classifiers. Relationships have cycles.
1) Yes, niche search engines are a probable outcome but if you consider that customization of indexing is in fact a means of creating a niche, Googling + situation/context would do that.
2) Over and over again, we've seen how easy it is to game the engines. A lot of intellect is working to solve that without tossing out 'the wisdom of crowds'. So far, not many people are willing to admit the obvious: classing IS an act of closing a system, not an act of opening it.
3) The question becomes one of rights over information retention. Once assigned to a class, are you stuck there? Should ChoicePoint or the other personal fact collection agencies be opened to inspection? They can cost you a job and ruin your life and you won't know it because no one asks you. Shouldn't they be open like your credit records at your request? Now you have to tackle identity management as well. Whoops! Here come the privacy advocates who think an authentication system means 666 and the Beast!
Technology outrunning culture and society make for good sci-fi, bad press, and work for the technologists. It is a good business: create a disaster then profit by cleaning it up.
There is some meaty and profitable litigation in the future of these companies. The trouble is, that won't solve the problem either. The web was fielded witlessly and we are stuck with it.
Re: niche search engines, Google has, of course, already started to tackle this with Google co-op (http://www.google.com/coop/). Plus, they've done Personalized search too! (http://www.google.com/psearch)
On the other hand, finding a search string that gets a reasonably good set of python (the language) stories out of Google News is practically impossible. You have to filter out a bunch of terms that deal with escaped snakes, swallowed pets, and comedians. But before you can eliminate all the false-positives, you run into length limits for the search string.
Which is not surprising. Topical vector indexing should do precisely that if the dominating term for the cluster is 'news'. In user terms, the languages dominate. In general publishing, snakes dominate.
There may be a social metaphor in there, but it's Saturday morning here and I've got avatars to build. :-)
hits=advertising money
if a google search for "python" returned the 2 or 3 sites on the internet that no one ever goes to anyway, google would lose all the money generated by the thousands of developers/wanna-be-developer/IT-professionals/IT-pretenders who search the word "python" over a billion times an hour. That would be a pretty stupid decision.
What Google gives you is a slice of the internet, ranking items by their approximate "mind share" of popular culture, or at least computer literate popular culture.
If you plug "Luke Skywalker" and "Darth Vader" in to Google Sets, you will get a much longer list of Star Wars characters than listing "Peter Wimsey" and "Harriet Vane" will give you of characters from Dorothy Sayers mysteries.
If the internet ever becomes self-aware, it will be robbing numbered accounts to buy Boba Fett figures off EBay, not trying to set off nukes to exterminate us a la SkyNet.
Google is taking the same approach as Microsoft did in the 80s i.e:
If attract developers, they world will follow - because it is actually the developers who decide on the success of a technology.
I have two questions.
1. What influence do you think google have on website developers?
2. What can developers do to improve their site rankings?