1 Trillion URLs – Google’s answer to Cuil?

Since the launching of Cuil has created much buzz in the internet, with many prominent netizens giving the newest search engine on the block positive reviews, there’s this consistent talk that it could give Google some serious competition.

After all, most of the folks behind Cuil are former Googlers and so this is something definitely worth taking seriously, especially for Google.

Cuil boasts that is has over 121 Billion web pages in its index and its new way of presenting search results is definitely something nice and refreshing. There are still bugs and kinks even spam in its search results but Cuil is just starting out and so things will be fixed and improved as time goes on and if the positive buzz around it grows into a community of fans and power users.

Sure Google still offers the best search results and the internet practically revolves around Google, but lately, the big G has been gaining some flak, like it’s perceived dominance of online advertising, it’s silent war against paid text link advertising (read: PR penalties), and most recently, its perceived attempt to retake the huge traffic to Wikipedia for itself and make huge money off it with Knol.

The collapse of the Microsoft-Yahoo acquisition has humiliated the former while the latter has compromised with Google, so alternatives like Cuil are becoming more and more attractive to the rest of us netizens.

However, Google seems to have taken notice and have sent out a strong message:
1 trillion (as in 1,000,000,000,000) unique URLs have now been indexed by their systems.

Note that they said unique URLs not web pages, giving the impression that web pages are not that significant a measure of how big a search index is, but the number of unique URLs. They even gave a lengthy explanation as to why:

We start at a set of well-connected initial pages and follow each of their links to new pages. Then we follow the links on those new pages to even more pages and so on, until we have a huge list of links. In fact, we found even more than 1 trillion individual links, but not all of them lead to unique web pages. Many pages have multiple URLs with exactly the same content or URLs that are auto-generated copies of each other. Even after removing those exact duplicates, we saw a trillion unique URLs, and the number of individual web pages out there is growing by several billion pages per day.

So Google continues to flex its muscles and uses its influence on the web, reminding all of us that Google has pwnd the internet!

Leave a Reply