Google Yahoo N MS to clean up web

In a rare instance of collaboration among otherwise fierce rivals, Google, Yahoo and Microsoft said Thursday that they would support a new Web standard that will allow millions of Web publishers to remove duplicate pages from their Web sites. As a result of the effort, search engines should be able to find and index more Web pages, making their search results more comprehensive.

?There is a lot of clutter on the Web and with this, publishers will be able to clean up a lot of junk,? said Matt Cutts, an engineer who heads Google?s spam fighting efforts. ?I think it is going to gain traction pretty quickly.?

The problem is the following: Many Web publishers, especially those that have voluminous sites, like e-commerce companies, have multiple URLs that all point to the same page. This confuses search engines, sometimes causing them to index the same page multiple times. As much as 20 percent of URLs on the Web may be duplicates, according to some estimates.

Engineers at Google came up with a simple way for Web publishers to indicate when a URL is a duplicate, and if so, which is the principal, or ?canonical,? URL that search engines should be indexing. Yahoo and Microsoft, the No. 2 and No. 3 search engines, have agreed to support the same standard.

?We are happy that everyone is going to support the same implementation,? said Nathan Buggia, a lead program manager at Microsoft. ?This is a clear benefit for publishers as it gives them an opportunity to get more exposure through search engines.?

All search engines have developed technologies to detect duplicates that are more or less effective. The so-called Canonical Link Tag, as the standard is known, should make it easier for both publishers and search engines to address the problem. ?It is an important step because all the search engines are coming out with it,? said Priyank Garg, director of product management for Web search at Yahoo.

Click to expand...

http://bits.blogs.nytimes.com/2009/02/12/google-yahoo-and-microsoft-collaborate-to-clean-up-web/?pagemode=print