16th November 2007 3:06 pm

University bods at Penn State have performed a study of robots.txt files from over 7500 websites and found that website have a bais towards Google.

Really; whats their next study - Is the sun hot?

On the face of things this study doesn’t provide us with anything that is of interest; however dig a little deeper and you will find some fastinating statistics that are useful for websmasters and marketers alike.

For example - Robots Exclusion Protocol (the standard that governs robots.txt’s file) is adotped most in America 45.93% of sites in the USA adopted the protocaol compared to European (37.8%) and Asian (15.4%) sites.

93.8% of robots.txt files have rules for the universal robots

72.4% of the named robots appeared only once or twice

46.02% of newspaper websites currently have implemented robots.txt files

“We expected that robots.txt files would treat all search engines equally or maybe disfavor certain obnoxious bots, so we were surprised to discover a strong correlation between the robots favored and the search engines’ market share.”

“Robots.txt files are written by Web policy makers and administrators who have to intentionally specify Google as the favored search engine,” said C. Lee Giles, the David Reese Professor of Information Sciences and Technology at Penn State.

Excellent research and one I hope they continue to develop. If you would like to more about the project goto Penn State Live.

Want to harness the power of your brand and gain more exposure on the Internet? Want to get more bang for your buck from your Pay-Per-Click (PPC) budget? then contact HitSearch.  Remember, its a big world outthere, make sure you become visible.

Subscribe To Our RSS Feed!

Terms | SEO Liverpool | PPC Liverpool | SEO and PPC Company | Reputation Management | Online Press Distribution