by Andrew Redfern @ 16th November 2007 3:06 pm

University bods at Penn State have performed a study of robots.txt files from over 7500 websites and found that website have a bais towards Google.

Really; whats their next study - Is the sun hot?

On the face of things this study doesn’t provide us with anything that is of interest; however dig a little deeper and you will find some fastinating statistics that are useful for websmasters and marketers alike.

For example - Robots Exclusion Protocol (the standard that governs robots.txt’s file) is adotped most in America 45.93% of sites in the USA adopted the protocaol compared to European (37.8%) and Asian (15.4%) sites.

93.8% of robots.txt files have rules for the universal robots

72.4% of the named robots appeared only once or twice

46.02% of newspaper websites currently have implemented robots.txt files

“We expected that robots.txt files would treat all search engines equally or maybe disfavor certain obnoxious bots, so we were surprised to discover a strong correlation between the robots favored and the search engines’ market share.”

“Robots.txt files are written by Web policy makers and administrators who have to intentionally specify Google as the favored search engine,” said C. Lee Giles, the David Reese Professor of Information Sciences and Technology at Penn State.

Excellent research and one I hope they continue to develop. If you would like to more about the project goto Penn State Live.

Want to harness the power of your brand and gain more exposure on the Internet? Want to get more bang for your buck from your Pay-Per-Click (PPC) budget? then contact HitSearch.  Remember, its a big world outthere, make sure you become visible.

Other Related Stories That May Be Of Interest:

1. Google Indexing Bing Shopping Data In Search Results
2. Google deliver two upgrades for webmasters and webmistresses
3. Google’s embedded site search now allows webmasters tailor the results
4. SEO research - Most people use 2 and 3 word phrases in search engines
5. Internet research - China’s growing tech-savvy youth
6. Google Announces Research Awards
7. New Research: How teenagers use their mobile phones
8. Google to Present at the Citi Investment Research Technology Conference
9. Google, Yahoo, Microsoft Search Advertising to Decide President?
10. Gigablast’s “beats Google” in recent search engine comparison study (by Gigablast)

Terms | SEO | PPC | Contacting Hit Search | Online Reputation Management | Online Press Distribution