Blocking Bots Based on User-Agent
12/12/2011 Normally as SEOs our focus is on giving Google and other search engines as much access to the sites we are working on as possible. There are times however in which privacy is a good thing. We have found blocking bots based on the user-agent very useful for development servers where you might be hosting multiple sites which you do not want crawled or indexed.