Robots Exclusion Protocol 101
1/17/2008 <p>The Robots Exclusion Protocol (REP) is a conglomerate of standards that regulate Web robot behavior and search engine indexing. Despite the "Exclusion" in its name, the REP covers mechanisms for inclusion too. The REP consists of</p><ol><li>The original REP from <a href="http://www.robotstxt.org/orig.html">1994</a>, extended <a href=&qu...