Robots.txt information
Posted: Wed Dec 05, 2007 4:14 pmI had a user ask a question about robots.txt files and found this useful information with a lot of additional reference links:
http://en.wikipedia.org/wiki/Robots.txt
http://en.wikipedia.org/wiki/Robots.txt