Robots.txt information

[Albert Wiersch]

I had a user ask a question about robots.txt files and found this useful information with a lot of additional reference links:
http://en.wikipedia.org/wiki/Robots.txt