I had a user ask a question about robots.txt files and found this useful information with a lot of additional reference links:
https://en.wikipedia.org/wiki/Robots_exclusion_standard
Robots.txt information
- Albert Wiersch
- Site Admin
- Posts: 3785
- Joined: Sat Dec 11, 2004 9:23 am
- Location: Near Dallas, TX
- Contact:
Robots.txt information
Albert Wiersch, CSS HTML Validator Developer • Download CSS HTML Validator FREE Trial
-
- Rank I - Novice
- Posts: 10
- Joined: Thu Oct 01, 2015 5:37 am
- Contact:
Re: Robots.txt information
Nice informaiton on Robots.txt files and really good reference links, i like it.