Robots.txt information

For general web development questions that are not specifically related to CSS HTML Validator. This includes (but is not limited to) general HTML, CSS, Accessibility, JavaScript, and SEO questions.
User avatar
Albert Wiersch
Site Admin
Posts: 3785
Joined: Sat Dec 11, 2004 9:23 am
Location: Near Dallas, TX

Robots.txt information

Post by Albert Wiersch »

I had a user ask a question about robots.txt files and found this useful information with a lot of additional reference links:
https://en.wikipedia.org/wiki/Robots_exclusion_standard
Albert Wiersch, CSS HTML Validator Developer • Download CSS HTML Validator FREE Trial
cadomagenge
Rank I - Novice
Posts: 10
Joined: Thu Oct 01, 2015 5:37 am

Re: Robots.txt information

Post by cadomagenge »

Nice informaiton on Robots.txt files and really good reference links, i like it.