Page 1 of 1

Check your robots.txt file with OnlineWebCheck.com

Posted: Fri Mar 20, 2015 8:36 am
by Albert Wiersch
I'm working on adding a robots.txt validator to the next update of CSE HTML Validator.

Please try it now at http://www.OnlineWebCheck.com/. Just check http://(yourdomain)/robots.txt and let me know how if you find any bugs or problems. or have any suggestions or comments.

Of course this assumes you have a robots.txt file!

Here is a link to more info about robots.txt files:
Robots.txt Specifications

Thanks!

Re: Check your robots.txt file with OnlineWebCheck.com

Posted: Fri Mar 20, 2015 2:06 pm
by Lou
roots.txt in 4 of 6 domains, found one error (mine). nice addition.

Re: Check your robots.txt file with OnlineWebCheck.com

Posted: Fri Mar 20, 2015 2:58 pm
by Albert Wiersch
Thanks Lou. If it's not too much trouble, can you check again? I've very recently made some changes and additions to the robots.txt checker. It should find more issues now.

Re: Check your robots.txt file with OnlineWebCheck.com

Posted: Sat Mar 21, 2015 2:59 pm
by Lou
Albert, I'm still good.

Re: Check your robots.txt file with OnlineWebCheck.com

Posted: Sat Mar 21, 2015 3:29 pm
by Albert Wiersch
Great. Glad to hear it (and no bug reports with the new checker :) ).

Re: Check your robots.txt file with OnlineWebCheck.com

Posted: Tue May 19, 2015 10:48 am
by RoyReed
Worked perfectly for me. No errors found.

Re: Check your robots.txt file with OnlineWebCheck.com

Posted: Wed Jun 03, 2015 10:06 am
by RSteinwand
Worked fine for me. No errors found.

Re: Check your robots.txt file with OnlineWebCheck.com

Posted: Thu Jun 02, 2016 4:15 pm
by RSteinwand
Thanks for this, Albert. I rechecked my robots.txt file and found one issue, easily fixed by adding a blank line between search engines. I guess that's what happens when you make a change without retesting.