Check your robots.txt file with OnlineWebCheck.com

Post here if your message doesn't fit into another forum but is still about web development. Includes site critiques, web hosting and server questions, helpful software and resources, and more.
Post Reply
User avatar
Albert Wiersch
Site Admin
Site Admin
Posts: 3236
Joined: Sat Dec 11, 2004 9:23 am
Location: Near Dallas, TX
Contact:

Check your robots.txt file with OnlineWebCheck.com

Post by Albert Wiersch » Fri Mar 20, 2015 8:36 am

I'm working on adding a robots.txt validator to the next update of CSE HTML Validator.

Please try it now at http://www.OnlineWebCheck.com/. Just check http://(yourdomain)/robots.txt and let me know how if you find any bugs or problems. or have any suggestions or comments.

Of course this assumes you have a robots.txt file!

Here is a link to more info about robots.txt files:
Robots.txt Specifications

Thanks!
Image
Albert Wiersch

User avatar
Lou
Rank V - Professional
Rank V - Professional
Posts: 246
Joined: Fri Jul 29, 2005 5:55 pm
Location: CO
Contact:

Re: Check your robots.txt file with OnlineWebCheck.com

Post by Lou » Fri Mar 20, 2015 2:06 pm

roots.txt in 4 of 6 domains, found one error (mine). nice addition.
Lou
Say what you will about Sisyphus. He always has work.

User avatar
Albert Wiersch
Site Admin
Site Admin
Posts: 3236
Joined: Sat Dec 11, 2004 9:23 am
Location: Near Dallas, TX
Contact:

Re: Check your robots.txt file with OnlineWebCheck.com

Post by Albert Wiersch » Fri Mar 20, 2015 2:58 pm

Thanks Lou. If it's not too much trouble, can you check again? I've very recently made some changes and additions to the robots.txt checker. It should find more issues now.
Image
Albert Wiersch

User avatar
Lou
Rank V - Professional
Rank V - Professional
Posts: 246
Joined: Fri Jul 29, 2005 5:55 pm
Location: CO
Contact:

Re: Check your robots.txt file with OnlineWebCheck.com

Post by Lou » Sat Mar 21, 2015 2:59 pm

Albert, I'm still good.
Lou
Say what you will about Sisyphus. He always has work.

User avatar
Albert Wiersch
Site Admin
Site Admin
Posts: 3236
Joined: Sat Dec 11, 2004 9:23 am
Location: Near Dallas, TX
Contact:

Re: Check your robots.txt file with OnlineWebCheck.com

Post by Albert Wiersch » Sat Mar 21, 2015 3:29 pm

Great. Glad to hear it (and no bug reports with the new checker :) ).
Image
Albert Wiersch

RoyReed
Rank I - Novice
Rank I - Novice
Posts: 15
Joined: Mon Nov 05, 2007 9:27 am
Location: London, UK
Contact:

Re: Check your robots.txt file with OnlineWebCheck.com

Post by RoyReed » Tue May 19, 2015 10:48 am

Worked perfectly for me. No errors found.

User avatar
RSteinwand
Rank VI - Professional
Rank VI - Professional
Posts: 562
Joined: Mon Jun 09, 2008 2:12 pm
Location: Fargo, ND
Contact:

Re: Check your robots.txt file with OnlineWebCheck.com

Post by RSteinwand » Wed Jun 03, 2015 10:06 am

Worked fine for me. No errors found.
Rick

User avatar
RSteinwand
Rank VI - Professional
Rank VI - Professional
Posts: 562
Joined: Mon Jun 09, 2008 2:12 pm
Location: Fargo, ND
Contact:

Re: Check your robots.txt file with OnlineWebCheck.com

Post by RSteinwand » Thu Jun 02, 2016 4:15 pm

Thanks for this, Albert. I rechecked my robots.txt file and found one issue, easily fixed by adding a blank line between search engines. I guess that's what happens when you make a change without retesting.
Rick

Post Reply