robots.txt permissions

[Dandello2000]

Edited: sorry, I have .htaccess on the brain at the moment.
I have some sub-directories blocked from being spidered - but that also means that CSE is blocked from accessing those sub-directories as well when checking the site online, so it throws 401 errors all over the place.
How is CSE Html identified as a spider so I can give it the proper permissions in my robots.txt file?

[Albert Wiersch]

Hello,

CSE HTML Validator does not pay attention to the robots.txt file, so it doesn't matter what is in there.

You can set the 'Agent' that CSE HTML Validator uses when making HTTP requests, however. By default it is "CSE HTML Validator Professional (http://www.htmlvalidator.com/)". You should be able to determine the request is from CSE HTML Validator by using that. It can be set in a target's properties or when adding a URL target to the Batch Wizard.

I hope this helps.

[Dandello2000]

I was right the first time *grin* - it's an .htaccess issue
I'll try it and see how it goes.