robot.txt question

RavenX

Newbie
Joined
Oct 11, 2009
Messages
38
Reaction score
5
I have never touched the robot.txt file, it was generated automatically and the sitemap was added in by a plugin....but does this mean the sitemap is not indexed by any spider? or is it good because it's not on the same line as disallow??
Code:
User-agent: *
Disallow:

Sitemap: http://www.siteexample.com/sitemap.xml.gz
 
If you want to block your sitemap do this
Code:
User-agent: *
Disallow: /sitemap.xml.gz
If you want to point out where your sitemap is do this;
Code:
Sitemap: http://www.siteexample.com/sitemap.xml.gz

^ this. everything looks in order to me unless you're attempting to cloak or deindexfrom google.
 
Back
Top