Is my robots.txt incorrect?

ArmoredMedia

Junior Member
Joined
Sep 29, 2013
Messages
129
Reaction score
40
So last night I submitted a sitemap.xml for a site of mine, and I just checked it a few minutes ago (following day), and it says 15 errors on all 15 links.

In reality there is just 3 warnings on webmasters.
Webmasters:
http://puu.sh/5gGDy.png
Robots:
http://puu.sh/5gGFk.png

I can understand why the about-me is blocked since I am still doing the page and I don't want it crawable/see-able yet.

So what might be the issue? The other two things webmasters listed are crawable... Do I need a robots.txt on every directory or something.
 
So last night I submitted a sitemap.xml for a site of mine, and I just checked it a few minutes ago (following day), and it says 15 errors on all 15 links.

In reality there is just 3 warnings on webmasters.
Webmasters:
http://puu.sh/5gGDy.png
Robots:
http://puu.sh/5gGFk.png

I can understand why the about-me is blocked since I am still doing the page and I don't want it crawable/see-able yet.

So what might be the issue? The other two things webmasters listed are crawable... Do I need a robots.txt on every directory or something.

You can check the robots.txt section of webmaster tools to monitor how changes effect how the site is crawled. So maybe have a play with that and see what works.
 
Back
Top