Sitemap, questions on errors

seoguy81

Senior Member
Joined
May 18, 2011
Messages
1,077
Reaction score
287
I've been noticing a consistent errors in my sitemaps. My money site uses Wordpress and using the Google XML Sitemaps plugin.

My stats show the following crawl errors:

In Sitemaps ‎(15)‎
Not found ‎(182)‎
Restricted by robots.txt ‎(1,922)‎
Unreachable ‎(34)‎

Would anyone with experience in dealing with sitemaps let me know what can be done in order to reduce the number of errors? Also does such high level of errors be affecting my site in anyway, say from an SEO ranking or sitelinks perspective?

Thanks.
 
Restricted by robots.txt ‎(1,922)‎

It seems like you rely a bit to much on your robot.txt.

You need to correct the errors where they are created and have minimal limitations in your robots.txt!
 
yes this is a big error because having lot of 404 errors are no make happy you visitors and big G
 
Thanks.. taking it one step at a time. Discovered that my pagination doesn't work beyond a few pages.

Besides that, I use the wpmp mobile switcher.. most of the 1k not found is because all the URL's have a ?wpmp_switcher=mobile appended to the end of the URL.

Also, I disabled atom feeds and another bunch of errors shown due to the url's being appended by /atom

So i'm using the
Disallow: /*atom*
Disallow: ?wpmp_switcher=mobile

Would this be the right syntax?
 
Last edited:
If your links are not deep like max 4 clicks deep you dont need a sitemap and unless you actually want to block pages from google you do not need a robots.txt. If your pages ever struggle to be indexed build a link to them or edit the content because often they decide to ignore rubbish low quality, but most of the time they take it.
 
Figured out the not founds and unreachables.. was a pagination issue. Now got to wait and see if the errors are reduced.
 
Hi guys.. so managed to fix all the errors and 404's. I just wanted check is 'Restricted by robots' is ok?

Presently I have 3500 url's restricted by robots. This is because my site uses th baap mobile theme switcher.. so all urls are appended with ?wpmp_switcher=mobile

My robots.txt uses the following
PHP:
Disallow: ?wpmp_switcher=mobile
to block the mobile switcher.

Appreciate any inputs on this.

Edit: I used the Google URL parameters and set it to not crawl wpmp_switcher. I'm assuming this should do the job??
 
Last edited:
Back
Top