1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Sitemap, questions on errors

Discussion in 'White Hat SEO' started by seoguy81, Jul 2, 2011.

  1. seoguy81

    seoguy81 Power Member

    Joined:
    May 18, 2011
    Messages:
    779
    Likes Received:
    163
    Occupation:
    Full time IM
    Location:
    My Desk
    I've been noticing a consistent errors in my sitemaps. My money site uses Wordpress and using the Google XML Sitemaps plugin.

    My stats show the following crawl errors:

    In Sitemaps ‎(15)‎
    Not found ‎(182)‎
    Restricted by robots.txt ‎(1,922)‎
    Unreachable ‎(34)‎

    Would anyone with experience in dealing with sitemaps let me know what can be done in order to reduce the number of errors? Also does such high level of errors be affecting my site in anyway, say from an SEO ranking or sitelinks perspective?

    Thanks.
     
  2. Chrisss

    Chrisss Senior Member

    Joined:
    Feb 22, 2011
    Messages:
    842
    Likes Received:
    172
    Location:
    BHW
    It seems like you rely a bit to much on your robot.txt.

    You need to correct the errors where they are created and have minimal limitations in your robots.txt!
     
  3. hurn

    hurn Power Member

    Joined:
    Jan 21, 2009
    Messages:
    692
    Likes Received:
    191
    yes this is a big error because having lot of 404 errors are no make happy you visitors and big G
     
  4. seoguy81

    seoguy81 Power Member

    Joined:
    May 18, 2011
    Messages:
    779
    Likes Received:
    163
    Occupation:
    Full time IM
    Location:
    My Desk
    Thanks.. taking it one step at a time. Discovered that my pagination doesn't work beyond a few pages.

    Besides that, I use the wpmp mobile switcher.. most of the 1k not found is because all the URL's have a ?wpmp_switcher=mobile appended to the end of the URL.

    Also, I disabled atom feeds and another bunch of errors shown due to the url's being appended by /atom

    So i'm using the
    Disallow: /*atom*
    Disallow: ?wpmp_switcher=mobile

    Would this be the right syntax?
     
    Last edited: Jul 6, 2011
  5. N1CHOLAS

    N1CHOLAS Regular Member

    Joined:
    Dec 26, 2010
    Messages:
    231
    Likes Received:
    45
    Location:
    Birmingham , United Kingdom
    If your links are not deep like max 4 clicks deep you dont need a sitemap and unless you actually want to block pages from google you do not need a robots.txt. If your pages ever struggle to be indexed build a link to them or edit the content because often they decide to ignore rubbish low quality, but most of the time they take it.
     
  6. seoguy81

    seoguy81 Power Member

    Joined:
    May 18, 2011
    Messages:
    779
    Likes Received:
    163
    Occupation:
    Full time IM
    Location:
    My Desk
    Figured out the not founds and unreachables.. was a pagination issue. Now got to wait and see if the errors are reduced.
     
  7. seoguy81

    seoguy81 Power Member

    Joined:
    May 18, 2011
    Messages:
    779
    Likes Received:
    163
    Occupation:
    Full time IM
    Location:
    My Desk
    Hi guys.. so managed to fix all the errors and 404's. I just wanted check is 'Restricted by robots' is ok?

    Presently I have 3500 url's restricted by robots. This is because my site uses th baap mobile theme switcher.. so all urls are appended with ?wpmp_switcher=mobile

    My robots.txt uses the following
    PHP:
    Disallow: ?wpmp_switcher=mobile
    to block the mobile switcher.

    Appreciate any inputs on this.

    Edit: I used the Google URL parameters and set it to not crawl wpmp_switcher. I'm assuming this should do the job??
     
    Last edited: Jul 29, 2011