1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Sitemap warnings

Discussion in 'Black Hat SEO' started by Xpand, May 21, 2013.

  1. Xpand

    Xpand Regular Member

    Joined:
    Jul 21, 2010
    Messages:
    226
    Likes Received:
    61
    Hey guys, so I created sitemap.xml with a plugin and submitted it to G webmaster tools (fresh WP website).

    In webmaster tools I got this: Error details: 0 Errors, 20 Warnings.

    Type Issue Description Issues count
    Warnings Url blocked by robots.txt. Sitemap contains urls which are blocked by robots.txt. 20


    Pages I have:
    About index, nofollow
    Contact index, nofollow
    Privacy noindex, nofollow
    Sitemap index, follow
    Home page (I guess this is index, follow by default)


    My robots.txt:

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-content/
    Disallow: /wp-includes/


    Anything I have to change/tweek?

    Thanks
     
  2. Endire

    Endire Elite Member Premium Member

    Joined:
    Mar 27, 2012
    Messages:
    1,756
    Likes Received:
    1,061
    Gender:
    Male
    Xpand,

    You will just want to look at the warnings and see what they are. Commonly these are URL's blocked by robots.txt so if this is the case, make sure that URLs you want indexed are not being blocked.

    As far as you rrobots file, if you want those directories to not show up in search, then you are fine. That looks pretty standard for a wordpress site. Just check up on the warnings and makes sure its nothing you want to fix. Warnings are necessarily bad unless something undesirable is happening.

    Best,

    Shawn
     
  3. MatthewWoodward

    MatthewWoodward Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 31, 2012
    Messages:
    992
    Likes Received:
    1,681
    Occupation:
    SEO
    Location:
    UK
    Home Page:
  4. Xpand

    Xpand Regular Member

    Joined:
    Jul 21, 2010
    Messages:
    226
    Likes Received:
    61
    Thanks for reply guys. Here is the entire error details:

    Issue - Url blocked by robots.txt.
    Description - Sitemap contains urls which are blocked by robots.txt.
    Issues count - 20

    Example
    Value: www(dot)mydomain(dot)com
    Value: www(dot)mydomain(dot)com(slash)about
    Value: www(dot)mydomain(dot)com(slash)contact
     
    Last edited: May 21, 2013
  5. Xpand

    Xpand Regular Member

    Joined:
    Jul 21, 2010
    Messages:
    226
    Likes Received:
    61
    Ok, one more thing, I just checked on google - site:www(dot)mydomain(dot)com and these pages were found (indexed) so far:
    About
    Contact
    Homepage, but it says in the meta description below link: "A description for this result is not available because of this site's robots.txt"

    Sitemap wasn't indexed, though it has index, follow tags
    Privacy won't be found because it has noindex, nofollow tags

    What's the catch? I didn't disallow my homepage or is meta description in robots.txt, I'm using All In One SEO, and I specified it there, and checked HTML code, is there. hmmm
     
  6. Xpand

    Xpand Regular Member

    Joined:
    Jul 21, 2010
    Messages:
    226
    Likes Received:
    61
    Solved!
    I re-edited permalink for sitemap (html version) and than resubmitted sitemap.xml link to webmaster tools and it shows no errors. Don't know why this worked lol


    But when I type site:www(dot)mydomain(dot).com in Google, it returns:
    Homepage
    About
    Contact

    but not Sitemap page, why is that? And there is no meta description on homepage result, instead it says:
    A description for this result is not available because of this site's robots.txt - learn more.

    Do I just have to wait for google to crawl my website again or?