1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Sitemap Submission Error WTF mate!?!!?

Discussion in 'Black Hat SEO' started by fatjack'sblackhat, Aug 31, 2013.

  1. fatjack'sblackhat

    fatjack'sblackhat Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 16, 2008
    Messages:
    611
    Likes Received:
    129
    Occupation:
    college student
    Location:
    Florida
    This is the error I keep getting when I test my sitemap in Google's webmaster tools:


    Error

    We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.

    1

    URL restricted by robots.txt


    Anyone know how to fix this problem? I have seen a bunch of people ask this question but havent found any answers so far
     
  2. ttrox

    ttrox Regular Member

    Joined:
    Jun 28, 2013
    Messages:
    217
    Likes Received:
    75
    Does your robots.txt deny the access to the root of the website / context where the sitemap is stored?

    Check out on myweb.com/robots.txt
     
    Last edited: Aug 31, 2013
  3. fatjack'sblackhat

    fatjack'sblackhat Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 16, 2008
    Messages:
    611
    Likes Received:
    129
    Occupation:
    college student
    Location:
    Florida
    this is what it says when I go there:
    # Added by SEO Ultimate's Link Mask Generator module
    User-agent: *
    Disallow: /go/
    # End Link Mask Generator output

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Sitemap: /sitemap.xml
     
  4. ttrox

    ttrox Regular Member

    Joined:
    Jun 28, 2013
    Messages:
    217
    Likes Received:
    75
    Mmm, that sounds strange. There doesn't seem to be any rule disallowing the access to the sitemap. If you are sure the sitemap is on sitemap.xml I would double check if the sitemap generated looks OK as maybe it's bad generated. This would seem like the problem.

    Other thing you might want to try, Google Webmaster tools provides a tool for testing the robots.txt (under Crawl -> Blocked Urls). Check that with the current robots.txt it can access the sitemap, else it will throw you an error with the line where it has a problem.

    Maybe this site can help you find an error: http://www.xml-sitemaps.com/validate-xml-sitemap.html
     
    Last edited: Aug 31, 2013
  5. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,112
    Likes Received:
    28,543
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
    I have had that happen to me before.
    Are you using Wordpress, if so what plugins do you have installed?
     
  6. fatjack'sblackhat

    fatjack'sblackhat Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 16, 2008
    Messages:
    611
    Likes Received:
    129
    Occupation:
    college student
    Location:
    Florida
    This is what happens when I test it there:

    Googlebot
    http://www.mysite.com
    Allowed
    Detected as a directory; specific files may have different restrictions

    Allowed
    Detected as a directory; specific files may have different restrictions
     
  7. ttrox

    ttrox Regular Member

    Joined:
    Jun 28, 2013
    Messages:
    217
    Likes Received:
    75
    That seems fine. Try checking again if it shows as allowed but against your mysite.com/sitemap.xml and also try getting your sitemap to the xml checker I wrote before to check for any problems.
     
  8. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,112
    Likes Received:
    28,543
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
    What are you using to create the Sitemap?
     
  9. fatjack'sblackhat

    fatjack'sblackhat Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 16, 2008
    Messages:
    611
    Likes Received:
    129
    Occupation:
    college student
    Location:
    Florida
    Its literally called: Google sitemap plugin
     
  10. ttrox

    ttrox Regular Member

    Joined:
    Jun 28, 2013
    Messages:
    217
    Likes Received:
    75
    Maybe it helps you, there's an option to rebuild the sitemap on Google's sitemap plugin tab.
     
  11. fatjack'sblackhat

    fatjack'sblackhat Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 16, 2008
    Messages:
    611
    Likes Received:
    129
    Occupation:
    college student
    Location:
    Florida
    I did this already and nothing happens
     
  12. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,112
    Likes Received:
    28,543
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
    Try this one
    http://wordpress.org/plugins/google-sitemap-generator/

    If it still doesn't work disable ALL plugins and try again.

    Sometimes if you have more than 1 SEO plugin it can cause conflicts.

    Edit: It looks like others are having problems with the plugin you are using.
    http://wordpress.org/support/topic/google-sitemap-doesnt-work
    http://wordpress.org/support/topic/sitemap-doesnt-show-up-in-webmaster-tools
    http://wordpress.org/support/plugin/google-sitemap-plugin
     
    Last edited: Aug 31, 2013
  13. fatjack'sblackhat

    fatjack'sblackhat Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 16, 2008
    Messages:
    611
    Likes Received:
    129
    Occupation:
    college student
    Location:
    Florida
    I deactivate all my plugins and then I activated to google xml sitemap that you recommended. Then I recreate my sitemap and try to submit it but I am still getting the same error message
     
  14. fatjack'sblackhat

    fatjack'sblackhat Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 16, 2008
    Messages:
    611
    Likes Received:
    129
    Occupation:
    college student
    Location:
    Florida
    Update: I downloaded a robots plugin that would let me add things to my robots file. I added in sitemap.xml and webmaster tools will complete the test without error but Im getting 23 warnings now that say:

    Warnings

    Url blocked by robots.txt.

    Sitemap contains urls which are blocked by robots.txt.

    23


    The URL they list are all my posts and pages including my homepage... The sitemap is detected but my robots is telling google not to crawl my pages. But when I go to mysite.robots.txt, everything appears to be in order. This is all very strange, is it not?
     
  15. ttrox

    ttrox Regular Member

    Joined:
    Jun 28, 2013
    Messages:
    217
    Likes Received:
    75
    Does any of those url point to mysite.com/go/post/whatever or something like that?
    My recommendation to eliminate more possibilities... comment the:

    # Added by SEO Ultimate's Link Mask Generator module
    User-agent: *
    Disallow: /go/
    # End Link Mask Generator output

    Just leave it like this to try again:

    # Added by SEO Ultimate's Link Mask Generator module
    #User-agent: *
    #Disallow: /go/
    # End Link Mask Generator output
     
  16. fatjack'sblackhat

    fatjack'sblackhat Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 16, 2008
    Messages:
    611
    Likes Received:
    129
    Occupation:
    college student
    Location:
    Florida
    You didnt change anything..
     
  17. dinkish

    dinkish Power Member

    Joined:
    Apr 19, 2013
    Messages:
    689
    Likes Received:
    159
    He commented the disallow to the go directory out, so it's accessible.
     
  18. ttrox

    ttrox Regular Member

    Joined:
    Jun 28, 2013
    Messages:
    217
    Likes Received:
    75
    Maybe I didn't express myself clearly. I meant leaving the first part like this:

    HTML:
    # Added by SEO Ultimate's Link Mask Generator module
    #User-agent: *
    #Disallow: /go/
    # End Link Mask Generator output
    so it's commented and it's accesible. Though, I find it hard that these links would be shown on the sitemap.
     
  19. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,112
    Likes Received:
    28,543
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
    If you are still stuck PM me the links of your sitemap and your robots text.
    I think I have an idea of whats wrong and why it is not working.
     
  20. fatjack'sblackhat

    fatjack'sblackhat Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 16, 2008
    Messages:
    611
    Likes Received:
    129
    Occupation:
    college student
    Location:
    Florida
    This is such a nightmare, nothing works. When I figure this out Im writing a tutorial for everyone on the internet because I cant find the answer anywhere