1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots.txt fetch error -- Google can't crawl my site!

Discussion in 'Black Hat SEO' started by AnthonyDG, Jan 11, 2014.

  1. AnthonyDG

    AnthonyDG Registered Member

    Joined:
    Oct 15, 2013
    Messages:
    63
    Likes Received:
    19
    I have my categories and tags as noindex, but I've been getting this error for a while. Anyone know how to fix this? Seeing as Google can't crawl my site, I figure it's important to get it resolved asap.

    Thanks!
     
  2. oxonbeef

    oxonbeef BANNED BANNED

    Joined:
    Jan 4, 2009
    Messages:
    2,242
    Likes Received:
    7,872
    Is it a Worpress site?
    Make sure you have option to make the site visible to search engines unchecked.
     
  3. seeplusplus

    seeplusplus Power Member

    Joined:
    Aug 18, 2008
    Messages:
    511
    Likes Received:
    163
    It is robots.txt and not Robots.txt isn't it?

    Should be in the domain's root folder.
     
  4. AnthonyDG

    AnthonyDG Registered Member

    Joined:
    Oct 15, 2013
    Messages:
    63
    Likes Received:
    19
    Yes it is a wordpress site. You mean "Discourage search engines from indexing this site?" Yes, that's unchecked.

    Yup it's robots.txt, not capitalized. I can even access it at mywebsite/robots.txt, so I know it's there...
     
  5. seeplusplus

    seeplusplus Power Member

    Joined:
    Aug 18, 2008
    Messages:
    511
    Likes Received:
    163
    In that case then as oxonbeef suggests, have a look in the source code of each page with your browser and check if WP is putting the meta tags to noindex, doesn't sound like a robots.txt issue (assuming it's wrote correctly).
     
  6. oxonbeef

    oxonbeef BANNED BANNED

    Joined:
    Jan 4, 2009
    Messages:
    2,242
    Likes Received:
    7,872
    http://productforums.google.com/forum/#!topic/webmasters/BrB1iTMhJMc

    http://wordpress.org/support/topic/robotstxt-8
     
    • Thanks Thanks x 1
  7. TimothyEyton

    TimothyEyton Junior Member

    Joined:
    Dec 30, 2013
    Messages:
    127
    Likes Received:
    62
    Are you sure that Google can't crawl your site? It's very normal for Google Webmaster Tools to give you errors with the robots.txt file. This is because the robots.txt is an optional file. It's a file that you can put in your root directory to specify which search engine spiders you want to allow/disallow to your site, or from specific directories on your site. If there is no robots.txt file in your root directory, search engines will simply assume that there are no restrictions, and proceed as normal.

    Since I can't see your site to know for sure, you could try checking to see if you have a robots.txt file, as well as ensuring that there's nothing in it that could be blocking your site. If you are not finding anything there, then I wouldn't worry about it. But if you do know for a fact that there's something going on, then maybe get back to us and we could try looking into it further. You could maybe even PM your domain.
     
  8. AnthonyDG

    AnthonyDG Registered Member

    Joined:
    Oct 15, 2013
    Messages:
    63
    Likes Received:
    19
    Yes I can see that everything I specified to be noindex, is in fact, noindex via the source code.

    I can't see it in my root directory, but I can access it by going to website.com/robots.txt for some reason.
     
  9. AnthonyDG

    AnthonyDG Registered Member

    Joined:
    Oct 15, 2013
    Messages:
    63
    Likes Received:
    19
    If it helps, I think I might know why...

    I have both Yoast and the XML Sitemap Generator... I might have configured settings in both "exclusion" settings and it could be causing a problem. I'll try disabling one of them and see.
     
  10. AnthonyDG

    AnthonyDG Registered Member

    Joined:
    Oct 15, 2013
    Messages:
    63
    Likes Received:
    19
    Sorry for the third post in a row, but I resolved the problem. It was the conflicting plugins. I disabled the XML sitemap generator.
    Moral of the story: when using YOAST or any all in one, check your other plugins if they have the same/similar functions as they may conflict.

    A big thanks to everyone that helped!
     
  11. tony_d

    tony_d Elite Member

    Joined:
    Jun 22, 2013
    Messages:
    2,581
    Likes Received:
    3,164
    Location:
    1600 Amphitheatre Parkway, Mountain View CA
    If you post your robots.txt here we'll be able to see what's happening...

    EDIT: I see you already resolved it :)