Robots.txt fetch error -- Google can't crawl my site!
- Thread starter AnthonyDG
- Start date
seeplusplus
Power Member
- Aug 18, 2008
- 525
- 179
It is robots.txt and not Robots.txt isn't it?
Should be in the domain's root folder.
Should be in the domain's root folder.
Yes it is a wordpress site. You mean "Discourage search engines from indexing this site?" Yes, that's unchecked.
Yup it's robots.txt, not capitalized. I can even access it at mywebsite/robots.txt, so I know it's there...
seeplusplus
Power Member
- Aug 18, 2008
- 525
- 179
In that case then as oxonbeef suggests, have a look in the source code of each page with your browser and check if WP is putting the meta tags to noindex, doesn't sound like a robots.txt issue (assuming it's wrote correctly).
TimothyEyton
Junior Member
- Dec 30, 2013
- 129
- 63
Are you sure that Google can't crawl your site? It's very normal for Google Webmaster Tools to give you errors with the robots.txt file. This is because the robots.txt is an optional file. It's a file that you can put in your root directory to specify which search engine spiders you want to allow/disallow to your site, or from specific directories on your site. If there is no robots.txt file in your root directory, search engines will simply assume that there are no restrictions, and proceed as normal.
Since I can't see your site to know for sure, you could try checking to see if you have a robots.txt file, as well as ensuring that there's nothing in it that could be blocking your site. If you are not finding anything there, then I wouldn't worry about it. But if you do know for a fact that there's something going on, then maybe get back to us and we could try looking into it further. You could maybe even PM your domain.
Since I can't see your site to know for sure, you could try checking to see if you have a robots.txt file, as well as ensuring that there's nothing in it that could be blocking your site. If you are not finding anything there, then I wouldn't worry about it. But if you do know for a fact that there's something going on, then maybe get back to us and we could try looking into it further. You could maybe even PM your domain.
Yes I can see that everything I specified to be noindex, is in fact, noindex via the source code.
I can't see it in my root directory, but I can access it by going to website.com/robots.txt for some reason.
If it helps, I think I might know why...
I have both Yoast and the XML Sitemap Generator... I might have configured settings in both "exclusion" settings and it could be causing a problem. I'll try disabling one of them and see.
I have both Yoast and the XML Sitemap Generator... I might have configured settings in both "exclusion" settings and it could be causing a problem. I'll try disabling one of them and see.
Sorry for the third post in a row, but I resolved the problem. It was the conflicting plugins. I disabled the XML sitemap generator.
Moral of the story: when using YOAST or any all in one, check your other plugins if they have the same/similar functions as they may conflict.
A big thanks to everyone that helped!
Moral of the story: when using YOAST or any all in one, check your other plugins if they have the same/similar functions as they may conflict.
A big thanks to everyone that helped!
