Google Webmaster: "URL restricted by robots.txt" -> What does this mean?

Hi guys,

I have set up a Google webmaster account for a fairly new domain. I was usually not wanting to get one as I read on here that generally, when using BH methods like using ScrapeBox etc you should not use it (for whatever reason)

Anyway, I signed up and went to submit my sitemap.

I have a WP site and I used to plug in that makes a sitemap for me in .xml

It is located at www.domainname.com/sitemap.xml

I told Google the location of this and it downloaded it but there is an "X" below it and when I click this X, it tells me:

" URL restricted by robots.txt "

This is likely the reason why all of my sites are taking so long to get indexed 'naturally' and require me to ping them ,etc to get them indexed (as they are all running on WP with the same install and same theme so I am assuming they all have the same problem)

I know that the robots.txt tells/sets instructions for robots.. but I have no idea on how to edit it??

Can anyone point me in the right direction, or perhaps a 'standard' robots.txt file that works well. I just want the search engines to index my stuff, nothing fancy.

Thank you and rep will be given for a truly helpful reply!

it means that google can't access the page because robotx.txt is blocking it

Do you know how I can fix it, please?

Your robots.txt is blocking google's spiders to fix this download a plugin named "KB Robots.txt " activate it and the go to the plugin panel and edit your robot.txt file(you can use and ftp client to edit robots.txt but with this plugin its much easier) to this:

I have done some research and this is the best robots.txt for SEO .

Dashboard>privacy settings>allow search engines

risefromdeath said:

Dashboard>privacy settings>allow search engines

Click to expand...

I think that problem is fixed after 2.7 , In the last verions search engines are allowed to spider the blog by default.

He must be using a pretty old WP version

What about if we using the blogspot...how to fix it please,

eddy jason said:

What about if we using the blogspot...how to fix it please,

Click to expand...

Blogger.com ---> Your Account ---> Your Blog ---> Click "Settings" Tab ---> Scroll down about half way ---> Set "Let Search Engines Find Your Blog?" to YES!

Done.

I'm using WP 3.0, the latest is 3.1 so it is not that old and I have my settings to allow Google, Yahoo etc to see my site. I will try the tips here and report back.

Actually, this is a new site and I DIDN'T have let search engines find my blog. I activated it now though. And I installed the recommended plug in and copy-pasted all the information into the plug in, and clicked save.


Then I went to webmaster tools, submitted my sitemap as www.domain.com/sitemap.xml and it is still showing the little X and saying the robots.txt is restricting it?

When I go to my www.domain.com/robots.txt I can see everything that Speedy5044 told me to paste in there?



"URL restricted by robots.txt
We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit."

when adding the sitemap did you used www. ,and when pointing your domain to http://www.yourdomain.com does it redirect you to yourdomain.com?
and when pointing to yourdomain.com/sitemap are you able to see your sitemap?
If you want you can PM the url of your website maybe i can help

Hi Speedy,

I have PM'ed you with my URL, if you could kindly take a look.

Thanks

It's always yes from the beginning...there's a few url that been restricted by it...can we just remove it?any solution

Speedy helped me fix it and now it is working.

It seems that in my WP the URL was http://www.site.com but Speedy told me to change it in WP settings to http://site.com instead, and I did that, and reuploaded sitemap.xml to Google and now it is working!