1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Google Webmaster: "URL restricted by robots.txt" -> What does this mean?

Discussion in 'Black Hat SEO' started by krzysiekz, Sep 23, 2010.

  1. krzysiekz

    krzysiekz Senior Member

    Joined:
    Jul 29, 2010
    Messages:
    953
    Likes Received:
    579
    Hi guys,

    I have set up a Google webmaster account for a fairly new domain. I was usually not wanting to get one as I read on here that generally, when using BH methods like using ScrapeBox etc you should not use it (for whatever reason)

    Anyway, I signed up and went to submit my sitemap.

    I have a WP site and I used to plug in that makes a sitemap for me in .xml

    It is located at www.domainname.com/sitemap.xml

    I told Google the location of this and it downloaded it but there is an "X" below it and when I click this X, it tells me:

    " URL restricted by robots.txt "

    This is likely the reason why all of my sites are taking so long to get indexed 'naturally' and require me to ping them ,etc to get them indexed (as they are all running on WP with the same install and same theme so I am assuming they all have the same problem)

    I know that the robots.txt tells/sets instructions for robots.. but I have no idea on how to edit it??

    Can anyone point me in the right direction, or perhaps a 'standard' robots.txt file that works well. I just want the search engines to index my stuff, nothing fancy.

    Thank you and rep will be given for a truly helpful reply!
     
  2. rambynas

    rambynas Newbie

    Joined:
    Nov 27, 2009
    Messages:
    19
    Likes Received:
    1
    it means that google can't access the page because robotx.txt is blocking it
     
    • Thanks Thanks x 1
  3. krzysiekz

    krzysiekz Senior Member

    Joined:
    Jul 29, 2010
    Messages:
    953
    Likes Received:
    579
    Do you know how I can fix it, please? :)
     
  4. speedy5044

    speedy5044 Regular Member

    Joined:
    Jul 29, 2008
    Messages:
    456
    Likes Received:
    994
    Occupation:
    IM
    Your robots.txt is blocking google's spiders to fix this download a plugin named "KB Robots.txt " activate it and the go to the plugin panel and edit your robot.txt file(you can use and ftp client to edit robots.txt but with this plugin its much easier) to this:
    Code:
    # This rule means it applies to all user-agents
    User-agent:  *
    Disallow: /wp-content/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-
    Disallow: /trackback/
    Disallow: /cgi-bin/
     
    # Disallow all monthly archive pages
    Disallow: /2005/0
    Disallow: /2005/1
    Disallow: /2006/0
    Disallow: /2006/1
    Disallow: /2007/0
    Disallow: /2007/1
     
    # The Googlebot is the main search bot for google
    User-agent: Googlebot
     
    # Disallow all files ending with these extensions
    Disallow: /*.php$
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.gz$
    Disallow: /*.wmv$
    Disallow: /*.tar$
    Disallow: /*.tgz$
    Disallow: /*.cgi$
    Disallow: /*.xhtml$
     
    # Disallow all files with ? in url
    Disallow: /*?*
    Disallow: /*?
     
    # Disallow all archived monthlies
    Disallow: /2006/0*
    Disallow: /2007/0*
    Disallow: /2006/1*
    Disallow: /2007/1*
    
    # The Googlebot-Image is the image bot for google
    User-agent: Googlebot-Image
     
    # Allow Everything
    Allow: /*
     
    # This is the ad bot for google
    User-agent: Mediapartners-Google*
     
    # Allow Everything
    Allow: /*
    I have done some research and this is the best robots.txt for SEO .
     
    • Thanks Thanks x 1
    Last edited: Sep 23, 2010
  5. risefromdeath

    risefromdeath BANNED BANNED

    Joined:
    Jul 1, 2009
    Messages:
    652
    Likes Received:
    107
    Dashboard>privacy settings>allow search engines :)
     
  6. speedy5044

    speedy5044 Regular Member

    Joined:
    Jul 29, 2008
    Messages:
    456
    Likes Received:
    994
    Occupation:
    IM
    I think that problem is fixed after 2.7 , In the last verions search engines are allowed to spider the blog by default.
     
  7. nufaman

    nufaman Elite Member

    Joined:
    May 29, 2009
    Messages:
    1,710
    Likes Received:
    1,193
    He must be using a pretty old WP version
     
  8. eddy jason

    eddy jason Newbie

    Joined:
    Sep 7, 2010
    Messages:
    4
    Likes Received:
    0
    What about if we using the blogspot...how to fix it please,
     
  9. Chronos

    Chronos Junior Member

    Joined:
    May 5, 2010
    Messages:
    126
    Likes Received:
    294

    Blogger.com ---> Your Account ---> Your Blog ---> Click "Settings" Tab ---> Scroll down about half way ---> Set "Let Search Engines Find Your Blog?" to YES!

    Done.
     
  10. krzysiekz

    krzysiekz Senior Member

    Joined:
    Jul 29, 2010
    Messages:
    953
    Likes Received:
    579
    I'm using WP 3.0, the latest is 3.1 so it is not that old and I have my settings to allow Google, Yahoo etc to see my site. I will try the tips here and report back.
     
  11. krzysiekz

    krzysiekz Senior Member

    Joined:
    Jul 29, 2010
    Messages:
    953
    Likes Received:
    579
    Actually, this is a new site and I DIDN'T have let search engines find my blog. I activated it now though. And I installed the recommended plug in and copy-pasted all the information into the plug in, and clicked save.


    Then I went to webmaster tools, submitted my sitemap as www.domain.com/sitemap.xml and it is still showing the little X and saying the robots.txt is restricting it?

    When I go to my www.domain.com/robots.txt I can see everything that Speedy5044 told me to paste in there?



    "URL restricted by robots.txt
    We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit."
     
  12. speedy5044

    speedy5044 Regular Member

    Joined:
    Jul 29, 2008
    Messages:
    456
    Likes Received:
    994
    Occupation:
    IM
    when adding the sitemap did you used www. ,and when pointing your domain to www.yourdomain.com does it redirect you to yourdomain.com?
    and when pointing to yourdomain.com/sitemap are you able to see your sitemap?
    If you want you can PM the url of your website maybe i can help
     
    • Thanks Thanks x 1
  13. krzysiekz

    krzysiekz Senior Member

    Joined:
    Jul 29, 2010
    Messages:
    953
    Likes Received:
    579
    Hi Speedy,

    I have PM'ed you with my URL, if you could kindly take a look.

    Thanks :)
     
  14. eddy jason

    eddy jason Newbie

    Joined:
    Sep 7, 2010
    Messages:
    4
    Likes Received:
    0
    It's always yes from the beginning...there's a few url that been restricted by it...can we just remove it?any solution
     
  15. krzysiekz

    krzysiekz Senior Member

    Joined:
    Jul 29, 2010
    Messages:
    953
    Likes Received:
    579
    Speedy helped me fix it and now it is working.

    It seems that in my WP the URL was http://www.site.com but Speedy told me to change it in WP settings to http://site.com instead, and I did that, and reuploaded sitemap.xml to Google and now it is working!