1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Googlebot can't access your site? (Robots file included)

Discussion in 'White Hat SEO' started by Patrick2014, Nov 25, 2015.

  1. Patrick2014

    Patrick2014 Junior Member

    Joined:
    Mar 7, 2014
    Messages:
    151
    Likes Received:
    24
    Googlebot can't access your site

    Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%.



    • If you are able to access it from your browser, then your site may be configured to deny access to googlebot. Check the configuration of your firewall and site to ensure that you are not denying access to googlebot.
    • If your robots.txt is a static page, verify that your web service has proper permissions to access the file.
    • If your robots.txt is dynamically generated, verify that the scripts that generate the robots.txt are properly configured and have permission to run. Check the logs for your website to see if your scripts are failing, and if so attempt to diagnose the cause of the failure.




    I received this error, I checked my robots file in browser and it is this..
    I have 4 other websites on server and they have had no problem?



    Code:
    User-agent: *
    Disallow: /wp-admin/
    
    Sitemap: http://www.******.org/sitemap.xml.gz
    
    Sitemap: http://www.******.org/sitemap.xml
     
  2. CrazyBuddy

    CrazyBuddy Regular Member

    Joined:
    Jun 23, 2013
    Messages:
    489
    Likes Received:
    126
    Occupation:
    Part Time IM , Full time Enjoyment
    Location:
    Earth
    if you are using the same robots.txt , then 100% it is due to server downtime..which hosting company are you with
     
  3. wesleyjf91

    wesleyjf91 Registered Member

    Joined:
    Sep 24, 2015
    Messages:
    58
    Likes Received:
    13
    Occupation:
    SEO Manager
    Location:
    San Diego
    Are you running any security plugins through a CMS?

    Those could be blocking bots. Could be server issues.
     
  4. rabbitking

    rabbitking Elite Member

    Joined:
    Sep 24, 2013
    Messages:
    1,712
    Likes Received:
    3,579
    I had this issue too because of a server issue like mentioned above. The only other time it happened was
    from a plugin that wasn't working properly. It also takes a bit of time for this issue to resolve on Googles side.