1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

googlebot

Discussion in 'Black Hat SEO' started by Ledlys, Dec 14, 2013.

  1. Ledlys

    Ledlys Regular Member

    Joined:
    Oct 3, 2011
    Messages:
    232
    Likes Received:
    0
    Got this message back in june: Headline Googlebot cannot acces your robot txt:

    Over the last 24 hours Googlebot found 15 errors while trying to access the robots.txt. To ensure that the pages listed in the file was not scanned, was crawling prone. The overall error rate for robots.txt on your website is 60.0%.

    I opende my robots.txt and found this:
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    Sitemap: http://www.mysite.com/sitemap.xml.gz


    How can I fix this? What should I include in my robot.txt?
     
  2. Ptrick125

    Ptrick125 Regular Member

    Joined:
    Mar 4, 2013
    Messages:
    428
    Likes Received:
    113
    Occupation:
    Going To School
    Location:
    Near Austin, Texas
    Home Page:
    Robots.txt tells search engines what pages to crawl and what pages not to crawl. The Yoast SEO plugin has a sitemap generator.

    Posted via Topify using iPhone/iPad
     
  3. tony_d

    tony_d Elite Member

    Joined:
    Jun 22, 2013
    Messages:
    2,581
    Likes Received:
    3,163
    Location:
    1600 Amphitheatre Parkway, Mountain View CA
    I think that's fine anyway - I wouldn't let G crawl those directories, because it's not something you want crawled or listed in the SERPs.
    I'm assuming your main page(s) appear in the SERPs?
     
  4. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    8,883
    Likes Received:
    7,481
    Occupation:
    ZLinky2Buy SEO Services
    Location:
    ⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩
    Home Page:
    Doesn't it give you any other details? What are the errors on robots.txt? There are normally descriptions next to the errors, mentioning what went wrong....

    From what you posted your robots.txt file looks alright.