1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

GoogleBot cannot access/index my blog

Discussion in 'Blogging' started by Oukast, Nov 26, 2012.

  1. Oukast

    Oukast Senior Member

    Joined:
    Jan 11, 2012
    Messages:
    832
    Likes Received:
    683
    Location:
    Under the palm tree
    So I set up a new blog today, on a same server I'm running two other blogs currently. Everything seemed to go fine, but I cannot fetch my site as GoogleBot, and also sitemap submission fails due to "URL restricted by robots.txt". Now I'm not using robots.txt (never have), and BingBot fetches the sites and sitemap just fine - so what am I doing wrong?

    GoogleBot can fetch the other blogs just fine, so shouldn't be a connectivity issue either. I wouldn't mind if it was just not getting WMT data, but it prevents the site to be indexed completely :crazy:

    Don't know if it's much of use, but you can check the source at fintattoodotcom. All help appreciated!
     
  2. Endire

    Endire Elite Member Premium Member

    Joined:
    Mar 27, 2012
    Messages:
    1,756
    Likes Received:
    1,061
    Gender:
    Male
    Very good question Oukast,

    Have you tried adding the following to the head section of your site,

    <meta name="robots" content="index,follow" />

    If that doesn't change anything, perhaps try removing this URL from your header (unless of course it is present on your other sites that are being crawled normally),

    <head profile="http://gmpg.org/xfn/11">


    Perhaps you can find some answers here,

    https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag

    Also, does this site redirect to any other sites that may use robots.txt?

    Hope that helps,

    Shawn
     
    • Thanks Thanks x 1
  3. umerjutt00

    umerjutt00 Jr. VIP Jr. VIP Premium Member

    Joined:
    Oct 28, 2011
    Messages:
    3,644
    Likes Received:
    1,904
    Occupation:
    Ninja
    Check your robot.txt file. I think the problem will be there. Add this in your head sectin:
    <meta name="robots" content="index,follow" />
     
  4. Oukast

    Oukast Senior Member

    Joined:
    Jan 11, 2012
    Messages:
    832
    Likes Received:
    683
    Location:
    Under the palm tree
    But it's not :)

    I've installed many self-hosted WP's with practically the same settings, and always just using the WP "virtual" robots.txt with the addition to my sitemap generator plugin, so the robots.txt looks as follows:
    I fetched the site as GoogleBot just moments ago, and it worked fine, but the sitemap failed. Then I removed the "<head profile="http://gmpg.org/xfn/11">" line and it seems that the sitemap was also fetched fine.So thanks a bunch, seems that this was the problem!