GWT says Robots.txt is blocking pages, but is it realy?

Discussion in 'White Hat SEO' started by loginname, May 19, 2012.

  1. loginname

    loginname Regular Member

    Joined:
    Oct 1, 2008
    Messages:
    405
    Likes Received:
    14
    Hi

    My robots.txt looks like this:
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: /wp-login.php
    Disallow: /wp-register.php
    Disallow: /*.php$
    Sitemap: http://www.mydomain.com/sitemap.xml.gz

    In Google Webmaster Toolkit I read that 3 posts have have not been indexed because robots.txt is blocking access. But is it? I'm using the same settings on other sites and there I don't have that issue. Maybe it's correct but google needs time to show that it's corrected...

    any suggestions?
     
  2. milian

    milian Power Member

    Joined:
    Jan 9, 2010
    Messages:
    515
    Likes Received:
    69
    Occupation:
    SEO stuff
    Location:
    Around
    What do the urls of these 3 pages end with? Im guessing its not .php ?
     
  3. loginname

    loginname Regular Member

    Joined:
    Oct 1, 2008
    Messages:
    405
    Likes Received:
    14
    How can I check what this pages are? I've checked in sitemap.xml and there it list all my posts like mydomain.com\mypost\
    (no page there ends with .php). Also notice that I have this domain registered twice on GWT, one with www and one without. That domain without www don't show this error, but the domain with www shows this error