1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

GWT says Robots.txt is blocking pages, but is it realy?

Discussion in 'White Hat SEO' started by loginname, May 19, 2012.

  1. loginname

    loginname Regular Member

    Joined:
    Oct 1, 2008
    Messages:
    371
    Likes Received:
    13
    Hi

    My robots.txt looks like this:
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: /wp-login.php
    Disallow: /wp-register.php
    Disallow: /*.php$
    Sitemap: http://www.mydomain.com/sitemap.xml.gz

    In Google Webmaster Toolkit I read that 3 posts have have not been indexed because robots.txt is blocking access. But is it? I'm using the same settings on other sites and there I don't have that issue. Maybe it's correct but google needs time to show that it's corrected...

    any suggestions?
     
  2. milian

    milian Power Member

    Joined:
    Jan 9, 2010
    Messages:
    515
    Likes Received:
    69
    Occupation:
    SEO stuff
    Location:
    Around
    What do the urls of these 3 pages end with? Im guessing its not .php ?
     
  3. loginname

    loginname Regular Member

    Joined:
    Oct 1, 2008
    Messages:
    371
    Likes Received:
    13
    How can I check what this pages are? I've checked in sitemap.xml and there it list all my posts like mydomain.com\mypost\
    (no page there ends with .php). Also notice that I have this domain registered twice on GWT, one with www and one without. That domain without www don't show this error, but the domain with www shows this error