1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Is my robots.txt incorrect?

Discussion in 'White Hat SEO' started by ArmoredMedia, Nov 13, 2013.

  1. ArmoredMedia

    ArmoredMedia Junior Member

    Joined:
    Sep 29, 2013
    Messages:
    130
    Likes Received:
    39
    Occupation:
    http://travisingram.net/
    Location:
    Washington, U.S
    Home Page:
    So last night I submitted a sitemap.xml for a site of mine, and I just checked it a few minutes ago (following day), and it says 15 errors on all 15 links.

    In reality there is just 3 warnings on webmasters.
    Webmasters:
    http://puu.sh/5gGDy.png
    Robots:
    http://puu.sh/5gGFk.png

    I can understand why the about-me is blocked since I am still doing the page and I don't want it crawable/see-able yet.

    So what might be the issue? The other two things webmasters listed are crawable... Do I need a robots.txt on every directory or something.
     
  2. gyaseen1986

    gyaseen1986 BANNED BANNED

    Joined:
    Oct 11, 2013
    Messages:
    56
    Likes Received:
    3
    just allow all things
     
  3. FutureProofSeo

    FutureProofSeo Jr. VIP Jr. VIP Premium Member UnGagged Attendee

    Joined:
    Jul 3, 2013
    Messages:
    921
    Likes Received:
    454
    Gender:
    Male
    Occupation:
    Google Warrior
    Location:
    UK
    Home Page:
    You can check the robots.txt section of webmaster tools to monitor how changes effect how the site is crawled. So maybe have a play with that and see what works.
     
  4. morzillon

    morzillon Newbie

    Joined:
    Nov 3, 2013
    Messages:
    15
    Likes Received:
    1
    Still having issues with the indexing?