Any body can help check my robots is right or not?

Discussion in 'White Hat SEO' started by leckie, Mar 12, 2012.

  1. leckie

    leckie Regular Member

    Joined:
    Oct 17, 2011
    Messages:
    219
    Likes Received:
    9
    Occupation:
    Software engineer
    Location:
    China
    I want disallow all html files in /category/ and /channels/
    allow all other html files
    my robots.txt like this:

    User-agent: *
    Allow: /
    Disallow: /category/
    Disallow: /channels/

    Could you do me a favor to check whether the robots.txt is right or not ?
    If not ,how to modify the code ?
     
    Last edited: Mar 12, 2012
  2. jamunkala

    jamunkala Regular Member

    Joined:
    Aug 14, 2010
    Messages:
    284
    Likes Received:
    55
    Location:
    India
    check yourself in google webmaster tools
     
  3. 45415

    45415 Junior Member

    Joined:
    Feb 5, 2009
    Messages:
    194
    Likes Received:
    92
    Looks correct.
     
  4. leckie

    leckie Regular Member

    Joined:
    Oct 17, 2011
    Messages:
    219
    Likes Received:
    9
    Occupation:
    Software engineer
    Location:
    China
    I did
    google said:
    Allowed by line 2: Allow: /
    Detected as a directory; specific files may have different restrictions
    is line3 and line4 code didn't work?
     
    Last edited: Mar 12, 2012
  5. at0mix

    at0mix Newbie

    Joined:
    Nov 27, 2011
    Messages:
    16
    Likes Received:
    1
    checkout "yeost" wordpress plugin (that is if you're using wp)
     
  6. leckie

    leckie Regular Member

    Joined:
    Oct 17, 2011
    Messages:
    219
    Likes Received:
    9
    Occupation:
    Software engineer
    Location:
    China
    Thank you but my site is not write by wordpress
     
  7. Orbit143

    Orbit143 Senior Member

    Joined:
    Aug 8, 2010
    Messages:
    895
    Likes Received:
    591
    Location:
    /home
    It's ok but I wouldn't put this

    Allow: /

    in the file. It is not neccessary and some bots can misinterpret it as Disallow. Google is smart enough to understand, but eg. slightly retarted Bing bot, I wouldnt be so sure ;)
     
  8. leckie

    leckie Regular Member

    Joined:
    Oct 17, 2011
    Messages:
    219
    Likes Received:
    9
    Occupation:
    Software engineer
    Location:
    China
    Thanks
    you mean google will automatic crawl every html file even no need tell Googlebot which file been allowed
     
  9. Orbit143

    Orbit143 Senior Member

    Joined:
    Aug 8, 2010
    Messages:
    895
    Likes Received:
    591
    Location:
    /home
    If you dont disallow it, google automatically expects it is allowed.
     
  10. leckie

    leckie Regular Member

    Joined:
    Oct 17, 2011
    Messages:
    219
    Likes Received:
    9
    Occupation:
    Software engineer
    Location:
    China
    thanks i remove Allow: / and test