1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Any body can help check my robots is right or not?

Discussion in 'White Hat SEO' started by leckie, Mar 12, 2012.

  1. leckie

    leckie Regular Member

    Joined:
    Oct 17, 2011
    Messages:
    219
    Likes Received:
    9
    Occupation:
    Software engineer
    Location:
    China
    I want disallow all html files in /category/ and /channels/
    allow all other html files
    my robots.txt like this:

    User-agent: *
    Allow: /
    Disallow: /category/
    Disallow: /channels/

    Could you do me a favor to check whether the robots.txt is right or not ?
    If not ,how to modify the code ?
     
    Last edited: Mar 12, 2012
  2. jamunkala

    jamunkala Regular Member

    Joined:
    Aug 14, 2010
    Messages:
    284
    Likes Received:
    55
    Location:
    India
    Home Page:
    check yourself in google webmaster tools
     
  3. 45415

    45415 Junior Member

    Joined:
    Feb 5, 2009
    Messages:
    194
    Likes Received:
    92
    Looks correct.
     
  4. leckie

    leckie Regular Member

    Joined:
    Oct 17, 2011
    Messages:
    219
    Likes Received:
    9
    Occupation:
    Software engineer
    Location:
    China
    I did
    google said:
    Allowed by line 2: Allow: /
    Detected as a directory; specific files may have different restrictions
    is line3 and line4 code didn't work?
     
    Last edited: Mar 12, 2012
  5. at0mix

    at0mix Newbie

    Joined:
    Nov 27, 2011
    Messages:
    9
    Likes Received:
    1
    checkout "yeost" wordpress plugin (that is if you're using wp)
     
  6. leckie

    leckie Regular Member

    Joined:
    Oct 17, 2011
    Messages:
    219
    Likes Received:
    9
    Occupation:
    Software engineer
    Location:
    China
    Thank you but my site is not write by wordpress
     
  7. Orbit143

    Orbit143 Senior Member

    Joined:
    Aug 8, 2010
    Messages:
    893
    Likes Received:
    588
    Location:
    /home
    It's ok but I wouldn't put this

    Allow: /

    in the file. It is not neccessary and some bots can misinterpret it as Disallow. Google is smart enough to understand, but eg. slightly retarted Bing bot, I wouldnt be so sure ;)
     
  8. leckie

    leckie Regular Member

    Joined:
    Oct 17, 2011
    Messages:
    219
    Likes Received:
    9
    Occupation:
    Software engineer
    Location:
    China
    Thanks
    you mean google will automatic crawl every html file even no need tell Googlebot which file been allowed
     
  9. Orbit143

    Orbit143 Senior Member

    Joined:
    Aug 8, 2010
    Messages:
    893
    Likes Received:
    588
    Location:
    /home
    If you dont disallow it, google automatically expects it is allowed.
     
  10. leckie

    leckie Regular Member

    Joined:
    Oct 17, 2011
    Messages:
    219
    Likes Received:
    9
    Occupation:
    Software engineer
    Location:
    China
    thanks i remove Allow: / and test