1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

robots.txt disallow without wildcard

Discussion in 'Web Design' started by blackseo1, Jun 16, 2010.

  1. blackseo1

    blackseo1 Newbie

    Joined:
    Jan 15, 2010
    Messages:
    16
    Likes Received:
    2
    I have the following code in my robots.txt
    Code:
    User-Agent: *
    Disallow: /tos
    Allow: /
    
    to prevent bots accessing hxxp://example.com/tos (terms of service).
    However this results googlebot complaining it is also blocking its access to hxxp://example.com/toscana even though there is no wildcard in disallow!
    How do I tell the bot to ignore /tos only?
     
  2. ourxp

    ourxp Newbie

    Joined:
    Jun 18, 2009
    Messages:
    28
    Likes Received:
    1
    how about trying:

    Disallow: /tos$

    or

    Disallow: /tos/
     
    • Thanks Thanks x 1
  3. blackseo1

    blackseo1 Newbie

    Joined:
    Jan 15, 2010
    Messages:
    16
    Likes Received:
    2
    thanks, $ at the end was what I needed