robots.txt disallow without wildcard

Discussion in 'Web Design' started by blackseo1, Jun 16, 2010.

  1. blackseo1

    blackseo1 Newbie

    Joined:
    Jan 15, 2010
    Messages:
    16
    Likes Received:
    2
    I have the following code in my robots.txt
    Code:
    User-Agent: *
    Disallow: /tos
    Allow: /
    
    to prevent bots accessing hxxp://example.com/tos (terms of service).
    However this results googlebot complaining it is also blocking its access to hxxp://example.com/toscana even though there is no wildcard in disallow!
    How do I tell the bot to ignore /tos only?
     
  2. ourxp

    ourxp Newbie

    Joined:
    Jun 18, 2009
    Messages:
    28
    Likes Received:
    1
    how about trying:

    Disallow: /tos$

    or

    Disallow: /tos/
     
    • Thanks Thanks x 1
  3. blackseo1

    blackseo1 Newbie

    Joined:
    Jan 15, 2010
    Messages:
    16
    Likes Received:
    2
    thanks, $ at the end was what I needed