I have the following code in my robots.txt
to prevent bots accessing hxxp://example.com/tos (terms of service).
However this results googlebot complaining it is also blocking its access to hxxp://example.com/toscana even though there is no wildcard in disallow!
How do I tell the bot to ignore /tos only?
Code:
User-Agent: *
Disallow: /tos
Allow: /
to prevent bots accessing hxxp://example.com/tos (terms of service).
However this results googlebot complaining it is also blocking its access to hxxp://example.com/toscana even though there is no wildcard in disallow!
How do I tell the bot to ignore /tos only?