Hi
I need help with this, currently my Robots.txt file is setup in this order as i dont want my website to be crawled in other directories other than the important ones?
User-agent: *
Disallow:
Disallow: /cgi-bin/
Disallow: /Admin/
Disallow: /images/
Disallow: /Account/
Disallow: /bin/...
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.