robot.txt needed?
- Thread starter reisuke
- Start date
edgematch
Elite Member
- May 24, 2010
- 2,545
- 1,971
Last edited:
yes you need a robots.txt
bots will crawl your site by default but you can use the robots to disallow bots crawling certain parts of your site like cgi/bin files or login pages that you dont want public or photo galleries that can cause infinite loops with bots.
You also use the robots.txt to list the sitemap.xml location on your site.
So yes its very much needed and very useful.
bots will crawl your site by default but you can use the robots to disallow bots crawling certain parts of your site like cgi/bin files or login pages that you dont want public or photo galleries that can cause infinite loops with bots.
You also use the robots.txt to list the sitemap.xml location on your site.
So yes its very much needed and very useful.
jlzandrad
Junior Member
- Apr 22, 2009
- 164
- 26
Wings[SENSE]
Regular Member
- Jan 10, 2010
- 372
- 180
Is it necessary for a wordpress site? You can manually decide which page you want to index/noindex in the Cpanel.
Alexfrance
Power Member
- Jul 26, 2010
- 685
- 165
yes, i also do it for the sites that have anything BH on it, for some other sites (all 100% WH), I dont do it cuz I'm too lazy
rohuhottie
Newbie
- Aug 11, 2010
- 32
- 4
There is no need or compulsion of a robots.txt file.. If you don't have a robots.txt file then the Google bot will have its default behavior on your site.. Robots.txt is helpful if you want to block or make the Google bot work in a specified way..