robot.txt needed?

[reisuke]

Hi SEO gurus,

May i know if there is a need to include a robot.txt?
From what i read , it's like disable other websites for google to crawl in...

If need , may can anyone teach me how to write a basic robot.txt in my website?

Thanks,
SEO newbie

 

[edgematch]

http://lmgtfy.com/?q=robots.txt

 

[Rwh909r]

yes you need a robots.txt
bots will crawl your site by default but you can use the robots to disallow bots crawling certain parts of your site like cgi/bin files or login pages that you dont want public or photo galleries that can cause infinite loops with bots.
You also use the robots.txt to list the sitemap.xml location on your site.

So yes its very much needed and very useful.

 

[jlzandrad]

I did'nt know about this... Creating mine now
Thanks

yes you need a robots.txt
bots will crawl your site by default but you can use the robots to disallow bots crawling certain parts of your site like cgi/bin files or login pages that you dont want public or photo galleries that can cause infinite loops with bots.
You also use the robots.txt to list the sitemap.xml location on your site.

So yes its very much needed and very useful.

 

[Wings[SENSE]]

Is it necessary for a wordpress site? You can manually decide which page you want to index/noindex in the Cpanel.

 

[Alexfrance]

yes, i also do it for the sites that have anything BH on it, for some other sites (all 100% WH), I dont do it cuz I'm too lazy

 

[rohuhottie]

There is no need or compulsion of a robots.txt file.. If you don't have a robots.txt file then the Google bot will have its default behavior on your site.. Robots.txt is helpful if you want to block or make the Google bot work in a specified way..