robots.txt

roborts.txt is to inform search engine spiders not to crawl certain pages. If search engine spiders crawl a web page it will be displayed in search engines for general public. People use roborts.txt option to block some personalized/private content.

below an example optimized robots.txt for my wordpress site (just my own perspective )

# Google Bot
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /search/
Disallow: /trackback/
Disallow: *?*
Disallow: */trackback/
Disallow: */date/*
Disallow: */page/*

# Google Image
User-agent: Googlebot-Image
Allow: /

# Google Mobile
User-agent: Googlebot-Mobile
Allow: /

# Google AdSense
User-agent: Mediapartners-Google*
Allow: /
Disallow: *?*
User-agent: Adsbot-Google
Allow: /

Sitemap: url/to/sitemap.xml

Click to expand...

any suggestion or better advice, let me know

I never use robots.txt. On new sites i create a blank one just to avoid all those 404 error being logged in my apache logs

You can do some nifty things with robots.txt.
One of my sites has a lot of dynamic pages that are updated frequently.
This all goes automatically. It's a mock-up of different api's.
1000's of urls, fresh content, the spiders go crawling like crazy.
Result? Api rate limits exceded.
(Yes, even with cached results. These damn spiders sometimes!)
So I added this to robots.txt:
Problem solved.
I also disallow a few bots.
But be carefull with robots.txt.
Especially if you don't know what you are doing.
Before you know it, you disallow every spider.
And you probably don't want that