You can do some nifty things with robots.txt.
One of my sites has a lot of dynamic pages that are updated frequently.
This all goes automatically. It's a mock-up of different api's.
1000's of urls, fresh content, the spiders go crawling like crazy.
Result? Api rate limits exceded.
(Yes, even...