How do I make robots.txt relate to the data in an SQL database? I have a site, and each person has an "inbox" which is a randomly generated 6 digit number/letter combination, so it would be something like domain.com/lol234. Recently, google crawled some of these links, and they can be seen when I list all crawled pages. THIS IS BAD!! All the info is stored in a mySQL database, so how would I make google not crawl these pages? Please help, thanks.