Get the correct code for your htaccess and allow it sometime for majestic and the like to implement your request.
Spyderspanker works for me, there is also a free alternative out there which works. I used it, forget the name, pretty sure someone will post the link here
htaccess works like a charme. there are several threads here showing the code. You dont need to pay for any plugin. Dont use robots.txt because it is a footprint and not as effective as htaccess
aussiejack is right, use .htaccess and it blocks them and don't use robots.txt because they'll ignore it.
