Hi all, I've been searching on this topic but can't really seem to find a decent answer. I want to hide my wordpress plugin page from being indexed. I also have a few pages I don't want indexed, like a download page and some pages with content for certain people only, and I don't want them indexed or found by other people. What are the best ways to go about this? For the humans I figured I'd use a login page. To stop indexing, I read that robots.txt isn't that trustworthy. G & Y bots follow it but other bots don't. How do I stop those others from indexing my pages? Is there a one-size-fits-all-solution? Do I have to combine robots.txt with .htaccess? Anything else? I'll be glad to learn from some more experienced members here.
Assuming you know how to use the robots.txt file, use the following user agents: Code: User-Agent: Googlebot As for Yahoo, I don't know what the User Agent for them is. Some other tips you could try are: -Make sure all links to those pages use the rel="nofollow" attribute -Make sure none of those pages are listed in your sitemap -Add a noindex, nofollow meta tag on all of those blocked pages
