1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

hiding pages from search engine bots & humans

Discussion in 'Blogging' started by pyrokinetic, Nov 27, 2011.

  1. pyrokinetic

    pyrokinetic Registered Member

    Joined:
    Apr 26, 2011
    Messages:
    97
    Likes Received:
    87
    Location:
    In a pretty dark place.
    Hi all,

    I've been searching on this topic but can't really seem to find a decent answer.
    I want to hide my wordpress plugin page from being indexed.
    I also have a few pages I don't want indexed, like a download page and some pages with content for certain people only, and I don't want them indexed or found by other people.

    What are the best ways to go about this?

    For the humans I figured I'd use a login page.

    To stop indexing, I read that robots.txt isn't that trustworthy. G & Y bots follow it but other bots don't.
    How do I stop those others from indexing my pages?

    Is there a one-size-fits-all-solution? Do I have to combine robots.txt with .htaccess? Anything else?

    I'll be glad to learn from some more experienced members here.
     
  2. pyrokinetic

    pyrokinetic Registered Member

    Joined:
    Apr 26, 2011
    Messages:
    97
    Likes Received:
    87
    Location:
    In a pretty dark place.
    68 views yet noone able to help me?
     
  3. Russ1395

    Russ1395 Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 16, 2010
    Messages:
    128
    Likes Received:
    108
    Home Page:
    Assuming you know how to use the robots.txt file, use the following user agents:

    Code:
    User-Agent: Googlebot
    
    As for Yahoo, I don't know what the User Agent for them is. Some other tips you could try are:

    -Make sure all links to those pages use the rel="nofollow" attribute
    -Make sure none of those pages are listed in your sitemap
    -Add a noindex, nofollow meta tag on all of those blocked pages