1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

blocking search engine spiders w/ robots.txt or .htaccess?

Discussion in 'Black Hat SEO' started by dirtbag, Aug 16, 2008.

  1. dirtbag

    dirtbag Senior Member

    Joined:
    Jul 24, 2008
    Messages:
    990
    Likes Received:
    525
    I have a page which I wish to keep hidden from the spiders and cannot have it listed in the serps, how can i do this? is it possible with .htaccess, or should i go with robots.txt, i'm not entirely sure how to do either really. what should i do?
     
  2. djedje70

    djedje70 Regular Member

    Joined:
    Feb 5, 2008
    Messages:
    239
    Likes Received:
    252
    Location:
    none of ur fr#$%ing biz dude!!!
    You could use this:

    <html>
    <head>

    <META name="robots" content="noindex, nofollow" >
     
  3. dirtbag

    dirtbag Senior Member

    Joined:
    Jul 24, 2008
    Messages:
    990
    Likes Received:
    525
    is that enough to keep me out of yahoo and google?
     
  4. djedje70

    djedje70 Regular Member

    Joined:
    Feb 5, 2008
    Messages:
    239
    Likes Received:
    252
    Location:
    none of ur fr#$%ing biz dude!!!
    Usually yes.
    This is a must for your ThankYou page if you're selling your own e-book.
    It prevents the web spiders from indexing your download pages.
     
  5. eponus

    eponus Registered Member

    Joined:
    Aug 13, 2008
    Messages:
    95
    Likes Received:
    55
    Occupation:
    Technical Consultant
    Location:
    Boston
    However, just to warn you, robots.txt only blocks "good" spiders like Gbot and Ybot; it won't stop the Al3xa bot or human readers from accessing the page.
     
  6. *zap*

    *zap* Regular Member

    Joined:
    Apr 7, 2008
    Messages:
    349
    Likes Received:
    98
    Occupation:
    none
    Location:
    Uk
    <META name="robots" content="noindex, nofollow" >

    A meta tag will not hide it from the spiders it's is telling the search engines not to index that page

    So is it hide it from the search engines Or stop it from being listed in the seach engine results?
     
  7. seeplusplus

    seeplusplus Power Member

    Joined:
    Aug 18, 2008
    Messages:
    511
    Likes Received:
    163
    Wouldn't using htaccess block All bots though?