1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

how to hide robots.txt

Discussion in 'Black Hat SEO' started by glasshalffull, Aug 3, 2009.

  1. glasshalffull

    glasshalffull Registered Member

    Joined:
    May 30, 2008
    Messages:
    76
    Likes Received:
    50
    when i add robots.txt to the end of my url i pull up some pages that are disallowed

    how do i stop that robots.txt from showing my hidden pages? I remember a thread aobut this in here but can't find it?

    thanks
     
  2. Kingfresh

    Kingfresh Regular Member

    Joined:
    Jul 8, 2009
    Messages:
    374
    Likes Received:
    295
  3. glasshalffull

    glasshalffull Registered Member

    Joined:
    May 30, 2008
    Messages:
    76
    Likes Received:
    50
    thanks King
     
  4. orangejuice

    orangejuice Registered Member

    Joined:
    Jul 25, 2009
    Messages:
    54
    Likes Received:
    16
    Remember that you can fake your way into your robots.txt (especially if you only use user-agent checking), so don't leave anything in there you never want people to see.
     
  5. kouxx

    kouxx Junior Member

    Joined:
    Aug 2, 2008
    Messages:
    160
    Likes Received:
    38
    you can use htaccess as said above. thats when you want to use a password.

    if you just dont want the spiders to crawl you pages you can just use noindex in your meta tags. use the searchengine of you choice to search for "noindex meta tag" ;)

    edit: when using meta tags you DO NOT need to edit yout robots.txt
     
  6. Kingfresh

    Kingfresh Regular Member

    Joined:
    Jul 8, 2009
    Messages:
    374
    Likes Received:
    295
    you are welcome
     
  7. glasshalffull

    glasshalffull Registered Member

    Joined:
    May 30, 2008
    Messages:
    76
    Likes Received:
    50
    basically i just don't want people that know how to use the robots.txt to be able to access any pages I don't want seen, so will htaccess be the easiest?

    thanks

     
  8. kouxx

    kouxx Junior Member

    Joined:
    Aug 2, 2008
    Messages:
    160
    Likes Received:
    38
    the easiest way would be not to upload a robots.txt
    you do not need the file at all.
    and for the SE's: as long as you to don't link to the sites, they will not crawl it. If you give a link to someone else and that person posts the link in a fourm or something you should delete the page anyway.