1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots.txt

Discussion in 'Black Hat SEO' started by simmonsmike7, Nov 28, 2008.

  1. simmonsmike7

    simmonsmike7 BANNED BANNED

    Joined:
    Jun 29, 2008
    Messages:
    123
    Likes Received:
    20
    I want to make this entire directory on my site not showing to any search engine, and I think you do this by adding a robots.txt file.

    1. Am I right?

    2. If so, where do I get this file?

    3. And finally, where do I place it.

    I have the directory installed on my website such as: http://www.mywebsite.com/directory

    Thank You.
     
  2. delmages

    delmages Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 24, 2008
    Messages:
    129
    Likes Received:
    130
    Occupation:
    Nursing
    Location:
    Northern Michigan
    You can easily create a robots.txt using an online generator. Try the following listed here:
    Code:
    http://www.mcanerin.com/en/search-engine/robots-txt.asp
     
    • Thanks Thanks x 1
  3. *zap*

    *zap* Regular Member

    Joined:
    Apr 7, 2008
    Messages:
    349
    Likes Received:
    98
    Occupation:
    none
    Location:
    Uk
    Don't rely on just a Robots.txt 4 this as it will fail
     
  4. jonshannow

    jonshannow Junior Member

    Joined:
    Jul 25, 2008
    Messages:
    128
    Likes Received:
    86
    Occupation:
    full time online business
    Location:
    Melbourne, Australia
    This is true. The best way to ensure spiders don't index a section is to make it password protected.
     
  5. jb01

    jb01 Registered Member

    Joined:
    Jan 10, 2008
    Messages:
    55
    Likes Received:
    30
    If you don't want the spiders to find something then do not
    have any internal or external links to it. All spiders do is
    follow links.
     
  6. dvdcowboy

    dvdcowboy Junior Member

    Joined:
    May 10, 2008
    Messages:
    188
    Likes Received:
    39
    Use robots.txt and also turn off indexing in cpanel.
     
  7. Tom__

    Tom__ Junior Member

    Joined:
    Nov 9, 2008
    Messages:
    134
    Likes Received:
    300
    Just blacklist all spiderIP's, their are websites who got lists of over 10.000 IP's from all sort of spiders. You could also just make everything in javascript (text output) and with some encryption. Spiders don't execute javascript so that means they won't see the text.
     
  8. java

    java Guest

    Also be aware that people can read the robots.txt file. It's a common place to look if people are trying to find hidden pages on your site.
     
  9. fatboy

    fatboy Elite Member

    Joined:
    Aug 13, 2008
    Messages:
    1,618
    Likes Received:
    3,227
    Occupation:
    Retired
    Location:
    Old Peoples Home
    and another thing to remember - robots.txt files only stops robots that obey the rules.
    Any scripted bot that doesn't obey the txt file will still stamp all over your site.
     
  10. istarapol

    istarapol Junior Member

    Joined:
    Jun 3, 2008
    Messages:
    110
    Likes Received:
    228
    Occupation:
    Graphic Designer
    Location:
    Under Your Bed
    Home Page:
    • use robots.txt - disallow all known robots heck even block voltes V, megatron and Voltron
    • php header redirect - when the script finds the referral/ IP is from google/ yahoo etc
    redirect them
    • put this between <head></head>
    <meta name="robots" content="noindex, nofollow" />
    • put it in frame
    • put a username and password (create an image for the visitors on how and what the username and passwords are)

    whatelse?
     
  11. mr4army

    mr4army Regular Member

    Joined:
    Nov 23, 2008
    Messages:
    459
    Likes Received:
    80
    Home Page:
    first make a robots.txt search google robots.txt generator
    second make it password protected with .httacsess and you will be fine (=
     
  12. dsizzle

    dsizzle Registered Member

    Joined:
    Mar 13, 2008
    Messages:
    52
    Likes Received:
    15
    jb01 is correct. This is all I do (or should I say, don't do) to protect my pages.
    No links means the spiders dont know about it.