1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

question about deindexing

Discussion in 'Black Hat SEO' started by rap8557, Dec 4, 2008.

  1. rap8557

    rap8557 Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 15, 2008
    Messages:
    1,081
    Likes Received:
    436
    Im planning to set up a buy and sell site.. now after purchasing the product the thank you page will appear with the download link right?!

    my question is how can i hide the thanks you page from search engine?
    i mean i dont want this page to be indexed how can i do this?

    thanks
     
  2. oldenstylehats

    oldenstylehats Elite Member Premium Member

    Joined:
    Apr 10, 2008
    Messages:
    1,893
    Likes Received:
    1,196
    Create a robots.txt file. Put this in it:
    Code:
    User-agent: *
    Disallow: /path/to/your/thankyou/page.html
    Save it and upload it to your root web directory. If you want to double check and make sure, Google has a tool in their Webmaster Tools services which will parse robots.txt and let you know where they will and will not go.

    For more info, go here:
    Code:
    http://www.robotstxt.org/robotstxt.html
     
    • Thanks Thanks x 1
    Last edited: Dec 4, 2008
  3. rap8557

    rap8557 Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 15, 2008
    Messages:
    1,081
    Likes Received:
    436
  4. oldenstylehats

    oldenstylehats Elite Member Premium Member

    Joined:
    Apr 10, 2008
    Messages:
    1,893
    Likes Received:
    1,196
    Robots.txt is the standard exclusion file for all spiders and robots on the web.
     
  5. rap8557

    rap8557 Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 15, 2008
    Messages:
    1,081
    Likes Received:
    436
  6. oldenstylehats

    oldenstylehats Elite Member Premium Member

    Joined:
    Apr 10, 2008
    Messages:
    1,893
    Likes Received:
    1,196
    No problem. Almost every single "good" robot is going to read robots.txt and follow the directions it gives and that includes all major search engine spiders.
     
  7. iSLaND

    iSLaND Registered Member

    Joined:
    Jan 28, 2008
    Messages:
    75
    Likes Received:
    40
    i think there is a vulnerability that contents of robots.txt file can be read as normal txt file.
    correct me if i m wrong. i m not expert in such things.
     
  8. iSLaND

    iSLaND Registered Member

    Joined:
    Jan 28, 2008
    Messages:
    75
    Likes Received:
    40
    well, sorry for double post(can't edit the one above) but what i meant is this:

    Code:
    http://www.6figurefreelancer.com/robots.txt
    ;)
     
  9. xbox360gurl70s

    xbox360gurl70s Elite Member

    Joined:
    Sep 28, 2008
    Messages:
    1,532
    Likes Received:
    349
    Location:
    In your wet dreams
    a simple log in required download page will make your files safe.
     
  10. *zap*

    *zap* Regular Member

    Joined:
    Apr 7, 2008
    Messages:
    349
    Likes Received:
    98
    Occupation:
    none
    Location:
    Uk
    Using a robots.txt file is very weak
    Many bots will bypass it
    .htaccess password protect a folder
    Then email out the password
    Do this right and at the same time make yourself a nice long list of email addresses that spend online
     
  11. scriptkiddy

    scriptkiddy Junior Member

    Joined:
    Jan 11, 2008
    Messages:
    104
    Likes Received:
    18
    If you put an entry in robots.txt saying don't access a page or directory, then well behaved robots will not spider it, BUT, you're telling the rest of the world here's something interesting please don't read it.

    Use some form of password protection.
     
  12. oldenstylehats

    oldenstylehats Elite Member Premium Member

    Joined:
    Apr 10, 2008
    Messages:
    1,893
    Likes Received:
    1,196
    Very good points.

    That said, there are a LOT Of reasons why you'd want to have public content on your site that you'd only be concerned about major search engine robots finding. I've *never* heard of an instance where a Google or Yahoo bot hasn't followed robots.txt. I'm not saying they don't fake UA and all of that good stuff, but a robots.txt file will keep content off the index and out of Archive.org.

    I don't know why I didn't think of it before, rap8557, but if you aren't too technically adept, you might want to look at a company like E-junkie to handle your files. Very easy to setup and not expensive at all. I think their first service level is only $5.00 a month. It's worth it and you can use either Google Checkout or PayPal as your payment processor. It's really easy to setup affiliate systems through them too.
     
  13. aftershock2020

    aftershock2020 Senior Member

    Joined:
    Oct 19, 2007
    Messages:
    981
    Likes Received:
    477
    Do it like this...

    1. Use a robot.txt , like Oldhat's suggested.

    2. Use a login/membership management script for your site to password protect it with sessions and redirects.

    3. On the page leading into the thank you page, use an individual " nofollow " code around download page code to protect it.


    Doing all of this will protect your page from being hit by anything short of a professional hacker handing you your server on a silver platter. If they are that desperate to steal your ebook, they have too much time on their hands.

    Hope that helps