1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Protecting websites

Discussion in 'Black Hat SEO' started by TheGamer, Jan 11, 2014.

  1. TheGamer

    TheGamer Regular Member

    Joined:
    Mar 25, 2013
    Messages:
    287
    Likes Received:
    46
    Location:
    UK
    Heya


    What would be the best way to protect my blog sites from 'prying eyes' :) So basically I am trying to block Moz, Majestic and Ahrefs crawlers from crawling these. I think
    the settings need to go into robots file and not htacess. But I am not 100% sure. If there is any plugin that makes the job easy, please let me know. If there are any other crawlers
    that I have missed other than the 3 (excluding G :D), please let me know.

    Cheers!
     
  2. innozemec

    innozemec Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2011
    Messages:
    5,288
    Likes Received:
    1,799
    Location:
    www.Indexification.com
    Home Page:
    robots.txt is not the best protection as one still can access anything on your website, no matter what is written in the robots.txt

    best way is to block their IP networks in your htaccess as that blocks any access from those IPs
     
  3. TheGamer

    TheGamer Regular Member

    Joined:
    Mar 25, 2013
    Messages:
    287
    Likes Received:
    46
    Location:
    UK
    Sorry for the late reply. Oh do you mean finding out which IPs are being used by MajesticSeo, Moz, and Ahrefs and blocking them in htAccess file. If so, how will I find the IPs of their crawlers? Also I dont think this would be a permanent solution since their IPs can change, cant they?

    I remember seeing someone sharing a few settings re robots file here, however I cant find the thread now.
     
  4. JoeMongan

    JoeMongan Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 24, 2013
    Messages:
    1,712
    Likes Received:
    3,553
    Location:
    International Waters
    Best way is to not block them from seeing your website but cloak what they can see.

    -RK
     
  5. TheGamer

    TheGamer Regular Member

    Joined:
    Mar 25, 2013
    Messages:
    287
    Likes Received:
    46
    Location:
    UK
    really? Can cloaking not get someone easily banned?
     
  6. 1+1=2

    1+1=2 Junior Member

    Joined:
    Jul 14, 2009
    Messages:
    198
    Likes Received:
    29
    • Thanks Thanks x 1