1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[HOW?] Blocking Linkcheckers from accessing my site

Discussion in 'Black Hat SEO' started by BrandingBaw, May 8, 2013.

  1. BrandingBaw

    BrandingBaw Power Member

    Joined:
    Jan 30, 2012
    Messages:
    667
    Likes Received:
    279
    Occupation:
    NA
    Location:
    BHW
    Hi there,

    I've noticed that a competitor of mine is copying my links 1-on-1. Therefore I'd like to block my site for all linkcheckers. I know all might not be possible, but a majority should be.

    Been searching all night on it on BHW, but can't find much more than some robots.txt files which isn't what I'm looking for.

    Can somebody tell me how to do this by .htaccess?

    Thanks a lot,

    BB
     
  2. rasmusk

    rasmusk Senior Member

    Joined:
    Oct 4, 2012
    Messages:
    905
    Likes Received:
    362
    Occupation:
    Bypassing IRA
    Location:
    Cayman Islands
    Subscribed. I would also like to do this for atleast Majestic, Ahrefs, Seomoz, Backlinkwatch and Siteexplorer
     
  3. BrandingBaw

    BrandingBaw Power Member

    Joined:
    Jan 30, 2012
    Messages:
    667
    Likes Received:
    279
    Occupation:
    NA
    Location:
    BHW
    Exactly the ones I'd liked to block from having access to my site :)
     
  4. coxi999

    coxi999 Junior Member

    Joined:
    Aug 14, 2007
    Messages:
    167
    Likes Received:
    49
    Location:
    UK
    Interesting, would like to know this also. I believe it is possible sure I have read somewhere that people do it..
     
  5. the_demon

    the_demon Jr. Executive VIP

    Joined:
    Nov 23, 2008
    Messages:
    3,177
    Likes Received:
    1,563
    Occupation:
    Search Engine Marketing
    Location:
    The Internet
    It's not something anything on your domain can do per-say. It's about stopping the link crawlers from accessing your link sources. My company has developed a solution for this called AI Link Guardian, but it's not something that's available to the public.
     
  6. rasmusk

    rasmusk Senior Member

    Joined:
    Oct 4, 2012
    Messages:
    905
    Likes Received:
    362
    Occupation:
    Bypassing IRA
    Location:
    Cayman Islands
    Hmm I'm sure I saw something on this forum.... Is'nt just to identify the crawlers from those sites and block them in .htacces or its not that simple?
     
  7. BrandingBaw

    BrandingBaw Power Member

    Joined:
    Jan 30, 2012
    Messages:
    667
    Likes Received:
    279
    Occupation:
    NA
    Location:
    BHW
    It's around here but I can't seem to find it. If anyone has a list of bots plus the .htaccess method, please share it!
     
  8. gorang

    gorang Elite Member

    Joined:
    Dec 6, 2008
    Messages:
    1,891
    Likes Received:
    1,650
    Occupation:
    SEO Consultant - Marketing Strategy
    Location:
    UK
    You can't, the bots from places like majestic don't need to access your site to see all the links pointing to it.
     
  9. cloakndagger2

    cloakndagger2 Regular Member

    Joined:
    Oct 30, 2012
    Messages:
    294
    Likes Received:
    88
    Here is the post,credit goes to the OP of the post not me.For majestic bot you need to add it to the robots.txt as well unfortunately it seems to be able to ignore the ht.acess file.Hopefully the op will chime in and take the deserved credit for their post.Also read the entire post and understand it.Monitor your stats they probably rename the bots so look out for them and change the file as needed.



    Here is my tip how to block competitors from spying on your links. If you build your private blog network or satellite sites linking to your money site it is very important to keep it as stealth as possible. What I use for my sites are two things:

    1. robots.txt
    2. .htaccess

    I will not explain how to create such files as it is easy even for me and I'm not a programmer

    Our main goal is to block robots such as Majestic seo, Ahrefs and Open site explorer. Very often those little bastards
    don't respect comands in robots.txt this is why you need to block them with .htaccess file.

    Here is how those files should look like:

    in robots.txt ( which is basically notepad file called robots) paste this:

    User-Agent: Googlebot
    Allow: /
    User-Agent: *
    Disallow: /

    This input will block access to your website to all bots apart of Google. In Theory. Many bots don't respect it so
    it is good idea to block them through .htaccess file.

    This is how my code looks like:

    version for Wordpress:

    Code:

    # BEGIN WordPress
    <IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /

    RewriteCond %{HTTP_USER_AGENT} .*Twice.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Yand.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Yahoo.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Voil.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*libw.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Java.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Sogou.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*psbot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Exabot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*boitho.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*ajSitemap.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Rankivabot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*DBLBot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*MJ1.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*ask.*
    RewriteCond %{HTTP_USER_AGENT} .*rogerbot [OR]
    RewriteCond %{HTTP_USER_AGENT} .*exabot [OR]
    RewriteCond %{HTTP_USER_AGENT} .*MJ12bot [OR]
    RewriteCond %{HTTP_USER_AGENT} .*dotbot [OR]
    RewriteCond %{HTTP_USER_AGENT} .*gigabot [OR]
    RewriteCond %{HTTP_USER_AGENT} .*AhrefsBot.*
    RewriteRule ^(.*)$ http://anysite.com/ [L,R=301]

    Order Allow,Deny
    Allow from all
    Deny from 110.0.0.0/8
    Deny from 111.0.0.0/8
    Deny from 112.0.0.0/5
    Deny from 120.0.0.0/6
    Deny from 124.0.0.0/8
    Deny from 125.0.0.0/8
    Deny from 147.0.0.0/8
    Deny from 169.208.0.0
    Deny from 175.0.0.0/8
    Deny from 180.0.0.0/8
    Deny from 182.0.0.0/8
    Deny from 183.0.0.0/8
    Deny from 202.0.0.0/8
    Deny from 203.0.0.0/8
    Deny from 210.0.0.0/8
    Deny from 211.0.0.0/8
    Deny from 218.0.0.0/8
    Deny from 219.0.0.0/8
    Deny from 220.0.0.0/8
    Deny from 221.0.0.0/8
    Deny from 222.0.0.0/8

    RewriteRule ^index\.php$ - [L]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /index.php [L]
    </IfModule>

    # END WordPress

    and for any other site:

    Code:

    RewriteEngine On
    RewriteBase /

    RewriteCond %{HTTP_USER_AGENT} .*Twice.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Yand.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Yahoo.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Voil.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*libw.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Java.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Sogou.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*psbot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Exabot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*boitho.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*ajSitemap.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Rankivabot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*DBLBot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*MJ1.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Rankivabot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*ask.*
    RewriteCond %{HTTP_USER_AGENT} .*AhrefsBot.*
    RewriteCond %{HTTP_USER_AGENT} .*rogerbot [OR]
    RewriteCond %{HTTP_USER_AGENT} .*exabot [OR]
    RewriteCond %{HTTP_USER_AGENT} .*MJ12bot [OR]
    RewriteCond %{HTTP_USER_AGENT} .*dotbot [OR]
    RewriteCond %{HTTP_USER_AGENT} .*gigabot [OR]
    RewriteRule ^(.*)$ http://anysite.com/ [L,R=301]

    Order Allow,Deny
    Allow from all
    Deny from 110.0.0.0/8
    Deny from 111.0.0.0/8
    Deny from 112.0.0.0/5
    Deny from 120.0.0.0/6
    Deny from 124.0.0.0/8
    Deny from 125.0.0.0/8
    Deny from 147.0.0.0/8
    Deny from 169.208.0.0
    Deny from 175.0.0.0/8
    Deny from 180.0.0.0/8
    Deny from 182.0.0.0/8
    Deny from 183.0.0.0/8
    Deny from 202.0.0.0/8
    Deny from 203.0.0.0/8
    Deny from 210.0.0.0/8
    Deny from 211.0.0.0/8
    Deny from 218.0.0.0/8
    Deny from 219.0.0.0/8
    Deny from 220.0.0.0/8
    Deny from 221.0.0.0/8
    Deny from 222.0.0.0/8

    Do not simply change your .htaccess files for the second option because you might have some important commands in your .htaccess already.
    However you would need to paste the code on top of this file. Version for Wordpress is copy- paste.

    Like you can see above I block also other bots like Yandex, mainly because they are using a lot of transfer from the website and sometimes it might cost you additional fees at your hosting provider. You can also add other bots to this list if they're pissing you off

    This rule:
    Code:

    RewriteRule ^(.*)$ http://anysite.com/ [L,R=301]

    will send the bastards from your website to any site- simply put there whatever url. You can put here also website of your competitors

    If you look at the ip's I'm blocking also countries such as China and Japan because I don't care about this traffic
    and those bots consuming only transfer. You need to also remember it is the best to implement those files before linking campaign. If you put it later, there still will be record however new links won't appear in those backlink tools..
    So this is my little input in the community and I hope it will help you to hide from grassers.
     
    • Thanks Thanks x 6
  10. BrandingBaw

    BrandingBaw Power Member

    Joined:
    Jan 30, 2012
    Messages:
    667
    Likes Received:
    279
    Occupation:
    NA
    Location:
    BHW
    Couldn't have asked for a better answer! Thanks and rep given.
     
  11. naweed

    naweed Junior Member

    Joined:
    Dec 25, 2011
    Messages:
    186
    Likes Received:
    37
    Ahh if only it was that simple!

    This will not do anything to prevent your competitors from viewing your backlinks. Because this will only prevent backlink checkers from acessing YOUR website. It will prevent links on your site to be index.

    Your backlinks are on other websites and to hide your backlinks, the above code need to be added on every website your backlink is found.
     
  12. cloakndagger2

    cloakndagger2 Regular Member

    Joined:
    Oct 30, 2012
    Messages:
    294
    Likes Received:
    88
    It's primarily used for network sites and your own,that's the purpose of it.It only prevents backlinks being indexed if you block the bots and you're clearly not blocking google.
    The code doesn't need to be added to every site you have links on,you would put it on your network sites,in the end all backlinks can still be found but it's hard work and 99% will not bother looking beyond the likes of majestic and ahefs.Even if you did you'd have a hard job finding a network if a site has 3k+ of backlinks on different types of sites.Thats the aim of this and to stop potential hacking traffic by blocking countries,no guarantees but it helps.
     
  13. flmommens

    flmommens Newbie

    Joined:
    May 16, 2013
    Messages:
    3
    Likes Received:
    0
    Occupation:
    Web entrepreneur
    Location:
    Amsterdam
    Home Page:
    BrandingBaw, what do you want exactly to achieve? If you want you competitor not to have access to your backlinks then Naweed is perfectly right, you'd have to stop link checkers from crawling every websites that has a link to your site.
     
  14. gorang

    gorang Elite Member

    Joined:
    Dec 6, 2008
    Messages:
    1,891
    Likes Received:
    1,650
    Occupation:
    SEO Consultant - Marketing Strategy
    Location:
    UK
    Exactly, its not possible.
     
  15. senty4love

    senty4love Junior Member

    Joined:
    May 17, 2010
    Messages:
    146
    Likes Received:
    31
    You can't completely block competitor accessing all your links..with above code u can block link crawlers accessing your site..but that's not always guaranteed..as some crawler has different spiders for crawling
     
  16. EmporioJamie

    EmporioJamie Newbie

    Joined:
    May 16, 2013
    Messages:
    5
    Likes Received:
    0
    Occupation:
    Businessman and a marketing specialist
    Location:
    USA
    Exciting, would like to know about it . I believe that it is possibly sure. I have studied somewhere that individuals do it.
     
  17. UrsuAke

    UrsuAke Power Member

    Joined:
    Sep 28, 2011
    Messages:
    700
    Likes Received:
    978
    Occupation:
    SEO Specialist.
    Location:
    Romania, land of choice
    Twist on this method to really screw with your competitors:

    1) Build a couple of irrelevant backlinks.
    2) Get them indexed.
    3) Check your site with lots of tools to get them to index you.
    4) Wait it out for a while (couple weeks)
    5) Add the .htacces restrictions
    6) Do proper link building.

    This method creates threads on BHW such as "HELP PLEASE!!!! How is this guy ranking with only 5 dudu backlinks?!?!?!?!"

    This method won't help you against the ol' fashioned way of backlink analysis, which is "doing it by hand" and searching for footprints yadda yadda. But the good part is, very few people are still doing this, or more importantly, few people really know how to identify backlinks by hand ;)
     
    • Thanks Thanks x 1
  18. gorang

    gorang Elite Member

    Joined:
    Dec 6, 2008
    Messages:
    1,891
    Likes Received:
    1,650
    Occupation:
    SEO Consultant - Marketing Strategy
    Location:
    UK
    This makes no sense. Tools like majesticseo.com will still show the links even with the htaccess restrictions. You can't stop people from using such tools to find your backlinks unless you can block majesticseo bots from every single website with a link to your website.

    It doesn't matter if majestic can't crawl your site, they don't need to.
     
  19. Sparrow

    Sparrow Newbie

    Joined:
    Jan 16, 2009
    Messages:
    45
    Likes Received:
    4
    I used this .htaccess method on each of the network of sites but MajesticSeo is reporting the same network backlinks. Though there where some differences in code I found as shown below:

    Sample of the code provided above:

    RewriteCond %{HTTP_USER_AGENT} .*Twice.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Yand.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Yahoo.* [OR]

    And Sample of the code I was using:

    RewriteCond %{HTTP_USER_AGENT} ^.*Twice.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Yand.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Yahoo.*$ [NC,OR]

    Could this be an issue? Am no coder and therefore don't know if there is a problem with the code I'm using above.

    thx.


     
  20. hpv222

    hpv222 Power Member

    Joined:
    Feb 8, 2010
    Messages:
    736
    Likes Received:
    274

    Agreed 100% - stopping the bots from spidering your own site, how does that prevent them from discovering your backlinks? It just doesn't