1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[Method] How to stop competitors spying on your links

Discussion in 'Black Hat SEO' started by notyourmatey, Oct 31, 2012.

  1. notyourmatey

    notyourmatey Power Member

    Joined:
    Jul 7, 2011
    Messages:
    573
    Likes Received:
    324
    Location:
    TOP3
    Here is my tip how to block competitors from spying on your links. If you build your private blog network or satellite sites linking to your money site it is very important to keep it as stealth as possible. What I use for my sites are two things:

    1. robots.txt
    2. .htaccess

    I will not explain how to create such files as it is easy even for me and I'm not a programmer:)

    Our main goal is to block robots such as Majestic seo, Ahrefs and Open site explorer. Very often those little bastards
    don't respect comands in robots.txt this is why you need to block them with .htaccess file.

    Here is how those files should look like:

    in robots.txt ( which is basically notepad file called robots) paste this:

    User-Agent: Googlebot
    Allow: /
    User-Agent: *
    Disallow: /

    This input will block access to your website to all bots apart of Google. In Theory. Many bots don't respect it so
    it is good idea to block them through .htaccess file.

    This is how my code looks like:

    version for Wordpress:

    Code:
    # BEGIN WordPress
    <IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /
    
    RewriteCond %{HTTP_USER_AGENT} .*Twice.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Yand.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Yahoo.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Voil.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*libw.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Java.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Sogou.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*psbot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Exabot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*boitho.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*ajSitemap.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Rankivabot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*DBLBot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*MJ1.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*ask.*
    RewriteCond %{HTTP_USER_AGENT} .*AhrefsBot.*
    RewriteRule ^(.*)$ http://anysite.com/ [L,R=301]
    
    Order Allow,Deny
    Allow from all
    Deny from 110.0.0.0/8
    Deny from 111.0.0.0/8
    Deny from 112.0.0.0/5
    Deny from 120.0.0.0/6
    Deny from 124.0.0.0/8
    Deny from 125.0.0.0/8
    Deny from 147.0.0.0/8
    Deny from 169.208.0.0
    Deny from 175.0.0.0/8
    Deny from 180.0.0.0/8
    Deny from 182.0.0.0/8
    Deny from 183.0.0.0/8
    Deny from 202.0.0.0/8
    Deny from 203.0.0.0/8
    Deny from 210.0.0.0/8
    Deny from 211.0.0.0/8
    Deny from 218.0.0.0/8
    Deny from 219.0.0.0/8
    Deny from 220.0.0.0/8
    Deny from 221.0.0.0/8
    Deny from 222.0.0.0/8
    
    RewriteRule ^index\.php$ - [L]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /index.php [L]
    </IfModule>
    
    # END WordPress
    
    and for any other site:

    Code:
    RewriteEngine On
    RewriteBase /
    
    RewriteCond %{HTTP_USER_AGENT} .*Twice.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Yand.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Yahoo.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Voil.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*libw.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Java.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Sogou.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*psbot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Exabot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*boitho.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*ajSitemap.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Rankivabot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*DBLBot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*MJ1.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*Rankivabot.* [OR]
    RewriteCond %{HTTP_USER_AGENT} .*ask.*
    RewriteCond %{HTTP_USER_AGENT} .*AhrefsBot.*
    RewriteRule ^(.*)$ http://anysite.com/ [L,R=301]
    
    Order Allow,Deny
    Allow from all
    Deny from 110.0.0.0/8
    Deny from 111.0.0.0/8
    Deny from 112.0.0.0/5
    Deny from 120.0.0.0/6
    Deny from 124.0.0.0/8
    Deny from 125.0.0.0/8
    Deny from 147.0.0.0/8
    Deny from 169.208.0.0
    Deny from 175.0.0.0/8
    Deny from 180.0.0.0/8
    Deny from 182.0.0.0/8
    Deny from 183.0.0.0/8
    Deny from 202.0.0.0/8
    Deny from 203.0.0.0/8
    Deny from 210.0.0.0/8
    Deny from 211.0.0.0/8
    Deny from 218.0.0.0/8
    Deny from 219.0.0.0/8
    Deny from 220.0.0.0/8
    Deny from 221.0.0.0/8
    Deny from 222.0.0.0/8
    
    Do not simply change your .htaccess files for the second option because you might have some important commands in your .htaccess already.
    However you would need to paste the code on top of this file. Version for Wordpress is copy- paste.

    Like you can see above I block also other bots like Yandex, mainly because they are using a lot of transfer from the website and sometimes it might cost you additional fees at your hosting provider. You can also add other bots to this list if they're pissing you off:)

    This rule:
    Code:
    RewriteRule ^(.*)$ [URL="http://bing.com/"]http://anysite.com/[/URL] [L,R=301]
    will send the bastards from your website to any site- simply put there whatever url. You can put here also website of your competitors :D

    If you look at the ip's I'm blocking also countries such as China and Japan because I don't care about this traffic
    and those bots consuming only transfer. You need to also remember it is the best to implement those files before linking campaign. If you put it later, there still will be record however new links won't appear in those backlink tools..
    So this is my little input in the community and I hope it will help you to hide from grassers.
     
    • Thanks Thanks x 51
    Last edited: Oct 31, 2012
  2. fanatic4k

    fanatic4k Junior Member

    Joined:
    Mar 28, 2012
    Messages:
    144
    Likes Received:
    48
    Location:
    BHW
    Thanks buddy for this tutorial :)

    One question I want to clarify. You're putting those things on the sites in your private network or on your money sites?
     
  3. Cshark

    Cshark Jr. VIP Jr. VIP

    Joined:
    Feb 25, 2011
    Messages:
    1,032
    Likes Received:
    134
    Gender:
    Male
    Occupation:
    Grinding
    Location:
    NYC
    Applicable for both private network and your money site.


    Great stuff OP. +Rep incoming.
     
  4. notyourmatey

    notyourmatey Power Member

    Joined:
    Jul 7, 2011
    Messages:
    573
    Likes Received:
    324
    Location:
    TOP3
    i use it to hide my links from satellite sites. just remember, robots.txt is not enough as some bots dont respect it
     
  5. fanatic4k

    fanatic4k Junior Member

    Joined:
    Mar 28, 2012
    Messages:
    144
    Likes Received:
    48
    Location:
    BHW
    So that means you put the things on your private network sites not your money sites, right?
     
  6. notyourmatey

    notyourmatey Power Member

    Joined:
    Jul 7, 2011
    Messages:
    573
    Likes Received:
    324
    Location:
    TOP3
    exactly
     
    • Thanks Thanks x 1
  7. BlackSEOBelt

    BlackSEOBelt Regular Member

    Joined:
    Jun 20, 2011
    Messages:
    213
    Likes Received:
    46
    Mmm, what about Bing? It now has this link explorer tool that's part of the webmaster tools. But then you might want to rank in Bing, huh?
    Anyhow, that Bing's tools is nowhere near YSE, if you ask me.
     
  8. notyourmatey

    notyourmatey Power Member

    Joined:
    Jul 7, 2011
    Messages:
    573
    Likes Received:
    324
    Location:
    TOP3
    if you block bing you wont be obviously indexed in it ( if you care about it)
     
  9. Lebron2727

    Lebron2727 Junior Member

    Joined:
    Nov 28, 2011
    Messages:
    179
    Likes Received:
    41
    awesome concept, rep added
     
  10. leena

    leena Jr. VIP Jr. VIP

    Joined:
    Apr 13, 2010
    Messages:
    1,979
    Likes Received:
    757
    Can anyone confirm if this code works fine and does not cause any malfunction for their websites? I am going to add it on my site using wordpress plugin so dont want any issues later

    thanks.
     
  11. Smart SEO

    Smart SEO Senior Member

    Joined:
    Apr 17, 2011
    Messages:
    857
    Likes Received:
    759
    Location:
    Google HQ
    Are you sure they only bot is GoogleBot, is there not a specific one for the UK or something? Great post btw
     
  12. bramantya

    bramantya Regular Member

    Joined:
    Oct 23, 2010
    Messages:
    244
    Likes Received:
    55
    great post, i have been wondering how to stop the spy lol
     
  13. No.RuleZ

    No.RuleZ Elite Member

    Joined:
    Jul 23, 2010
    Messages:
    1,732
    Likes Received:
    358
    great post, still working?
     
    • Thanks Thanks x 1
  14. imshinji

    imshinji Junior Member

    Joined:
    May 20, 2011
    Messages:
    113
    Likes Received:
    18
    Great tips.

    Any idea how to block Googlebot to access in "/tag/", and /go/ directories and redirect it to homepage ?
     
    • Thanks Thanks x 1
  15. bk071

    bk071 Jr. Executive VIP Jr. VIP Premium Member

    Joined:
    Nov 24, 2010
    Messages:
    3,105
    Likes Received:
    7,917
    Occupation:
    I don't have a job
    Location:
    .............
    If it is a wordpress blog, you could use Yoast SEO plugin and noindex/nofollow the tags / category pages (you can do anything you want with pretty much any page you want)
     
    • Thanks Thanks x 2
  16. jham1322

    jham1322 Newbie

    Joined:
    Oct 5, 2012
    Messages:
    25
    Likes Received:
    6
    great. ill try it
     
    • Thanks Thanks x 1
  17. kideze

    kideze Elite Member

    Joined:
    Jun 23, 2009
    Messages:
    1,719
    Likes Received:
    330
    Location:
    the GRAND valley
    great info
     
    • Thanks Thanks x 1
  18. jason2009

    jason2009 Senior Member

    Joined:
    Apr 23, 2010
    Messages:
    1,005
    Likes Received:
    206
    Occupation:
    Student
    Location:
    Earth
    Nice tutorial bro. I knew the robot info but for htaccess with lot of bot name..... its amazing. Thanks and +rep for you.
     
    • Thanks Thanks x 1
  19. keesky

    keesky Newbie

    Joined:
    Jun 4, 2011
    Messages:
    19
    Likes Received:
    6
    I want to try this method. But got a few questions:

    1. Do you use these files for the main money sites?
    2. Does this have any issues with Google UK?
    4. Does it block Bing from indexing my site, as I do get a good flow of traffic from them..
     
    Last edited: Feb 2, 2013
  20. mdmostfa

    mdmostfa Regular Member

    Joined:
    Apr 15, 2010
    Messages:
    200
    Likes Received:
    188
    Location:
    Where ever Money Is
    great method.. thank you