1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

need to block ahrefs

Discussion in 'Black Hat SEO' started by nonai, Feb 19, 2014.

  1. nonai

    nonai Power Member

    Joined:
    Oct 10, 2013
    Messages:
    524
    Likes Received:
    63
    using htaccess, I want to block as many backliink checking tools as possible. I have already done some research on this (including searching this forum) but I have not been able to find a solution. I have found several proposed solutions, but not one that's confirmed working by more than one person.

    I will post them here:

    solution 1:
    Code:
    BrowserMatchNoCase rogerbot bad_bot
    BrowserMatchNoCase exabot bad_bot
    BrowserMatchNoCase MJ12bot bad_bot
    BrowserMatchNoCase dotbot bad_bot
    BrowserMatchNoCase gigabot bad_bot
    BrowserMatchNoCase AhrefsBot bad_bot
    Order Deny,Allow
    Deny from env=bad_bot
    
    solution 2:
    Code:
    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} ^rogerbot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^exabot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^dotbot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^gigabot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot
    RewriteRule .* - [F]
    
    solution 3:
    Code:
    RewriteEngine On
    RewriteBase /
    RewriteCond %{HTTP_USER_AGENT} ^rogerbot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^exabot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^dotbot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^gigabot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot
    RewriteRule ^.* - [F,L]
    
    solution 4:
    Code:
    SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
    SetEnvIfNoCase User-Agent .*exabot.* bad_bot
    SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
    SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
    SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
    SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
    SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
    <Limit GET POST HEAD>
    Order Allow,Deny
    Allow from all
    Deny from env=bad_bot
    </Limit>
    
    In addition to these, I want to block
    BLEXbot
    Blekkobot
    SEOkicks-Robot

    I dont know anything about apache and these are codes that I just found online. I have no clue which one would work and which one would not. If you could just let me know which solution is best (and maybe your reasoning), that would be really appreciated.
     
  2. Apricot

    Apricot Administrator Staff Member Moderator

    Joined:
    Mar 26, 2013
    Messages:
    11,963
    Likes Received:
    6,444
    Gender:
    Female
    Occupation:
    BHW Moderator
    Location:
    London
    Home Page:
  3. innozemec

    innozemec Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2011
    Messages:
    5,288
    Likes Received:
    1,799
    Location:
    www.Indexification.com
    Home Page:
  4. JonnyBorneo

    JonnyBorneo Regular Member

    Joined:
    Feb 7, 2012
    Messages:
    388
    Likes Received:
    95
    Occupation:
    Affiliate
    robots.txt

    User-agent: *
    Disallow: /

    User-agent: Googlebot
    Allow: /


    This will block everything, except googlebot
     
  5. lover

    lover Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 25, 2007
    Messages:
    2,147
    Likes Received:
    811
    Location:
    BHW
    Plugins like spyderspanker can do this. Can block ahrefs + many more.
     
  6. innozemec

    innozemec Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2011
    Messages:
    5,288
    Likes Received:
    1,799
    Location:
    www.Indexification.com
    Home Page:
    that is not blocking anything, it is just telling the bot you don't want it to continue, however it may not obey it if it wants..
     
  7. BlueSteele

    BlueSteele Jr. VIP Jr. VIP Premium Member

    Joined:
    Apr 10, 2010
    Messages:
    282
    Likes Received:
    134
    Location:
    Vancouver, BC
    Ahrefs is annoying because it goes by different names. "ahrefsbot" and "sitebot" for example. I believe there are more too, but I don't know them. Here is the list that I block:

    AhrefsBot
    Alexibot
    Aqua_Products
    asterias
    b2w/0.1
    BackDoorBot/1.0
    BecomeBot
    BlekkoBot
    BlowFish/1.0
    Bookmark search tool
    BotALot
    BuiltBotTough
    Bullseye/1.0
    BunnySlippers
    CheeseBot
    CherryPicker
    Copernic
    CopyRightCheck
    cosmos
    Crescent
    Crescent Internet ToolPak HTTP OLE Control v.1.0
    DittoSpyder
    dotbot
    dumbot
    EmailCollector
    EmailSiphon
    EmailWolf
    Enterprise_Search
    EroCrawler
    es
    exabot
    ExtractorPro
    FairAd Client
    Flaming AttackBot
    Foobot
    Gaisbot
    GetRight/4.2
    gigabot
    grub
    grub-client
    Harvest/1.5
    Hatena Antenna
    hloader
    Pubcon | Pubcon Search, Social Media, Affiliate Marketing Conferences bot
    WebmasterWorld News and Discussion for the Web Professional bot
    httplib
    humanlinks
    ia_archiver
    InfoNaviRobot
    Iron33/1.0.2
    JennyBot
    Jetbot
    Kenjin Spider
    Keyword Density/0.9
    larbin
    LexiBot
    libWeb/clsHTTP
    LinkextractorPro
    LinkScan/8.1a Unix
    LinkWalker
    LNSpiderguy
    looksmart
    lwp-trivial
    lwp-trivial/1.34
    Mata Hari
    Microsoft URL Control
    MIIxpc
    Mister PiX
    MJ12bot
    moget
    MSIECrawler
    naver
    NetAnts
    NetMechanic
    NICErsPRO
    Nutch
    Offline Explorer
    Openbot
    Openfind
    Openfind data gathere
    Oracle Ultra Search
    PerMan
    ProPowerBot/2.14
    ProWebWalker
    psbot
    Python-urllib
    QueryN Metasearch
    Radiation Retriever 1.1
    RepoMonkey
    RepoMonkey Bait & Tackle/v1.01
    RMA
    rogerbot
    scooter
    searchpreview
    Sitebot
    SiteSnagger
    sootle
    SpankBot
    spanner
    Stanford
    Stanford Comp Sci
    Stanford CompClub
    Stanford CompSciClub
    Stanford Spiderboys
    suzuran
    Szukacz/1.4
    Teleport
    TeleportPro
    Telesoft
    Teoma
    The Intraformant
    TheNomad
    toCrawl/UrlDispatcher
    True_Robot
    turingos
    URL Control
    URL_Spider_Pro
    URLy Warning
    VCI
    VCI WebViewer VCI WebViewer Win32
    Web Image Collector
    WebAuto
    WebBandit
    WebCopier
    WebEnhancer
    WebmasterWorld Extractor
    WebmasterWorldForumBot
    WebSauger
    Website Quester
    Webster Pro
    WebStripper
    WebVac
    WebZip
    Wget
    WWW-Collector-E
    Xenu
    Xenu's
    Zeus

    Here is a pretty sweet plugin that will tell you what bots are visiting your site: http://wordpress.org/plugins/seo-crawlytics/
     
  8. DamageX

    DamageX Elite Member

    Joined:
    Sep 17, 2008
    Messages:
    2,692
    Likes Received:
    1,687
    Occupation:
    Unemployable
    Location:
    Former nomad
  9. elner

    elner BANNED BANNED

    Joined:
    May 23, 2012
    Messages:
    453
    Likes Received:
    53
    a noob qustion here. why did you guy want to block these ? You don't want competitor knows your site details ?
     
  10. just4ivaylo

    just4ivaylo Regular Member

    Joined:
    Jul 15, 2010
    Messages:
    248
    Likes Received:
    43
    Yes, you don't want to have a competitor checking out your backlink profile and so on. You can "steal" links that way and get a link on the same website that they're on. It works pretty well sometimes.
     
  11. BlueSteele

    BlueSteele Jr. VIP Jr. VIP Premium Member

    Joined:
    Apr 10, 2010
    Messages:
    282
    Likes Received:
    134
    Location:
    Vancouver, BC
    Could you block ahrefs from seeing only a part of your link profile...for example, just my social signals, press releases or haha guest posts. Fool them into thinking that is what is pushing me, instead of all the HPR backlinks. Just to fuck with them.
     
  12. DamageX

    DamageX Elite Member

    Joined:
    Sep 17, 2008
    Messages:
    2,692
    Likes Received:
    1,687
    Occupation:
    Unemployable
    Location:
    Former nomad
    You can only block it on sites you own.
     
  13. BlueSteele

    BlueSteele Jr. VIP Jr. VIP Premium Member

    Joined:
    Apr 10, 2010
    Messages:
    282
    Likes Received:
    134
    Location:
    Vancouver, BC
    Would very much be my own site. Any thoughts?

     
  14. DamageX

    DamageX Elite Member

    Joined:
    Sep 17, 2008
    Messages:
    2,692
    Likes Received:
    1,687
    Occupation:
    Unemployable
    Location:
    Former nomad
    • Thanks Thanks x 1
  15. sssworld

    sssworld Junior Member

    Joined:
    Jan 16, 2014
    Messages:
    144
    Likes Received:
    32
    Why you want to block the site through ahref ?
     
  16. ChokeStone

    ChokeStone Registered Member

    Joined:
    Mar 31, 2013
    Messages:
    55
    Likes Received:
    14
    Blocking bots doesn't block people from seeing your backlinks. It only blocks these bots from crawling sites that YOU own. That means the only links that you can potentially hide from your competitors are links from your OWN private network. everything else is being seen.