1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Any updated/working code to block ahref/majestic bots?

Discussion in 'Black Hat SEO' started by beingink, Jul 31, 2016.

  1. beingink

    beingink Senior Member

    Joined:
    Sep 19, 2013
    Messages:
    1,016
    Likes Received:
    219
    I tried one from here which doesn't seem to work since ahref picked up the PBN anyways. Can anyone share a solution pls? Thanks.
     
  2. DigitalCon

    DigitalCon Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 27, 2014
    Messages:
    518
    Likes Received:
    88
    Gender:
    Male
    Occupation:
    Internet Research
    Location:
    Home
    Home Page:
    This works good for all my PBN domains, hope it works for you too.



    Code:
    SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
    SetEnvIfNoCase User-Agent .*exabot.* bad_bot
    SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
    SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
    SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
    SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
    SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
    SetEnvIfNoCase User-Agent .*semrushbot.* bad_bot
    SetEnvIfNoCase User-Agent .*ia_archiver.* bad_bot
    SetEnvIfNoCase User-Agent .*searchmetricsbot.* bad_bot
    SetEnvIfNoCase User-Agent .*seokicks-robot.* bad_bot
    SetEnvIfNoCase User-Agent .*sistrix.* bad_bot
    SetEnvIfNoCase User-Agent .*lipperhey spider.* bad_bot
    SetEnvIfNoCase User-Agent .*ncbot.* bad_bot
    SetEnvIfNoCase User-Agent .*backlinkcrawler.* bad_bot
    SetEnvIfNoCase User-Agent .*archive.org_bot.* bad_bot
    SetEnvIfNoCase User-Agent .*meanpathbot.* bad_bot
    SetEnvIfNoCase User-Agent .*pagesinventory.* bad_bot
    SetEnvIfNoCase User-Agent .*aboundexbot.* bad_bot
    SetEnvIfNoCase User-Agent .*spbot.* bad_bot
    SetEnvIfNoCase User-Agent .*linkdexbot.* bad_bot
    SetEnvIfNoCase User-Agent .*nutch.* bad_bot
    SetEnvIfNoCase User-Agent .*blexbot.* bad_bot
    SetEnvIfNoCase User-Agent .*ezooms.* bad_bot
    SetEnvIfNoCase User-Agent .*scoutjet.* bad_bot
    SetEnvIfNoCase User-Agent .*majestic-12.* bad_bot
    SetEnvIfNoCase User-Agent .*majestic-seo.* bad_bot
    SetEnvIfNoCase User-Agent .*dsearch.* bad_bot
    SetEnvIfNoCase User-Agent .*blekkobo.* bad_bot
    SetEnvIfNoCase User-Agent .*screaming frog seo spider/*.* bad_bot
    SetEnvIfNoCase User-Agent .*PHPCrawl.* bad_bot
    SetEnvIfNoCase User-Agent .*gocrawl.* bad_bot
    SetEnvIfNoCase User-Agent .*DigExt.* bad_bot
    SetEnvIfNoCase User-Agent .*DomainSONOCrawler.* bad_bot
    SetEnvIfNoCase User-Agent .*TweetmemeBot.* bad_bot
    SetEnvIfNoCase User-Agent .*OpenHoseBot/2.1.* bad_bot
    SetEnvIfNoCase User-Agent .*Kraken/0.1.* bad_bot
    SetEnvIfNoCase User-Agent .*-Java-.* bad_bot
    SetEnvIfNoCase User-Agent .*ubermetrics.* bad_bot
    SetEnvIfNoCase User-Agent .*best-seo.* bad_bot
    SetEnvIfNoCase User-Agent .*Synapse.* bad_bot
    SetEnvIfNoCase User-Agent .*Harvest.* bad_bot
    SetEnvIfNoCase User-Agent .*Harvester.* bad_bot
    SetEnvIfNoCase User-Agent .*harvester.* bad_bot
    SetEnvIfNoCase User-Agent .*harvest.* bad_bot
    
    <Limit GET POST HEAD>
    
    Order Allow,Deny
    
    Allow from all
    
    Deny from env=bad_bot
    
    </Limit>
     
    • Thanks Thanks x 2
  3. beingink

    beingink Senior Member

    Joined:
    Sep 19, 2013
    Messages:
    1,016
    Likes Received:
    219
    Thanks will try it out.
     
  4. pauverinox

    pauverinox Newbie

    Joined:
    Apr 21, 2016
    Messages:
    18
    Likes Received:
    1
    This is for htaccess right?

    Where do I insert that code on my wordpress htaccess.

    Code:
    # BEGIN WordPress
    <IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /
    RewriteRule ^index\.php$ - [L]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /index.php [L]
    </IfModule>
    
    # END WordPress
    
     
  5. DigitalCon

    DigitalCon Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 27, 2014
    Messages:
    518
    Likes Received:
    88
    Gender:
    Male
    Occupation:
    Internet Research
    Location:
    Home
    Home Page:
    Yes, this for .htaccess.
    Paste if after the default wordpress .htaccess code.

    After this line
    # END WordPress
     
    • Thanks Thanks x 1