1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Blocking Spiders For PBN

Discussion in 'Black Hat SEO' started by MikeyMikey13, Aug 1, 2016.

  1. MikeyMikey13

    MikeyMikey13 Supreme Member

    Joined:
    May 25, 2014
    Messages:
    1,486
    Likes Received:
    423
    I've tried using both htaccess and the plugin link privacy but neither of them has worked.

    Ahrefs and Majestic SEO are still finding my network, any advice?
     
  2. DigitalCon

    DigitalCon Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 27, 2014
    Messages:
    518
    Likes Received:
    88
    Gender:
    Male
    Occupation:
    Internet Research
    Location:
    Home
    Home Page:
    Is your .htaccess code similar to the one below?
    I use it and it works fine for me.

    Code:
    SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
    SetEnvIfNoCase User-Agent .*exabot.* bad_bot
    SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
    SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
    SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
    SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
    SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
    SetEnvIfNoCase User-Agent .*semrushbot.* bad_bot
    SetEnvIfNoCase User-Agent .*ia_archiver.* bad_bot
    SetEnvIfNoCase User-Agent .*searchmetricsbot.* bad_bot
    SetEnvIfNoCase User-Agent .*seokicks-robot.* bad_bot
    SetEnvIfNoCase User-Agent .*sistrix.* bad_bot
    SetEnvIfNoCase User-Agent .*lipperhey spider.* bad_bot
    SetEnvIfNoCase User-Agent .*ncbot.* bad_bot
    SetEnvIfNoCase User-Agent .*backlinkcrawler.* bad_bot
    SetEnvIfNoCase User-Agent .*archive.org_bot.* bad_bot
    SetEnvIfNoCase User-Agent .*meanpathbot.* bad_bot
    SetEnvIfNoCase User-Agent .*pagesinventory.* bad_bot
    SetEnvIfNoCase User-Agent .*aboundexbot.* bad_bot
    SetEnvIfNoCase User-Agent .*spbot.* bad_bot
    SetEnvIfNoCase User-Agent .*linkdexbot.* bad_bot
    SetEnvIfNoCase User-Agent .*nutch.* bad_bot
    SetEnvIfNoCase User-Agent .*blexbot.* bad_bot
    SetEnvIfNoCase User-Agent .*ezooms.* bad_bot
    SetEnvIfNoCase User-Agent .*scoutjet.* bad_bot
    SetEnvIfNoCase User-Agent .*majestic-12.* bad_bot
    SetEnvIfNoCase User-Agent .*majestic-seo.* bad_bot
    SetEnvIfNoCase User-Agent .*dsearch.* bad_bot
    SetEnvIfNoCase User-Agent .*blekkobo.* bad_bot
    SetEnvIfNoCase User-Agent .*screaming frog seo spider/*.* bad_bot
    SetEnvIfNoCase User-Agent .*PHPCrawl.* bad_bot
    SetEnvIfNoCase User-Agent .*gocrawl.* bad_bot
    SetEnvIfNoCase User-Agent .*DigExt.* bad_bot
    SetEnvIfNoCase User-Agent .*DomainSONOCrawler.* bad_bot
    SetEnvIfNoCase User-Agent .*TweetmemeBot.* bad_bot
    SetEnvIfNoCase User-Agent .*OpenHoseBot/2.1.* bad_bot
    SetEnvIfNoCase User-Agent .*Kraken/0.1.* bad_bot
    SetEnvIfNoCase User-Agent .*-Java-.* bad_bot
    SetEnvIfNoCase User-Agent .*ubermetrics.* bad_bot
    SetEnvIfNoCase User-Agent .*best-seo.* bad_bot
    SetEnvIfNoCase User-Agent .*Synapse.* bad_bot
    SetEnvIfNoCase User-Agent .*Harvest.* bad_bot
    SetEnvIfNoCase User-Agent .*Harvester.* bad_bot
    SetEnvIfNoCase User-Agent .*harvester.* bad_bot
    SetEnvIfNoCase User-Agent .*harvest.* bad_bot
    
    <Limit GET POST HEAD>
    
    Order Allow,Deny
    
    Allow from all
    
    Deny from env=bad_bot
    
    </Limit>
    
     
  3. onlineonly

    onlineonly Power Member

    Joined:
    Jul 27, 2014
    Messages:
    614
    Likes Received:
    287
    Location:
    online
    If you using wordpress you can use a plugin called "link privacy". You probably using wrong code thats not updated.

    Edit: Just read you already tried link privacy. Then I don't know. Works fine for me.
     
  4. mbreezy

    mbreezy Jr. VIP Jr. VIP

    Joined:
    Jun 27, 2012
    Messages:
    495
    Likes Received:
    163
  5. ChrisX

    ChrisX Jr. VIP Jr. VIP

    Joined:
    Oct 8, 2011
    Messages:
    284
    Likes Received:
    141
    Gender:
    Male
    Home Page:
    If you're blocking bots at least do it at the .htaccess level instead of robots.txt so that google cannot see what you're blocking.
    Remember, normal websites don't block bots so it could be a red flag in google's eyes.
     
  6. MikeyMikey13

    MikeyMikey13 Supreme Member

    Joined:
    May 25, 2014
    Messages:
    1,486
    Likes Received:
    423
    I am doing it at htacess level, I don't know where you got the impression I wasn't.
    some sites it works fine on, then others have started to pop up, seems odd.