1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Help with Blocking Ahrefs, Majestic, etc? (PBN)

Discussion in 'Black Hat SEO' started by Joethemonkey101, Jun 30, 2015.

  1. Joethemonkey101

    Joethemonkey101 Junior Member

    Joined:
    Jul 23, 2013
    Messages:
    149
    Likes Received:
    28
    Hi, I'm working on building a PBN and I'm trying to block ahrefs and all the other backlink websites. I found some threads dealing with this but they are a couple of years old so I wanted to make sure everything is still the same.

    Firstly, I'm editing the htaccess code of my PBN sites, but how does this block ahrefs? Does it mean that people can't look at the backlinks pointed at my PBN site or the PBN backlink pointed at my money site?

    Secondly, is my htaccess file OK?
    Code:
    # BEGIN WordPress
    <IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /
    RewriteRule ^index\.php$ - [L]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /index.php [L]
    </IfModule>
    
    
    SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
    SetEnvIfNoCase User-Agent .*exabot.* bad_bot
    SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
    SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
    SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
    SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
    SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
    <Limit GET POST HEAD>
    Order Allow,Deny
    Allow from all
    Deny from env=bad_bot
    </Limit>
    
    
    # END WordPress
    
    
    
    And lastly, how can I edit my robots.txt file?

    Thanks in advanced
     
  2. deancow

    deancow Jr. VIP Jr. VIP

    Joined:
    Jul 8, 2009
    Messages:
    682
    Likes Received:
    244
    htaccess looks fine, I wouldn't bother with a robots.txt file, if you block the same bots on every robots.txt it's a footprint and they might just ignore them anyway.
     
  3. Joethemonkey101

    Joethemonkey101 Junior Member

    Joined:
    Jul 23, 2013
    Messages:
    149
    Likes Received:
    28
    Is there a way to check if it's working?
     
  4. bahus

    bahus Regular Member

    Joined:
    Jun 4, 2014
    Messages:
    342
    Likes Received:
    93
    Gender:
    Male
    If you're on Wordpress you can use KB Robots.txt plugin to edit your robots.txt file from within Wordpress.
     
  5. Joethemonkey101

    Joethemonkey101 Junior Member

    Joined:
    Jul 23, 2013
    Messages:
    149
    Likes Received:
    28
    thanks. Do you know how I can check if the htaccess code worked?
     
  6. jmcgoose

    jmcgoose Junior Member

    Joined:
    Jun 20, 2013
    Messages:
    143
    Likes Received:
    36
    build some backlinks and see if they show up
     
  7. JustUs

    JustUs Power Member

    Joined:
    May 6, 2012
    Messages:
    626
    Likes Received:
    585
    Change your user agent to a string incorporating the string of a banned bot and try to access the site.

    Example:
    banned: *mj12bot*
    change UA to micemj12bot2 and access the site.
     
  8. thelartman

    thelartman Newbie

    Joined:
    Jul 24, 2015
    Messages:
    3
    Likes Received:
    1
    You can check whether you site is blocking bots on wanna browser


    By using the majestic user agent (look it up cos cant post the URL in here) it replicates the majestic bot.

    I have found in recent weeks that blocking bots with .htaccess has stopped working and I have you use a plugin, which id rather not.

    Does anyone have any definite working code?