Blocking major backlinks crwawlers on PBN or on Monsy site?

Discussion in 'Black Hat SEO' started by frankstan, Sep 9, 2014.

  1. frankstan

    frankstan Regular Member

    Joined:
    Jan 14, 2014
    Messages:
    352
    Likes Received:
    44
    I'm building some PBN recently, I'm going to block some crwawlers like MOZ, MAjestic, I heard there is a plugin can help me to do it, the problem is do I need to install this plugin on every PBN or I just need to install it on my money site?
    I don't want to install in on every site since I'm afraid it is a footprint.
    So does it work if I only install the plugin on my money site?
     
  2. SEO Power

    SEO Power Elite Member

    Joined:
    Jul 14, 2014
    Messages:
    2,642
    Likes Received:
    683
    Occupation:
    Self employed
    Location:
    Houston, TX
  3. FitnessFred

    FitnessFred Newbie

    Joined:
    Jul 31, 2014
    Messages:
    13
    Likes Received:
    3
    Yes unfortunately you will have to install on each of your pbn site, if you want to avoid a plugin footprint, just use robots or htaccess to block crawlers.
     
  4. tony_d

    tony_d Elite Member

    Joined:
    Jun 22, 2013
    Messages:
    2,584
    Likes Received:
    3,183
    Location:
    1600 Amphitheatre Parkway, Mountain View CA
    I use a robots.txt file. To address the 'footprint' issue, I have 100+ variations of robots.txt that I randomly upload to the hosting account, in each case blocking the standard variety (ahrefs, majestic, et al) and plenty of other random bots that I don't really care about (just to mix it up and make the files different).

    There's still a footprint to the extent that the sites have a robots.txt file, but each file is randomly different. I'm not sure that the mere presence of a robots.txt file is anywhere near a significant footprint.
     
  5. DannyZhang

    DannyZhang Regular Member

    Joined:
    Apr 2, 2014
    Messages:
    233
    Likes Received:
    69
    This seems like a better solution that having plugins.

    Where did you get the list of bots to block?
     
  6. tony_d

    tony_d Elite Member

    Joined:
    Jun 22, 2013
    Messages:
    2,584
    Likes Received:
    3,183
    Location:
    1600 Amphitheatre Parkway, Mountain View CA
    An owner-submitted list of bots is available at robotstxt.org (direct link here)

    There are 300+ bots on the list, so many tens of thousands of possible combinations for a robots.txt file, even when you consider the 'constants' (ahrefs, majestic, et al).
     
  7. frankstan

    frankstan Regular Member

    Joined:
    Jan 14, 2014
    Messages:
    352
    Likes Received:
    44
    Thanks everyone, I will use robot.txt to block them
     
  8. hatblackguy

    hatblackguy Regular Member

    Joined:
    Jun 8, 2011
    Messages:
    252
    Likes Received:
    61
    Occupation:
    kicking Big G's fat Mug
    Location:
    1% Percent of wealthiest people
    Interesting how do you successfully block ahrefs?
    I can never get them to piss off when doing through Robots.txt