1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Blocking major backlinks crwawlers on PBN or on Monsy site?

Discussion in 'Black Hat SEO' started by frankstan, Sep 9, 2014.

  1. frankstan

    frankstan Regular Member

    Joined:
    Jan 14, 2014
    Messages:
    350
    Likes Received:
    44
    I'm building some PBN recently, I'm going to block some crwawlers like MOZ, MAjestic, I heard there is a plugin can help me to do it, the problem is do I need to install this plugin on every PBN or I just need to install it on my money site?
    I don't want to install in on every site since I'm afraid it is a footprint.
    So does it work if I only install the plugin on my money site?
     
  2. SEO Power

    SEO Power Elite Member

    Joined:
    Jul 14, 2014
    Messages:
    2,637
    Likes Received:
    680
    Occupation:
    Self employed
    Location:
    Houston, TX
  3. FitnessFred

    FitnessFred Newbie

    Joined:
    Jul 31, 2014
    Messages:
    13
    Likes Received:
    3
    Yes unfortunately you will have to install on each of your pbn site, if you want to avoid a plugin footprint, just use robots or htaccess to block crawlers.
     
  4. tony_d

    tony_d Elite Member

    Joined:
    Jun 22, 2013
    Messages:
    2,581
    Likes Received:
    3,163
    Location:
    1600 Amphitheatre Parkway, Mountain View CA
    I use a robots.txt file. To address the 'footprint' issue, I have 100+ variations of robots.txt that I randomly upload to the hosting account, in each case blocking the standard variety (ahrefs, majestic, et al) and plenty of other random bots that I don't really care about (just to mix it up and make the files different).

    There's still a footprint to the extent that the sites have a robots.txt file, but each file is randomly different. I'm not sure that the mere presence of a robots.txt file is anywhere near a significant footprint.
     
  5. DannyZhang

    DannyZhang Regular Member

    Joined:
    Apr 2, 2014
    Messages:
    233
    Likes Received:
    69
    This seems like a better solution that having plugins.

    Where did you get the list of bots to block?
     
  6. tony_d

    tony_d Elite Member

    Joined:
    Jun 22, 2013
    Messages:
    2,581
    Likes Received:
    3,163
    Location:
    1600 Amphitheatre Parkway, Mountain View CA
    An owner-submitted list of bots is available at robotstxt.org (direct link here)

    There are 300+ bots on the list, so many tens of thousands of possible combinations for a robots.txt file, even when you consider the 'constants' (ahrefs, majestic, et al).
     
  7. frankstan

    frankstan Regular Member

    Joined:
    Jan 14, 2014
    Messages:
    350
    Likes Received:
    44
    Thanks everyone, I will use robot.txt to block them
     
  8. hatblackguy

    hatblackguy Regular Member

    Joined:
    Jun 8, 2011
    Messages:
    252
    Likes Received:
    61
    Occupation:
    kicking Big G's fat Mug
    Location:
    1% Percent of wealthiest people
    Interesting how do you successfully block ahrefs?
    I can never get them to piss off when doing through Robots.txt