1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Question about blocking the Ahrefs, Majestic and Moz bots in htaccess for PBN

Discussion in 'Black Hat SEO' started by karupoiss, Jun 21, 2014.

  1. karupoiss

    karupoiss Elite Member

    Joined:
    May 16, 2012
    Messages:
    1,635
    Likes Received:
    515
    Gender:
    Male
    Location:
    Europe
    Home Page:
    If I block Ahrefs, Majestic and Moz bots in my .htaccess file then will google be able to see that and can that in itself leave a footprint?
     
    • Thanks Thanks x 3
  2. bigballin6161

    bigballin6161 Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 16, 2011
    Messages:
    1,084
    Likes Received:
    420
    I was thinking the same thing. Who blocks bots except blog network owners? If your whole network is blocked then I think this is a huge footprint. I would like to see some answers on this.
     
  3. WPRipper

    WPRipper Supreme Member

    Joined:
    Mar 24, 2010
    Messages:
    1,379
    Likes Received:
    1,493
    Location:
    Proudly romanian
    These are third party tools, i dont think will be a problem. Maybe i'm wrong.
     
  4. farkasb

    farkasb Junior Member

    Joined:
    Oct 12, 2012
    Messages:
    105
    Likes Received:
    52
    I'm not a networking expert, but I doubt anybody could actually download/see the content (as plain text) of a .htaccess file without having access to the site itself (by ftp for example).
    So, how could they tell what is blocked than?
     
  5. THUNDERELVI

    THUNDERELVI Elite Member

    Joined:
    Sep 12, 2009
    Messages:
    2,202
    Likes Received:
    1,732
    Gender:
    Male
    Location:
    W3
    Not only blog networks block those crawlers. If Google used this "footprint" to penalize us, it would also penalize some other legit sites.
     
  6. Expertpeon

    Expertpeon Elite Member

    Joined:
    Apr 22, 2011
    Messages:
    1,959
    Likes Received:
    1,187
    I see this kind of claim fairly regularly, but why does anyone think that Google cares if they punish legit sites during their large "anti-spam" updates? I've seen no real evidence they care.
     
    • Thanks Thanks x 1
  7. the_demon

    the_demon Jr. Executive VIP

    Joined:
    Nov 23, 2008
    Messages:
    3,177
    Likes Received:
    1,563
    Occupation:
    Search Engine Marketing
    Location:
    The Internet
    It's possible they could detect it using a brute force approach, however that would likely be a waste of their computational resources as they would have to check every site on the web and rather than make one request make half a dozen or more requests. Also, it could potentially cause their crawler to get blocked for abuse if they did it from the same IP. DNS Security solutions like CloudFlare and Distil Networks might determine Google Bot to be a bad bot/crawler. If that were to happen Google might start to come under fire for abusive web requests on people's domains or could end up blocked by a significant number of sites utilizing the aforementioned security solutions.

    With that said, for the time being a .htaccess or DNS level filtering solution is probably the safest route versus blocking bots via robots.txt, which Google could flag sites as suspicious that are blocking link crawlers like Ahrefs or Majestic.
     
    • Thanks Thanks x 1
  8. THUNDERELVI

    THUNDERELVI Elite Member

    Joined:
    Sep 12, 2009
    Messages:
    2,202
    Likes Received:
    1,732
    Gender:
    Male
    Location:
    W3
    You are right. Google does not care. All they care about is how to push Adwords even further to all webmasters.
    However, for the sake of relevant search results (which is the core that keeps Google alive and #1), the algorithm is designed to target as many sites as possible with little collateral damage. Unless done manually, I cannot understand how the algorithm can penalize your site if you block certain crawlers. It is your right to do after all.
     
  9. karupoiss

    karupoiss Elite Member

    Joined:
    May 16, 2012
    Messages:
    1,635
    Likes Received:
    515
    Gender:
    Male
    Location:
    Europe
    Home Page:
    So Google should not see what my .htaccess file has in it and does not know that I block the crawlers ~ does not see a footprint?
     
  10. praetserge

    praetserge Power Member

    Joined:
    Apr 4, 2014
    Messages:
    643
    Likes Received:
    167
    Nobody can see what's in htaccess file... you need FTP access and such to read that file.
     
  11. mk810

    mk810 Newbie

    Joined:
    Sep 4, 2014
    Messages:
    26
    Likes Received:
    2
    Right now I haven't heard any issues at all with this, however it definitely could be a signal if all of your links come from sites with them blocked.
     
  12. DjinnNTonik

    DjinnNTonik Newbie

    Joined:
    Feb 3, 2015
    Messages:
    16
    Likes Received:
    5
    Location:
    The LAMP
    Blocking unwanted bots with .htaccess is a good way to help prevent getting your PBN spotted in SEO tools like MajesticSEO and Ahrefs. Unless you specifically block Googlebot (and who would do that if trying to rank in Google is the goal?), Google will never notice your handiwork. I personally block unwanted bots from everything. Once you've identified a good domain for your PBN and purchased it, looking up your stats about it in the various SEO tools is just vanity and could lead to your competitors finding you and punishing you with various blackhat techniques to diminish your rankings.

    The only question is how to write the code for your particular server configuration. Some people use RewriteCond, but that requires mod_rewrite to be turned on. Otherwise it can be accomplished with SetEnvIfNoCase or BrowserMatchNoCase. Check your server config, then google for those commands to learn how to use them. There are also threads on BHW with code examples.

    You can check your server config by querying DomainTools with your domain name.
     
    Last edited: Feb 18, 2015
  13. CharliePug

    CharliePug Newbie

    Joined:
    May 4, 2014
    Messages:
    26
    Likes Received:
    3
    Yes, but you can fake your useragent and check if the website is blocking you or not, which, as said, seems an enormous effort to do with each website on Internet.