1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How many normal sites block majestic / ahrefs?

Discussion in 'Black Hat SEO' started by MrBeastsOnToast, Mar 26, 2016.

  1. MrBeastsOnToast

    MrBeastsOnToast Jr. VIP Jr. VIP

    Joined:
    Dec 17, 2011
    Messages:
    1,049
    Likes Received:
    569
    Location:
    The Internetz
    I use a bunch of PBN's on various sites - they work great currently, i am not disputing that. But i also view those sites as churn and burn. Any site with a majestic / ahrefs blocked majority is going to get slapped - unless i am missing something here....

    I would bet not many sites that are not PBNS block majestic, isn't this a huge and super easy footprint for google to find?

    Here watch me destroy our sites with 2 lines of code.

    if (percenatageOfSitesLinkingInWithMajesticBlocked > AverageAmountOfSitesWithMajesticBlocked * amountOfACuntGoogleWantsToBe )
    {
    callMattCuttsBackFromVacation();
    }

    Prove me wrong. Please
     
  2. agent-x

    agent-x Junior Member

    Joined:
    Apr 4, 2012
    Messages:
    173
    Likes Received:
    34
    Gender:
    Male
    Block them in htaccess and then have a rule blocking anyone from seeing your htaccess too - impossible to detect then.
     
  3. agent-x

    agent-x Junior Member

    Joined:
    Apr 4, 2012
    Messages:
    173
    Likes Received:
    34
    Gender:
    Male
    Actually I did used to block both of them and others like Baidu spider, simply because they weren't very intelligent and kept hitting the same pages over and over again wasting server resources. Baidu spider still does this and it annoys the fuck out of me, Majestic and Ahrefs have improved their crawling technique lately.
     
  4. MrBeastsOnToast

    MrBeastsOnToast Jr. VIP Jr. VIP

    Joined:
    Dec 17, 2011
    Messages:
    1,049
    Likes Received:
    569
    Location:
    The Internetz
    Good answer. /endThread
     
  5. OutOfContext

    OutOfContext Junior Member

    Joined:
    Oct 31, 2015
    Messages:
    100
    Likes Received:
    17
    Well actually you can still detect blocking via htaccess by merely pretending to be the Majestic or whatever robot - simply set the user agent. We can only hope that Google doesn't engage in this type of behaviour. On the other hand they could engage in it only in special cases when they suspect​ a PBN, simply to prove themselves right.
     
  6. agent-x

    agent-x Junior Member

    Joined:
    Apr 4, 2012
    Messages:
    173
    Likes Received:
    34
    Gender:
    Male
    This is true and since Majestic state that they don't use consecutive IP blocks it's hard to tell a real MJ12 bot from a fake one, same may well be true for Ahrefs. The only thing about that would be that if a bot pretending to be a Majestic or Ahrefs bot gets a 403 error, it doesn't necessarily know that it's the user-agent that is being blocked, it could be another reason.
     
  7. agent-x

    agent-x Junior Member

    Joined:
    Apr 4, 2012
    Messages:
    173
    Likes Received:
    34
    Gender:
    Male
    I don't know about Google yet but Bing are certainly not afraid of using fake user-agents in their bots...
     
  8. OutOfContext

    OutOfContext Junior Member

    Joined:
    Oct 31, 2015
    Messages:
    100
    Likes Received:
    17
    Really? Didn't know that. Do you think they do it to detect cloaking?


    As for Google, I know that I often get visits from Mountain View IP addresses even on very new websites that are not yet getting any search engine traffic and these visits don't have Googlebot in their user agent string. The websites don't have Adsense on them either. So these are either Google's bots masquerading as normal users or manual reviewers.
     
  9. agent-x

    agent-x Junior Member

    Joined:
    Apr 4, 2012
    Messages:
    173
    Likes Received:
    34
    Gender:
    Male
    Here's a typical one that visited me this morning:

    65.55.218.58 () en-us Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; Trident/5.0)

    that IP resolves to: msnbot-65-55-218-58.search.msn.com

    I expect it is indeed to catch cloaking.
     
    Last edited: Mar 27, 2016
  10. RuthSam

    RuthSam Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 19, 2010
    Messages:
    3,813
    Likes Received:
    976
    Gender:
    Male
    Home Page:
    how do you set this up?
     
  11. agent-x

    agent-x Junior Member

    Joined:
    Apr 4, 2012
    Messages:
    173
    Likes Received:
    34
    Gender:
    Male
    This is a handy tool for htaccess rules http://www.htaccessredirect.net/

    So you'd end up with something like this in your htaccess file:

    //Block Ahrefs and Majestic
    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} MJ12bot [OR]
    RewriteCond %{HTTP_USER_AGENT} AhrefsBot
    RewriteRule ^.* - [F,L]

    //Prevent viewing of .htaccess file
    <Files .htaccess>
    order allow,deny
    deny from all
    </Files>
     
  12. nikita12

    nikita12 Jr. VIP Jr. VIP

    Joined:
    Nov 15, 2012
    Messages:
    595
    Likes Received:
    190
    Occupation:
    DomCop Guru
    Location:
    DomCop
    Home Page:

    You are not wrong. If Google wants to make an example of any PBN, this is a super easy way to do that. In fact, in this case, they would not de-index the PBN - they would simply mark those links as ineffective and wait for people to just give up, because their PBNs did not work.

    Anything you implement, that you would not normally implement for a normal website becomes a footprint. The one important thing to focus on when building a PBN is this - if any person sees the link from your PBN to your money site, they do not think twice about whether this link is justified to exist or not. Go visit the top 5 back links to any of your successful competitors. Now ask yourself - is this a PBN link - if you cannot tell for sure, then that's exactly the kind of PBN back link you want.
     
  13. accelerator_dd

    accelerator_dd Jr. VIP Jr. VIP

    Joined:
    May 14, 2010
    Messages:
    2,448
    Likes Received:
    1,010
    Occupation:
    SEO
    Location:
    IM Wonderland
    The only way Google can notice a site is blocking ahrefs/majestic is either if you block via robots.txt (bad idea) or if Google uses majestic's/ahrefs' useragent (which it doesn't, i've been tracking the IPs of those crawlers vs Google's).

    You can read about blocking via majestic/ahrefs at http://www.blackhatworld.com/blackh...os-why-doing-wrong-deindexing.html?highlight=

    Also, there are a lot of legitimate reasons to block those bots though - saving bandwidth/resources for example.