1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

majestic seo has found my PBN

Discussion in 'Black Hat SEO' started by nonai, May 15, 2014.

  1. nonai

    nonai Power Member

    Joined:
    Oct 10, 2013
    Messages:
    524
    Likes Received:
    63
    I blocked many bots including ^MJ12bot using htaccess
    it looks like ahrefs has not found my PBN but majesticseo did
    how do I block majestic? please, this is really urgent.
     
  2. lucasbraydon

    lucasbraydon Supreme Member

    Joined:
    Nov 9, 2013
    Messages:
    1,231
    Likes Received:
    1,561
    Location:
    Here and there
    • Thanks Thanks x 3
  3. FBGuru

    FBGuru Senior Member

    Joined:
    Sep 22, 2013
    Messages:
    928
    Likes Received:
    1,171
    Location:
    Personality Type : ESTP
    There are so many versions of majestic bots crawling around and sucking behemoth amount of bandwidth so you may have entered just a single user agent and missed the others. Please post the user agent code that you have on your .htaccess to block the MJ12bot. Also, the easiest way to identify them is by checking out the raw hit logs/awstats on your cPanel for your PB.
     
    • Thanks Thanks x 1
  4. nonai

    nonai Power Member

    Joined:
    Oct 10, 2013
    Messages:
    524
    Likes Received:
    63
    I am not running wordpress so plugins wont work for me, I need to block them through htaccess.
    out of all the things I am blocking, the only majestic seo is MJ12bot
    I didn't know there are many bot names for majestic, I thought that's their only name.
    I will try and see what i can do as far as logs go.
     
  5. BuildMoreLinks

    BuildMoreLinks Jr. VIP Jr. VIP Premium Member

    Joined:
    Jun 7, 2012
    Messages:
    1,910
    Likes Received:
    655
    Location:
    17.3660° N, 78.4760° E
    blocking majectic bit is next to impossible in my tests, they are very sneaky and have lot of variations from a range of IP's
     
  6. FBGuru

    FBGuru Senior Member

    Joined:
    Sep 22, 2013
    Messages:
    928
    Likes Received:
    1,171
    Location:
    Personality Type : ESTP
    Yes it's virtually impossible to block them through IP's as their crawlers are crowdsourced and crawl webpages by the end users like you and me.

    OP, put this in your robots.txt

    User-agent: MJ12bot
    Disallow: /

    and shoot an email with the hit logs to bot@majestic12.co.uk and request them to stop crawling your site.
     
  7. twilightofidols

    twilightofidols Power Member

    Joined:
    Nov 11, 2011
    Messages:
    675
    Likes Received:
    223
    Majestic is a huge pain the ass. You could also disallow all bots except Google, Bing/Yahoo, Yandex and hope they follow it.
     
  8. stugz

    stugz Junior Member

    Joined:
    Apr 14, 2013
    Messages:
    154
    Likes Received:
    33
    Majestic is a distributed bot run from all over the world. Look at majestic12.co.uk for more detail. Also, look in the Majestic FAQ:

    How can I see more detailed information for my site?

    By registering with us you can add your own websites to your profile to see detailed analysis of backlinks and anchor text. This information is prioritised to show the most important backlinks first, rather than showing a random sample (this helps gives prominence to the more important backlinks). The catch is that in exchange for free basic information about anchor text and backlinks for your site you will have to allow our bot (MJ12bot) to crawl your site for at least 12 months since your last usage of reports. In the vast majority of cases (95%+) you already do that, so in this case there is no catch really: only those who specifically disallowed our bot in their robots.txt or only allow a handful of other bots to crawl and disallow all the rest, will need to decide if they want to allow us crawl them in exchange for the free information that they can get. We hope your decision will be positive!
     
  9. YESEO

    YESEO Newbie

    Joined:
    Apr 7, 2014
    Messages:
    16
    Likes Received:
    1
    I block majestic without any problem with "MJ12bot" tag and some php code but htaccess should work too.
    If you did not block them at the beginning, I think you can't do anything now because they already crawled you :-(
     
  10. spanishduez

    spanishduez Newbie

    Joined:
    Apr 1, 2014
    Messages:
    21
    Likes Received:
    1
    Check your logs for the actual crawler from majestic so you could block it if you have not already done so. At the end of the day, it is their choice whether or not to crawl your sites.
     
  11. ezines

    ezines Power Member

    Joined:
    Jan 3, 2011
    Messages:
    712
    Likes Received:
    216
    Occupation:
    Online/Offline
    Location:
    Somewhere On Earth
    True, they even published a blog post about it here;
    Code:
    http://blog.majesticseo.com/general/verifying-site-majesticseo-every-seo-audit/
    I guess blocking it at both .htaccess and robots.txt would do?
     
  12. kboing

    kboing Newbie

    Joined:
    Sep 19, 2013
    Messages:
    24
    Likes Received:
    4
    hello

    maybe this help you add this rule in .htaccess

    RewriteCond %{HTTP:User-Agent} (?:Owlinbo|BLEXBot|XoviBot|SEOkicks|meanpathbot|360Spider|SemrushBot|genieo|spbot|MJ12bot|AhrefsBot|SearchmetricsBot|Zeus|DotBot|Mail.RU_Bot|Ezooms|rogerbot|James) [NC]
    RewriteRule .? - [F]


    block majestic , all mozbot variations , ahref and others

    best
     
    • Thanks Thanks x 1
  13. eck340

    eck340 Registered Member

    Joined:
    Jan 12, 2013
    Messages:
    57
    Likes Received:
    5
    Occupation:
    SEO, Graphic Design and Web Design
    Location:
    Canada
    Hi guys, I had the same issue this week, I blocked majestic bots on my .htaccess file but it got resetted for what ever reason and now my PBN site is showing as a reffering domains when I analyse my money site with Majestic.

    Do you guys know if it's possible to delete it from their listing if so how ? Can we contact them to request it ?

    I bought my PBN domain like 350$ so it's really pissing me of, because I dont want to delete it.

    What do you guys think ?

    Thanks
     
  14. SupremeBoner

    SupremeBoner Junior Member

    Joined:
    Jan 8, 2015
    Messages:
    114
    Likes Received:
    19
    Stats of pbn? tbh i wanna know what a 350$ pbn looks like nowadays

    You cant ask them to remove it from what i know

    Better idea is to make the pbn real fancy so it fools any seos looking at it.
     
  15. eck340

    eck340 Registered Member

    Joined:
    Jan 12, 2013
    Messages:
    57
    Likes Received:
    5
    Occupation:
    SEO, Graphic Design and Web Design
    Location:
    Canada
    Thank for your reply

    There is now way they can drop the domain from their database ? If when they crawl back my PBN and they get a 404 error or Server Drop Out Code ?

    Since I reblocked the bots with a pluging instead of the htaccess, what will happen to the backlink if they are not able to crawl back my website ? the backlink will stay alive, will be marked as removed or they will remove it ? (The last option would be awesome)

    What do you guys think ?
     
  16. infoking1

    infoking1 Junior Member

    Joined:
    Sep 16, 2010
    Messages:
    182
    Likes Received:
    24
    Home Page:
    But once they crawled i think no use now and your old page will be in their server. Only thing you can do is to avoid any future content crawled by them. Are you sure they crawled after you added .htaccess condition?

    Also check your .htaccess condition, in general it is case sensitive, to skip case you have to use [NC]. may be you can create own mjbot useragent like mjbot and try access your site and see .htaccess can able to correctly identify and block it. you can do this using "inspect element" in chrome or mozilla.
     
  17. mainceaft

    mainceaft Regular Member

    Joined:
    Apr 10, 2013
    Messages:
    373
    Likes Received:
    39
    j majestic are screamers
    I catch their bot uses Googlebot User Agent . even they are using proxies to hide there real IP's . I already reported on them to one of ISP in Europe after use their IP in sneaky way . Personally I hope someone nuke their entire network with some multi-gigabyte DDos Attack:(
     
  18. ThopHayt

    ThopHayt Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 25, 2011
    Messages:
    5,396
    Likes Received:
    1,644
    This is not as big of deal as you think. If a competitor were really after you they don't need to know your network, they just negative seo you directly. They will copy your link profile if they are smart, that is Majestic's only real use.