How to block Majestic bot?

Hetneo · Jun 7, 2019

jyesdi said:
Won't google identify the cloaker and then you have the potential of setting up a footprint?

Possible, but a hell of a lot better than f**king majestic listing your PBNs for all your competitors to see (and report to Google).

Majestic especially are no longer playing by the rules, they are sending unidentified bots to check websites.

JOSourcing · Jun 9, 2019

I don't use a PBN, but I do block 93% of bots via .htaccess for other reasons. The .htaccess advice given in https://www.blackhatworld.com/seo/how-to-block-majestic-bot.1091978/#post-11775939 will work. Don't even bother with the robots.txt file. If .htaccess blocking doesn't work, you've screwed up something in there.

You can also block a bot via PHP by checking the user agent of each visit.

You basically want to put something like this on your pages:

Code:

<?php
If (strpos($_SERVER['HTTP_USER_AGENT'], 'mj12_or_whatever_the_hell_its_called') !== false) {
 // Insert code to kick them to the curb
}
?>

Hetneo · Jun 9, 2019

JOSourcing said:
If .htaccess blocking doesn't work, you've screwed up something in there.

Not really, as we've already covered in this thread, Majestic (and some others) are not playing by the rules anymore.

Your code is assuming that these analytics sites are using their correct user-agent when visiting your website. They do use their user-agent on the first round of visiting, however, they then revisit the website without their user-agent to double check you're not tricking them. Then, if the version they find on the second turn is different, they will use that version to post on their website.

fb3003 · Jun 9, 2019

none of them play by the rules anymore: semrush, ahref and majestic send other unknown bots to websites to figure things out. just don't bother anymore.

JOSourcing · Jun 9, 2019

Hetneo said:
Not really, as we've already covered in this thread, Majestic (and some others) are not playing by the rules anymore.

Your code is assuming that these analytics sites are using their correct user-agent when visiting your website. They do use their user-agent on the first round of visiting, however, they then revisit the website without their user-agent to double check you're not tricking them. Then, if the version they find on the second turn is different, they will use that version to post on their website.

They'll switch IP numbers too. That's why you have to actively watch your traffic logs and watch their behavior.

ilhivy · Feb 8, 2022

Hello all, if I just want to prevent competitors from spying on my keywords, is blocking semrush, ahref and Majestic bots at robots.txt going to help at all? Thanks

How to block Majestic bot?

Hetneo

Power Member

JOSourcing

Regular Member

Hetneo

Power Member

fb3003

Elite Member

JOSourcing

Regular Member

ilhivy

Registered Member

Main Menu

Marketplace

Making Money

BlackHat World