How to block Majestic bot?

[Narix]

Hi community!
I have a question regarding PBN's and hiding them from Majestic (MJ12bot) bot.
I have a PBN, actually a lot of PBN's. I was hiding them using robots.txt method known by everyone. But then in one sunny day Majestic started to ignore my robots.txt and now others can see my PBN sites when crawling my money site. Please note, that before nothing was visible...

So I have tried various blocking methods afterwards, like:

• various entries in robots.txt
• blocking through .htaccess
• tried to block URL from all bots where is placed link to money site
• tried to block URL's instead of whole domain;
• used some available plugins for WordPress
• then there was combinations of everything mentioned above
• and so on...

When using htaccess method I saw that even if bot get's not found error from site - it still got's crawled.

My tests were performed for a months and if some of them would go through - I would definitely not missed that.

I'm like in panic mode a bit

Community - have you experienced something like that? Maybe someone have solution for this?

This problem is large enough to consider to pay someone for information. So if you have something to offer - please drop me PM.

Ah, forgot to mention that I have contacted Majestic support for several times. They wasn't able to provide me any help or solution.

 

[tappu]

It will help you https://www.blackhatworld.com/seo/anybody-with-the-code-to-block-majestic-ahrefs-and-all-others-from-crawling-a-site.584470/

 

[Narix]

It will help you

Thank's for your reply, but unfortunately everything described in that post has been tried already and it doesn't work.

 

[Narix]

BUMP.

Anyone please?

 

[jamie3000]

Can you see in your logs what user agent they use? Even if you block them though there are still systems that can find hidden pbn links ;-)

 

[Narix]

As far as I have investigated they use Mozilla user agent. You can see that below:



And nothing can block that evil Majestic thing :/

 

[jamie3000]

As far as I have investigated they use Mozilla user agent. You can see that below:

View attachment 112845

And nothing can block that evil Majestic thing :/

A htaccess should block that

 

[Narix]

A htaccess should block that

Unfortunately it's don't. Have tried all methods can be found over the web.

 

[AngelSeo]

Unfortunately it's don't. Have tried all methods can be found over the web.

Then you either are doing it wrong, or you have wrong expectations. For example if you block the bots, they still will show the collected info they already had, with the exception of Archive - if you drop them e-mail they will remove you completely from their archive. All the rest will show the info up to the date of blocking..

 

[Narix]

Then you either are doing it wrong, or you have wrong expectations. For example if you block the bots, they still will show the collected info they already had, with the exception of Archive - if you drop them e-mail they will remove you completely from their archive. All the rest will show the info up to the date of blocking..

It was like all PBN domains was blocked using robots.txt and they didn't show up in Majesctic as referring domains, but then suddenly they started to show. Nothing was changed for domains. It was definitely something from Majestic side.

 

[Spam Boy]

Then you either are doing it wrong, or you have wrong expectations. For example if you block the bots, they still will show the collected info they already had, with the exception of Archive - if you drop them e-mail they will remove you completely from their archive. All the rest will show the info up to the date of blocking..

I have a question in that, if i block the bots in my wordpress site what about the majestic tf and cf score, it affects or not

 

[samrox]

Are there any one found proper solution for this one, i have same matter

 

[BHSEONewb]

I did a quick Google, here's what I found:

How can I block MJ12bot?
MJ12bot adheres to the robots.txt standard. If you want the bot to prevent website from being crawled then add the following text to your robots.txt:

User-agent: MJ12bot
Disallow: /
Please do not block our bot via IP in htaccess - we do not use any consecutive IP blocks as we are a community based distributed crawler. Please always make sure the bot can actually retrieve robots.txt itself. If it can't then it will assume that it is okay to crawl your site.

If you have reason to believe that MJ12bot did NOT obey your robots.txt commands, then please let us know via email: [email protected]. Please provide URL to your website and log entries showing bot trying to retrieve pages that it was not supposed to.

 

[Hetneo]

I did a quick Google, here's what I found:

Appreciate the Googling, but I think you may have missed the point here. This issue has been ongoing.

What Majestic are doing is a little sinister and they've been doing it for months now. Basically, they know that PBN owners can't use the same robots.txt file across their network (footprint). Therefore, most smart PBN owners attempt to block bots like majestic through htaccess.

However, we can only block them through htaccess via checking their user-agent. The way we block these bots is either sending the bot a 403 (disallow) or a 301 (redirect). Almost all bots obey this.... except f**king majestic. When they receive these responses from your htaccess file, it triggers their system to send new bots without user-agents. Those bots will double check the 403/301 and confirm if it's true or not.

If not true, majestic will use the new bots to crawl your site and index it on their system.

 

[BHSEONewb]

Appreciate the Googling, but I think you may have missed the point here. This issue has been ongoing.

What Majestic are doing is a little sinister and they've been doing it for months now. Basically, they know that PBN owners can't use the same robots.txt file across their network (footprint). Therefore, most smart PBN owners attempt to block bots like majestic through htaccess.

However, we can only block them through htaccess via checking their user-agent. The way we block these bots is either sending the bot a 403 (disallow) or a 301 (redirect). Almost all bots obey this.... except f**king majestic. When they receive these responses from your htaccess file, it triggers their system to send new bots without user-agents. Those bots will double check the 403/301 and confirm if it's true or not.

If not true, majestic will use the new bots to crawl your site and index it on their system.

That is bloody sneaky of them! Thanks for the explanation.

 

[Seo Licious]

if you block bot your site will deindex soon ? may I right?

 

[brazi]

Thx for the awesome replíes!

Just to understand it correctly shouldn`t I use the same robot.txt file within my PBN. Basically, it looks the same for every wordpress site created so for a PBN this should not be too suspicious?

 

[bartosimpsonio]

Best way to block these spiders is through a cloaker.

robots.txt requires you to trust the crawlers and most data gathering operations have bots running under fake user agents, they end up crawling your data anyway.

With a cloaker you can select user agent based content and show their crawlers whatever you want.

So if you detect majestic or ahrefs you can present them with a page full of links to disney just for kicks.

 

[jyesdi]

Best way to block these spiders is through a cloaker.

robots.txt requires you to trust the crawlers and most data gathering operations have bots running under fake user agents, they end up crawling your data anyway.

With a cloaker you can select user agent based content and show their crawlers whatever you want.

So if you detect majestic or ahrefs you can present them with a page full of links to disney just for kicks.

Won't google identify the cloaker and then you have the potential of setting up a footprint?

 

[naaz aniqua]

Either by IP or by the user agent we will block majestic bot.