Is it really impossible to block majestic bot? Or would this work?

[taketheride]

I've been trying on several sites with different combinations of code to block majestic crawlers through htaccess, but it just doesn't work. I'm having no problem with the other bots like ahrefs and OSE. Don't want to leave a footprint by using neither the robots file, nor spyderspanker/other plugins right now. But I was thinking, wouldn't it be possible to block all bots except for those coming from search engines? I've been looking around and trying to learn how to put this into code, but can't seem to figure it out. So I'm wondering:

- Do you think this would be possible without obstructing the bots you want to have crawling the site?

- Is there a way to distinguish between bots and real people/browsers in htaccess?

 

[ghostlyman]

Hey, I know you said that you don't want to use SpyderSpanker plugin but I talked to few people who have decent sized PBNs and they said that they have SS installed on all sites and there hasn't been any problems with their network yet. I don't think the plugin itself leaves the footprint. The footprint would be the fact that crawlers are blocked, plugin wouldn't be a footprint if that makes sense

 

[mikpel]

What code are you using to block majestic? Just add this to you .htaccess and majestic crawler gets blocked:

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} Majestic-12 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Majestic-SEO [NC]
RewriteRule .* - [F]

There is no need to pay 39$ for such a simple task as blocking bots (spyder spanker). Just add as many unwanted user agents in your .htaccess as you want.

 

[taketheride]

What code are you using to block majestic? Just add this to you .htaccess and majestic crawler gets blocked:

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} Majestic-12 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Majestic-SEO [NC]
RewriteRule .* - [F]

There is no need to pay 39$ for such a simple task as blocking bots (spyder spanker). Just add as many unwanted user agents in your .htaccess as you want.

That's similar to what I've been using, except I redirect it to another website. Don't think I included the user agent with "majestic-seo" before though, just mj12, will try that out.