Disallow all crawlers except Google

[D0llarBillz]

Hi,

How to disallow all crawlers (Ahrefs,Yahoobot,Bingbot,MajesticSEObot and etc..) except Google with .htaccess? I know how to do with robots.txt but I need with .htaccess.

Thank you!

 

[irdeto]

+1 here for this request.
I haven't done much research on it and would also like to know.

 

[Final Infinity]

I am not sure how to do it with .htaccess but if you are using wp you can use a plugin called Spyder Spanker. Costs a little bit of money but well worth it imo. You can choose any kind of bots you want to let access your website and those that you don't. Hope that helps at least a little bit.

 

[tony_d]

You did specify htaccess, but a robots.txt file with the following will do the trick nicely

User-agent: Googlebot
Disallow:

User-agent: msnbot
Disallow:

User-agent: Slurp
Disallow:

User-agent: Teoma
Disallow:

User-agent: rogerbot
Disallow: /

User-agent: exabot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: gigabot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: *
Disallow: /

 

[D0llarBillz]

You did specify htaccess, but a robots.txt file with the following will do the trick nicely

User-agent: Googlebot
Disallow:

User-agent: msnbot
Disallow:

User-agent: Slurp
Disallow:

User-agent: Teoma
Disallow:

User-agent: rogerbot
Disallow: /

User-agent: exabot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: gigabot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: *
Disallow: /

Yes, I know how to block with robots.txt but I need block with .htaccess (not all bots checks robots.txt) I need to block them from my all website.

 

[the_demon]

There are many ways... Just Google: block robots with .htaccess

Found this site w/ one Google Search: http://www.thesitewizard.com/apache/block-bots-with-htaccess.shtml

 

[D0llarBillz]

There are many ways... Just Google: block robots with .htaccess

Found this site w/ one Google Search: http://www.thesitewizard.com/apache/block-bots-with-htaccess.shtml

I googled. I didn't found that I wanted.

 

[cashcorp]

Use the .htaccess, robots.txt is useless for Majestic/Ahrefs etc.

Further, even "Botblocking" via htaccess fails. You'll want to drop in the relevant IP ranges (When you can find them) as well.

SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
SetEnvIfNoCase User-Agent .*exabot.* bad_bot
SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>
Order Deny,Allow
deny from 5.10.83.0/25
deny from 217.69.128.0/21
deny from 213.138.96.0/19
deny from 213.186.112.0/20
deny from 212.113.32.0/21
deny from 173.199.115.112/29
deny from 188.200.0.0/13
deny from 89.108.96.0/21
deny from 173.192.34.64/26
deny from 173.11.97.112/28
deny from 5.10.83.0/25
deny from 213.186.96.0/19
deny from 188.138.0.0/17
Deny from 212.113.35.162
Deny from 212.113.37.105
Deny from 212.113.37.106
Deny from 213.186.119.134
Deny from 213.186.120.196
Deny from 213.186.122.2
Deny from 213.186.122.3
Deny from 213.186.127.3
Deny from 173.199.115.115
Deny from 5.10.83.70

 

[domainmadness]

You can use this, just redirect them to google.

RewriteEngine On
RewriteCond %{REQUEST_URI} !/robots.txt$
RewriteCond %{HTTP_USER_AGENT} ^.*BLEXBot.*$ [NC,OR]
..... more bots .....
RewriteRule ^.*.* http://google.com/ [L]

I hope this helps.

 

[D0llarBillz]

Thanks! I will try this soon.

 

[TropicalSun]

I've also had a problem blocking access to ahrefs/majestic, but especially ahrefs.

Any idea why Ahrefs is being so difficult?