Disallow all crawlers except Google

D0llarBillz · Jun 10, 2014

Hi,

How to disallow all crawlers (Ahrefs,Yahoobot,Bingbot,MajesticSEObot and etc..) except Google with .htaccess? I know how to do with robots.txt but I need with .htaccess.

Thank you!

irdeto · Jun 10, 2014

+1 here for this request.
I haven't done much research on it and would also like to know.

Final Infinity · Jun 10, 2014

I am not sure how to do it with .htaccess but if you are using wp you can use a plugin called Spyder Spanker. Costs a little bit of money but well worth it imo. You can choose any kind of bots you want to let access your website and those that you don't. Hope that helps at least a little bit.

tony_d · Jun 10, 2014

You did specify htaccess, but a robots.txt file with the following will do the trick nicely

Code:

User-agent: Googlebot
Disallow:

User-agent: msnbot
Disallow:

User-agent: Slurp
Disallow:

User-agent: Teoma
Disallow:

User-agent: rogerbot
Disallow: /

User-agent: exabot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: gigabot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: *
Disallow: /

D0llarBillz · Jun 10, 2014

tony_d said:

You did specify htaccess, but a robots.txt file with the following will do the trick nicely

Code:

User-agent: Googlebot
Disallow:

User-agent: msnbot
Disallow:

User-agent: Slurp
Disallow:

User-agent: Teoma
Disallow:

User-agent: rogerbot
Disallow: /

User-agent: exabot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: gigabot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: *
Disallow: /

Yes, I know how to block with robots.txt but I need block with .htaccess (not all bots checks robots.txt) I need to block them from my all website.

the_demon · Jun 11, 2014

There are many ways... Just Google: block robots with .htaccess

Found this site w/ one Google Search: http://www.thesitewizard.com/apache/block-bots-with-htaccess.shtml

D0llarBillz · Jun 13, 2014

the_demon said:
There are many ways... Just Google: block robots with .htaccess

Found this site w/ one Google Search: http://www.thesitewizard.com/apache/block-bots-with-htaccess.shtml

I googled. I didn't found that I wanted.

cashcorp · Jun 13, 2014

Use the .htaccess, robots.txt is useless for Majestic/Ahrefs etc.

Further, even "Botblocking" via htaccess fails. You'll want to drop in the relevant IP ranges (When you can find them) as well.

SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
SetEnvIfNoCase User-Agent .*exabot.* bad_bot
SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>
Order Deny,Allow
deny from 5.10.83.0/25
deny from 217.69.128.0/21
deny from 213.138.96.0/19
deny from 213.186.112.0/20
deny from 212.113.32.0/21
deny from 173.199.115.112/29
deny from 188.200.0.0/13
deny from 89.108.96.0/21
deny from 173.192.34.64/26
deny from 173.11.97.112/28
deny from 5.10.83.0/25
deny from 213.186.96.0/19
deny from 188.138.0.0/17
Deny from 212.113.35.162
Deny from 212.113.37.105
Deny from 212.113.37.106
Deny from 213.186.119.134
Deny from 213.186.120.196
Deny from 213.186.122.2
Deny from 213.186.122.3
Deny from 213.186.127.3
Deny from 173.199.115.115
Deny from 5.10.83.70

domainmadness · Jun 13, 2014

You can use this, just redirect them to google.

RewriteEngine On
RewriteCond %{REQUEST_URI} !/robots.txt$
RewriteCond %{HTTP_USER_AGENT} ^.*BLEXBot.*$ [NC,OR]
..... more bots .....
RewriteRule ^.*.* http://google.com/ [L]

I hope this helps.

D0llarBillz · Jun 13, 2014

Thanks! I will try this soon.

TropicalSun · Jun 13, 2014

I've also had a problem blocking access to ahrefs/majestic, but especially ahrefs.

Any idea why Ahrefs is being so difficult?

Disallow all crawlers except Google

D0llarBillz

Regular Member

irdeto

Regular Member

Final Infinity

Junior Member

tony_d

Elite Member

D0llarBillz

Regular Member

the_demon

Elite Member

D0llarBillz

Regular Member

cashcorp

Regular Member

domainmadness

Supreme Member

D0llarBillz

Regular Member

TropicalSun

Regular Member

Main Menu

Marketplace

Making Money

BlackHat World