Looking for updated htaccess code to block crawlers

[Openside]

Hi guys,

I was using the code off of this old thread but it seems everything is getting picked up now by various crawlers:
http://www.blackhatworld.com/seo/howto-hiding-your-pbn-like-the-pros-and-why-doing-it-wrong-deindexing.747598/page-3

Has anyone got an updated htaccess code to block bots which would be much appreciated!

Thanks in advance.

 

[NulledCode]

Hi guys,

I was using the code off of this old thread but it seems everything is getting picked up now by various crawlers:
http://www.blackhatworld.com/seo/ho...d-why-doing-it-wrong-deindexing.747598/page-3

Has anyone got an updated htaccess code to block bots which would be much appreciated!

Thanks in advance.

What bots are getting through? Why not add them to your list?

Also, htaccess and robots file are not sure fire ways to stop bots. It's "best practice" to identify your not but nothing can stop a scripted bot from accessing your site unless you block by IP/ IP range.

 

[pauverinox]

Got this from a thread here.

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress


SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
SetEnvIfNoCase User-Agent .*exabot.* bad_bot
SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
SetEnvIfNoCase User-Agent .*semrushbot.* bad_bot
SetEnvIfNoCase User-Agent .*ia_archiver.* bad_bot
SetEnvIfNoCase User-Agent .*searchmetricsbot.* bad_bot
SetEnvIfNoCase User-Agent .*seokicks-robot.* bad_bot
SetEnvIfNoCase User-Agent .*sistrix.* bad_bot
SetEnvIfNoCase User-Agent .*lipperhey spider.* bad_bot
SetEnvIfNoCase User-Agent .*ncbot.* bad_bot
SetEnvIfNoCase User-Agent .*backlinkcrawler.* bad_bot
SetEnvIfNoCase User-Agent .*archive.org_bot.* bad_bot
SetEnvIfNoCase User-Agent .*meanpathbot.* bad_bot
SetEnvIfNoCase User-Agent .*pagesinventory.* bad_bot
SetEnvIfNoCase User-Agent .*aboundexbot.* bad_bot
SetEnvIfNoCase User-Agent .*spbot.* bad_bot
SetEnvIfNoCase User-Agent .*linkdexbot.* bad_bot
SetEnvIfNoCase User-Agent .*nutch.* bad_bot
SetEnvIfNoCase User-Agent .*blexbot.* bad_bot
SetEnvIfNoCase User-Agent .*ezooms.* bad_bot
SetEnvIfNoCase User-Agent .*scoutjet.* bad_bot
SetEnvIfNoCase User-Agent .*majestic-12.* bad_bot
SetEnvIfNoCase User-Agent .*majestic-seo.* bad_bot
SetEnvIfNoCase User-Agent .*dsearch.* bad_bot
SetEnvIfNoCase User-Agent .*blekkobo.* bad_bot
SetEnvIfNoCase User-Agent .*screaming frog seo spider/*.* bad_bot
SetEnvIfNoCase User-Agent .*PHPCrawl.* bad_bot
SetEnvIfNoCase User-Agent .*gocrawl.* bad_bot
SetEnvIfNoCase User-Agent .*DigExt.* bad_bot
SetEnvIfNoCase User-Agent .*DomainSONOCrawler.* bad_bot
SetEnvIfNoCase User-Agent .*TweetmemeBot.* bad_bot
SetEnvIfNoCase User-Agent .*OpenHoseBot/2.1.* bad_bot
SetEnvIfNoCase User-Agent .*Kraken/0.1.* bad_bot
SetEnvIfNoCase User-Agent .*-Java-.* bad_bot
SetEnvIfNoCase User-Agent .*ubermetrics.* bad_bot
SetEnvIfNoCase User-Agent .*best-seo.* bad_bot
SetEnvIfNoCase User-Agent .*Synapse.* bad_bot
SetEnvIfNoCase User-Agent .*Harvest.* bad_bot
SetEnvIfNoCase User-Agent .*Harvester.* bad_bot
SetEnvIfNoCase User-Agent .*harvester.* bad_bot
SetEnvIfNoCase User-Agent .*harvest.* bad_bot

<Limit GET POST HEAD>

Order Allow,Deny

Allow from all

Deny from env=bad_bot

</Limit>

 

[StSignal]

What bots are getting through? Why not add them to your list?

Also, htaccess and robots file are not sure fire ways to stop bots. It's "best practice" to identify your not but nothing can stop a scripted bot from accessing your site unless you block by IP/ IP range.

Exactly,nothing can stop it after activation.

 

[Openside]

I have pretty much the same code as above...just weird that all of a sudden everything is showing when 6 months ago nothing was showing up!