are badbots ignoring htacces ?

onebluesummer

onebluesummer

Junior Member
Joined
Feb 18, 2016
Messages
103
Reaction score
60
Anyone else see this happening in the past couple weeks?

I guess I should also say common blocking plugins?
 
Last edited:
Advertise on BHW
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteRule ^wp-content/plugins/spiderblocker/index.php /index.php%{REQUEST_URI} [QSA,L]
RewriteRule ^wp-content/plugins/spiderblocker/readme.txt /index.php%{REQUEST_URI} [QSA,L]
RewriteRule ^wp-content/plugins/spiderblocker/ /index.php%{REQUEST_URI} [QSA,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

# BEGIN NiteowebSpiderBlocker
SetEnvIfNoCase User-Agent "AhrefsBot" block_bot
SetEnvIfNoCase User-Agent "MJ12bot" block_bot
SetEnvIfNoCase User-Agent "Rogerbot" block_bot
SetEnvIfNoCase User-Agent "SemrushBot" block_bot
SetEnvIfNoCase User-Agent "ia_archiver" block_bot
SetEnvIfNoCase User-Agent "ScoutJet" block_bot
SetEnvIfNoCase User-Agent "sistrix" block_bot
SetEnvIfNoCase User-Agent "SearchmetricsBot" block_bot
SetEnvIfNoCase User-Agent "SEOkicks-Robot" block_bot
SetEnvIfNoCase User-Agent "Lipperhey Spider" block_bot
SetEnvIfNoCase User-Agent "Exabot" block_bot
SetEnvIfNoCase User-Agent "NCBot" block_bot
SetEnvIfNoCase User-Agent "BacklinkCrawler" block_bot
SetEnvIfNoCase User-Agent "archive.org_bot" block_bot
SetEnvIfNoCase User-Agent "meanpathbot" block_bot
SetEnvIfNoCase User-Agent "PagesInventory" block_bot
SetEnvIfNoCase User-Agent "Aboundexbot" block_bot
SetEnvIfNoCase User-Agent "spbot" block_bot
SetEnvIfNoCase User-Agent "linkdexbot" block_bot
SetEnvIfNoCase User-Agent "Gigabot" block_bot
SetEnvIfNoCase User-Agent "dotbot" block_bot
SetEnvIfNoCase User-Agent "Nutch" block_bot
SetEnvIfNoCase User-Agent "BLEXBot" block_bot
SetEnvIfNoCase User-Agent "Ezooms" block_bot
SetEnvIfNoCase User-Agent "Majestic-12" block_bot
SetEnvIfNoCase User-Agent "Majestic-SEO" block_bot
SetEnvIfNoCase User-Agent "DSearch" block_bot
SetEnvIfNoCase User-Agent "BlekkoBot" block_bot
SetEnvIfNoCase User-Agent "NerdyBot" block_bot
SetEnvIfNoCase User-Agent "JamesBOT" block_bot
SetEnvIfNoCase User-Agent "TinEye" block_bot
SetEnvIfNoCase User-Agent "Baiduspider" block_bot
SetEnvIfNoCase User-Agent "serpstat" block_bot
SetEnvIfNoCase User-Agent "spyfu" block_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=block_bot
</Limit>
# END NiteowebSpiderBlocker
 
The following should be placed before your wordpress redirect code...

# BEGIN NiteowebSpiderBlocker
....stuffs
# END NiteowebSpiderBlocker

as the [L] command in RewriteRule . /index.php [L] tells your server not to process any more of the htaccess rules.

So your complete .htaccess should be as follows....

# BEGIN NiteowebSpiderBlocker
SetEnvIfNoCase User-Agent "AhrefsBot" block_bot
SetEnvIfNoCase User-Agent "MJ12bot" block_bot
SetEnvIfNoCase User-Agent "Rogerbot" block_bot
SetEnvIfNoCase User-Agent "SemrushBot" block_bot
SetEnvIfNoCase User-Agent "ia_archiver" block_bot
SetEnvIfNoCase User-Agent "ScoutJet" block_bot
SetEnvIfNoCase User-Agent "sistrix" block_bot
SetEnvIfNoCase User-Agent "SearchmetricsBot" block_bot
SetEnvIfNoCase User-Agent "SEOkicks-Robot" block_bot
SetEnvIfNoCase User-Agent "Lipperhey Spider" block_bot
SetEnvIfNoCase User-Agent "Exabot" block_bot
SetEnvIfNoCase User-Agent "NCBot" block_bot
SetEnvIfNoCase User-Agent "BacklinkCrawler" block_bot
SetEnvIfNoCase User-Agent "archive.org_bot" block_bot
SetEnvIfNoCase User-Agent "meanpathbot" block_bot
SetEnvIfNoCase User-Agent "PagesInventory" block_bot
SetEnvIfNoCase User-Agent "Aboundexbot" block_bot
SetEnvIfNoCase User-Agent "spbot" block_bot
SetEnvIfNoCase User-Agent "linkdexbot" block_bot
SetEnvIfNoCase User-Agent "Gigabot" block_bot
SetEnvIfNoCase User-Agent "dotbot" block_bot
SetEnvIfNoCase User-Agent "Nutch" block_bot
SetEnvIfNoCase User-Agent "BLEXBot" block_bot
SetEnvIfNoCase User-Agent "Ezooms" block_bot
SetEnvIfNoCase User-Agent "Majestic-12" block_bot
SetEnvIfNoCase User-Agent "Majestic-SEO" block_bot
SetEnvIfNoCase User-Agent "DSearch" block_bot
SetEnvIfNoCase User-Agent "BlekkoBot" block_bot
SetEnvIfNoCase User-Agent "NerdyBot" block_bot
SetEnvIfNoCase User-Agent "JamesBOT" block_bot
SetEnvIfNoCase User-Agent "TinEye" block_bot
SetEnvIfNoCase User-Agent "Baiduspider" block_bot
SetEnvIfNoCase User-Agent "serpstat" block_bot
SetEnvIfNoCase User-Agent "spyfu" block_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=block_bot
</Limit>
# END NiteowebSpiderBlocker

RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteRule ^wp-content/plugins/spiderblocker/index.php /index.php%{REQUEST_URI} [QSA,L]
RewriteRule ^wp-content/plugins/spiderblocker/readme.txt /index.php%{REQUEST_URI} [QSA,L]
RewriteRule ^wp-content/plugins/spiderblocker/ /index.php%{REQUEST_URI} [QSA,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress
 
wow, thanks... i just realized i didn't copy the whole thing so it did have the begin ... so here is how it is current:
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteRule ^wp-content/plugins/spiderblocker/index.php /index.php%{REQUEST_URI} [QSA,L]
RewriteRule ^wp-content/plugins/spiderblocker/readme.txt /index.php%{REQUEST_URI} [QSA,L]
RewriteRule ^wp-content/plugins/spiderblocker/ /index.php%{REQUEST_URI} [QSA,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

# BEGIN NiteowebSpiderBlocker
SetEnvIfNoCase User-Agent "AhrefsBot" block_bot
SetEnvIfNoCase User-Agent "MJ12bot" block_bot
SetEnvIfNoCase User-Agent "Rogerbot" block_bot
SetEnvIfNoCase User-Agent "SemrushBot" block_bot
SetEnvIfNoCase User-Agent "ia_archiver" block_bot
SetEnvIfNoCase User-Agent "ScoutJet" block_bot
SetEnvIfNoCase User-Agent "sistrix" block_bot
SetEnvIfNoCase User-Agent "SearchmetricsBot" block_bot
SetEnvIfNoCase User-Agent "SEOkicks-Robot" block_bot
SetEnvIfNoCase User-Agent "Lipperhey Spider" block_bot
SetEnvIfNoCase User-Agent "Exabot" block_bot
SetEnvIfNoCase User-Agent "NCBot" block_bot
SetEnvIfNoCase User-Agent "BacklinkCrawler" block_bot
SetEnvIfNoCase User-Agent "archive.org_bot" block_bot
SetEnvIfNoCase User-Agent "meanpathbot" block_bot
SetEnvIfNoCase User-Agent "PagesInventory" block_bot
SetEnvIfNoCase User-Agent "Aboundexbot" block_bot
SetEnvIfNoCase User-Agent "spbot" block_bot
SetEnvIfNoCase User-Agent "linkdexbot" block_bot
SetEnvIfNoCase User-Agent "Gigabot" block_bot
SetEnvIfNoCase User-Agent "dotbot" block_bot
SetEnvIfNoCase User-Agent "Nutch" block_bot
SetEnvIfNoCase User-Agent "BLEXBot" block_bot
SetEnvIfNoCase User-Agent "Ezooms" block_bot
SetEnvIfNoCase User-Agent "Majestic-12" block_bot
SetEnvIfNoCase User-Agent "Majestic-SEO" block_bot
SetEnvIfNoCase User-Agent "DSearch" block_bot
SetEnvIfNoCase User-Agent "BlekkoBot" block_bot
SetEnvIfNoCase User-Agent "NerdyBot" block_bot
SetEnvIfNoCase User-Agent "JamesBOT" block_bot
SetEnvIfNoCase User-Agent "TinEye" block_bot
SetEnvIfNoCase User-Agent "Baiduspider" block_bot
SetEnvIfNoCase User-Agent "serpstat" block_bot
SetEnvIfNoCase User-Agent "spyfu" block_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=block_bot
</Limit>
# END NiteowebSpiderBlocker
 
but will change it to your revition with the L

So changing that wont mess up the plugin or anything?

Or I guess I dont need the plugin now?
 
A bit cannot ignore htaccess like it could with robots.txt file, they could change their User-agent but then I don't see much that you could do about that tbh
 
.htaccess is read on the server side so the external request is forced to abide by it.
You're probably thinking of robots.txt which is read on client side and is completely optional to follow.
 
onebluesummer said:
but will change it to your revition with the L

So changing that wont mess up the plugin or anything?

Or I guess I dont need the plugin now?
Click to expand...

Just make sure your spider/bot blocking code is the very first thing in your .htaccess and you should be good to go. Though please keep in mind that every spider and its dog has been able to crawl your site so far, until you make this change, so that domain is now a liability if you planned using it for a PBN. Hopefully, it wasn't too expensive a lesson to learn!

As for the plugin, I'm not familiar with it personally so probably best to just keep it installed in case it updates the bot list in the future.
 
ok so... I can put the following first on any htacces then just put a space and leave the rest of the htaccess alone, and it will block everything

# BEGIN NiteowebSpiderBlocker
SetEnvIfNoCase User-Agent "AhrefsBot" block_bot
SetEnvIfNoCase User-Agent "MJ12bot" block_bot
SetEnvIfNoCase User-Agent "Rogerbot" block_bot
SetEnvIfNoCase User-Agent "SemrushBot" block_bot
SetEnvIfNoCase User-Agent "ia_archiver" block_bot
SetEnvIfNoCase User-Agent "ScoutJet" block_bot
SetEnvIfNoCase User-Agent "sistrix" block_bot
SetEnvIfNoCase User-Agent "SearchmetricsBot" block_bot
SetEnvIfNoCase User-Agent "SEOkicks-Robot" block_bot
SetEnvIfNoCase User-Agent "Lipperhey Spider" block_bot
SetEnvIfNoCase User-Agent "Exabot" block_bot
SetEnvIfNoCase User-Agent "NCBot" block_bot
SetEnvIfNoCase User-Agent "BacklinkCrawler" block_bot
SetEnvIfNoCase User-Agent "archive.org_bot" block_bot
SetEnvIfNoCase User-Agent "meanpathbot" block_bot
SetEnvIfNoCase User-Agent "PagesInventory" block_bot
SetEnvIfNoCase User-Agent "Aboundexbot" block_bot
SetEnvIfNoCase User-Agent "spbot" block_bot
SetEnvIfNoCase User-Agent "linkdexbot" block_bot
SetEnvIfNoCase User-Agent "Gigabot" block_bot
SetEnvIfNoCase User-Agent "dotbot" block_bot
SetEnvIfNoCase User-Agent "Nutch" block_bot
SetEnvIfNoCase User-Agent "BLEXBot" block_bot
SetEnvIfNoCase User-Agent "Ezooms" block_bot
SetEnvIfNoCase User-Agent "Majestic-12" block_bot
SetEnvIfNoCase User-Agent "Majestic-SEO" block_bot
SetEnvIfNoCase User-Agent "DSearch" block_bot
SetEnvIfNoCase User-Agent "BlekkoBot" block_bot
SetEnvIfNoCase User-Agent "NerdyBot" block_bot
SetEnvIfNoCase User-Agent "JamesBOT" block_bot
SetEnvIfNoCase User-Agent "TinEye" block_bot
SetEnvIfNoCase User-Agent "Baiduspider" block_bot
SetEnvIfNoCase User-Agent "serpstat" block_bot
SetEnvIfNoCase User-Agent "spyfu" block_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=block_bot
</Limit>
# END NiteowebSpiderBlocker
 
onebluesummer said:
Anyone else see this happening in the past couple weeks?

I guess I should also say common blocking plugins?
Click to expand...

That is the most stupid to block bots, by checking their user-agent. There are other ways to stop bots such as specific javascript libraries and making your content dynamic. Bots often run on HTTP requests and will avoid runing executing javascript since it slows them down a lot. There are libraries out there and scripts you can add to your site.
 
DupM said:
That is the most stupid to block bots, by checking their user-agent. There are other ways to stop bots such as specific javascript libraries and making your content dynamic. Bots often run on HTTP requests and will avoid runing executing javascript since it slows them down a lot. There are libraries out there and scripts you can add to your site.
Click to expand...

Javascript blocking/rendering is an incredibly poor choice as all it takes is for one of the crawlers to implement javascript rendering to their bot and your backlinks are exposed. Currently, blocking via user-agent remains the most effective.

Please note, recently Ahrefs added javascript rendering to their bot ( https://ahrefs.com/blog/crawling-javascript/ ), so expect other crawlers to do so in time too!
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top