are badbots ignoring htacces ?

Anyone else see this happening in the past couple weeks?

I guess I should also say common blocking plugins?

RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteRule ^wp-content/plugins/spiderblocker/index.php /index.php%{REQUEST_URI} [QSA,L]
RewriteRule ^wp-content/plugins/spiderblocker/readme.txt /index.php%{REQUEST_URI} [QSA,L]
RewriteRule ^wp-content/plugins/spiderblocker/ /index.php%{REQUEST_URI} [QSA,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

# BEGIN NiteowebSpiderBlocker
SetEnvIfNoCase User-Agent "AhrefsBot" block_bot
SetEnvIfNoCase User-Agent "MJ12bot" block_bot
SetEnvIfNoCase User-Agent "Rogerbot" block_bot
SetEnvIfNoCase User-Agent "SemrushBot" block_bot
SetEnvIfNoCase User-Agent "ia_archiver" block_bot
SetEnvIfNoCase User-Agent "ScoutJet" block_bot
SetEnvIfNoCase User-Agent "sistrix" block_bot
SetEnvIfNoCase User-Agent "SearchmetricsBot" block_bot
SetEnvIfNoCase User-Agent "SEOkicks-Robot" block_bot
SetEnvIfNoCase User-Agent "Lipperhey Spider" block_bot
SetEnvIfNoCase User-Agent "Exabot" block_bot
SetEnvIfNoCase User-Agent "NCBot" block_bot
SetEnvIfNoCase User-Agent "BacklinkCrawler" block_bot
SetEnvIfNoCase User-Agent "archive.org_bot" block_bot
SetEnvIfNoCase User-Agent "meanpathbot" block_bot
SetEnvIfNoCase User-Agent "PagesInventory" block_bot
SetEnvIfNoCase User-Agent "Aboundexbot" block_bot
SetEnvIfNoCase User-Agent "spbot" block_bot
SetEnvIfNoCase User-Agent "linkdexbot" block_bot
SetEnvIfNoCase User-Agent "Gigabot" block_bot
SetEnvIfNoCase User-Agent "dotbot" block_bot
SetEnvIfNoCase User-Agent "Nutch" block_bot
SetEnvIfNoCase User-Agent "BLEXBot" block_bot
SetEnvIfNoCase User-Agent "Ezooms" block_bot
SetEnvIfNoCase User-Agent "Majestic-12" block_bot
SetEnvIfNoCase User-Agent "Majestic-SEO" block_bot
SetEnvIfNoCase User-Agent "DSearch" block_bot
SetEnvIfNoCase User-Agent "BlekkoBot" block_bot
SetEnvIfNoCase User-Agent "NerdyBot" block_bot
SetEnvIfNoCase User-Agent "JamesBOT" block_bot
SetEnvIfNoCase User-Agent "TinEye" block_bot
SetEnvIfNoCase User-Agent "Baiduspider" block_bot
SetEnvIfNoCase User-Agent "serpstat" block_bot
SetEnvIfNoCase User-Agent "spyfu" block_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=block_bot
</Limit>
# END NiteowebSpiderBlocker

ahrefs and majestic

haven't checked the others

The following should be placed before your wordpress redirect code...

# BEGIN NiteowebSpiderBlocker
....stuffs
# END NiteowebSpiderBlocker

as the [L] command in RewriteRule . /index.php [L] tells your server not to process any more of the htaccess rules.

So your complete .htaccess should be as follows....

# BEGIN NiteowebSpiderBlocker
SetEnvIfNoCase User-Agent "AhrefsBot" block_bot
SetEnvIfNoCase User-Agent "MJ12bot" block_bot
SetEnvIfNoCase User-Agent "Rogerbot" block_bot
SetEnvIfNoCase User-Agent "SemrushBot" block_bot
SetEnvIfNoCase User-Agent "ia_archiver" block_bot
SetEnvIfNoCase User-Agent "ScoutJet" block_bot
SetEnvIfNoCase User-Agent "sistrix" block_bot
SetEnvIfNoCase User-Agent "SearchmetricsBot" block_bot
SetEnvIfNoCase User-Agent "SEOkicks-Robot" block_bot
SetEnvIfNoCase User-Agent "Lipperhey Spider" block_bot
SetEnvIfNoCase User-Agent "Exabot" block_bot
SetEnvIfNoCase User-Agent "NCBot" block_bot
SetEnvIfNoCase User-Agent "BacklinkCrawler" block_bot
SetEnvIfNoCase User-Agent "archive.org_bot" block_bot
SetEnvIfNoCase User-Agent "meanpathbot" block_bot
SetEnvIfNoCase User-Agent "PagesInventory" block_bot
SetEnvIfNoCase User-Agent "Aboundexbot" block_bot
SetEnvIfNoCase User-Agent "spbot" block_bot
SetEnvIfNoCase User-Agent "linkdexbot" block_bot
SetEnvIfNoCase User-Agent "Gigabot" block_bot
SetEnvIfNoCase User-Agent "dotbot" block_bot
SetEnvIfNoCase User-Agent "Nutch" block_bot
SetEnvIfNoCase User-Agent "BLEXBot" block_bot
SetEnvIfNoCase User-Agent "Ezooms" block_bot
SetEnvIfNoCase User-Agent "Majestic-12" block_bot
SetEnvIfNoCase User-Agent "Majestic-SEO" block_bot
SetEnvIfNoCase User-Agent "DSearch" block_bot
SetEnvIfNoCase User-Agent "BlekkoBot" block_bot
SetEnvIfNoCase User-Agent "NerdyBot" block_bot
SetEnvIfNoCase User-Agent "JamesBOT" block_bot
SetEnvIfNoCase User-Agent "TinEye" block_bot
SetEnvIfNoCase User-Agent "Baiduspider" block_bot
SetEnvIfNoCase User-Agent "serpstat" block_bot
SetEnvIfNoCase User-Agent "spyfu" block_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=block_bot
</Limit>
# END NiteowebSpiderBlocker

RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteRule ^wp-content/plugins/spiderblocker/index.php /index.php%{REQUEST_URI} [QSA,L]
RewriteRule ^wp-content/plugins/spiderblocker/readme.txt /index.php%{REQUEST_URI} [QSA,L]
RewriteRule ^wp-content/plugins/spiderblocker/ /index.php%{REQUEST_URI} [QSA,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

wow, thanks... i just realized i didn't copy the whole thing so it did have the begin ... so here is how it is current:
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteRule ^wp-content/plugins/spiderblocker/index.php /index.php%{REQUEST_URI} [QSA,L]
RewriteRule ^wp-content/plugins/spiderblocker/readme.txt /index.php%{REQUEST_URI} [QSA,L]
RewriteRule ^wp-content/plugins/spiderblocker/ /index.php%{REQUEST_URI} [QSA,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

# BEGIN NiteowebSpiderBlocker
SetEnvIfNoCase User-Agent "AhrefsBot" block_bot
SetEnvIfNoCase User-Agent "MJ12bot" block_bot
SetEnvIfNoCase User-Agent "Rogerbot" block_bot
SetEnvIfNoCase User-Agent "SemrushBot" block_bot
SetEnvIfNoCase User-Agent "ia_archiver" block_bot
SetEnvIfNoCase User-Agent "ScoutJet" block_bot
SetEnvIfNoCase User-Agent "sistrix" block_bot
SetEnvIfNoCase User-Agent "SearchmetricsBot" block_bot
SetEnvIfNoCase User-Agent "SEOkicks-Robot" block_bot
SetEnvIfNoCase User-Agent "Lipperhey Spider" block_bot
SetEnvIfNoCase User-Agent "Exabot" block_bot
SetEnvIfNoCase User-Agent "NCBot" block_bot
SetEnvIfNoCase User-Agent "BacklinkCrawler" block_bot
SetEnvIfNoCase User-Agent "archive.org_bot" block_bot
SetEnvIfNoCase User-Agent "meanpathbot" block_bot
SetEnvIfNoCase User-Agent "PagesInventory" block_bot
SetEnvIfNoCase User-Agent "Aboundexbot" block_bot
SetEnvIfNoCase User-Agent "spbot" block_bot
SetEnvIfNoCase User-Agent "linkdexbot" block_bot
SetEnvIfNoCase User-Agent "Gigabot" block_bot
SetEnvIfNoCase User-Agent "dotbot" block_bot
SetEnvIfNoCase User-Agent "Nutch" block_bot
SetEnvIfNoCase User-Agent "BLEXBot" block_bot
SetEnvIfNoCase User-Agent "Ezooms" block_bot
SetEnvIfNoCase User-Agent "Majestic-12" block_bot
SetEnvIfNoCase User-Agent "Majestic-SEO" block_bot
SetEnvIfNoCase User-Agent "DSearch" block_bot
SetEnvIfNoCase User-Agent "BlekkoBot" block_bot
SetEnvIfNoCase User-Agent "NerdyBot" block_bot
SetEnvIfNoCase User-Agent "JamesBOT" block_bot
SetEnvIfNoCase User-Agent "TinEye" block_bot
SetEnvIfNoCase User-Agent "Baiduspider" block_bot
SetEnvIfNoCase User-Agent "serpstat" block_bot
SetEnvIfNoCase User-Agent "spyfu" block_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=block_bot
</Limit>
# END NiteowebSpiderBlocker

but will change it to your revition with the L

So changing that wont mess up the plugin or anything?

Or I guess I dont need the plugin now?

A bit cannot ignore htaccess like it could with robots.txt file, they could change their User-agent but then I don't see much that you could do about that tbh

A bot*

.htaccess is read on the server side so the external request is forced to abide by it.
You're probably thinking of robots.txt which is read on client side and is completely optional to follow.

you can block everything from htacess

onebluesummer said:

but will change it to your revition with the L

So changing that wont mess up the plugin or anything?

Or I guess I dont need the plugin now?

Click to expand...

Just make sure your spider/bot blocking code is the very first thing in your .htaccess and you should be good to go. Though please keep in mind that every spider and its dog has been able to crawl your site so far, until you make this change, so that domain is now a liability if you planned using it for a PBN. Hopefully, it wasn't too expensive a lesson to learn!

As for the plugin, I'm not familiar with it personally so probably best to just keep it installed in case it updates the bot list in the future.

ok so... I can put the following first on any htacces then just put a space and leave the rest of the htaccess alone, and it will block everything

# BEGIN NiteowebSpiderBlocker
SetEnvIfNoCase User-Agent "AhrefsBot" block_bot
SetEnvIfNoCase User-Agent "MJ12bot" block_bot
SetEnvIfNoCase User-Agent "Rogerbot" block_bot
SetEnvIfNoCase User-Agent "SemrushBot" block_bot
SetEnvIfNoCase User-Agent "ia_archiver" block_bot
SetEnvIfNoCase User-Agent "ScoutJet" block_bot
SetEnvIfNoCase User-Agent "sistrix" block_bot
SetEnvIfNoCase User-Agent "SearchmetricsBot" block_bot
SetEnvIfNoCase User-Agent "SEOkicks-Robot" block_bot
SetEnvIfNoCase User-Agent "Lipperhey Spider" block_bot
SetEnvIfNoCase User-Agent "Exabot" block_bot
SetEnvIfNoCase User-Agent "NCBot" block_bot
SetEnvIfNoCase User-Agent "BacklinkCrawler" block_bot
SetEnvIfNoCase User-Agent "archive.org_bot" block_bot
SetEnvIfNoCase User-Agent "meanpathbot" block_bot
SetEnvIfNoCase User-Agent "PagesInventory" block_bot
SetEnvIfNoCase User-Agent "Aboundexbot" block_bot
SetEnvIfNoCase User-Agent "spbot" block_bot
SetEnvIfNoCase User-Agent "linkdexbot" block_bot
SetEnvIfNoCase User-Agent "Gigabot" block_bot
SetEnvIfNoCase User-Agent "dotbot" block_bot
SetEnvIfNoCase User-Agent "Nutch" block_bot
SetEnvIfNoCase User-Agent "BLEXBot" block_bot
SetEnvIfNoCase User-Agent "Ezooms" block_bot
SetEnvIfNoCase User-Agent "Majestic-12" block_bot
SetEnvIfNoCase User-Agent "Majestic-SEO" block_bot
SetEnvIfNoCase User-Agent "DSearch" block_bot
SetEnvIfNoCase User-Agent "BlekkoBot" block_bot
SetEnvIfNoCase User-Agent "NerdyBot" block_bot
SetEnvIfNoCase User-Agent "JamesBOT" block_bot
SetEnvIfNoCase User-Agent "TinEye" block_bot
SetEnvIfNoCase User-Agent "Baiduspider" block_bot
SetEnvIfNoCase User-Agent "serpstat" block_bot
SetEnvIfNoCase User-Agent "spyfu" block_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=block_bot
</Limit>
# END NiteowebSpiderBlocker

onebluesummer said:

Anyone else see this happening in the past couple weeks?

I guess I should also say common blocking plugins?

Click to expand...

That is the most stupid to block bots, by checking their user-agent. There are other ways to stop bots such as specific javascript libraries and making your content dynamic. Bots often run on HTTP requests and will avoid runing executing javascript since it slows them down a lot. There are libraries out there and scripts you can add to your site.

onebluesummer said:

ok so... I can put the following first on any htacces then just put a space and leave the rest of the htaccess alone, and it will block everything

Click to expand...

Correct

DupM said:

That is the most stupid to block bots, by checking their user-agent. There are other ways to stop bots such as specific javascript libraries and making your content dynamic. Bots often run on HTTP requests and will avoid runing executing javascript since it slows them down a lot. There are libraries out there and scripts you can add to your site.

Click to expand...

Javascript blocking/rendering is an incredibly poor choice as all it takes is for one of the crawlers to implement javascript rendering to their bot and your backlinks are exposed. Currently, blocking via user-agent remains the most effective.

Please note, recently Ahrefs added javascript rendering to their bot ( https://ahrefs.com/blog/crawling-javascript/ ), so expect other crawlers to do so in time too!