dee_emm_tee
Junior Member
- Oct 6, 2016
- 129
- 53
When blocking crawlers from PBNs most people blacklist specific crawlers in the .htaccess file. This is a perfectly valid way of blocking unwanted crawlers from your sites, but it can leave you vulnerable. Here’s why:
There are many small, lesser known backlink tools out there, and unless you can determine their user-agent and block it, there is a chance they will crawl your PBN. Not only this, but every time there is a new backlink tool with an unkown user agent you risk your competitors discovering your PBN.
Rather than allowing all UA’s by default and only blacklisting certain ones, my solution to this is to disallow all UAs and only whitelist the one’s I want to be able to access my site. I haven’t seen this type of .htaccess code shared here before, so I figured I’d post it:
Putting this code in your .htaccess file will block all user agents your from your site by default, and only allow those user agents designated as “good_agents”.
This means that all existing backlink crawlers with unknown user agents as well as any new ones in the future will all be given an error 403 when they try to crawl your PBN.
Of course, User Agents are easy to spoof, so it is still possible that some backlink crawlers could crawl your site while masquerading as the Googlebot, but whatcha gonna do.
If the code above causes a 500 internal server error, remove this line: “Order Deny, allow”.
If anyone has any suggestions/contributions to this code I’d love to hear them!
There are many small, lesser known backlink tools out there, and unless you can determine their user-agent and block it, there is a chance they will crawl your PBN. Not only this, but every time there is a new backlink tool with an unkown user agent you risk your competitors discovering your PBN.
Rather than allowing all UA’s by default and only blacklisting certain ones, my solution to this is to disallow all UAs and only whitelist the one’s I want to be able to access my site. I haven’t seen this type of .htaccess code shared here before, so I figured I’d post it:
Code:
SetEnvIfNoCase User-Agent .*Googlebot.* good_agent
SetEnvIfNoCase User-Agent .*MSIE.* good_bot
SetEnvIfNoCase User-Agent .*Safari mobile.* good_agent
SetEnvIfNoCase User-Agent .*Safari.* good_agent
SetEnvIfNoCase User-Agent .*Firefox.* good_agent
SetEnvIfNoCase User-Agent .*Firefox (BonEcho).* good_agent
SetEnvIfNoCase User-Agent .*Firefox (GranParadiso).* good_agent
SetEnvIfNoCase User-Agent .*Firefox (Lorentz).* good_agent
SetEnvIfNoCase User-Agent .*Firefox (Minefield).* good_agent
SetEnvIfNoCase User-Agent .*Firefox (Namoroka).* good_agent
SetEnvIfNoCase User-Agent .*Firefox (Shiretoko).* good_agent
SetEnvIfNoCase User-Agent .*Firefox mobile.* good_agent
SetEnvIfNoCase User-Agent .*DuckDuckGo Mobile.* good_agent
SetEnvIfNoCase User-Agent .*Chrome Mobile.* good_agent
SetEnvIfNoCase User-Agent .*Chrome.* good_agent
SetEnvIfNoCase User-Agent .*Opera.* good_agent
SetEnvIfNoCase User-Agent .*Opera Mini.* good_agent
SetEnvIfNoCase User-Agent .*Opera Mobile.* good_agent
SetEnvIfNoCase User-Agent .*Yahoo.* good_agent
SetEnvIfNoCase User-Agent .*bingbot.* good_agent
SetEnvIfNoCase User-Agent .*DuckDuckBot.* good_agent
SetEnvIfNoCase User-Agent .*UC Browser.* good_agent
SetEnvIfNoCase User-Agent .*UC Browser mobile.* good_agent
SetEnvIfNoCase User-Agent .*Iron.* good_agent
SetEnvIfNoCase User-Agent .*Iron mobile.* good_agent
SetEnvIfNoCase User-Agent .*Android browser.* good_agent
<Limit GET POST HEAD>
Order Deny, allow
Deny from all
Allow from env=good_agent
</Limit>Putting this code in your .htaccess file will block all user agents your from your site by default, and only allow those user agents designated as “good_agents”.
This means that all existing backlink crawlers with unknown user agents as well as any new ones in the future will all be given an error 403 when they try to crawl your PBN.
Of course, User Agents are easy to spoof, so it is still possible that some backlink crawlers could crawl your site while masquerading as the Googlebot, but whatcha gonna do.
If the code above causes a 500 internal server error, remove this line: “Order Deny, allow”.
If anyone has any suggestions/contributions to this code I’d love to hear them!
Last edited:
