How to block bot and crawlers from visiting your website for free?

TFuhrer

Junior Member
Joined
Oct 26, 2023
Messages
116
Reaction score
23
Hello

Is there any method (a free one) to prevent bots and crawlers from accessing your website?

I've heard that Ngrok can do that with a little of coding but I don't know how to do that. I f anyone here know an effective method to do it please share.

Thank you.
 
You can put this code in your .htaccess file:


Code:
<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} (Amazonbot|PetalBot|bingbot|DataForSeoBot) [NC]
    RewriteRule ^ - [F]
</IfModule>
 
You can put this code in your .htaccess file:


Code:
<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} (Amazonbot|PetalBot|bingbot|DataForSeoBot) [NC]
    RewriteRule ^ - [F]
</IfModule>
thank you so muchh does it really prevent all kinds of bots and crawlers from accessing ur site?
 
You can't block crawlers perfectly except for search engine bot.

But there are some ways to make scraping your site difficult.


ex) javascript obfuscator with dev tool blocking, cloudflare JS challenge, recaptcha, referer check, block well-known ISP&VPN ASN etc...
 
Back
Top
AdBlock Detected

We get it, advertisements are annoying!

Sure, ad-blocking software does a great job at blocking ads, but it also blocks useful features and essential functions on BlackHatWorld and other forums. These functions are unrelated to ads, such as internal links and images. For the best site experience please disable your AdBlocker.

I've Disabled AdBlock