How to block bot and crawlers from visiting your website for free?

T

TFuhrer

Junior Member
Oct 26, 2023
134
47
Hello

Is there any method (a free one) to prevent bots and crawlers from accessing your website?

I've heard that Ngrok can do that with a little of coding but I don't know how to do that. I f anyone here know an effective method to do it please share.

Thank you.
 
Advertise on BHW
You can put this code in your .htaccess file:


Code: 
<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} (Amazonbot|PetalBot|bingbot|DataForSeoBot) [NC]
    RewriteRule ^ - [F]
</IfModule>
 
irinakessler7777 said:
You can put this code in your .htaccess file:


Code: 
<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} (Amazonbot|PetalBot|bingbot|DataForSeoBot) [NC]
    RewriteRule ^ - [F]
</IfModule>
Click to expand...
thank you so muchh does it really prevent all kinds of bots and crawlers from accessing ur site?
 
You can't block crawlers perfectly except for search engine bot.

But there are some ways to make scraping your site difficult.


ex) javascript obfuscator with dev tool blocking, cloudflare JS challenge, recaptcha, referer check, block well-known ISP&VPN ASN etc...
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Networking Downloads

Marketplace

Content / Copywriting Hosting Images, Logos & Videos Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design Misc

Making Money

Affiliate Programs Hire a Freelancer Making Money Pay Per Click Site Flipping

BlackHatWorld

BHW Merch Forum Suggestions & Feedback Introductions Lounge Dispute Resolution

Other

Domain Name Forum IM Journeys Web Hosting Newsletter
Back
Top
AdBlock Detected

We get it, advertisements are annoying!

Sure, ad-blocking software does a great job at blocking ads, but it also blocks useful features and essential functions on BlackHatWorld and other forums. These functions are unrelated to ads, such as internal links and images. For the best site experience please disable your AdBlocker.

I've Disabled AdBlock