How to stop ChatGPT from Crawling your website?

@tazarbm

Yandex indeed uses alot of resources if you use mass pages on share hosting for example you get easier 500 status codes... , so i blocked those bots on some sites.

There are more than one Yandex bots. If you want to block all Bots then use:

User-agent: Yandex
Disallow: /

If you only want to block the main indexing Bot then ise:

User-agent: YandexBot
Disallow: /

:)
 
Advertise on BHW
Raoul_Duke said:
@tazarbm

Yandex indeed uses alot of resources if you use mass pages on share hosting for example you get easier 500 status codes... , so i blocked those bots on some sites.

There are more than one Yandex bots. If you want to block all Bots then use:

User-agent: Yandex
Disallow: /

If you only want to block the main indexing Bot then ise:

User-agent: YandexBot
Disallow: /

:)
Click to expand...
ok, but doesn't the wildcard (*) already block yandex?? And which bot should I block?
 
yeah with wildcard (*) you speak to every bot. but with your robtos.txt you allow bots to get your content and stuff.

you could block for example ahrefs, semrush and other seo tools if you want. i wouldnt block those search engines where you want to rank.

this could help: https://www.blackhatworld.com/seo/m...the-radar-of-the-competition-to-make.1429804/ a post by sartre

and also a search on the forum will give you other possible robots.txt setups :)
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top