How I fight back the crawlers!? 2Q

[zelosn]

I have a website I need to fight those crawlers such as Ashraf, SEMrush, bing, google, etc. I don't want my site to index at all. Is there a way to do that?
I have an adlut I want to send some traffic from it to a regular site.

The other question is there a Banner show how much impression my site gets with the countries' visitors in the banner so if one wants to buy the banner, know the details in the banner.

 

[SunnyLeon]

You can block the known crawlers. There are a lot of lists of crawlers on google for free.. you can block them in robots.txt ..

There are widgets that show third party analytics, but I wouldn't give out all that info on public access.. That's why most people make a simple "buy this ad space" banner and send the analytics in private conversations.. To keep the website info secret

 

[dvlhr]

You should search robots.txt

 

[zelosn]

Some crawlers don't respect anything; that's why I need to ban them.

Thank you for answering <3

 

[SunnyLeon]

Some crawlers don't respect anything; that's why I need to ban them.

Thank you for answering <3

You can use plugins like wordfence if you're using wordpress to block specific useragents. I think cloudflare also has some options for that. Ultimately you can go server side and ban crawlers from there, but it is harder and you must have access to the server features

 

[martyparty]

Check stop bad bots plugin if you are using WP

 

[MikeBurics873]

block the known crawlers.

 

[vigyavan]

If you are using cloudflare CDN then use this expression in firewall to block bad bots

(http.user_agent contains "Yandex") or (http.user_agent contains "muckrack") or (http.user_agent contains "Qwantify") or (http.user_agent contains "Sogou") or (http.user_agent contains "BUbiNG") or (http.user_agent contains "knowledge") or (http.user_agent contains "CFNetwork") or (http.user_agent contains "Scrapy") or (http.user_agent contains "SemrushBot") or (http.user_agent contains "AhrefsBot") or (http.user_agent contains "Baiduspider") or (http.user_agent contains "python-requests") or (http.user_agent contains "crawl" and not cf.client.bot) or (http.user_agent contains "Crawl" and not cf.client.bot) or (http.user_agent contains "bot" and not http.user_agent contains "bingbot" and not http.user_agent contains "Google" and not http.user_agent contains "Twitter" and not cf.client.bot) or (http.user_agent contains "Bot" and not http.user_agent contains "Google" and not cf.client.bot) or (http.user_agent contains "Spider" and not cf.client.bot) or (http.user_agent contains "spider" and not cf.client.bot)