how can Knew the bad boot

goolfx

Newbie
Joined
Oct 15, 2021
Messages
3
Reaction score
0
hello
The average visitor stay on my site has decreased from 2 minutes to 8 seconds, what should I do?
this is a boot list
what should I block and how, please help me

0​
64 different robots
MJ12bot
Googlebot
rogerbot
Unknown robot identified by bot\*
bingbot
AhrefsBot
SemrushBot
oBot
feed
Curl
Sogou web spider
BLEXBot
A PHP script
Screaming Frog SEO Spider
Applebot
LinkpadBot
YandexBot
Feedfetcher-Google
CCBot
facebookexternalhit
Mediapartners-Google
crawl
link
SeznamBot
empty user agent string
robot
Firefox version 10 and lower - various robots
Baidu ( catchall )
TweetmemeBot
Googlebot-Image
AdsBot-Google
BingPreview
360Spider
Mail.RU Bot
archive.org_bot
Unknown robot (identified by hit on robots.txt)
DotBot
SurdotlyBot
YisouSpider
Incutio XML
LinkedInBot
DuckDuckGo-Favicons-Bot
Apache-HttpClient
PHP/5.2.8
Google-Adwords-Instant
discover
PhantomJS
MojeekBot
Baiduspider
GrapeshotCrawler
Wget
linkdexbot
SafeDNSBot
CFNetwork
yacybot
spider
nutch ( catchall )
checker
semanticbot
fetcher
Konqueror
netEstate NE Crawler
Java ( catchall )
nbot
 
Most of these bots firstly look into the robots.txt file, and if you tell them not to scrape your website they won't.

But most of these bots are good, they add your site to search results, etc...

For bad bots protection aka custom made scripts that scrape website content,
Cloudflare paid plan, Datadome, etc ...

But this is a cat and mouse game because scrapper writers are finding ways to bypass these protections.

100% way to disable scrappers is to:

#1 Disable your site for all visitors :)

#2 Add some sort of registration or paywall and kick accounts that have red flags: too many pages visit per day,
on multiple pages leaving them in a short time, etc... But this also can be bypassed with a list of accounts, a list of proxies, and
if each bot behaves more human-like without triggering any flags.

As I said cat and mouse game. Prevention is possible to a certain level.
 
Back
Top