[Question] How to find out that which crawlers visited my website?

4ry4n

4ry4n

Power Member
Joined
Dec 25, 2021
Messages
650
Reaction score
450
Greetings BHW community,
I hope that you all are doing great!

A question or actually two if I may:

1- How to find out which crawler bots visit my website? Like crawlers from google, bing, yahoo and...

2- Have you ever experienced slow website caused by too many active crawlers? Can my host CPU be caught up by all the non-wanted crawlers?


Thank you all,
 
Advertise on BHW
A plugin? I don't know, almost every thing has a plug in or you can check server-side logs.
 
Castamir said:
A plugin? I don't know, almost every thing has a plug in or you can check server-side logs.
Click to expand...
I thought about server logs, I am not professional enough to analyze that kind of data.
 
4ry4n said:
I thought about server logs, I am not professional enough to analyze that kind of data.
Click to expand...
yep, the server logs will show you. And it's not hard, either, Awstats is probably the best log analyzer. Just open it up for the domain you're interested in and I can bet that you'll be able to read it within minutes :)
 
tazarbm said:
yep, the server logs will show you. And it's not hard, either, Awstats is probably the best log analyzer. Just open it up for the domain you're interested in and I can bet that you'll be able to read it within minutes :)
Click to expand...
Thanks I always appreciate your professional input!
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top