how to find search engine ip`s

D

devaeclectic

Newbie
Joined
Mar 30, 2010
Messages
9
Reaction score
0
Hey Guys,

We would like to collect the bot ip`s and UA`s of search engines which has crawled our sites, check and add new ip`s and UA`s to the existing list. If new bot is found then it has to be informed thru alerts may be via email. Please help us on this. Thank you in Advance!
 
Advertise on BHW
Setup a honey pot or "spider trap".

Create a link on your site that's invisible to humans but crawlable by spiders and webbots. Link to a php file that, when executed, places the client's useragent and ip into a mysql database. The php file will also send an email with that information using sendmail.

Problem solved :cool:

Now get coding...
 
old but ok: hxxp://www.iplists.com
 
Last edited:
I had the fantomaster IP list... any reason to go on with it? Or is that stuff on iplist.com halfway up-to-date?
 
You can use reverse DNS to lookup and add all the IP to DB
 
Grizzy said:
Setup a honey pot or "spider trap".

Create a link on your site that's invisible to humans but crawlable by spiders and webbots. Link to a php file that, when executed, places the client's useragent and ip into a mysql database. The php file will also send an email with that information using sendmail.

Problem solved :cool:

Now get coding...
Click to expand...

Did you do it? How well does it compared to fantomaster's ip list? They are presumably doing something similar, but still: how long to wait until your list is somehow up-to-date?

To be sure, that a bot was trapped, you would maybe also have to cross-check against another ip-base. And you would have to provide a domain which already high pr-rank to ensure, that links are followed and a full crawl takes place. Could be an interesting project for the BHW community.
 
Last edited:
cooooookies said:
I had the fantomaster IP list... any reason to go on with it? Or is that stuff on iplist.com halfway up-to-date?
Click to expand...

It's not bad but there are a few missing.

Bear in mind, there will never be a complete list.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top