how to find search engine ip`s

[devaeclectic]

Hey Guys,

We would like to collect the bot ip`s and UA`s of search engines which has crawled our sites, check and add new ip`s and UA`s to the existing list. If new bot is found then it has to be informed thru alerts may be via email. Please help us on this. Thank you in Advance!

 

[Grizzy]

Setup a honey pot or "spider trap".

Create a link on your site that's invisible to humans but crawlable by spiders and webbots. Link to a php file that, when executed, places the client's useragent and ip into a mysql database. The php file will also send an email with that information using sendmail.

Problem solved

Now get coding...

 

[russaus]

old but ok: hxxp://www.iplists.com

 

[cooooookies]

I had the fantomaster IP list... any reason to go on with it? Or is that stuff on iplist.com halfway up-to-date?

 

[dwpg002]

You can use reverse DNS to lookup and add all the IP to DB

 

[cooooookies]

Setup a honey pot or "spider trap".

Create a link on your site that's invisible to humans but crawlable by spiders and webbots. Link to a php file that, when executed, places the client's useragent and ip into a mysql database. The php file will also send an email with that information using sendmail.

Problem solved

Now get coding...

Did you do it? How well does it compared to fantomaster's ip list? They are presumably doing something similar, but still: how long to wait until your list is somehow up-to-date?

To be sure, that a bot was trapped, you would maybe also have to cross-check against another ip-base. And you would have to provide a domain which already high pr-rank to ensure, that links are followed and a full crawl takes place. Could be an interesting project for the BHW community.

 

[russaus]

I had the fantomaster IP list... any reason to go on with it? Or is that stuff on iplist.com halfway up-to-date?

It's not bad but there are a few missing.

Bear in mind, there will never be a complete list.