1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

how to find search engine ip`s

Discussion in 'Cloaking and Content Generators' started by devaeclectic, Mar 30, 2010.

  1. devaeclectic

    devaeclectic Newbie

    Joined:
    Mar 30, 2010
    Messages:
    9
    Likes Received:
    0
    Hey Guys,

    We would like to collect the bot ip`s and UA`s of search engines which has crawled our sites, check and add new ip`s and UA`s to the existing list. If new bot is found then it has to be informed thru alerts may be via email. Please help us on this. Thank you in Advance!
     
  2. Grizzy

    Grizzy Senior Member

    Joined:
    Nov 11, 2008
    Messages:
    919
    Likes Received:
    999
    Setup a honey pot or "spider trap".

    Create a link on your site that's invisible to humans but crawlable by spiders and webbots. Link to a php file that, when executed, places the client's useragent and ip into a mysql database. The php file will also send an email with that information using sendmail.

    Problem solved :cool:

    Now get coding...
     
    • Thanks Thanks x 2
  3. russaus

    russaus Junior Member

    Joined:
    Mar 5, 2010
    Messages:
    164
    Likes Received:
    17
    Occupation:
    software engineer
    Location:
    /dev/null & canada
    old but ok: hxxp://www.iplists.com
     
    Last edited: Apr 1, 2010
  4. cooooookies

    cooooookies Senior Member

    Joined:
    Oct 6, 2008
    Messages:
    1,008
    Likes Received:
    216
    I had the fantomaster IP list... any reason to go on with it? Or is that stuff on iplist.com halfway up-to-date?
     
  5. dwpg002

    dwpg002 Senior Member

    Joined:
    Dec 29, 2008
    Messages:
    919
    Likes Received:
    47
    You can use reverse DNS to lookup and add all the IP to DB
     
  6. cooooookies

    cooooookies Senior Member

    Joined:
    Oct 6, 2008
    Messages:
    1,008
    Likes Received:
    216
    Did you do it? How well does it compared to fantomaster's ip list? They are presumably doing something similar, but still: how long to wait until your list is somehow up-to-date?

    To be sure, that a bot was trapped, you would maybe also have to cross-check against another ip-base. And you would have to provide a domain which already high pr-rank to ensure, that links are followed and a full crawl takes place. Could be an interesting project for the BHW community.
     
    Last edited: Apr 5, 2010
  7. russaus

    russaus Junior Member

    Joined:
    Mar 5, 2010
    Messages:
    164
    Likes Received:
    17
    Occupation:
    software engineer
    Location:
    /dev/null & canada
    It's not bad but there are a few missing.

    Bear in mind, there will never be a complete list.