1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

whois scraper/harvester (?)

Discussion in 'Black Hat SEO Tools' started by dre2027, Nov 14, 2009.

  1. dre2027

    dre2027 Newbie

    Joined:
    Jul 21, 2008
    Messages:
    13
    Likes Received:
    0
    I've been looking for a good whois scraper/harvester.

    For a start, I need something that can check 10k specific domains w/o getting blocked every 8-15 records. Or at least can change its ip address after every 5-10 records pulled.

    hidemyass.c0m/vpn/ can change my ip every xx minutes. That's as close as i've come to finding a way to do this. But still, xx minutes isn't good enough. I still get blocked. I need to change more frequently than by xx minutes. I need to change by the # of records or cycle through multiple whois systems(?).

    I may be going about this all wrong.

    But if not, do you have any idea of how to go about doing whois lookups w/o getting banned/blocked every handful of search results? Or know of an all-in-one bh tool that can do the harvesting for me en masse?

    Oh. Eventually I will need to lookup 250k+ specific domains. This initial 10k test group will determine if is worth the effort or not to continue with the idea.

    Thanks
     
  2. howdoyou

    howdoyou Regular Member

    Joined:
    Nov 17, 2008
    Messages:
    284
    Likes Received:
    57
    Occupation:
    Programming
    Location:
    Kentucky
    I can program you a small application that will do this. PM me if your interested.
     
  3. dre2027

    dre2027 Newbie

    Joined:
    Jul 21, 2008
    Messages:
    13
    Likes Received:
    0
    thanks. Pm sent.
     
  4. doctorwar

    doctorwar Registered Member

    Joined:
    May 25, 2009
    Messages:
    61
    Likes Received:
    28
    • Thanks Thanks x 1
  5. howdoyou

    howdoyou Regular Member

    Joined:
    Nov 17, 2008
    Messages:
    284
    Likes Received:
    57
    Occupation:
    Programming
    Location:
    Kentucky
    The software works now, i just need a few details from you.
    Like how you want the list format to be.
    it changes to a new proxy built in every search so there is no chance of being locked out.

    If the one posted above works thats fine, just let me know what you want me to do - if you still want the one i coded let me know, and any additional features you want added.
     
    • Thanks Thanks x 1
    Last edited: Nov 16, 2009
  6. dre2027

    dre2027 Newbie

    Joined:
    Jul 21, 2008
    Messages:
    13
    Likes Received:
    0
    Hey doctorwar , Just realized i had grabbed your whois scraper from the other thread a bit ago (blackhatworld.c0m/blackhat-seo/member-downloads/124531-get-onlinemarketresults-whois-email-scraper-v1.html#post1168895). It gets blocked too, at least for me :(

    New version would be killer if it could

    1) work with list of proxy addresses instead of a single proxy address and

    2) could rotate ip address every 4 or so requests, since 4 requests per minute per ip seems to be the max allowed for most registrars.

    Rotating the ip every third or fourth request would bypass the 4 per minute limitation. Could then grab whois records as fast as your connection speed and pc could handle. Would work provided the ip changes every 3-4 requests or cycles in a way that the same ip is never used more than 3-4 times in any given minute.

    That would be a killer whois scraper!!!
     
  7. dre2027

    dre2027 Newbie

    Joined:
    Jul 21, 2008
    Messages:
    13
    Likes Received:
    0
    Hey doctorwar, just realized i had grabbed your whois scraper from the other thread a bit ago (blackhatworld.c0m/blackhat-seo/member-downloads/124531-get-onlinemarketresults-whois-email-scraper-v1.html#post1168895). It gets blocked too, at least for me :(

    New version would be killer if it could

    1) work with list of proxy addresses instead of a single proxy address and

    2) could rotate ip address every 4 or so requests, since 4 requests per minute per ip seems to be the max allowed for most registrars.

    Rotating the ip every third or fourth request would bypass the 4 per minute limitation. Could then grab whois records as fast as your connection speed and pc could handle. Would work provided the ip changes every 3-4 requests or cycles in a way that the same ip is never used more than 3-4 times in any given minute.

    That would be a killer whois scraper!!!
     
  8. dre2027

    dre2027 Newbie

    Joined:
    Jul 21, 2008
    Messages:
    13
    Likes Received:
    0
    Earlier I wrote:

    "hidemyass.c0m/vpn/ can change my ip every xx minutes. That's as close as i've come to finding a way to do this. But still, xx minutes isn't good enough. I still get blocked."

    I meant to type "I'd still get blocked" instead of "I".
     
  9. doctorwar

    doctorwar Registered Member

    Joined:
    May 25, 2009
    Messages:
    61
    Likes Received:
    28
    When I get a few free minutes I can update this whois scraper to use proxy lists with delays/rotation
     
  10. badd

    badd Regular Member

    Joined:
    Jan 22, 2008
    Messages:
    360
    Likes Received:
    51
    Occupation:
    Innovator
    Location:
    Earth
    i can put u in the right direction if u r still looking for this tupe of bot
     
  11. doctorwar

    doctorwar Registered Member

    Joined:
    May 25, 2009
    Messages:
    61
    Likes Received:
    28
    In the mean time, what about using proxyfirewall to rotate proxies?
     
  12. dre2027

    dre2027 Newbie

    Joined:
    Jul 21, 2008
    Messages:
    13
    Likes Received:
    0
    Seems something more custom, more specific to whois searches is needed [like your software and howdoyou's software].

    On my end i'm finding registrars limiting searches to 4 per minute before temporarily banning or delaying my ip. So maybe cycling connections not by the minute but by the number of whois requests will get past autobanning.

    Also, I can't find a setting to make proxyfirewall or hidemy@$$ cycle per # of requests. Too bad. Because I want access to that 'free' whois data!!! lol

    I thought of maybe requesting the data direct from registrars so i could setup a new search engine but really with the goal of 'hacking' my own database remotely so i could claim someone else did it -hehe - but I bet i'd unknowingly leave a trail and still get caught. Oh well. Gotta do things legit...by scraping.

    Maybe i'm overlooking something(?).
     
    Last edited: Nov 18, 2009
  13. yazzou

    yazzou Registered Member

    Joined:
    Dec 19, 2008
    Messages:
    61
    Likes Received:
    7
    Interesting
     
  14. dre2027

    dre2027 Newbie

    Joined:
    Jul 21, 2008
    Messages:
    13
    Likes Received:
    0
    d@mn! Not finding a scraper that can avoid getting banned every fourth request is costing me millions in possible opportunity losses. Too bad I'm too dang broke to do anything about it - lol

    Maybe i will setup my own search engine and create a 'reason' for internic to share all that private data with me. Ok. Bad idea.

    If anyone has or knows of a tool that can extract whois contact data into csv format and won't time out/get blocked after 3 or 4 requests, please hook a brotha up :)

    thx
     
    Last edited: Dec 12, 2009
  15. bradsteves

    bradsteves Registered Member

    Joined:
    Nov 23, 2009
    Messages:
    79
    Likes Received:
    12
  16. Vinchenz

    Vinchenz Newbie

    Joined:
    Jan 12, 2010
    Messages:
    2
    Likes Received:
    0
    I'm looking for something similar as well. Anyone have any new ideas about getting unlimited who is info? I would appreciate it.

    Since this is an old thread I thought I mention it here again.

    Thank you
     
  17. eb0la

    eb0la Newbie

    Joined:
    Apr 6, 2011
    Messages:
    33
    Likes Received:
    8
    Have your tried TOR (the onion router).
    I use it at work to scrape piratebay and works great.
     
  18. gregstereo

    gregstereo Elite Member

    Joined:
    Oct 5, 2009
    Messages:
    1,833
    Likes Received:
    1,027
    Occupation:
    I'm known to locate certain things from time to ti
    Location:
    Moose Factory, ON
    TOR is very risky and hideous software. I don't recommend using it - there are way better alternatives for anonymity.

    ScrapeBox has a whois scraper in it.
     
  19. kibitz

    kibitz Newbie

    Joined:
    Jul 7, 2012
    Messages:
    12
    Likes Received:
    1
    has anyone used bestseosuite's whois scrapper?
     
  20. jkwilson78

    jkwilson78 Regular Member Premium Member

    Joined:
    Jun 24, 2010
    Messages:
    224
    Likes Received:
    311
    Google "Atomic WHOIS Explorer". May be what you are looking for.