WHOIS scraper?

Discussion in 'Black Hat SEO' started by Zidane10, Nov 30, 2014.

  1. Zidane10

    Zidane10 Newbie

    Joined:
    Feb 19, 2014
    Messages:
    25
    Likes Received:
    6
    Hi. Does anyone know of a good whois scraper? The one on scrapebox isn't really good. I need one that will get all of the emails for a domain.
     
  2. CyberAlien

    CyberAlien Power Member

    Joined:
    Apr 14, 2010
    Messages:
    505
    Likes Received:
    248
    You can whois all of the popular domains through command line. The fastest way would be to download a registry list of all the .com's and then use Perl to whois all of them and output the results to a file. You could then simply extract the emails from that file. If you're planning on scraping millions of domains, this would be the fastest way.
     
  3. Zidane10

    Zidane10 Newbie

    Joined:
    Feb 19, 2014
    Messages:
    25
    Likes Received:
    6
    I'm talking about 8000-10000 domains. I'm not familiar with perl. Is there a program that can do this?
     
  4. lord1027

    lord1027 Elite Member

    Joined:
    Sep 20, 2013
    Messages:
    3,181
    Likes Received:
    2,241
    You can do it in linux easily, with a small bash script. But I'm not sure exactly how to make it multithreaded.
     
  5. Mercury_Hg

    Mercury_Hg Registered Member

    Joined:
    Aug 23, 2010
    Messages:
    88
    Likes Received:
    19
    It doesn't sound like it'd be very difficult to accomplish in C++. Stream domains from file -> Query [multiple] whois domain -> retain emails -> stream results out to file

    It'd be pretty trivial to filter out masked emails too, i.e. those who have whois protection.
     
  6. ronalde

    ronalde Registered Member

    Joined:
    Aug 1, 2014
    Messages:
    98
    Likes Received:
    23
    Great custom footprints and then scrape with Gscraper.
     
  7. Zidane10

    Zidane10 Newbie

    Joined:
    Feb 19, 2014
    Messages:
    25
    Likes Received:
    6
    Can anyone create a program that will do this?
     
  8. Mercury_Hg

    Mercury_Hg Registered Member

    Joined:
    Aug 23, 2010
    Messages:
    88
    Likes Received:
    19
    I could. Do you have the domain names you need already?

    EDIT: This wouldn't need multithreading. If it's only 10,000 queries, you could accomplish them at a conservative speed of one query / sec in under three hours. That way you don't get IP banned for essentially DoSing the site.
     
    Last edited: Dec 1, 2014