WHOIS scraper?
- Thread starter Zidane10
- Start date
CyberAlien
BANNED
- Apr 14, 2010
- 596
- 289
You can whois all of the popular domains through command line. The fastest way would be to download a registry list of all the .com's and then use Perl to whois all of them and output the results to a file. You could then simply extract the emails from that file. If you're planning on scraping millions of domains, this would be the fastest way.
I'm talking about 8000-10000 domains. I'm not familiar with perl. Is there a program that can do this?
lord1027
Elite Member
- Sep 20, 2013
- 3,218
- 2,342
You can do it in linux easily, with a small bash script. But I'm not sure exactly how to make it multithreaded.
Mercury_Hg
Registered Member
- Aug 23, 2010
- 88
- 19
It doesn't sound like it'd be very difficult to accomplish in C++. Stream domains from file -> Query [multiple] whois domain -> retain emails -> stream results out to file
It'd be pretty trivial to filter out masked emails too, i.e. those who have whois protection.
It'd be pretty trivial to filter out masked emails too, i.e. those who have whois protection.
Mercury_Hg
Registered Member
- Aug 23, 2010
- 88
- 19
I could. Do you have the domain names you need already?
EDIT: This wouldn't need multithreading. If it's only 10,000 queries, you could accomplish them at a conservative speed of one query / sec in under three hours. That way you don't get IP banned for essentially DoSing the site.
EDIT: This wouldn't need multithreading. If it's only 10,000 queries, you could accomplish them at a conservative speed of one query / sec in under three hours. That way you don't get IP banned for essentially DoSing the site.
Last edited: