I want to scrape whois age of 50K domains daily

serpnames

Regular Member
Jr. VIP
Joined
Jan 21, 2021
Messages
312
Reaction score
87
Hello,

I'm looking to scrape domain age of 50K domains from whois or other methods.

I looked at apilayer but it takes around 3 min+ for 100 domains and it's too slow.

What's the best and fastest way to scrape domain age?

Please help.

Thank you.
 
What's the best and fastest way to scrape domain age?
Get a custom coded bot/script that scrapes the info from websites showing domain whois data.
You will also need a quality proxy pool, captcha service and a fast VPS.
I used to scrape lot more than 50k in good old days when finding quality expired domains was not that difficult.
Use multiple sources/websites to scrape the data to make sure that if one source isn't working there are other options.
 
50k per day using a paid API service is going to be expensive. My route would be a custom script.

I would research some scripts from codecanyon and similar sites that are popular and provide a scrapeable whois service.
Then I'd find the footprints of these scripts and scrape for as many installations as I could find.
Then I'd use proxies and multithreaded requests to spread the whois requests over these sites.

This is off the top of my head but should work (so long as you can find a whois script you can target). You'll need to be able to parse whois results to get registration date - this library should be what you need: https://github.com/joepie91/python-whois

Edit: actually, here's a footprint for a 'domain age checker' script that you could just scrape, and seems to have a few hundred installations:
Code:
"Enter more information about the Domain Age Checker tool!"
 
Scraping is mostly a good script and hiding your traces.

Use Good proxies (Residential from Storm o else) and Captcha solving (Death By Captcha, etc)
 
Back
Top