Millions of urls - where ?

jameslarkin

Junior Member
Joined
Jun 19, 2013
Messages
171
Reaction score
14
where do i find a list (in the millions) of just domains ?

at the moment im scraping using scrapebox, but if there is a list some where this would be more efficient
 
D

Deleted member 1456969

Guest
Only scrapebox is still efficient in scraping domains for me
 
Joined
Dec 2, 2018
Messages
45
Reaction score
24
Do you know about "Project Sonar "?! (Project Sonar is a security research project by Rapid7 that conducts internet-wide surveys across different services and protocols to gain insights into global exposure to common vulnerabilities. The data collected is available to the public in an effort to enable security research.)
This is much more like only the domain names....the fresh (1-2 weeks old) database uncompressed size about 70+GB, there is need some work to take apart and collect only the domain names.
But there you can find information also about a domain have "A" or "AAA" DNS record.

The whole (free) database size is 42+TB.
You can find here: https://opendata.rapid7.com/about/
 

jameslarkin

Junior Member
Joined
Jun 19, 2013
Messages
171
Reaction score
14
Do you know about "Project Sonar "?! (Project Sonar is a security research project by Rapid7 that conducts internet-wide surveys across different services and protocols to gain insights into global exposure to common vulnerabilities. The data collected is available to the public in an effort to enable security research.)
This is much more like only the domain names....the fresh (1-2 weeks old) database uncompressed size about 70+GB, there is need some work to take apart and collect only the domain names.
But there you can find information also about a domain have "A" or "AAA" DNS record.

The whole (free) database size is 42+TB.
You can find here: https://opendata.rapid7.com/about/
wow that is great info thank you so much
 
Top