Scrape the Internet FOR Websites

I'm wondering if there is a way (be it a tool of some sort) to scrape the Internet for Websites. Maybe based on Keywords, Country, and/or content type (video, article).

Mind you, I'm not referring to scraping already known websites FOR content. I'm talking about scraping the internet to discover new websites altogether. And at a large scale, thousands, maybe 10's of thousands.

Maybe I would require someone to build this type of bot for me? What do you guys think.

check for example ahrefs and see their data stats, then calculate if you can handle it as it is very expensive to store all that data.

So, you're looking to create a bot that discovers websites that aren't already indexed in another database, such as Google?

pick something easier to make money doing, this is just a pie in the sky project.

itz_styx said:

check for example ahrefs and see their data stats, then calculate if you can handle it as it is very expensive to store all that data.

Click to expand...

I'll look into that. Primarily, I'd just want the website names themselves. So just simple, raw data in a sense.

phr0zt said:

So, you're looking to create a bot that discovers websites that aren't already indexed in another database, such as Google?

Click to expand...

Yea something like that. However, of course the websites would ideally be indexed, as it would be nearly impossible to "discover" un-indexed sites.

Panther28 said:

pick something easier to make money doing, this is just a pie in the sky project.

Click to expand...

Aha "pie in the sky". It's curiosity first and ideation second. I want to see what opportunities i can create with the data and use it in a creative way. I wouldn't say it's solely based on revenue just yet.

There's some services out there that gives you a listing of newly registered domains.