Scrape the Internet FOR Websites

[aduseia]

I'm wondering if there is a way (be it a tool of some sort) to scrape the Internet for Websites. Maybe based on Keywords, Country, and/or content type (video, article).

Mind you, I'm not referring to scraping already known websites FOR content. I'm talking about scraping the internet to discover new websites altogether. And at a large scale, thousands, maybe 10's of thousands.

Maybe I would require someone to build this type of bot for me? What do you guys think.

 

[itz_styx]

check for example ahrefs and see their data stats, then calculate if you can handle it as it is very expensive to store all that data.

 

[X]

So, you're looking to create a bot that discovers websites that aren't already indexed in another database, such as Google?

 

[Panther28]

pick something easier to make money doing, this is just a pie in the sky project.

 

[aduseia]

check for example ahrefs and see their data stats, then calculate if you can handle it as it is very expensive to store all that data.

I'll look into that. Primarily, I'd just want the website names themselves. So just simple, raw data in a sense.

So, you're looking to create a bot that discovers websites that aren't already indexed in another database, such as Google?

Yea something like that. However, of course the websites would ideally be indexed, as it would be nearly impossible to "discover" un-indexed sites.

pick something easier to make money doing, this is just a pie in the sky project.

Aha "pie in the sky". It's curiosity first and ideation second. I want to see what opportunities i can create with the data and use it in a creative way. I wouldn't say it's solely based on revenue just yet.

 

[affk2011]

There's some services out there that gives you a listing of newly registered domains.