Mining infinite & actionable niche ideas (Is it possible ?)

Hello bhw members,
Firstly, I would like to share my best wishes for all of you in 2020 !

Last months, I've been thinking a lot about a piece of code which could be easily build. The pitch is kinda simple.

We all want to find niches with nice search volume and almost none competition, putting some piece of content with affiliate links or ads on, and banking on a passive way right ? Finding these kind of niche is quit easy with tools like ahref.

But, most of us don't have writing skills, time or money to build content for "virgin" niche. Most of us (This is what I do too) take content from same niche website in another language, translate it, spin it etc.

In fact, the perfect scenario for me, would be to find a niche where there is NO to LOW competition in my native language but content in english or another language which I could translate would exist.

Here is then what I would wish to do :
1 - Having a list of all website (more than 1.5 bilion website in 2018, does such a list exist ?)
2 - Parsing these websites (does the website match my buisiness model ? would be easy to identify amazon affiliate website by counting number of links by example)
3 - Parsing websites which match my buisiness model to infer main topic (the niche)
4 - Does competition exist for such a niche in my native language ?
5 - If not, reproduce it in your native language.

This pipeline could be easily implemented in python. I just don't know if a such a fresh list of ALL website exist (Alexa rank ?)

If you are as enthusiast as me about this pipeline, let's discuss about it. I could put some effort on this & share the code/result here since I have some decent coding skills.

Apologies for my low english, hope it wasn't a pain in the ass to read.

Bumping the brainstorm with few hints :
* Found a service which could provide a list of 250,000,000 domain which is 1/6 domain the total amount that exist.
* Tested a plan B where I can use google search to find amazon affiliate website with this kind of query :
I'll test this strategy in coming days, I dunno if I can manage to get a lot of result with this.

Outch. It seems that the brainstorm didn't happend.

I found a solution by using ahrefs, Im now filtering this list. Once my approach is validated by my tests, I could write a guide for my 100' post or even before.
Cheers.

I can answer to a few questions. In order to do this you must build a domain list for each tld. 1.5 billion? No. Probably around 400-450 million. Just root domain names, not subdomains. Alexa rank, the list that is on the Internet has only 1 million and is outdated. Ahrefs the same, but is updated daily. Majestic has one too. This 1 million is good to test, then you can scale things.

I used the ahrefs list : 3.5 millions domains which has at least 1 link to amazon.com .

I'm now working on the filter which will give me an actionable list currently 50k feom the original 3.5 millions. Want to add the number of indexed page of each domain so I wrote a little selenium script to query google for the site:domains.com count.

Good work! Also, what I've noticed lately, site:domain.com, is not longer accurate, or it will show different results at each search. If you have time, I'd suggest you also add Yahoo! and Bing (they have different results for this query).

Thanx for the advice. The indexed page metric seem bugged/spoofed for me too. Sometimes it show 56 on first serp while only 12 on the 2nd serp.

What I want to accomplish is to filter my list of domain with this metric so I ll be able to find site with (eg: 10 articles & organic traff > 5k) which are perfect candidate if this micro niche is not exploited in my native language.

What do you think about using the sitemap.xml to count the number of "indexed" page ?

The most eficient way would be to use ahref or any tool whuch come with an api. But it would cost a lot of money to process 100k domain.

One more thing I would like to extract is the overall word count...

Well, sitemap.xml will do, but depends if is not standard. You have to look at robots.txt, parse the sitemapindex, etc. Only this way you can count the words. And is best to keep each url you download, so you can re-run things on the page source, in case something wrong happens, so you won't have to re-scrape everything.