Tool to categorize millions of websites?

I'm after a tool that will take in many millions of websites, and check them all, categorizing each one as a main subject. (Art, Sport etc)

Majestic only seems to do 50,000 so that's a no-go.

Scrapebox can do it on platform and extension, but not the actual category/subject of the site.

BeautifulSoup for parsing HTML and XML documents.

Aviel said:

BeautifulSoup for parsing HTML and XML documents.

Click to expand...

That has no other use than just reading the website(s) HTML codes in this case,
How to filter & categorize them is the real work,

OP, there are multiple services like Majestic, Categorizer & Klazify,
But, that would cost you around $1k per 1 million website at least

ChatGPT or LLama can do a decent job

I've tried several of the paid services, all seem to work pretty well but most are not cost-effective at scale. This one might offer you the best price, but you'll be limited to a set number of API calls per day: https://www.uclassify.com/pricing

Also, are you sure you need all those 'many millions of websites' categorizing? Is there any other sort of easy filtering (eg. checking server response to make sure the site is actually alive) you can do before trying to categorize them?