I need a large database of Articles.

[iffiz]

Does anybody have access to a large sql database of articles?

 

[HustleTong]

Why not use open ai's API and plug it into whatever you're building?

 

[jimbobo2779]

Any db will have had it's contents dumped to a thousand websites already. There will be no possibilty of getting those articles ranked.

Now, if you are looking to train an LLM or something then maybe it would be worthwhile.

 

[iffiz]

Any db will have had it's contents dumped to a thousand websites already. There will be no possibilty of getting those articles ranked.

Now, if you are looking to train an LLM or something then maybe it would be worthwhile.

You hit the nail on the head sir! Any ideas on where I may able to get my hands on it?

 

[noneisnone]

i have one in the music niche i think somewhere on my old laptop i used it to populate a website with articles maybe around 200k articles or so would you be interested should i start looking for it ?

 

[iffiz]

i have one in the music niche i think somewhere on my old laptop i used it to populate a website with articles maybe around 200k articles or so would you be interested should i start looking for it ?

Thanks for offering this but unfortunately that wouldn't be close to what I need. I need articles in the high millions to billions.

 

[noneisnone]

Thanks for offering this but unfortunately that wouldn't be close to what I need. I need articles in the high millions to billions.

the headache of dealing with a database this big I'm glad i don't deal with large databases anymore lol best of luck regardless broski

 

[KuromiA]

If you want to scrape and build your own, a relatively simple way to do that and keep the quality decently high would be to crawl a list of the top 10M sites, and if the WordPress API is enabled, just dump all of the articles, save them to a data-lake/mongo/etc. and then afterwards run some quality-control and populate an SQL db with them.

Any experienced developer could create something in a few hours to do it.