I need a large database of Articles.
- Thread starter iffiz
- Start date
HustleTong
Elite Member
- May 30, 2019
- 13,466
- 10,660
Why not use open ai's API and plug it into whatever you're building?
- Sep 17, 2008
- 6,037
- 3,974
Any db will have had it's contents dumped to a thousand websites already. There will be no possibilty of getting those articles ranked.
Now, if you are looking to train an LLM or something then maybe it would be worthwhile.
Now, if you are looking to train an LLM or something then maybe it would be worthwhile.
You hit the nail on the head sir! Any ideas on where I may able to get my hands on it?
- Aug 1, 2021
- 2,046
- 376
i have one in the music niche i think somewhere on my old laptop i used it to populate a website with articles maybe around 200k articles or so would you be interested should i start looking for it ?
Thanks for offering this but unfortunately that wouldn't be close to what I need. I need articles in the high millions to billions.
- Aug 1, 2021
- 2,046
- 376
the headache of dealing with a database this big I'm glad i don't deal with large databases anymore lol best of luck regardless broski
KuromiA
Newbie
- Oct 2, 2024
- 18
- 7
If you want to scrape and build your own, a relatively simple way to do that and keep the quality decently high would be to crawl a list of the top 10M sites, and if the WordPress API is enabled, just dump all of the articles, save them to a data-lake/mongo/etc. and then afterwards run some quality-control and populate an SQL db with them.
Any experienced developer could create something in a few hours to do it.
Any experienced developer could create something in a few hours to do it.