How do you manage and store scraped data?
- Thread starter Mynovel
- Start date
hideath
Elite Member
- Jun 16, 2019
- 6,620
- 7,350
all depends on the data value, how much did you invest into, in terms of money/time/efforts,
for easy to scrape data, i tend to leave it wherever it was scraped or wherever i'm planning to use (PC/VPS),
for important data, i simply save it in some drive (g drive for example), just compress and upload
for easy to scrape data, i tend to leave it wherever it was scraped or wherever i'm planning to use (PC/VPS),
for important data, i simply save it in some drive (g drive for example), just compress and upload
myplaysoft
Junior Member
- Feb 18, 2014
- 112
- 43
Using WP with mysql seems ok, just make sure you backup the scraped data table on regular basis, then you can copy backups on an external drive, for instance. Thus, you will be able to periodically refresh that db table.
- Aug 9, 2017
- 5,595
- 6,893
I have a VPS scraping various data 24/7, which either gets inserted straight into mysql tables, saved as csvs, or dumped as plain text files for other scripts to parse through later.
D
Deleted member 752298
Guest
In what format is your data stored?
Jack Rodding
Regular Member
- Feb 5, 2018
- 271
- 72
AWS S3
- Apr 25, 2014
- 6,918
- 12,420
Most of mine is saved on a flash drive and locally on my scraping laptop.
It really depends on what that data is being scraped for. I normally keep it in a CSV.
It really depends on what that data is being scraped for. I normally keep it in a CSV.
arenis
BANNED
- Jul 27, 2016
- 495
- 154
i think local storage might not handle very large datasets well cause disk space and performance can become issues. for most individual or small projects, local storage is often sufficient and simpler to manage but for larger or collaborative projects, leveraging WordPress with MySQL can provide more robust solutions.
- May 27, 2016
- 11,528
- 2,643
I would use Wordpress and MySQL, provided I do several back ups in order not to loose any of the data.
- Sep 10, 2010
- 11,829
- 26,671
You don’t need wordpress for this. Just install mysql and phpmyadmin, create some schema through phpmyadmin and store data in that. Depending on the schema you could store a hell lot of data and query the data as you need. Csv is fine for upto 100k tows perhaps, it becomes painful after that.
Also, you could do it locally as well. Install docker desktop and use the gui to add a mysql and a phpmyadmin container.
Also, you could do it locally as well. Install docker desktop and use the gui to add a mysql and a phpmyadmin container.
dandan594594
Elite Member
- Jan 31, 2013
- 1,664
- 1,238
CSV on external storage. Easy to import format.