Combining a data channel with Wordpress

Scenario:

1. I have a data channel that sends me string lines continously.
2. I have a function that parse that lines and can get 15 types of content.
3. I need to store that content in a database and I would like to store this in a Wordpress database, but here is where the question starts.

My first idea, is to create a Custom Post Type with 15 meta fields one for each type of content. Everytime I receive a string line, I create a new post and set 1 or more of those meta fields after parsing the string.

The problem is that I may be receiving around 10K strings per day, for a total of 300K strings per month.

Knowing how much data each WP_Post saves, I think that my wp_posts + wp_postmeta databases will be >1Gb in the first month, for just 300K pieces of light text content which is probably absurd.

So I'm looking for some pro advice, in order to optimize this a little bit if possible

Only one caveat: I need to do this under WordPress because I need the WordPress architecture to design afterwards a nice front end without a lot of extra job (I always have a hard time with front end coding)

Store it off wordprss, create a simple plugin to pull the data in a post? I wouldent pull that much data in a wordpress database and use it internaly.

Uhm... I could create an alt DB and batch process and update the information with a wordpress cron script or something in order to put the information accordingly to certain WP_Post that store less information...

I have to think how may I structure this.

Anyway I'm still open for which solutions for big chucks of data have you been using under wordpress, more considering that there are plenty of Wordpress stats plugins out there (that definitely store huge amount of info)

SirLouen said:

Uhm... I could create an alt DB and batch process and update the information with a wordpress cron script or something in order to put the information accordingly to certain WP_Post that store less information...

I have to think how may I structure this.

Anyway I'm still open for which solutions for big chucks of data have you been using under wordpress, more considering that there are plenty of Wordpress stats plugins out there (that definitely store huge amount of info)

Click to expand...

Why dont you go with custome cms tho?

Who says you have to use the posts table to store this information? I would store this in its own table.

cnick79 said:

Who says you have to use the posts table to store this information? I would store this in its own table.

Click to expand...

Yes I was thinking so.
The big problem is that they won't be easily accessible from certain plugins like Elementor which I need to show the information in a fashioned matter. This is why I'm thinking on batch processing or something
But the big deal is that I need something the simplest as possible, because it is a micro-job which I don't want to invest a huge amount of time and I have not had any experience with
Any references that I can read to go straight forward?
Thanks in advance

I’m not exactly sure what you are trying to achieve here, but trying to display 10k strings of data is a lot of information that isn’t going to be useful for anyone, so I don’t see the purpose other than bots.

Not trying to display. I get that info process it / parse it and then display the processed information

Example:

Line 1: "Mr. Fox from Colorado wants to see the film The Lion King next monday in the cinema"
Line 2: "Ms Smith from Arizona wants to see the tv show Friends tonight in her apartment"
...
Then I get some info from this lines like
Name: "Mr. Fox, Ms. Smith"...
Location: "Colorado, Arizona",...
Type of visualization: "film, tv show"
Name of the show: "Lion King, Friends,..."
Proposed day: ...
Location of the visualization

Then for example I show this info in page based on Wordpress with something like Elementor or Divi adequately formated

Something like:

People from Colorado: 10
% of type of visualizaton: "30% films 70% tv shows"
Most watched shows: "1. Lion King, 2. ..."

Do you understand now?

I get it now. You want to aggregate the data and then report on it. I would still store the data in a separate table so you have it, then create a post that contains the aggregated data you want to display such as percentages and counts. I would aggregate the data on the fly as you are receiving the data and store it with the post.
Your flow would be:

1) receive data channel
2) fetch post and meta data
3) update meta data with received data
4) save post with updated data

How would you organize the CPT?
1 post per day with all the combined data, namely "daily_data" for example?
I'm thinking on creating 1 post x 1 name x 1 day
There are a total of 30 names max per day, therefore max 30 posts per day = 10.950 posts per year. Not few, but not many also. Or maybe yes. Not sure how much is considered a lot of posts for an average VPS with 2Gb RAM when dealing with them

I would organize it however you need to consume it. If you need to display daily stats then daily posting like you said seems reasonable.

Would creating a post per name and updating the data daily per post work for you?

cnick79 said:

Would creating a post per name and updating the data daily per post work for you?

Click to expand...

Not valid, because i need historic data