1. This website uses cookies to improve service and provide a tailored user experience. By using this site, you agree to this use. See our Cookie Policy.
    Dismiss Notice

Combining a data channel with Wordpress

Discussion in 'Programming' started by SirLouen, Jul 11, 2019.

Tags:
  1. SirLouen

    SirLouen Jr. VIP Jr. VIP

    Joined:
    Jan 17, 2015
    Messages:
    623
    Likes Received:
    189
    Gender:
    Male
    Home Page:
    Scenario:

    1. I have a data channel that sends me string lines continously.
    2. I have a function that parse that lines and can get 15 types of content.
    3. I need to store that content in a database and I would like to store this in a Wordpress database, but here is where the question starts.

    My first idea, is to create a Custom Post Type with 15 meta fields one for each type of content. Everytime I receive a string line, I create a new post and set 1 or more of those meta fields after parsing the string.

    The problem is that I may be receiving around 10K strings per day, for a total of 300K strings per month.

    Knowing how much data each WP_Post saves, I think that my wp_posts + wp_postmeta databases will be >1Gb in the first month, for just 300K pieces of light text content which is probably absurd.

    So I'm looking for some pro advice, in order to optimize this a little bit if possible

    Only one caveat: I need to do this under WordPress because I need the WordPress architecture to design afterwards a nice front end without a lot of extra job (I always have a hard time with front end coding)
     
  2. ILearnSEO

    ILearnSEO Jr. VIP Jr. VIP

    Joined:
    Apr 4, 2015
    Messages:
    477
    Likes Received:
    143
    Gender:
    Male
    Store it off wordprss, create a simple plugin to pull the data in a post? I wouldent pull that much data in a wordpress database and use it internaly.
     
  3. SirLouen

    SirLouen Jr. VIP Jr. VIP

    Joined:
    Jan 17, 2015
    Messages:
    623
    Likes Received:
    189
    Gender:
    Male
    Home Page:
    Uhm... I could create an alt DB and batch process and update the information with a wordpress cron script or something in order to put the information accordingly to certain WP_Post that store less information...

    I have to think how may I structure this.

    Anyway I'm still open for which solutions for big chucks of data have you been using under wordpress, more considering that there are plenty of Wordpress stats plugins out there (that definitely store huge amount of info)
     
  4. ILearnSEO

    ILearnSEO Jr. VIP Jr. VIP

    Joined:
    Apr 4, 2015
    Messages:
    477
    Likes Received:
    143
    Gender:
    Male
    Why dont you go with custome cms tho?
     
  5. cnick79

    cnick79 Senior Member

    Joined:
    Jun 10, 2010
    Messages:
    854
    Likes Received:
    446
    Location:
    Wandering
    Who says you have to use the posts table to store this information? I would store this in its own table.
     
  6. SirLouen

    SirLouen Jr. VIP Jr. VIP

    Joined:
    Jan 17, 2015
    Messages:
    623
    Likes Received:
    189
    Gender:
    Male
    Home Page:
    Yes I was thinking so.
    The big problem is that they won't be easily accessible from certain plugins like Elementor which I need to show the information in a fashioned matter. This is why I'm thinking on batch processing or something
    But the big deal is that I need something the simplest as possible, because it is a micro-job which I don't want to invest a huge amount of time and I have not had any experience with
    Any references that I can read to go straight forward?
    Thanks in advance
     
  7. cnick79

    cnick79 Senior Member

    Joined:
    Jun 10, 2010
    Messages:
    854
    Likes Received:
    446
    Location:
    Wandering
    I’m not exactly sure what you are trying to achieve here, but trying to display 10k strings of data is a lot of information that isn’t going to be useful for anyone, so I don’t see the purpose other than bots.
     
  8. SirLouen

    SirLouen Jr. VIP Jr. VIP

    Joined:
    Jan 17, 2015
    Messages:
    623
    Likes Received:
    189
    Gender:
    Male
    Home Page:
    Not trying to display. I get that info process it / parse it and then display the processed information

    Example:

    Line 1: "Mr. Fox from Colorado wants to see the film The Lion King next monday in the cinema"
    Line 2: "Ms Smith from Arizona wants to see the tv show Friends tonight in her apartment"
    ...
    Then I get some info from this lines like
    Name: "Mr. Fox, Ms. Smith"...
    Location: "Colorado, Arizona",...
    Type of visualization: "film, tv show"
    Name of the show: "Lion King, Friends,..."
    Proposed day: ...
    Location of the visualization

    Then for example I show this info in page based on Wordpress with something like Elementor or Divi adequately formated

    Something like:

    People from Colorado: 10
    % of type of visualizaton: "30% films 70% tv shows"
    Most watched shows: "1. Lion King, 2. ..."

    Do you understand now?
     
    • Thanks Thanks x 1
  9. cnick79

    cnick79 Senior Member

    Joined:
    Jun 10, 2010
    Messages:
    854
    Likes Received:
    446
    Location:
    Wandering
    I get it now. You want to aggregate the data and then report on it. I would still store the data in a separate table so you have it, then create a post that contains the aggregated data you want to display such as percentages and counts. I would aggregate the data on the fly as you are receiving the data and store it with the post.
    Your flow would be:

    1) receive data channel
    2) fetch post and meta data
    3) update meta data with received data
    4) save post with updated data
     
  10. SirLouen

    SirLouen Jr. VIP Jr. VIP

    Joined:
    Jan 17, 2015
    Messages:
    623
    Likes Received:
    189
    Gender:
    Male
    Home Page:
    How would you organize the CPT?
    1 post per day with all the combined data, namely "daily_data" for example?
    I'm thinking on creating 1 post x 1 name x 1 day
    There are a total of 30 names max per day, therefore max 30 posts per day = 10.950 posts per year. Not few, but not many also. Or maybe yes. Not sure how much is considered a lot of posts for an average VPS with 2Gb RAM when dealing with them
     
  11. cnick79

    cnick79 Senior Member

    Joined:
    Jun 10, 2010
    Messages:
    854
    Likes Received:
    446
    Location:
    Wandering
    I would organize it however you need to consume it. If you need to display daily stats then daily posting like you said seems reasonable.

    Would creating a post per name and updating the data daily per post work for you?
     
    Last edited: Jul 13, 2019
  12. SirLouen

    SirLouen Jr. VIP Jr. VIP

    Joined:
    Jan 17, 2015
    Messages:
    623
    Likes Received:
    189
    Gender:
    Male
    Home Page:
    Not valid, because i need historic data :p