1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Whats the biggest wordpress site you've ever successfully ran?

Discussion in 'BlackHat Lounge' started by HenryHavoc, Aug 6, 2017.

  1. HenryHavoc

    HenryHavoc Jr. VIP Jr. VIP

    Joined:
    Mar 24, 2008
    Messages:
    877
    Likes Received:
    1,647
    Occupation:
    Hustler
    Location:
    Cincinnati
    I'm currently working on a pretty massive project that will be creating wordpress posts via CSV files with my own data and I will probably end up with over 500,000 posts. Possibly a million. I know wordpress has its limitations with how large a site you can build (mainly because every time a post is viewed, it needs to query a database, and the more posts you have, the larger that database becomes).

    Obviously shared hosts are out of the question. I was recommended to use Kinsta. Anyone have other recommendations?

    So whats the biggest site you've built? I'd love to get some input on how you go about managing a site this big (server speed optimization, getting the site indexed, SEO, etc)
     
  2. roadhamster

    roadhamster Regular Member

    Joined:
    Mar 12, 2012
    Messages:
    335
    Likes Received:
    242
    The pain in the ass with wordpress, and more CMS'es/ applications using databases, the more database entries (posts), the slower your site will become. Depending on your hardware sooner or later your website will become unresponsive. But for every problem there's a solution. When it's gonna be huge, get yourself a neat LinuxVPS, install Sphinxsearch and let Sphinx get the stress from the database. Wordpress has a Sphinx plugin btw, https://wordpress.org/plugins/wordpress-sphinx-plugin/ and, if configured right, the power of this tool is amazing. Drop on top of that Redis server (http://redis.io) with the Redis wordpress plugin for WP https://wordpress.org/plugins/search/redis/ and you're good to go to the next 500,000 posts and your users won't notice a difference in performance.
    SphinxSearch is doing much much better than the wordpress standard search, so SEO point of view this is a good startingpoint.

    I'm using Sphinx/ Redis on a website (non-wordpress) with a database with over 20 million records 6.5GB in size with full search capability and it's lightning fast. Sphinxsearch: http://sphinxsearch.com.
     
    • Thanks Thanks x 3
  3. HenryHavoc

    HenryHavoc Jr. VIP Jr. VIP

    Joined:
    Mar 24, 2008
    Messages:
    877
    Likes Received:
    1,647
    Occupation:
    Hustler
    Location:
    Cincinnati
    You are a god amongst men. What would you say is the most user friendly way to set up this combo? I talked with a developer from Kinsta which runs on google cloud and they said as long as I had caching enabled I would be able to host the site with them. I'm pretty familiar with WordPress and front end stuff but back end server and database stuff is still relatively new to me. It seems like you have some experience with stuff so I'd love to hear your input on getting this up and running.
     
  4. rafark

    rafark Regular Member

    Joined:
    Jan 15, 2013
    Messages:
    403
    Likes Received:
    189
    Gender:
    Male
    Occupation:
    Moderador
    Location:
    Noble and Heroic MC
    The database is not the problem, in this case is wordpress that is terribly optimized.

    Assuming you're not on a tight budget you should get a medium vps with enough ram and then run an in memory cache storage tool such as redis. You should also use op cache provided you have full control of th server. (VPS or dedicated).

    Also, are you importing the posts? Are a programmer? If so, don't read all data into memory at once when importing, stream it line by line.
     
  5. HenryHavoc

    HenryHavoc Jr. VIP Jr. VIP

    Joined:
    Mar 24, 2008
    Messages:
    877
    Likes Received:
    1,647
    Occupation:
    Hustler
    Location:
    Cincinnati
    Are there any hosting solutions that offer this type of setup right out of the box?

    I'm importing the posts from csv files via WP All Import. I'll probably end up breaking it all up with cron jobs so it's not running all at once.
     
  6. roadhamster

    roadhamster Regular Member

    Joined:
    Mar 12, 2012
    Messages:
    335
    Likes Received:
    242
    I checked Kinsta, looks nice, but for the 100$ a month you can get yourself a very decent VPS/ dedicated machine and you have full control over your setup. Put somehting like webmin on it to control your websites/ webserver setup.
    Importing posts with wp-allimport, I tried it but it failed with csv files with lots of entries. For this I'm using Mysql cmmand line, I can import a huge csv files in no time straight into the database.
    What I'm trying to say: every choice you make has its pro's & cons. A fully self-managed VPS or dedicated can be a lot of work, but when it's running, you have full control over all the aspects of your server. When you go for a managed option, you depend on the hosting company and may hope that they will be cooperative in your wishes, it's up to you.