1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

What blog system for HUGE amount of posts?

Discussion in 'Blogging' started by catchdoe, Jan 24, 2010.

  1. catchdoe

    catchdoe Newbie

    Joined:
    Jan 17, 2010
    Messages:
    34
    Likes Received:
    6
    As above, I'm looking for a blog system which can handle a lot of posts. WordPress keeps killing my dedicated server with 100% usage of mysql after adding around of 40,000 posts. Any ideas?
     
  2. penguin

    penguin Junior Member

    Joined:
    Nov 20, 2009
    Messages:
    149
    Likes Received:
    60
    Occupation:
    SEO Manager
    Home Page:
    have you tried silver stripe ? may be that will end your search. www.[B]silverstripe[/B].com/
     
  3. jaguarslug

    jaguarslug BANNED BANNED

    Joined:
    Dec 29, 2009
    Messages:
    339
    Likes Received:
    51
    How about splitting 40k posts in 2 wordpress?
     
  4. catchdoe

    catchdoe Newbie

    Joined:
    Jan 17, 2010
    Messages:
    34
    Likes Received:
    6
    Thanks for reply. However, I'm looking for something a little more popular - I need good described API or script's functions. I've got about 1 300 000 rewrited articles to add and I'm not going to do it manually :)

    @jaguarslug: I've been thinking about splitting content to multiple databases and running blogs on subdomains. That's my backup way of finishing this project.
     
    Last edited: Jan 24, 2010
  5. Razorbl

    Razorbl Jr. VIP Jr. VIP Premium Member

    Joined:
    Jun 25, 2008
    Messages:
    441
    Likes Received:
    121
    ok use wordpress with plugin called wp super cache . This plugin generates static html files from your blog. After files are created , blog wont query database .
    I'm talking about this plugin
    Code:
    http://wordpress.org/extend/plugins/wp-super-cache/
     
  6. catchdoe

    catchdoe Newbie

    Joined:
    Jan 17, 2010
    Messages:
    34
    Likes Received:
    6
    Doesn't really work well, I was checking this plugin before. MySQL still is pretty much loaded.
     
  7. bangkai

    bangkai Junior Member

    Joined:
    Dec 2, 2008
    Messages:
    173
    Likes Received:
    18
    what about installing these plugins: hyper-cache and db cache reloaded ?
     
  8. Razorbl

    Razorbl Jr. VIP Jr. VIP Premium Member

    Joined:
    Jun 25, 2008
    Messages:
    441
    Likes Received:
    121
    Thats just be at the beggining , as wp super cache uses database to create html pages , after all pages are creating everything should be fine . You can also use plug ins mentioned in post above
     
  9. seomum

    seomum Newbie

    Joined:
    Dec 7, 2009
    Messages:
    32
    Likes Received:
    1
    After adding 40,000 posts your blog didn't got banned from Google and other pining services as WP ping automatically to pinging services and Google by default.

    Have you removed the pinging service url from WP admin panel from here : http://www.yourdomain.com/wp-admin/options-writing.php

    or

    you are using any other kind of plugin or way to post so much large no. of posts in wordpress blog?
     
  10. judson

    judson Power Member

    Joined:
    Nov 29, 2009
    Messages:
    530
    Likes Received:
    319
    Occupation:
    Fulltime Newbie IM
    Location:
    Sub Ubi
    Drupal.

    Am yet to hear of anyone maxing it out yet. Plus, if you are just serving static PLR articles, using the Drupal Pressflow distro will speed up your site considerably.
     
  11. AltaVega

    AltaVega Newbie

    Joined:
    Dec 23, 2009
    Messages:
    23
    Likes Received:
    8
    @catchdoe: I am using Wordpress with Super Cache plugin to publish nearly 160 000 articles for about 8 months now and there is no problem on my server. It had it's problems when I first used super cache but everything works fine now.

    @seomum: Yes, I've had disabled the pinging services at that time. enabled it again after importing all the articles. And I used a plugin called "Zip Poster" at that time to import articles in bulk and to set the publishing interval.
     
  12. seomum

    seomum Newbie

    Joined:
    Dec 7, 2009
    Messages:
    32
    Likes Received:
    1
    Thanks @ AltaVega for your reply

    wow so much huge posts in short period of time, am doing the same thing but in different manner, need your suggestion as you have you did in past :)

    Here is my question:

    Am targeting one niche and i have made around more than 2,000 articles for that niche.

    1. I have created wordpress blogs
    2. I have removed the date from from theme so it should look like a website not blog
    3. I will be posting 2,000 articles in one day or 2
    4. So do you think, posting such huge amount of articles in one day at dateless wordpress blog will still hurt my blog even after date code has been removed
    5. Or do you think, i should remove every thing from "Ping" text area under "setting" option so it won't ping every time i will post the content
    6. Or is there any more options that i have to turned off, so it should not ping the Google, every time i do the posting.
    7. I have inserted following meta in theme also:
    <meta name="Revisit-after" content="5 days">
    <meta name="Robots" content="Follow">
    <meta name="Rating" content="General">
    so robots won't come at my site before 5 days
    do you think it will help me to block the search engine?

    My main concern is, my blog should look like a static website and Google shouldn't slap me for huge amount of content posting in one day

    if this will help than i will slap more 30k to 40k content in 1 week :D

    but before all of this action i take i need your suggestion seriously, i will be greatly thankful of yours :)
     
  13. AltaVega

    AltaVega Newbie

    Joined:
    Dec 23, 2009
    Messages:
    23
    Likes Received:
    8
    Hi there seomum.

    The first post of this thread was a question about server load in this is why I didn't mention SEO Stuff.
    For articles I mentioned the time period of posting them was something around two weeks, and that was an experiment dealing with how search engines deals with duplicate content and those blogs were not money sites. in fact, they still sit there with that amount of content without being monetized at all.
    Now, as I said in another post in another thread here, I don't know what to say about posting huge amount of content in a short time. My experience was that every single site of mine got through the google dance, but none of them got deindexed from google. they still are indexed and being served by google as search result, but some of them with hig rankings, some others in page 150 and more of search results.
    I am a newbie in the SEO area so I think it's best to hear another expert's opinion here.
    But I know for sure some pinging service will actually ban you for submitting that amount in that short time, so, best to disable the pinging options (removing everything from "Ping")
    Removing the date from theme options or permalink URL's to my opinion does not help nor damage. What I've seen is that search engines don't give a damn about the date you post on your posts. But, even if you remove the date from the theme options, remember that the date is still present in the RSS feeds.
    As I said, remove the ping list, if you have xml sitemaps installed disable it for the time you post articles and as for the metas for robots, I think just make your blog private to search engines using robots.txt protocol, or just use the option make private in wordpress and it will do all the job for you. next, when you are done with posting enable pinging, enable xml-sitemaps, and make your blog public.

    My idea and blueprint to do this is:

    install wordpress, theme, plugins etc and configure them.
    use "WP Auto Tagger" plugin to automatically finds tags based on your post content as you post them.
    use wp-posterperiodic of the "WPTraffic with videos" package to post your articles. it has many options to post them all in a moment or to delay each article in the time period you set. I found the package here in the BHW, just search for it. If you don't find it let me know and I'll find it for you.
    use "cbnet Ping Optimizer" to manage your pinging stuff sou you don't get tagged as ping spammer.
    use "Robots Meta" . Name says it all.
    don't forget "WP Super Cache" and "WP-DBManager" :)
    next use "SEO No Duplicate" - This plugin helps you manage your search engine duplicate content, by setting your post page's canonical to the permalink.
    finally, activate "Google XML Sitemaps", crate your sitemap, make your blog public, configure rss options - lets say, to display 10 posts, fill in the pninging options, submit your sitemap to google and write and publish manually just one article.

    This is what I did in my case, but as I said at the beginning my experiment was just for the duplicate content idea.

    For the SEO stuff better just let's ask some SEO Expert here.
     
  14. seomum

    seomum Newbie

    Joined:
    Dec 7, 2009
    Messages:
    32
    Likes Received:
    1
    I have those 2,000 articles as complete re-written content, so i think it will help me?
     
  15. AltaVega

    AltaVega Newbie

    Joined:
    Dec 23, 2009
    Messages:
    23
    Likes Received:
    8
    tried to respond to your PM but as it seems you can send PM's but you've chosen not to receive PM's. lol
     
  16. cooooookies

    cooooookies Senior Member

    Joined:
    Oct 6, 2008
    Messages:
    1,008
    Likes Received:
    216
    Here my thoughts:

    I uploaded once using WPMU 100x 500-1000 posts, so 50-100k individual posts. But... wpmu stores data for each domain separately. So if you really have trouble then the problem is obviously long searches, maybe some columns are not indexed (if you have some mysql knowledge you know what I am speaking from). Maybe caching plugins can help you, I really would assume that.

    For huge sites, you might want to check the wikipedia engine. I played a little around with it and yes, obviously, this engine is thought to deal with a lot of posts.

    But anyhow, rethink maybe your strategy and use wpmu and split to several domains==individual tables for each blog... there are some auto-submitters floating around here.
     
  17. HookUPcom2

    HookUPcom2 Jr. VIP Jr. VIP

    Joined:
    Dec 15, 2009
    Messages:
    252
    Likes Received:
    111
    Occupation:
    Founder
    Location:
    Colorado
    Home Page:
    I've just gone through something similar.

    For me it wasn't the pages but the number of bots trying to hit it. The Cuil bot, (a totally worthless search engine in my opinion) was trying to spider 20,000- 40,000 pages a day.

    Block that spider and I bet you'll be ok.
    I would also turn pinging off.

    And last but not least, I got 16 Gig Ram, 8 Core Processor machine for like $250 a month.

    but I have alot more content than you have stated.