How to generate sitemap for big autoblog?

Discussion in 'White Hat SEO' started by tonib, Mar 18, 2014.

  1. tonib

    tonib Power Member

    Joined:
    May 1, 2009
    Messages:
    541
    Likes Received:
    397
    First I should mention that I use WordPress.

    When I started the autoblog it was all working fine with a couple of hundred posts, but now I have 80k+ posts and I am adding 5-10k every day. So far I have tested about 10 plugins that should generate sitemaps and I always receive an error that pretty much tells me that I do not have enough PHP memory. I have tried editing the `wp-config` file to add more memory (300MB), but that didn't help either. Manual sitemap creation is out of the question as I am in no mood to type down 100k+ lines. :D

    I could ask the hosting provider for more PHP memory, but I expect my memory consumption to rise in the future (with every post), so I have to find another way around this.

    The only other option I though about was creating my own script to scrape all links and create sitemap which I could later upload, but I'd prefer another, easier solution.

    Question is - Is there a way to generate a sitemap without using too much PHP memory?

    At it's current speed Google indexes about ~2,000 posts/day. I just put a plugin for automatic interlinking, but I am not sure if that will help much.
     
  2. innozemec

    innozemec Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2011
    Messages:
    5,365
    Likes Received:
    1,806
    Location:
    www.Indexification.com
    Home Page:
    make a simple php script in a loop to read lets say 2k records from the mysql database in wp_posts table, generate a sitemap and save. That way you will end up with several sitemaps and you won't hit the memory limits.. I can't think up anything simpler than that..