1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to generate sitemap for big autoblog?

Discussion in 'White Hat SEO' started by tonib, Mar 18, 2014.

  1. tonib

    tonib Power Member

    Joined:
    May 1, 2009
    Messages:
    541
    Likes Received:
    395
    First I should mention that I use WordPress.

    When I started the autoblog it was all working fine with a couple of hundred posts, but now I have 80k+ posts and I am adding 5-10k every day. So far I have tested about 10 plugins that should generate sitemaps and I always receive an error that pretty much tells me that I do not have enough PHP memory. I have tried editing the `wp-config` file to add more memory (300MB), but that didn't help either. Manual sitemap creation is out of the question as I am in no mood to type down 100k+ lines. :D

    I could ask the hosting provider for more PHP memory, but I expect my memory consumption to rise in the future (with every post), so I have to find another way around this.

    The only other option I though about was creating my own script to scrape all links and create sitemap which I could later upload, but I'd prefer another, easier solution.

    Question is - Is there a way to generate a sitemap without using too much PHP memory?

    At it's current speed Google indexes about ~2,000 posts/day. I just put a plugin for automatic interlinking, but I am not sure if that will help much.
     
  2. innozemec

    innozemec Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2011
    Messages:
    5,288
    Likes Received:
    1,799
    Location:
    www.Indexification.com
    Home Page:
    make a simple php script in a loop to read lets say 2k records from the mysql database in wp_posts table, generate a sitemap and save. That way you will end up with several sitemaps and you won't hit the memory limits.. I can't think up anything simpler than that..