1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How would you structure this sitemap?

Discussion in 'White Hat SEO' started by youtalk, Dec 15, 2013.

  1. youtalk

    youtalk Regular Member

    Joined:
    Jul 5, 2012
    Messages:
    337
    Likes Received:
    6
    Occupation:
    Owner
    Location:
    I don't even know anymore
    I have over 8 million URLS on my site, and I'm trying to get a better understand of how everyone would structure the sitemaps for these?

    I understand Google will only accept 50,000 URLS per sitemap, so that would be 160 sitemaps that I would need to create.

    What does everyone think is the most effective way to do this?
     
  2. Freeopkiller

    Freeopkiller Junior Member

    Joined:
    Dec 30, 2009
    Messages:
    117
    Likes Received:
    47
    Location:
    Montana
    If your pages are interlinked, why sitemap the whole thing. Google will crawl the links regardless, right. You could just sitemap your most recent, categories, tags, or ?
    I don't know what type, or how your site is structured.
    It would be difficult to 'structure' without that data..

    Here is a resource link that might help :
    dynamical.biz/blog/seo-technical/sitemap-strategy-large-sites-17.html

    Just snatch some robot.txt from some large sites with billions of links, get some ideas from the structure(s) they are using.. You might think about the server load when generating a sitemap that large everytime it updates.

    Hope it help's a bit..
     
  3. youtalk

    youtalk Regular Member

    Joined:
    Jul 5, 2012
    Messages:
    337
    Likes Received:
    6
    Occupation:
    Owner
    Location:
    I don't even know anymore
    Thank you for the information. Can you explain a little about what you mean above?
     
  4. Freeopkiller

    Freeopkiller Junior Member

    Joined:
    Dec 30, 2009
    Messages:
    117
    Likes Received:
    47
    Location:
    Montana
    Your server is going to generate the sitemaps when they change. Your talking about 8mil links, generally you would want to refresh and update your sitemap when new content is created/generated. I'd generate daily with a cron in the middle of the night, about 100k links on one of my sites.
    It would slow my server down a bit when generating the file.

    Again I don't know what type of site you have, I was just informing you to think about it when rebuilding a sitemap of that size..