1. This website uses cookies to improve service and provide a tailored user experience. By using this site, you agree to this use. See our Cookie Policy.
    Dismiss Notice

How would you structure this sitemap?

Discussion in 'White Hat SEO' started by youtalk, Dec 15, 2013.

  1. youtalk

    youtalk Regular Member

    Joined:
    Jul 5, 2012
    Messages:
    353
    Likes Received:
    6
    Occupation:
    Marketing
    Location:
    Closet
    I have over 8 million URLS on my site, and I'm trying to get a better understand of how everyone would structure the sitemaps for these?

    I understand Google will only accept 50,000 URLS per sitemap, so that would be 160 sitemaps that I would need to create.

    What does everyone think is the most effective way to do this?
     
  2. Freeopkiller

    Freeopkiller Jr. VIP Jr. VIP

    Joined:
    Dec 30, 2009
    Messages:
    130
    Likes Received:
    53
    Location:
    Montana
    If your pages are interlinked, why sitemap the whole thing. Google will crawl the links regardless, right. You could just sitemap your most recent, categories, tags, or ?
    I don't know what type, or how your site is structured.
    It would be difficult to 'structure' without that data..

    Here is a resource link that might help :
    dynamical.biz/blog/seo-technical/sitemap-strategy-large-sites-17.html

    Just snatch some robot.txt from some large sites with billions of links, get some ideas from the structure(s) they are using.. You might think about the server load when generating a sitemap that large everytime it updates.

    Hope it help's a bit..
     
  3. youtalk

    youtalk Regular Member

    Joined:
    Jul 5, 2012
    Messages:
    353
    Likes Received:
    6
    Occupation:
    Marketing
    Location:
    Closet
    Thank you for the information. Can you explain a little about what you mean above?
     
  4. Freeopkiller

    Freeopkiller Jr. VIP Jr. VIP

    Joined:
    Dec 30, 2009
    Messages:
    130
    Likes Received:
    53
    Location:
    Montana
    Your server is going to generate the sitemaps when they change. Your talking about 8mil links, generally you would want to refresh and update your sitemap when new content is created/generated. I'd generate daily with a cron in the middle of the night, about 100k links on one of my sites.
    It would slow my server down a bit when generating the file.

    Again I don't know what type of site you have, I was just informing you to think about it when rebuilding a sitemap of that size..