1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Recommendation to create a sitemap for a large site

Discussion in 'White Hat SEO' started by alexsaiz, Jun 13, 2013.

  1. alexsaiz

    alexsaiz Newbie

    Joined:
    Nov 6, 2012
    Messages:
    18
    Likes Received:
    2
    I have a site with adult content, very similar to Pinterest. I have around +30,000 pages and growing. What is the best tool you recommended to create a sitemap? I have installed Google Sitemap Generator on my server, however is very limited as it is not supported by Google anymore since several years ago.

    Do you any other tools for large websites? (30,000 to 200,000 pages or more). Thanks!
     
  2. HeRBaR

    HeRBaR Supreme Member

    Joined:
    Aug 15, 2011
    Messages:
    1,216
    Likes Received:
    981
    Occupation:
    Sleeping
    Location:
    localhost
    Home Page:
    Take a look here: support.google.com/webmasters/bin/answer.py?hl=en&answer=34634
    This is Google tool for the exact thing that you are asking for, it is in BETA release but I think that it should do the work OK.
    And if you are on Wordpress Platform like me, I use this plugin here: arnebrachhold.de/projects/wordpress-plugins/google-xml-sitemaps-generator/
    I didn't test it with 200k pages or posts, but till now it is handling very well a News Site with 8k+ posts.
     
  3. alexsaiz

    alexsaiz Newbie

    Joined:
    Nov 6, 2012
    Messages:
    18
    Likes Received:
    2
    I am currently using the tool you suggest (Google Sitemal Generator). However this is not a good tool as it is very limited and it is not supported by Google anymore. Do you recommend any others? My site is a custom development in php, it is not based on WP. Thanks!
     
  4. innozemec

    innozemec Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2011
    Messages:
    5,305
    Likes Received:
    1,799
    Location:
    www.Indexification.com
    Home Page:
    i am not sure what tool to use for such a large site, probably a custom one you would need to get. Just don't forget that for such a large website you will need to created several sitemaps, as there's a sitemap limit in size and urls that can be contained into one when submitting to google
     
  5. ContentLockPro

    ContentLockPro Power Member Premium Member

    Joined:
    Nov 7, 2012
    Messages:
    723
    Likes Received:
    129
    There is a tool called Sitemap generator. It's a desktop soft. It allows you to enable it for a couple of hours and then it grabs all the information from your site, building a sitemap. Has lot's of features to play with.
     
  6. alexsaiz

    alexsaiz Newbie

    Joined:
    Nov 6, 2012
    Messages:
    18
    Likes Received:
    2
    Thanks. What's the website? I have found lots of results at Google with "Sitemap Generator". On the other side, is there anything for linux servers?
     
  7. Raffy

    Raffy Regular Member

    Joined:
    Nov 30, 2012
    Messages:
    212
    Likes Received:
    615
    Since you're using a custom site and not wordpress, you'll probably have to modify a sitemap generator to fit your needs. This is one of the first things that popped up in google, maybe it'll help https://github.com/pawelantczak/php-sitemap-generator By the way if you go to 200k+ pages you won't be able to put them all into a valid sitemap because Google limits sitemaps to 50k URLs.
     
  8. -Jericho-

    -Jericho- Jr. Executive VIP

    Joined:
    Jan 10, 2010
    Messages:
    2,849
    Likes Received:
    1,704
    Location:
    Stalking My Ex-Wife
    Another option is if you have scrapebox and your pages are indexed you can scrape them that way. You would just need to use multiple search strings to get the links.

    site:domain.com/category-1
    site:domain.com/category-2
    site:domain.com/category-3

    Just saw this on cnet too: http://download.cnet.com/Inspyder-Sitemap-Creator/3000-18485_4-10635412.html

    Costs $40 though. No idea if it's any good. I know there's some other ones out there because I've had this same problem before and I can't remember how I did it for my site.
     
  9. This Web Guy

    This Web Guy Junior Member

    Joined:
    Jun 21, 2010
    Messages:
    128
    Likes Received:
    35
    I've got a large site, with about the same amount of pages. I used Screaming From (it creates sitemaps too) - it's desktop software.

    Though I did have to leave it to run for several hours and overnight... done the job though, I found alot of the other tools didn't do the full job or crashed, if you find something better, it'll be good to know!
     
  10. Techxan

    Techxan Elite Member

    Joined:
    Dec 7, 2011
    Messages:
    3,093
    Likes Received:
    3,585
    Occupation:
    Local SEOist
    Location:
    TEXAS (you have to yell, its the law.)