Recommendation to create a sitemap for a large site

A

alexsaiz

Newbie
Joined
Nov 6, 2012
Messages
19
Reaction score
2
I have a site with adult content, very similar to Pinterest. I have around +30,000 pages and growing. What is the best tool you recommended to create a sitemap? I have installed Google Sitemap Generator on my server, however is very limited as it is not supported by Google anymore since several years ago.

Do you any other tools for large websites? (30,000 to 200,000 pages or more). Thanks!
 
Advertise on BHW
Take a look here: support.google.com/webmasters/bin/answer.py?hl=en&answer=34634
This is Google tool for the exact thing that you are asking for, it is in BETA release but I think that it should do the work OK.
And if you are on Wordpress Platform like me, I use this plugin here: arnebrachhold.de/projects/wordpress-plugins/google-xml-sitemaps-generator/
I didn't test it with 200k pages or posts, but till now it is handling very well a News Site with 8k+ posts.
 
I am currently using the tool you suggest (Google Sitemal Generator). However this is not a good tool as it is very limited and it is not supported by Google anymore. Do you recommend any others? My site is a custom development in php, it is not based on WP. Thanks!
 
i am not sure what tool to use for such a large site, probably a custom one you would need to get. Just don't forget that for such a large website you will need to created several sitemaps, as there's a sitemap limit in size and urls that can be contained into one when submitting to google
 
There is a tool called Sitemap generator. It's a desktop soft. It allows you to enable it for a couple of hours and then it grabs all the information from your site, building a sitemap. Has lot's of features to play with.
 
Thanks. What's the website? I have found lots of results at Google with "Sitemap Generator". On the other side, is there anything for linux servers?
 
Since you're using a custom site and not wordpress, you'll probably have to modify a sitemap generator to fit your needs. This is one of the first things that popped up in google, maybe it'll help https://github.com/pawelantczak/php-sitemap-generator By the way if you go to 200k+ pages you won't be able to put them all into a valid sitemap because Google limits sitemaps to 50k URLs.
 
Another option is if you have scrapebox and your pages are indexed you can scrape them that way. You would just need to use multiple search strings to get the links.

site:domain.com/category-1
site:domain.com/category-2
site:domain.com/category-3

Just saw this on cnet too: http://download.cnet.com/Inspyder-Sitemap-Creator/3000-18485_4-10635412.html

Costs $40 though. No idea if it's any good. I know there's some other ones out there because I've had this same problem before and I can't remember how I did it for my site.
 
I've got a large site, with about the same amount of pages. I used Screaming From (it creates sitemaps too) - it's desktop software.

Though I did have to leave it to run for several hours and overnight... done the job though, I found alot of the other tools didn't do the full job or crashed, if you find something better, it'll be good to know!
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Networking Downloads

Marketplace

Content / Copywriting Hosting Images, Logos & Videos Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design Misc

Making Money

Affiliate Programs Hire a Freelancer Making Money Pay Per Click Site Flipping

BlackHatWorld

BHW Merch Forum Suggestions & Feedback Introductions Lounge Dispute Resolution

Other

Domain Name Forum IM Journeys Web Hosting Newsletter
Back
Top