1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

sitemap generator needed for 300k+ links

Discussion in 'White Hat SEO' started by nicker, Apr 21, 2008.

  1. nicker

    nicker Registered Member

    Joined:
    Mar 16, 2008
    Messages:
    66
    Likes Received:
    144
    i was trying a1sitemap sitemap builder.net and some other warez softs but all of them freez when spidering and building index of pages over 300.000 site


    i have good enough pc for sure...
     
  2. dellzip

    dellzip Jr. VIP Jr. VIP Premium Member

    Joined:
    Jan 25, 2008
    Messages:
    285
    Likes Received:
    29
    I with you I need one myself 450,000 pages
     
  3. nicker

    nicker Registered Member

    Joined:
    Mar 16, 2008
    Messages:
    66
    Likes Received:
    144
    your site is based on htaccess or legit full hosted?


    if full hosted you can use "xenu links sleught" then generate sitemap from accesslog with python google sitemap generator its really cool and fast but i have some htaccess files that are not user friendly with all links checkers so it wont work in my case :(
     
  4. rannybo

    rannybo Junior Member

    Joined:
    Jul 7, 2007
    Messages:
    175
    Likes Received:
    54
  5. Hardmaster

    Hardmaster BANNED BANNED

    Joined:
    Mar 23, 2008
    Messages:
    96
    Likes Received:
    24
    i need such sitemap generator for my doorway pages too...
     
  6. Subject

    Subject Power Member

    Joined:
    Dec 26, 2007
    Messages:
    656
    Likes Received:
    303
    Location:
    Living With Articles!!
    Try phpSitemapNG
    http://enarion.net/google/
     
  7. nishanthe

    nishanthe Newbie

    Joined:
    Jan 26, 2008
    Messages:
    25
    Likes Received:
    9
    I used custom sitemap generator built with PHP for my lyrics site. http://www.48g.net with Sitemap Index, because Google not recommends includng more than 50K in a single sitemap file.

    for 300K link you will definetely need to use sitemap index file, and I advice you to not to use all the URL once. because it might cause your site to de-index from google index.
     
  8. nicker

    nicker Registered Member

    Joined:
    Mar 16, 2008
    Messages:
    66
    Likes Received:
    144
    every program i ve tested is creating 50k sitemaps


    for 300k links i use it sitemap index and use it and ping before creating page - its helping mi get indexed faster.
     
  9. rannybo

    rannybo Junior Member

    Joined:
    Jul 7, 2007
    Messages:
    175
    Likes Received:
    54

    The script works very well for me but I'm on a dedicated server so I can tweak my memory and php execution time on the server for the php script to accomodate the long delay of developing the sitemaps. For a website this guy who created this thread is talking about then a shared hosting WON'T WORK! You will get your account closed or script disabled immediately.

    I have alot of pages also and it takes about a day and a half to generate the sitemaps. The really cool part about the script is that I can set the sitemap to only generate only 30,000 urls per sitemap, which puts me under the 10mb size limit and 50,000 urls per sitemap limit that Google has for sitemaps. It also has cron job capability.

    Plus the technical support is good and forum support is pretty active. For 29.99, it's a good buy.
     
  10. nicker

    nicker Registered Member

    Joined:
    Mar 16, 2008
    Messages:
    66
    Likes Received:
    144

    But it is crawling method or using phpfiles/mysql to generate sitemap?
     
  11. rannybo

    rannybo Junior Member

    Joined:
    Jul 7, 2007
    Messages:
    175
    Likes Received:
    54
    It's not using mysql only php. It crawls every page on your site and allows you to exclude certain directories whether static or dynamic. It's good for doorway pages as long as the urls can be found within your website.
     
  12. Tofast4u

    Tofast4u Newbie

    Joined:
    Mar 25, 2008
    Messages:
    49
    Likes Received:
    28
    Occupation:
    it specialist
    Location:
    well living on the edge
    Home Page:
    here is the newest Sitemap.Generator.v1.6.6 works for me even with big sites......


    hxxp://rapidshare.com/files/109655639/sitemap.rar.html

    pw: tofast4u
     
  13. samloron

    samloron Junior Member

    Joined:
    Apr 16, 2008
    Messages:
    184
    Likes Received:
    201
    Location:
    Õ°°°°Ø
    I want to get a similar situation accomplished so I'm in your same predicament I have found an answer here.
     
  14. nicker

    nicker Registered Member

    Joined:
    Mar 16, 2008
    Messages:
    66
    Likes Received:
    144
    it wont work on htaccess parsed content
     
  15. nicker

    nicker Registered Member

    Joined:
    Mar 16, 2008
    Messages:
    66
    Likes Received:
    144
    as you can see in my topic:

    /on the begining/

    "i was trying a1sitemap..."

    and your link is a1 sitemap ....
     
  16. Tofast4u

    Tofast4u Newbie

    Joined:
    Mar 25, 2008
    Messages:
    49
    Likes Received:
    28
    Occupation:
    it specialist
    Location:
    well living on the edge
    Home Page:
    yes. its the newest version +patch (which sould fix your problem not to safe the sitemap)
    its also working on big sites. and it doesnt freeze. it maybe looks like because your mouse is working but you can notice its still collectiong more links and going to generate it. so on a 300k DB you prolly will need some hours to generate a sitemap at all.