1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

indexing a 500,000 page blogger blogspot blog

Discussion in 'Blogging' started by n3wbseo, Dec 30, 2011.

  1. n3wbseo

    n3wbseo Junior Member

    Joined:
    Mar 22, 2011
    Messages:
    117
    Likes Received:
    23
    help! I scraped 500,000 posts using wordpress/feedwordpress and exported/converted/uploaded them a category at a time into a new blogger blog spot blog. google will only visit 2000 times a day or 20Megs in total bandwidth per day it seems. while indexed pages are going up by a couple thousand a day using site:site.blogspot.com only 30 pages a day are showing up under top pages. is there any way to speed this process up?
     
  2. n3wbseo

    n3wbseo Junior Member

    Joined:
    Mar 22, 2011
    Messages:
    117
    Likes Received:
    23
    no one has any idea how to speed up page recognition process in wmt?
     
  3. bigeazy

    bigeazy Regular Member

    Joined:
    Feb 20, 2011
    Messages:
    315
    Likes Received:
    268
    Location:
    #1 to #9 on your fav SE
    Have you thought about building backlinks to the pages that are not indexed?
     
    • Thanks Thanks x 1
  4. Rushdie

    Rushdie BANNED BANNED

    Joined:
    Feb 2, 2009
    Messages:
    1,378
    Likes Received:
    1,720
    is there really a limit of 2000 visits/day? is it only to blogspot?
     
  5. spsfinest

    spsfinest Junior Member

    Joined:
    Jul 25, 2009
    Messages:
    192
    Likes Received:
    26
    Occupation:
    IM
    Location:
    U.S.
    He's talking about Google's robot for indexing, he say's it comes around about 2k times per day.
     
  6. n3wbseo

    n3wbseo Junior Member

    Joined:
    Mar 22, 2011
    Messages:
    117
    Likes Received:
    23
    yes this seems correct a few weeks later. im not sure yet if its a blogger blogspot only thing so i copied and pasted it to a few dozen new free web 2.0's and see soon hopefully! hope i can get around this technical hurdle so i can start releasing larger websites on free web 2.0s! =)
     
  7. IamNRE

    IamNRE Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 18, 2010
    Messages:
    4,663
    Likes Received:
    7,108
    Occupation:
    Generate Leads With FB Ads For Just $1
    Home Page:
    Try posting more unique content.
     
  8. MandLoys

    MandLoys Regular Member Premium Member

    Joined:
    Apr 18, 2011
    Messages:
    450
    Likes Received:
    71
    Occupation:
    SEO & PPC Campaign Management
    Location:
    http://bit.ly/UTpXg3
    Home Page:
    Use profile backlinks or whatever you can get in masses and deeplink all the content on your blog, it will help (a lot) getting everything indexed.
     
  9. thachcaodotcom

    thachcaodotcom Junior Member

    Joined:
    Oct 21, 2009
    Messages:
    121
    Likes Received:
    151
    You can consider to increase bot visit rate in google Webmaster Tool
     
  10. schwagoo

    schwagoo Jr. Executive VIP Jr. VIP Premium Member

    Joined:
    Nov 8, 2010
    Messages:
    807
    Likes Received:
    737
    Location:
    Midwest, USA
    Yes you can increase it...but it doesn't improve more than 10-15% over the standard.

    I have a similar project that targets millions of keywords in a fully automated system creating keywords and dynamic PHP content pages from a huge database of spintax articles that dynamically insert my target keywords.

    In two months, the google bot has hit the domain over 300,000 times...but only about 50K pages are indexed when I search site:domain.com

    Might be a quality issue...as the quality is most certainly low. I only set it up for testing purposes. If it generates revenue someday that is great. But I just wanted to see what it would do.

    Another thought I had (but I haven't tested) is splitting a huge site into 10K page chunks via subdomain or cheap .info sites. site1.domain.com site2.domain.com and doing some blasts to each. Technically these should be seen as separate sites and could all be indexed at 2K/day rate max. Then once your page are all indexed, 301 redirect each of your subdomains to your main site and insert the articles.

    Good luck on your project

     
  11. truongduong201

    truongduong201 Newbie

    Joined:
    Jun 7, 2011
    Messages:
    12
    Likes Received:
    0
    Home Page:
    i added 2000 links, now my blog disappear on Google, how can i solve this problem my blog: xuonginoffset.blogspot.com, pls give me some advise
     
  12. n3wbseo

    n3wbseo Junior Member

    Joined:
    Mar 22, 2011
    Messages:
    117
    Likes Received:
    23
    just a quick follow up: google starts indexing more pages faster over time without any backlinks to atleast a sub domain like blogspot.com.
     
  13. markhenry121

    markhenry121 Elite Member

    Joined:
    Oct 14, 2011
    Messages:
    2,149
    Likes Received:
    239
    You can start building backlinks.
     
  14. pYraTe

    pYraTe Junior Member

    Joined:
    Jun 10, 2011
    Messages:
    111
    Likes Received:
    49
    You can change it here:

    Code:
    http://www.google.com/webmasters/