1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

import 26M urls into gscaper and crashing

Discussion in 'Black Hat SEO' started by aarongoldsten, Aug 7, 2015.

  1. aarongoldsten

    aarongoldsten Junior Member

    Joined:
    Oct 22, 2014
    Messages:
    177
    Likes Received:
    63
    I keep trying to import 26 million urls into gscraper and it keeps freezing.

    also memory on vps is 90% disk c is 100%. I am wondering if the list I trying import to big? that why keep freezing?

    or what I can do to fix it? i just need to import list 26M urls into gscraper
     
  2. ekapek

    ekapek Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 2, 2010
    Messages:
    266
    Likes Received:
    47
    Home Page:
    Upgrade vps to have more memory or just split yours list. Why can't you just split it?
     
  3. davids355

    davids355 Jr. VIP Jr. VIP

    Joined:
    Apr 25, 2011
    Messages:
    9,833
    Likes Received:
    7,440
    Home Page:
    Sounds like you need more power. As above, upgrade your VPS or split the lists.
    You can use scrapebox if you have it to split the list into 10 or something and then try again.
     
  4. watley

    watley Power Member

    Joined:
    Oct 17, 2012
    Messages:
    524
    Likes Received:
    237
    Location:
    San Francisco
    This comes as a surprise? You need a better VPS.
     
  5. soccerlover

    soccerlover Jr. VIP Jr. VIP

    Joined:
    Jun 12, 2014
    Messages:
    3,181
    Likes Received:
    1,682
    Gender:
    Male
    Occupation:
    Seo Analyst :D
    Location:
    ♥♥♥ BHW ♥♥♥
    Home Page:
    26m is a huge amount, you need to upgrade current vps or go on with the dedicated stuff :)