1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

import 26M urls into gscaper and crashing

Discussion in 'Black Hat SEO' started by aarongoldsten, Aug 7, 2015.

  1. aarongoldsten

    aarongoldsten Junior Member

    Joined:
    Oct 22, 2014
    Messages:
    177
    Likes Received:
    63
    I keep trying to import 26 million urls into gscraper and it keeps freezing.

    also memory on vps is 90% disk c is 100%. I am wondering if the list I trying import to big? that why keep freezing?

    or what I can do to fix it? i just need to import list 26M urls into gscraper
     
  2. ekapek

    ekapek Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 2, 2010
    Messages:
    282
    Likes Received:
    49
    Home Page:
    Upgrade vps to have more memory or just split yours list. Why can't you just split it?
     
  3. davids355

    davids355 Moderator-In-Training Jr. VIP

    Joined:
    Apr 25, 2011
    Messages:
    11,194
    Likes Received:
    8,927
    Location:
    Scotland
    Home Page:
    Sounds like you need more power. As above, upgrade your VPS or split the lists.
    You can use scrapebox if you have it to split the list into 10 or something and then try again.
     
  4. watley

    watley Power Member

    Joined:
    Oct 17, 2012
    Messages:
    525
    Likes Received:
    238
    Location:
    San Francisco
    This comes as a surprise? You need a better VPS.
     
  5. soccerlover

    soccerlover Moderator-In-Training Jr. VIP

    Joined:
    Jun 12, 2014
    Messages:
    3,529
    Likes Received:
    1,875
    Gender:
    Male
    Occupation:
    Managing BHW :D
    Location:
    ♥♥♥ BHW ♥♥♥
    Home Page:
    26m is a huge amount, you need to upgrade current vps or go on with the dedicated stuff :)