1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

import 26M urls into gscaper and crashing

Discussion in 'Black Hat SEO' started by aarongoldsten, Aug 7, 2015.

  1. aarongoldsten

    aarongoldsten Junior Member

    Joined:
    Oct 22, 2014
    Messages:
    177
    Likes Received:
    63
    I keep trying to import 26 million urls into gscraper and it keeps freezing.

    also memory on vps is 90% disk c is 100%. I am wondering if the list I trying import to big? that why keep freezing?

    or what I can do to fix it? i just need to import list 26M urls into gscraper
     
  2. ekapek

    ekapek Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 2, 2010
    Messages:
    277
    Likes Received:
    48
    Home Page:
    Upgrade vps to have more memory or just split yours list. Why can't you just split it?
     
  3. davids355

    davids355 Jr. VIP Jr. VIP

    Joined:
    Apr 25, 2011
    Messages:
    10,401
    Likes Received:
    8,105
    Sounds like you need more power. As above, upgrade your VPS or split the lists.
    You can use scrapebox if you have it to split the list into 10 or something and then try again.
     
  4. watley

    watley Power Member

    Joined:
    Oct 17, 2012
    Messages:
    525
    Likes Received:
    238
    Location:
    San Francisco
    This comes as a surprise? You need a better VPS.
     
  5. soccerlover

    soccerlover Jr. VIP Jr. VIP

    Joined:
    Jun 12, 2014
    Messages:
    3,399
    Likes Received:
    1,773
    Gender:
    Male
    Occupation:
    Seo Analyst :D
    Location:
    ♥♥♥ BHW ♥♥♥
    Home Page:
    26m is a huge amount, you need to upgrade current vps or go on with the dedicated stuff :)