1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Over a billion scraped URLs

Discussion in 'Black Hat SEO' started by Snckr, Mar 12, 2014.

  1. Snckr

    Snckr BANNED BANNED

    Joined:
    Dec 27, 2013
    Messages:
    582
    Likes Received:
    381
    So, so far GScraper has scraped 0.00% and it has around 10mb of lists already.

    I will have over a billion URLs when done, but my dedi can't process that. Any ideas?
     
  2. mirrorer

    mirrorer Jr. VIP Jr. VIP

    Joined:
    Jan 30, 2009
    Messages:
    1,163
    Likes Received:
    1,030
    Split it into smaller pieces . SB can help with that. This is how i was doing when i had the same problem.
     
    • Thanks Thanks x 1
  3. Snckr

    Snckr BANNED BANNED

    Joined:
    Dec 27, 2013
    Messages:
    582
    Likes Received:
    381
    SB can't even load a 10 GB file without crashing, GSCraper can do it, but it takes time. :/
     
  4. Veyron

    Veyron BANNED BANNED

    Joined:
    Jun 3, 2013
    Messages:
    1,452
    Likes Received:
    1,297
    Wow, that'll be quite a list. Good luck!