1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Importing Lists Exceeding 1 Million W/ Scrapebox

Discussion in 'Black Hat SEO Tools' started by Dan Da Man, Jan 13, 2012.

  1. Dan Da Man

    Dan Da Man Elite Member Premium Member

    Joined:
    May 31, 2011
    Messages:
    1,850
    Likes Received:
    937
    Occupation:
    Duh
    Location:
    San Diego
    Home Page:
    I was harvesting a huge list and it crashed on me. Went back to the harvester sessions and was importing the lists so that I can remove dups but can't exceed 1 million URLs.

    Is there a way that I can import more then a mil? Or is there anything I can do with .txt file?

    Thx
     
  2. Dan Da Man

    Dan Da Man Elite Member Premium Member

    Joined:
    May 31, 2011
    Messages:
    1,850
    Likes Received:
    937
    Occupation:
    Duh
    Location:
    San Diego
    Home Page:
    No one????
     
  3. ILoveCSharp

    ILoveCSharp Regular Member

    Joined:
    Nov 2, 2011
    Messages:
    368
    Likes Received:
    45
    There are addons for that job . Look them.
     
    • Thanks Thanks x 1
  4. Kaistar

    Kaistar Power Member

    Joined:
    Jan 9, 2010
    Messages:
    562
    Likes Received:
    333
    Split the list down to <1M chunks and filter/save each one then import the filtered lists.
     
  5. rulez05

    rulez05 Power Member

    Joined:
    Feb 3, 2011
    Messages:
    745
    Likes Received:
    142
    yeah justsplit the list to less than 1 mil
     
  6. jb2008

    jb2008 Senior Member

    Joined:
    Jul 15, 2010
    Messages:
    1,158
    Likes Received:
    972
    Occupation:
    Scraping, Harvesting in the Corn Fields
    Location:
    On my VPS servers
    Use scrapebox duperemove separate app.
     
  7. thedon23

    thedon23 Elite Member

    Joined:
    Dec 21, 2009
    Messages:
    1,759
    Likes Received:
    1,268
    Yeah, split them up bro. I had the same problem when dealing with lists over 1 million.
     
  8. mindmaster

    mindmaster Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 16, 2010
    Messages:
    2,501
    Likes Received:
    1,135
    Location:
    at my new office
    As guys said already.
    Use Dupe/Remove Addon.
     
  9. woot123

    woot123 Junior Member

    Joined:
    Jan 10, 2012
    Messages:
    113
    Likes Received:
    27
    Yes that´s really doing its job well! You cann merge all the lists from the harvester sessions(should be at least 2 txt files if you had over 1mio urls) and then check it for duplicate urls. It can process a really big amount of URLs in almost no time :)
     
  10. GoldenGlovez

    GoldenGlovez Moderator Staff Member Moderator Jr. VIP

    Joined:
    Mar 23, 2011
    Messages:
    701
    Likes Received:
    1,713
    Location:
    Guangdong, China
    Home Page:
    +1 for using the Scrapebox Dup Remover add-on.

    Another utility that can be used is Loopline's Domain Cleaner tool. Search for it here on the forums.