1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

SB: Weird Remove Duplicates Freeze

Discussion in 'Black Hat SEO' started by jb2008, Nov 23, 2010.

  1. jb2008

    jb2008 Senior Member

    Joined:
    Jul 15, 2010
    Messages:
    1,158
    Likes Received:
    972
    Occupation:
    Scraping, Harvesting in the Corn Fields
    Location:
    On my VPS servers
    Ok, so I harvest, remove duplicates and everything is fine.

    However, if I add a txt file of urls to that list (previously harvested from SB), it ALWAYS freezes on me 100% of the time. I've let it wait about an hour to test if it was serious. I am pretty sure this didn't used to happen to me before. WTF?

    Is anyone else having this problem?

    I need a way to remove duplicates but all programs e.g. gvim so far don't work for me (they freeze).

    I am doing a supermassive scrape akin to the construction of the large hadron collider in Switzerland, and desperately, DESPERATELY need to remove duplicates on an industrial scale :eek: