Mass duplicate lines remover

Discussion in 'Black Hat SEO Tools' started by MethShow, Mar 22, 2011.

  1. MethShow

    MethShow Registered Member

    Joined:
    Jan 24, 2011
    Messages:
    57
    Likes Received:
    2
    Hey guys,

    I have 100 txt file, each of them is around 5gb, I want to remove duplicate lines in all these files, what's the best solution.

    Now, I'm using scrapebox dupe remover, but it's taking like forever to remove just 5-10 file.

    Do you know any better solution, if not, do u know what's the ideal file size for scrapebox dupe remover, so that I would split it and remove manually.

    Thanks
     
  2. algski

    algski Junior Member

    Joined:
    Nov 26, 2008
    Messages:
    141
    Likes Received:
    53
    just a quick seach on google , I tried it and it work