1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Mass duplicate lines remover

Discussion in 'Black Hat SEO Tools' started by MethShow, Mar 22, 2011.

  1. MethShow

    MethShow Registered Member

    Joined:
    Jan 24, 2011
    Messages:
    57
    Likes Received:
    2
    Hey guys,

    I have 100 txt file, each of them is around 5gb, I want to remove duplicate lines in all these files, what's the best solution.

    Now, I'm using scrapebox dupe remover, but it's taking like forever to remove just 5-10 file.

    Do you know any better solution, if not, do u know what's the ideal file size for scrapebox dupe remover, so that I would split it and remove manually.

    Thanks
     
  2. algski

    algski Junior Member

    Joined:
    Nov 26, 2008
    Messages:
    130
    Likes Received:
    49
    just a quick seach on google , I tried it and it work