how the hell do to that? remove duplicates .txt

[The Blackhat Guy]

Notepad++ crash as hell.
I have a 26m lines .txt (476mb). I want to remove duplicates only, how can I do that?

 

[AlexIonescu]

Select all - TextFX- TextFX Tools - Sort lines case sensitive or insensitive

TextFX is a plugin for notepad++. If you don`t have it, you'll need to download it.

Hope this helps

 

[The Blackhat Guy]

Select all - TextFX- TextFX Tools - Sort lines case sensitive or insensitive

TextFX is a plugin for notepad++. If you don`t have it, you'll need to download it.

Hope this helps

Dude, I just said that notepadd crash on such a large file

 

[AlexIonescu]

Try this

Hopefully browser will not crash.

Or just try to separate the file in multiple files

 

[The Blackhat Guy]

Try this

Hopefully browser will not crash.

Or just try to separate the file in multiple files

It doesnt work.
And how am I supposed to remove duplicates, if the all lines aren't on the same .txt? :s

 

[AlexIonescu]

Select de first million, remove duplicates from that one, than the next...and eventually files will get smaller.
And then combine them step by step. Needs some work but can be done. I made this way huge files

 

[RalphSEO]

vim text editor or command line

 

[rexstacy]

Go the programming way
Search google for python script to remove duplicate lines it will be job done in a couple of minutes
of course you will have to install python.

 

[qrazy]

Try Editplus. It does work well for me in the past. Just make sure you use 64-bit version as 32-bit applications have a memory limit of 2GB.

 

[trex6187]

try this php script, don't know if it will work for such a large file...
<?php
$array = file("yourfile.txt");
$array = array_unique($array);
file_put_contents('yourfileunique.txt', implode($array));
?>

 

[The Blackhat Guy]

Try Editplus. It does work well for me in the past. Just make sure you use 64-bit version as 32-bit applications have a memory limit of 2GB.

Just downloaded editplus, what a shit...

 

[The Blackhat Guy]

Anyone could help me, please?

 

[Automated]

Split the file into smaller lists, remove the duplicates from each of the smaller lists, then combine the lists and dedupe the main list...

 

[The Blackhat Guy]

Split the file into smaller lists, remove the duplicates from each of the smaller lists, then combine the lists and dedupe the main list...

from 26m only 3-4m are duplicates. it would't work

 

[avi619]

Your best bet is to split the file into various smaller files of 100mb each, remove duplicates with notepad++ and then merge.
I believe scrapebox can help you with these operations.

Edit: Just read the above post - Read this - http://stackoverflow.com/questions/22623472/remove-all-duplicates-in-a-large-text-file

 

[The Blackhat Guy]

Doesn't work.

Help me guys please

 

[berkay1907]

Scrapebox can solve your problem if you got it.

 

[jazzc]

On a Linux console:

cd to the folder that has the file

sort filename | uniq

replace "filename" with the actual filename

 

[gashead]

Scrapebox has a free tool http://www.scrapebox.com/free-dupe-remove

 

[rere003]

You can use scrapebox dup remove addons.