remove ALL duplicates (2x) on big files?

The Blackhat Guy

Elite Member

Oct 15, 2014

2,096

936

• Mar 8, 2015

• #1

Are many tools for duplicating remove, what I need is:

Let's say I have a big list (3m users).
First lines
A
B
C
C
D

I don't want to delete only "C", I want to delete 2xC, the final result to be something like:
A
B
D

Any website/tool for this, please?

 

Brad100

Supreme Member

Nov 9, 2014

1,346

1,001

• Mar 8, 2015

• #2

Notepad++ use its find and replace feature to do it. Put the "C" in the "find" box and leave the replace box emtry and you're done.

 

R

rashedhns

Junior Member

Aug 13, 2014

127

9

• Mar 8, 2015

• #3

Brad100 said:

Notepad++ use its find and replace feature to do it. Put the "C" in the "find" box and leave the replace box emtry and you're done.

Click to expand...

Maybe its a long words not only the first letter

 

The Blackhat Guy

Elite Member

Oct 15, 2014

2,096

936

• Mar 8, 2015
• Thread Starter Thread Starter

• #4

Brad100 said:

Notepad++ use its find and replace feature to do it. Put the "C" in the "find" box and leave the replace box emtry and you're done.

Click to expand...

Dude...I said "3m lines", not only ABCD
And no, I dont know which line is duplicate.

And no, I won't take any line 1 by 1, to ctrl+f and replace. It would take me like 10 years

 

Brad100

Supreme Member

Nov 9, 2014

1,346

1,001

• Mar 8, 2015

• #5

BigTroll said:

Dude...I said "3m lines", not only ABCD
And no, I dont know which line is duplicate.

And no, I won't take any line 1 by 1, to ctrl+f and replace. It would take me like 10 years

Click to expand...

Yeah, it'll remove all 3 millions, try it, its notepad++ not notepad.

 

The Blackhat Guy

Elite Member

Oct 15, 2014

2,096

936

• Mar 9, 2015
• Thread Starter Thread Starter

• #6

Brad100 said:

Yeah, it'll remove all 3 millions, try it, its notepad++ not notepad.

Click to expand...

Man, you don't understand.
I dont know which is "C" on this 3m list. I don't know what to search on "find" box.
I need a tool to IDENTIFY ALL duplicates and remove them.

 

T

ttrox

Regular Member

Jun 28, 2013

217

76

• Mar 9, 2015

• #7

Grab TextFX Plugin for Notepad++.
Sort by Ascending/descending order.
Pull out a regex to remove duplicate lines that are separated by at least one line.

 

K

kcollier63

Registered Member

Nov 23, 2010

66

29

• Mar 9, 2015

• #8

Look up TextPipe. It is a pretty good program for working with files. Saved my ass a bunch of times...

 

Nitros

Power Member

Jan 30, 2009

592

302

• Mar 9, 2015

• #9

Brad is right, Notepad++ will do this work just fine. It supports regex expressions.

 

conrulez

Power Member

Dec 29, 2009

539

434

• Mar 9, 2015

• #10

Here you go:

http://onlinetexttools.com/Remove-Duplicate-Lines-from-Text.html

Some popup comes up. Just close it and you should be fine. I've used this tool numerous times.

Good luck.

 

HelloInsomnia

Elite Member

Mar 1, 2009

1,801

3,042

• Mar 9, 2015

• #11

In Notepad++

Go to edit -> line operations -> sort ascending

Hit Ctrl + H for find and replace

Top box put this: ^(.*)(\r?\n\1)+$

Second box put nothing

Check regular expression (bottom left)

Find and replace all

Then: edit -> line operations -> remove empty lines

That's it!

 

Last edited: Mar 9, 2015

HelloInsomnia

Elite Member

Mar 1, 2009

1,801

3,042

• Mar 9, 2015

• #12

conrulez said:

Some popup comes up. Just close it and you should be fine. I've used this tool numerous times.

Click to expand...

That site doesn't do what he needs.

And that popup tries to get you to install a virus. Don't use that site anymore, try this one next time: http://textmechanic.com/