How to remove duplicate lines ?

Salamouna · Mar 23, 2019

Hello i hope this will be the best section of the thread
i want to remove duplicate lines from notepad text those line can be text or numbers
i don't want just remove the duplicate lines but both .

Means :
1
2
1

results 2
so i get only the unique lines

Aty · Mar 23, 2019

ctrl+h
1
'nothing'
enter

Naroha · Mar 23, 2019

Well you can copy it to excel, highlight duplicate lines and delete the highlighted ones

alain12 · Mar 23, 2019

Code:

cd c:\where\your\file\is
type yourfile.txt | sort /unique > newfile.txt

In the command prompt

kveldulv · Mar 23, 2019

https://www.emeditor.com/

Then Ctrl-Shift-D

Salamouna · Mar 24, 2019

alain12 said:
Code:

cd c:\where\your\file\is type yourfile.txt | sort /unique > newfile.txt

In the command prompt

It gived me 1 and 2 not only 2

bkenn · Mar 24, 2019

Notepad++ is your best bet when working with text files.

HoNeYBiRD · Mar 24, 2019

Notepad++ has a downloadable addon called TextFX, which is pretty useful for other tasks too. Here's how to remove duplicate lines with it: https://stackoverflow.com/a/3958364

Edit: forget the addon, i just realized that's not exactly what you want. To achieve what you want, follow this: https://stackoverflow.com/a/35350261. Still Notepad++ btw.

IGFella · Mar 24, 2019

http://www.dedupelist.com

Salamouna · Mar 24, 2019

kveldulv said:
https://www.emeditor.com/

Then Ctrl-Shift-D

View attachment 114133

i used this thank you but not that command cause it will give 1 and 2
i used delete/bookmark

Akermi · Apr 11, 2019

Hi majid, if you still looking for a solution, I could make custom python script for you.

verrocchio · Apr 15, 2019

I'm bored and yes, this is a backwards one liner but it was kind of a stream of consciousness

Code:

somehost:~$ cat a
This line will be removed
This line will remain
This line too
Because they are unique
However
This line will be removed
And this is different of a raw uniq because it removes both repeated lines

Code:

somehost:~$ cat a | sort | uniq -c | grep  -v -E  '^\s+[2-9]' | sed 's/^\s*[0-9]*\s*//'
And this is different of a raw uniq because it removes both repeated lines
Because they are unique
However
This line too
This line will remain
somehost:~$

How to remove duplicate lines ?

Salamouna

Elite Member

Aty

Elite Member

Naroha

Elite Member

alain12

Registered Member

kveldulv

Regular Member

Salamouna

Elite Member

bkenn

Banned - selling outside of the marketplace.

HoNeYBiRD

Elite Member

IGFella

Regular Member

Salamouna

Elite Member

Akermi

Registered Member

verrocchio

Newbie

Main Menu

Marketplace

Making Money

BlackHat World