Script/software that deletes multiple lines in .txt files

R

raybeam

Newbie

Jan 12, 2011

41

2

• Feb 23, 2011

• #1

The word in the title should be "duplicate", then you might understand me better.
It's a simple little task, but haven't found any program that does the job..

What I want is a program that finds multiples and lets me delete them:
email1
email2
email3
email1
email2
email4
-------------
In my case above result would be

email1
email2
email3
email4

 

B

badpete

Newbie

Dec 18, 2009

13

4

• Feb 23, 2011

• #2

Import data into excel and remove duplicates then copy and paste back to a .txt

 

R

RootLocus

Newbie

Jan 5, 2011

3

1

• Feb 23, 2011

• #3

Loop each line of the txt loading them one by one in a List.
If the entrance is already present in that List, skip the current line.
Pseudo-code will be.

List li;
For Each line in txt
For Each element in li
If line == element
li.add(element);

Easy

 

C

chris456

Regular Member

May 17, 2010

281

572

• Feb 23, 2011

• #4

TextPipe is best - search for it here , it has 1000 functions ready to use ,
Extract link or Link Extractor (I don't remember the name) is also removing duplicate lines automaticaly , you can find it here too.
many other text editors do that too , very simple task for every editor . almost everywhere the function should name : remove duplicate lines

 

R

raybeam

Newbie

Jan 12, 2011

41

2

• Feb 24, 2011
• Thread Starter Thread Starter

• #5

Thanks for your replies, will start taking a look at textpipe then.
Love this forum, so much knowledge.

 

R

raybeam

Newbie

Jan 12, 2011

41

2

• Feb 24, 2011
• Thread Starter Thread Starter

• #6

I can recommend using TextPipe Pro if anyone else has the same issue in the future.

 

alaltaierii

Supreme Member

Jun 11, 2010

1,494

388

• Feb 24, 2011

• #7

I'm using this script: http://textmechanic.com/Remove-Duplicate-Lines.html

 

fatboy

Elite Member

Aug 13, 2008

1,615

3,262

• Feb 24, 2011

• #8

If you are using Linux, type something like:

Code:

uniq original.list.txt > dedupe.list.txt

That should help you without installing anything else

 

corstenroam

Junior Member

Jun 10, 2010

106

10

• Feb 24, 2011

• #9

what about textpipe. I have a keygen if u need it