have any tool for compare and remove duplicate email in many text file ?

C

chaiya555

Newbie

May 28, 2018

3

0

• May 28, 2018

• #1

have any tool for compare and remove duplicate email in many text file ?

see example imgur.com/3YEPsVx

thank you very very much for help me. god bless you .

 

botrockets

Regular Member

Mar 16, 2013

389

647

• May 30, 2018

• #2

Hey Dude,

Use this python script to do your job. It's pretty easy !

Code:

import os

currentDirectory = os.getcwd()
textFiles = []
for f in os.listdir(currentDirectory):
    if (f.endswith('.txt') or f.endswith('.TXT')) and (f != 'output.txt'):
        textFiles.append(f)

d = {}
for tf in textFiles:
    cf = open(tf, "r")
    for line in cf:
        line = line.strip('\n') + '\n'
        d[line] = 0
    cf.close()

wf = open("output.txt", "w")
wf.writelines(d)

 

HoNeYBiRD

Elite Member

May 1, 2009

10,187

12,580

• May 30, 2018

• #3

If you have the input like on the picture, one email address per line, it's easy with Notepad++ and TextFX plugin: https://stackoverflow.com/a/3958364

 

M

mickyfu

Banned due to many warnings

Dec 14, 2011

9,430

27,944

• May 30, 2018

• #4

Yeah, notepad. Use find and replace. Plus there are many online text tools to do the same.

 

CoolNicks

Registered Member

Aug 21, 2016

55

23

• May 30, 2018

• #5

Use cat (linux) or type. (windows) to concatenate all the files, paste into
Excel or notepad plus plus

MS excel, notepad plus plus have remove duplicate entries feature.

 

C

chaiya555

Newbie

May 28, 2018

3

0

• May 30, 2018
• Thread Starter Thread Starter

• #6

botrockets said:

Hey Dude,

Use this python script to do your job. It's pretty easy !

Code:

import os

currentDirectory = os.getcwd()
textFiles = []
for f in os.listdir(currentDirectory):
    if (f.endswith('.txt') or f.endswith('.TXT')) and (f != 'output.txt'):
        textFiles.append(f)

d = {}
for tf in textFiles:
    cf = open(tf, "r")
    for line in cf:
        line = line.strip('\n') + '\n'
        d[line] = 0
    cf.close()

wf = open("output.txt", "w")
wf.writelines(d)

Click to expand...

i don't have knowledge about programing i am sad sir.
but if it is javascript i think i can do it.

 

C

chaiya555

Newbie

May 28, 2018

3

0

• May 30, 2018
• Thread Starter Thread Starter

• #7

hello all sir, now i found tool for resolve my problem it is here youtube. com/watch?v=KDA04sVIdNg
thank a lot for your comment. GOD BLESS YOU.

 

dabandit

Junior Member

Feb 21, 2008

159

65

• May 30, 2018

• #8

You don't need anything complex. Use the sort and uniq commands. There's plenty of online dedup pages.

Code:

sort textfile | uniq