[NEED] Bot to Remove duplicate Lines.

S

socketul

Newbie

Jul 7, 2014

29

5

• Dec 13, 2014

• #1

Hello,

So let's say I have a .txt file with 50k Lines. I need a bot that will search duplicate LINES and remove them.
Lines length are small like:

ht TTp://Dog love cats.com/Lovecats

 

Whateverest

Elite Member

Apr 20, 2012

2,036

1,642

• Dec 13, 2014

• #2

I will save you some time and money

Code:

http://textmechanic.com/Remove-Duplicate-Lines.html

 

Brad100

Supreme Member

Nov 9, 2014

1,346

1,001

• Dec 13, 2014

• #3

get Notepad++ and use the find and replace feature to do this.

 

HoNeYBiRD

Elite Member

May 1, 2009

10,187

12,580

• Dec 13, 2014

• #4

another solution would be notepad++ with textfx addon, the addon has the function to do this with a click of a button

 

ibins

Junior Member

May 9, 2011

199

43

• Dec 13, 2014

• #5

If you want to build your own bot, you can use the free https://www.autoitscript.com/site/autoit/ scripting language and build your exe file, here's something that will get you started:

Code:

#include <Array.au3>
#include <File.au3>
#include <FileConstants.au3>
Local $out, $i, $unique, $arr, $in = "c:\test.txt"


_FileReadToArray($in, $arr)
$unique = _ArrayUnique($arr)
$out = FileOpen("c:\test_unique.txt",$FO_OVERWRITE)


For $i = 0 To UBound($unique) - 1
if ($i >1) Then
 FileWrite($out, $unique[$i] & @CRLF)
EndIf
Next


FileClose($out)

 

A

aliachmad

Newbie

Apr 5, 2014

43

8

• Dec 13, 2014

• #6

get Notepad++ another solution

 

botrockets

Regular Member

Mar 16, 2013

389

647

• Dec 14, 2014

• #7

If you want to remove duplicate lines from billions or trillions of lines of text file I have created a program for that. If interested I can show you a demo.

 

pishty

Regular Member

Dec 17, 2012

220

68

• Dec 14, 2014

• #8

I have a quick solution for you. Just send me the file.

 

The Blackhat Guy

Elite Member

Oct 15, 2014

2,096

936

• Dec 14, 2014

• #9

http://www.blackhatworld.com/blackh...all-duplicate-original-lines-please-help.html
any help here?

 

SEODEMON

Regular Member

Sep 15, 2010

251

78

• Dec 14, 2014

• #10

PHP or .bat Script real easy just do the research. No spoon feeding.

 

H

hackedd

Junior Member

Aug 11, 2010

147

178

• Dec 16, 2014

• #11

Open with Excel - go to DATA tab -> Remove duplicate lines. Easy as ABC