1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[NEED] Bot to Remove duplicate Lines.

Discussion in 'Hire a Freelancer' started by socketul, Dec 13, 2014.

  1. socketul

    socketul Newbie

    Joined:
    Jul 7, 2014
    Messages:
    29
    Likes Received:
    4
    Hello,

    So let's say I have a .txt file with 50k Lines. I need a bot that will search duplicate LINES and remove them.
    Lines length are small like:

    ht TTp://Dog love cats.com/Lovecats
     
    • Thanks Thanks x 1
  2. iPinner

    iPinner Senior Member Premium Member

    Joined:
    Apr 20, 2012
    Messages:
    1,062
    Likes Received:
    1,039
    I will save you some time and money
    Code:
    http://textmechanic.com/Remove-Duplicate-Lines.html
     
  3. Brad100

    Brad100 Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 9, 2014
    Messages:
    1,347
    Likes Received:
    958
    Gender:
    Male
    get Notepad++ and use the find and replace feature to do this.
     
  4. HoNeYBiRD

    HoNeYBiRD Jr. VIP Jr. VIP

    Joined:
    May 1, 2009
    Messages:
    5,888
    Likes Received:
    7,127
    Gender:
    Male
    Occupation:
    Geographer, Tourism Manager
    Location:
    Ghosted
  5. ibins

    ibins Junior Member

    Joined:
    May 9, 2011
    Messages:
    176
    Likes Received:
    39
    Location:
    really world wide
    If you want to build your own bot, you can use the free https://www.autoitscript.com/site/autoit/ scripting language and build your exe file, here's something that will get you started:

    Code:
    #include <Array.au3>
    #include <File.au3>
    #include <FileConstants.au3>
    Local $out, $i, $unique, $arr, $in = "c:\test.txt"
    
    
    _FileReadToArray($in, $arr)
    $unique = _ArrayUnique($arr)
    $out = FileOpen("c:\test_unique.txt",$FO_OVERWRITE)
    
    
    For $i = 0 To UBound($unique) - 1
    if ($i >1) Then
     FileWrite($out, $unique[$i] & @CRLF)
    EndIf
    Next
    
    
    FileClose($out)
    
     
  6. aliachmad

    aliachmad Newbie

    Joined:
    Apr 5, 2014
    Messages:
    42
    Likes Received:
    8
    get Notepad++ another solution
     
  7. botrockets

    botrockets Regular Member

    Joined:
    Mar 16, 2013
    Messages:
    272
    Likes Received:
    463
    Occupation:
    Software Developer
    Location:
    Saint T.N.
    If you want to remove duplicate lines from billions or trillions of lines of text file I have created a program for that. If interested I can show you a demo.
     
  8. pishty

    pishty Junior Member

    Joined:
    Dec 17, 2012
    Messages:
    132
    Likes Received:
    26
    Location:
    Bucharest
    I have a quick solution for you. Just send me the file.
     
  9. BigTroll

    BigTroll Elite Member

    Joined:
    Oct 15, 2014
    Messages:
    1,703
    Likes Received:
    623
    Occupation:
    CPA
    Location:
    2009
  10. SEODEMON

    SEODEMON Regular Member

    Joined:
    Sep 15, 2010
    Messages:
    220
    Likes Received:
    64
    PHP or .bat Script real easy just do the research. No spoon feeding.
     
  11. hackedd

    hackedd Junior Member

    Joined:
    Aug 11, 2010
    Messages:
    138
    Likes Received:
    172
    Gender:
    Male
    Open with Excel - go to DATA tab -> Remove duplicate lines. Easy as ABC