1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Need to remove duplicates from excel or txt file...what software to use?

Discussion in 'BlackHat Lounge' started by soberealtor, Nov 5, 2010.

  1. soberealtor

    soberealtor Junior Member

    Joined:
    Jun 8, 2009
    Messages:
    134
    Likes Received:
    34
    Cannot find this. Tried ablebits for excel free trial...is there any other program?
     
  2. popcrdom29

    popcrdom29 Jr. VIP Jr. VIP Premium Member

    Joined:
    May 20, 2008
    Messages:
    807
    Likes Received:
    518
    Excel 2007 or 2010 is much easier to use. If you don't have it you can download it from some file sharing sites.
     
    • Thanks Thanks x 1
  3. soberealtor

    soberealtor Junior Member

    Joined:
    Jun 8, 2009
    Messages:
    134
    Likes Received:
    34
    iam assuming it had a duplicate removal stock now...might as well upgrade now...thanks
     
  4. blackjack

    blackjack Regular Member

    Joined:
    Nov 23, 2007
    Messages:
    276
    Likes Received:
    51
    Excel 2003 has the option too.

    Under Data>>>Filters>>>Advance Filter>>>Unique Only
     
    • Thanks Thanks x 1
  5. xpleet

    xpleet Regular Member

    Joined:
    Jan 18, 2010
    Messages:
    377
    Likes Received:
    327
    Location:
    Morocco
    You can use Notepad++
     
  6. steveJD

    steveJD Junior Member

    Joined:
    Nov 29, 2008
    Messages:
    146
    Likes Received:
    142
    Quick way to do this with a formula in any version of excel:

    if the data is in column A then first sort the column, then in cell B1 put the formula =EXACT(A1:A2) and copy the formula to the end of the data. You can remove data that is TRUE in column B.
     
    • Thanks Thanks x 1
  7. falconer

    falconer Registered Member

    Joined:
    Dec 17, 2009
    Messages:
    92
    Likes Received:
    116
    This is useful -

    h**p://textop.us/Lines-tools/Delete-Duplicate-Lines
     
    • Thanks Thanks x 2
  8. darkmobius

    darkmobius Regular Member

    Joined:
    Jul 16, 2008
    Messages:
    238
    Likes Received:
    227
    Occupation:
    software developer
    Location:
    canada
    Home Page:
    textpad can sort and remove dups for you and its free
     
    • Thanks Thanks x 2
  9. sdtopensied

    sdtopensied Regular Member

    Joined:
    Sep 1, 2009
    Messages:
    211
    Likes Received:
    113
    Occupation:
    Software Developer
    Location:
    Western Hemisphere
    I'm assuming that since you have Microsoft Excel, you have (or can get) Microsoft Access.

    Import the files into an Access MDB.

    Create a new query, and go to the SQL view...

    SELECT DISTINCT
    FIELD1
    ,FIELD2
    ,FIELD3
    ,.and so on
    FROM yourtablename

    The result will be a de-duplicated version of the original table. "SELECT DISTINCT" is ANSI SQL, so it will also work in MySQL, SQL Server Express, or any other flavor of SQL you might have or any product that has a SQL interpreter (FoxPro, etc.).

    Feel free to PM me if you need some more help.

    -SDT
     
  10. phatzilla

    phatzilla Supreme Member

    Joined:
    Apr 9, 2009
    Messages:
    1,366
    Likes Received:
    1,017
    i like poweredit
     
  11. Dreama

    Dreama Newbie

    Joined:
    Jun 30, 2010
    Messages:
    22
    Likes Received:
    1
    This came in handy, thanks! Need to remove duplicates asap to start posting. :p
     
    • Thanks Thanks x 1
  12. ermiz

    ermiz Newbie

    Joined:
    Nov 11, 2007
    Messages:
    30
    Likes Received:
    10
    You can use ASAP utilities a plugin for for Excel
     
  13. darkmobius

    darkmobius Regular Member

    Joined:
    Jul 16, 2008
    Messages:
    238
    Likes Received:
    227
    Occupation:
    software developer
    Location:
    canada
    Home Page:
    I'm glad it's handy to you, I use this both at work and at home.