1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Excel tools - Need some help with huge volumes of data

Discussion in 'BlackHat Lounge' started by mrmarchuk, Jul 25, 2014.

Tags:
  1. mrmarchuk

    mrmarchuk Registered Member

    Joined:
    Jun 4, 2012
    Messages:
    65
    Likes Received:
    30
    Occupation:
    Working From Home
    Location:
    Portland, OR
    Hey, so I've got ~27 excel files ( I do an export each week) with data in each column.

    Each file has ~100k rows and ~50 columns of data.

    In order to be able to use the data in a more efficient way, I need to be able to
    1. extrapolate certain columns of data into a new workbook
    2. check for duplicate rows
    3. remove duplicate rows (leaving only one of the duplicates)
    4. and save that as an excel worksheet/workbook.

    Does anyone else use anything that might help me with this? because as it stands, I've got over 2.7million rows of data and I need about 10 columns from it (aka lots of manual labor), not to mention checking for dupes.

    Thx ahead.
     
  2. sohom

    sohom Senior Member

    Joined:
    May 26, 2013
    Messages:
    981
    Likes Received:
    175
    Location:
    not in Past
    if by some how in excel ,above thing possible then good
    otherwise,if you need,I can make a Bot to do that job for you :)
     
  3. trevormorley

    trevormorley Junior Member

    Joined:
    Feb 25, 2010
    Messages:
    170
    Likes Received:
    46
    try asap utilities for excel (free download) - it can remove duplicates and just leave one instance, should do the trick for you
     
  4. mrmarchuk

    mrmarchuk Registered Member

    Joined:
    Jun 4, 2012
    Messages:
    65
    Likes Received:
    30
    Occupation:
    Working From Home
    Location:
    Portland, OR
    I'll check it out, thanks.. I know I can remove dupes using Excel itself, but will asap utilities then combine the remaining data?