1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Vim for large .txt file split & duplicated URLs removal

Discussion in 'Black Hat SEO' started by observer, Mar 7, 2011.

  1. observer

    observer Power Member

    Joined:
    Apr 7, 2010
    Messages:
    731
    Likes Received:
    22
    Hi guys,
    I've just downloaded gVim, and it seems it can remove duplicated and split large files. I just can't figure out how removal works. I tried to follow some blog's instruction, no luck.

    Anybody used this program?
     
  2. observer

    observer Power Member

    Joined:
    Apr 7, 2010
    Messages:
    731
    Likes Received:
    22
    Ok, I seem to figure out the duplicates removal procedure with gVim. Now, how do I split these huge files? with gVim or without :)
     
  3. Autumn

    Autumn Elite Member

    Joined:
    Nov 18, 2010
    Messages:
    2,197
    Likes Received:
    3,041
    Occupation:
    I figure out ways to make money online and then au
    Location:
    Spamville
    Look at the unix "split" command.
     
  4. observer

    observer Power Member

    Joined:
    Apr 7, 2010
    Messages:
    731
    Likes Received:
    22
    Sorry, could you please point in details? Not sure where to look and for what exactly. (not a techie)
     
  5. Autumn

    Autumn Elite Member

    Joined:
    Nov 18, 2010
    Messages:
    2,197
    Likes Received:
    3,041
    Occupation:
    I figure out ways to make money online and then au
    Location:
    Spamville
    I assume if you're using gvim then you're using linux or os x? Type in "man split" in a terminal or the shell prompt on your server for documentation.

    If you want to split file.txt into chunks of 100 lines, you would use:

    Code:
    split -l 100 file.txt
    
    Here's an online version of the docs:
    http://unixhelp.ed.ac.uk/CGI/man-cgi?split