[Method] Split large txt/csv files in multiple pieces

Discussion in 'Black Hat SEO' started by kappa84, Feb 6, 2011.

  1. kappa84

    kappa84 Power Member

    Joined:
    May 19, 2010
    Messages:
    736
    Likes Received:
    337
    Location:
    Bath, UK
    I do not know if this belong here, however here it is. I needed this to split a .txt keywords list (about 4 million keywords) into small 10k pieces and had no idea how to do this (splitting lists by numbers), until I found gsplit...

    SO:

    1. Download gsplit.

    2. On "original file", click browse and choose your txt/csv large file.

    3. Click "Destination folder", "Browse" and select where you want to output your newly created files.

    4. Click "Type and size", click on "blocked pieces" and select from the drop-down ?I want to split after the nth occurrence of a specified pattern?; then choose ?Split after the occurrence number? and enter the number of lines you want your new files to contain (I've chosen 10,000 because I wanted x .txt files containing 10,000 keywords, so you choose whatever no. you want)

    5. Click "filenames" and on "piece name mask" write "{ofw}_{num}{ore}" (without the quotes) - this will name you new created files like kwds1.txt, kwds2.txt etc.

    6. Click "other properties" and check ?do not add tags to piece files?.

    This is it, it also works with very large text files with URLs and so on, hope it will help you. I searched a lot for this "trick" and once I found it I thought to share it with you (works great with ScrapeBox).

    Best!
     
    • Thanks Thanks x 8
  2. BugFixed

    BugFixed Junior Member

    Joined:
    Sep 24, 2010
    Messages:
    130
    Likes Received:
    40
    Another one: small, fast, and free File Splitter (for Windows)
    Code:
    http://www.filesplitter.org
     
    • Thanks Thanks x 1
  3. blackberry

    blackberry Power Member

    Joined:
    Apr 26, 2009
    Messages:
    718
    Likes Received:
    241
    Occupation:
    Making money
    Location:
    Planet Earth
    This will do exactly what your looking to do:
    Code:
    http://download.cnet.com/Npust-Email-List-Manager/3000-2369_4-10170841.html
     
    • Thanks Thanks x 1
  4. dichotom

    dichotom Jr. VIP Jr. VIP

    Joined:
    Dec 9, 2008
    Messages:
    2,114
    Likes Received:
    560
    Two I have used are Text Wedge (good for small files and does many cool things) and HJsplit (not many options, but it can handle GIGANTIC files that all the other splitters choke on)
     
    • Thanks Thanks x 1
  5. Anubis1980

    Anubis1980 Regular Member

    Joined:
    Mar 20, 2010
    Messages:
    276
    Likes Received:
    81
    Occupation:
    webmaster and father
    Old thread here
     
    • Thanks Thanks x 1
  6. djmonster

    djmonster Regular Member

    Joined:
    Jan 16, 2010
    Messages:
    202
    Likes Received:
    40
    This is what I need .... Thank For the tutorial Op
     
  7. danfolt

    danfolt Newbie

    Joined:
    Nov 23, 2013
    Messages:
    4
    Likes Received:
    0
    Gender:
    Male
    Location:
    Italy
    Somebody knows how to split a big(about 300K) email mailing list into multiple files (create them)based by keywords or best append them to(already created txt files with keyword names)?
    Example:
    [email protected]
    [email protected]
    [email protected]
    [email protected]
    [email protected]

    and I want to split this big file with those extracted emails into 2 and more files as hotel.txt and restaurant.txt etc and constantly build(append to) them.
    There is already a scrapebox function to split harvested urls into tld multiple files but I also want to separate them by keywords
     
    Last edited: May 5, 2018
  8. HoNeYBiRD

    HoNeYBiRD Jr. VIP Jr. VIP

    Joined:
    May 1, 2009
    Messages:
    8,067
    Likes Received:
    8,917
    Gender:
    Male
    Occupation:
    Geographer, Tourism Manager
    Location:
    Ghosted
    Well, there's a tool called Send-Safe List Manager. It has plenty of filtering options, you can try that.
     
    • Thanks Thanks x 2
  9. danfolt

    danfolt Newbie

    Joined:
    Nov 23, 2013
    Messages:
    4
    Likes Received:
    0
    Gender:
    Male
    Location:
    Italy
    Downloaded and it looks very good, seems like filter addresses should do that, I will try it and inform You,
    many thanks Honeybird!
     
  10. ComputerJunkie

    ComputerJunkie Regular Member

    Joined:
    Oct 9, 2012
    Messages:
    274
    Likes Received:
    85
    simple .bat file can do any file;

    @echo off
    setLocal EnableDelayedExpansion

    set limit=500000
    set file=raw_urls.csv
    set lineCounter=1
    set filenameCounter=1

    set name=
    set extension=
    for %%a in (%file%) do (
    set "name=%%~na"
    set "extension=%%~xa"
    )

    for /f "tokens=*" %%a in (%file%) do (
    set splitFile=!name!-part!filenameCounter!!extension!
    if !lineCounter! gtr !limit! (
    set /a filenameCounter=!filenameCounter! + 1
    set lineCounter=1
    echo Created !splitFile!.
    )
    echo %%a>> !splitFile!

    set /a lineCounter=!lineCounter! + 1
    )

    copy the code, paste it into notepad and save as a .bat file, put it in a folder with the file you want to split, edit the file size (set limit="$") and change the (set file="$filename") to your file name you want to split, then double click the bat file.
     
    • Thanks Thanks x 1