Please help me to import large data files into mySQL database

I am trying to load the infamous 2006 AOL leak data into a db for use in a custom keyword selection toolkit based on MySQL and Workbench, with maybe an extra OOObase layer for fancy reports

the problem I am having is to find tools to handle the rather large data files. The CSVs are too big to import using either PHPmyadmin or Workbench. If I use a file splitting tool to make the file size manageable then I get column misalignment ruining any chance for an easy, error free import.

I am grateful for any suggestions..

If you have access to php configuration files, you can tweak it to increase the memory & timeout limits and still use phpmyadmin to import the files.

have you tried bigdump?

You have to zip or gzip the file, MySQL can import compressed files.

Sounds like you want to import to a database on a remote server. Just SSH in and do it from the cli:

http://dev.mysql.com/doc/refman/5.6/en/mysqlimport.html

Bigdump was a big help. I used a filesplitter go get it down to workable chunks and cleaned the files down to only one column (I only wanted the query data). I simply saved all of the cleaned files as csv files in the bigdump directory, told silkuli to delete the completed documents and go to the next with the visual cue of file completion dialog from bigdump. About 50 5 meg csvs to load and am at about 25.

I have to say that my SQL is non existent and I am now stuck with a db of keywords that duplicate badly. Is is currently at 20 million and climbing.

I am researching and have found to build a new table, query for uniques and transfer data, then removing old table. The silver spoonfeeding question of the day is this...Once I remove all of the dupes. Is there anything I need to do to make all records that I enter from now that go into other tables relate to the kw data, such as key ID or such?

I will be downloading and using bigdump to enter data into other tables for use in formulas that tie in with each keyword.I am thinking that I might do this with MySQL data and an OpenOffice Base frontend, or maybe Workbench. If you have a suggestion for this also from experience I would appreciate it.This DB is going to be the engine for an Automated Adsense Network Keyword Research System. If all goes as planned, I might post updates on it in a journey thread.Thanks for the help to all that offered advice.

You 're going to use 7 years old data for an adsense keyword research system??

maybe u should learn basic mysql or whatever DBMS u plan to use... each one has commands for this. for example in mysql you can do a LOAD DATA INFILE command and various other stuff.

jazzc said:

You 're going to use 7 years old data for an adsense keyword research system??

Click to expand...

Yeah, I plan on using the queries as seeds for the system.
Now that the seeds are entered, the first stop is still GAWT suggestions for traffic and value.

"maybe u should learn basic mysql or whatever DBMS u plan to use... each one has commands for this." I agree. DBMS are my weak point.

madoctopus said:

maybe u should learn basic mysql or whatever DBMS u plan to use... each one has commands for this. for example in mysql you can do a LOAD DATA INFILE command and various other stuff.

Click to expand...

I went a slightly different route (I understand it takes more resources, but I am on a local system and would rather not ssh in).

I used:

I created a clone table (only one column, so simple for me to do 0

then I


It is still working on the migration, so my fingers are crossed for success.