[HELP!]Extract URLs from 4GB CSV File. HOW?!

HealeyV3

HealeyV3

Power Member
Joined
Mar 4, 2009
Messages
521
Reaction score
348
So, I'm lost.
I can't open a 4gb csv or txt file in any program.
Every URL Extractor I've tried doesn't work since everything is formatted in .csv,
IE: It has "www.whatever.com" quite a bit.

If I could open the file in notepad, I'd just do a find/replace and then be okay, but I can't.

I've tried splitting the file into 100 meg versions, but it still locks up my notepad. (I'm running Windows 7, 9 Gigs Ram, i7 Proc)

I'm out of ideas. If you had to extract urls from a 4gb file, what would YOU do?
 
Advertise on BHW
Asians....

No seriously. That seems to suck...
What about renaming it to 4gigfile.txt and try the url extractors again?
 
wkrappen91 said:
Asians....

No seriously. That seems to suck...
What about renaming it to 4gigfile.txt and try the url extractors again?
Click to expand...

Heh, whether it's in .csv for .txt format, it still reads the text the same.
No URL extractor I've used can do it.
 
Import it into Access (or another database).

If you aren't comfortable using databases, split it into smaller files and open it in Excel.
 
Split that into 10 megs files, then run a parser to extract links from all files in given directory. I used to have such a parser like 5 years ago, can't remember the name, sorry, but it exists. On a second thought... Put those 10 megs files in different directories, like 10 in each, so the parser doesn't freeze. Huh ... hard task.
 
Try ornj[dot]net/araneae/ or guugle for araneae. It is free and is like notepad on steroids, plus lots of features and very min mem usage. Results will depend on your system, but I have openned 1 gig sql files on my circa 2007 XP clunker.
 
Thanks for all the help. Ill let you guys know.
I really don't understand excel though... I split it into 100 meg files and it still says something like" unable to open full file"
My original idea was to go into each excel, and then just delete every other unnecessary column.

Hmm.
Posted via Mobile Device
 
Use a converter.. csv to txt (if there is any out there)
then split the files into 100 pieces.
and theeeeeeeen you run it with an url extractor. there is standalone versions, or online versions.
 
write a code, read line by line, not 100% sure it will work. but worth try.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top