1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[HELP!]Extract URLs from 4GB CSV File. HOW?!

Discussion in 'Black Hat SEO Tools' started by HealeyV3, Dec 13, 2010.

  1. HealeyV3

    HealeyV3 Power Member

    Joined:
    Mar 4, 2009
    Messages:
    521
    Likes Received:
    344
    So, I'm lost.
    I can't open a 4gb csv or txt file in any program.
    Every URL Extractor I've tried doesn't work since everything is formatted in .csv,
    IE: It has "www.whatever.com" quite a bit.

    If I could open the file in notepad, I'd just do a find/replace and then be okay, but I can't.

    I've tried splitting the file into 100 meg versions, but it still locks up my notepad. (I'm running Windows 7, 9 Gigs Ram, i7 Proc)

    I'm out of ideas. If you had to extract urls from a 4gb file, what would YOU do?
     
  2. wkrappen91

    wkrappen91 Power Member

    Joined:
    Sep 9, 2010
    Messages:
    588
    Likes Received:
    720
    Location:
    127.0.0.1
    Asians....

    No seriously. That seems to suck...
    What about renaming it to 4gigfile.txt and try the url extractors again?
     
  3. HealeyV3

    HealeyV3 Power Member

    Joined:
    Mar 4, 2009
    Messages:
    521
    Likes Received:
    344
    Heh, whether it's in .csv for .txt format, it still reads the text the same.
    No URL extractor I've used can do it.
     
  4. detox1978

    detox1978 Junior Member

    Joined:
    Jan 16, 2009
    Messages:
    110
    Likes Received:
    33
    Import it into Access (or another database).

    If you aren't comfortable using databases, split it into smaller files and open it in Excel.
     
  5. nambooooo

    nambooooo Regular Member

    Joined:
    May 21, 2009
    Messages:
    219
    Likes Received:
    71
    Location:
    GreenMTN
    Split that into 10 megs files, then run a parser to extract links from all files in given directory. I used to have such a parser like 5 years ago, can't remember the name, sorry, but it exists. On a second thought... Put those 10 megs files in different directories, like 10 in each, so the parser doesn't freeze. Huh ... hard task.
     
  6. MarketerMac

    MarketerMac Regular Member

    Joined:
    Oct 26, 2009
    Messages:
    247
    Likes Received:
    101
  7. seolaptop

    seolaptop Power Member

    Joined:
    Aug 9, 2010
    Messages:
    504
    Likes Received:
    55
    4GB CSV URLs. Are you kidding?
     
  8. gapster

    gapster Registered Member

    Joined:
    Aug 6, 2010
    Messages:
    61
    Likes Received:
    32
    Occupation:
    IM full time
    Location:
    USA NW
    Try ornj[dot]net/araneae/ or guugle for araneae. It is free and is like notepad on steroids, plus lots of features and very min mem usage. Results will depend on your system, but I have openned 1 gig sql files on my circa 2007 XP clunker.
     
  9. donduck

    donduck Junior Member

    Joined:
    May 9, 2007
    Messages:
    162
    Likes Received:
    17
    Location:
    Australia
    use office 2007 --- xl
     
  10. HealeyV3

    HealeyV3 Power Member

    Joined:
    Mar 4, 2009
    Messages:
    521
    Likes Received:
    344
    Thanks for all the help. Ill let you guys know.
    I really don't understand excel though... I split it into 100 meg files and it still says something like" unable to open full file"
    My original idea was to go into each excel, and then just delete every other unnecessary column.

    Hmm.
    Posted via Mobile Device
     
  11. ericsson

    ericsson Elite Member Premium Member

    Joined:
    Apr 25, 2009
    Messages:
    2,642
    Likes Received:
    8,132
    Occupation:
    www
    Location:
    Swe
    Home Page:
    Use a converter.. csv to txt (if there is any out there)
    then split the files into 100 pieces.
    and theeeeeeeen you run it with an url extractor. there is standalone versions, or online versions.
     
  12. Pikachoo

    Pikachoo Junior Member

    Joined:
    Apr 16, 2010
    Messages:
    151
    Likes Received:
    121
    Location:
    UK
    I think Ultra Edit will cope with a 2GB file
     
  13. walker

    walker Junior Member

    Joined:
    Feb 19, 2009
    Messages:
    146
    Likes Received:
    49
    write a code, read line by line, not 100% sure it will work. but worth try.