1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Need a program to download a bunch of URLs (web pages)

Discussion in 'Black Hat SEO Tools' started by mrmarchuk, Jul 29, 2014.

  1. mrmarchuk

    mrmarchuk Registered Member

    Joined:
    Jun 4, 2012
    Messages:
    65
    Likes Received:
    30
    Occupation:
    Working From Home
    Location:
    Portland, OR
    So I'm wanting to scrape some data off a site that has a ton of information on it, I've tried using HTTrack, but it's having issues doing what I need it to.

    If someone knows of a script or a program that can download a list of html files and pictures (and save them as the same filenames), I'd love to hear about it.

    p.s. When I loaded a list of 50k URLs into HTTrack, I was able to get it to work, but the larger (parent list of 1.01million URLs) one won't work for some reason. Tried splitting it to lists of 500k URLs, but that's not working either.

    p.p.s. Also, the reason I want to download them to the PC is so that I can easier scrape/refer back to the files later - the site has temporary listings (think like craigslist postings - they expire after X days, I want to be able to store the data for later).
     
  2. MikeyMikey13

    MikeyMikey13 Supreme Member

    Joined:
    May 25, 2014
    Messages:
    1,393
    Likes Received:
    384
    This may be of help - I use it for downloading files. Personally have not explored it too much.

    - Jaydownloader
     
    • Thanks Thanks x 1
  3. coloredhatnew

    coloredhatnew Registered Member

    Joined:
    Aug 17, 2013
    Messages:
    69
    Likes Received:
    9
    Look for web reaper.
     
    • Thanks Thanks x 1
  4. blackhat777

    blackhat777 Elite Member

    Joined:
    Jun 25, 2011
    Messages:
    1,784
    Likes Received:
    653
    i think you will need some custom coded bot.
     
  5. rpkrocks

    rpkrocks Senior Member

    Joined:
    Feb 4, 2010
    Messages:
    986
    Likes Received:
    935
    Gender:
    Male
    Location:
    Bḥārata
    Home Page:
    Did you tried IDM site grabber?
     
    • Thanks Thanks x 1
  6. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,644
    Likes Received:
    11,311
    Occupation:
    CHEAP
    Location:
    DATASETS
    Home Page:
    All you need is included in a very expensive program called wget.

    By the way, wget is free.
     
    • Thanks Thanks x 1
  7. SlickMax

    SlickMax Newbie

    Joined:
    Aug 23, 2013
    Messages:
    30
    Likes Received:
    3
    Occupation:
    SEO Services
    Location:
    USA
    Home Page:
    Web Data Extraction software from automationanywhere is a great software, pricey but very good. A few of my friends use it and are very happy with it. I have not personally tried it yet though.