Need a program to download a bunch of URLs (web pages)

So I'm wanting to scrape some data off a site that has a ton of information on it, I've tried using HTTrack, but it's having issues doing what I need it to.

If someone knows of a script or a program that can download a list of html files and pictures (and save them as the same filenames), I'd love to hear about it.

p.s. When I loaded a list of 50k URLs into HTTrack, I was able to get it to work, but the larger (parent list of 1.01million URLs) one won't work for some reason. Tried splitting it to lists of 500k URLs, but that's not working either.

p.p.s. Also, the reason I want to download them to the PC is so that I can easier scrape/refer back to the files later - the site has temporary listings (think like craigslist postings - they expire after X days, I want to be able to store the data for later).

This may be of help - I use it for downloading files. Personally have not explored it too much.

- Jaydownloader

Look for web reaper.

i think you will need some custom coded bot.

Did you tried IDM site grabber?

All you need is included in a very expensive program called wget.

By the way, wget is free.

Web Data Extraction software from automationanywhere is a great software, pricey but very good. A few of my friends use it and are very happy with it. I have not personally tried it yet though.