auto download script

Hello, I need to check a http url and download any new files that are listed in a directory listing, every minute.
I don't care if it downloads all files repeatedly, as long as it opens them for append so it doesnt have 5000 copies of the same file.


Does anyone know of a script that can do this?


PM me please.

Have a cron job run the following wget command every minute:

Once the files are downloaded you can merge them together using a bash script