1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

auto download script

Discussion in 'General Programming Chat' started by NutraCash, Dec 5, 2014.

  1. NutraCash

    NutraCash Regular Member Premium Member

    Joined:
    Nov 7, 2014
    Messages:
    230
    Likes Received:
    45
    Occupation:
    Network Operator
    Location:
    Delaware
    Home Page:
    Hello, I need to check a http url and download any new files that are listed in a directory listing, every minute.
    I don't care if it downloads all files repeatedly, as long as it opens them for append so it doesnt have 5000 copies of the same file.


    Does anyone know of a script that can do this?


    PM me please.
     
  2. MrBlue

    MrBlue Senior Member

    Joined:
    Dec 18, 2009
    Messages:
    956
    Likes Received:
    667
    Occupation:
    Web/Bot Developer
    Have a cron job run the following wget command every minute:
    Code:
    wget -r --no-parent http://example.com/folder/
    Once the files are downloaded you can merge them together using a bash script
    Code:
    for i in /path/to/downloaded/files/*
    do
    cat $i >> bigfile
    done