[BETA] Wayback Machine Downloader

[moneyguru2013]

Thanks for this share, i wouldn't advise using these articles on your main money sites, unless you are totally blackhat. I have seen lawsuits and Google striking down websites for stolen content.

 

[anssoft]

Interested in this. Keep moving forward.

 

[NulledCode]

I've not used it yet, but is there any way in which it could be used to scrape just the expired content, as opposed to the whole web files?

Realised you might have just slightly answered this above. It can take a considerable amount of time to do that manually, especially if a site has a lot of good content.

At the moment the restore process will download the entire website. I will be adding new ways to download such as downloading all files in a certain date range, or downloading single pages.

@cnick79, thank you! You have a great tool - but it's not there yet.

check this out: https://github.com/erlange/wbm-dl

this was working very well until a few weeks ago - maybe you will find some inspiration there.

it was enough to give the input as "domain.com" - and that tool was downloading every page with the correct structure.

I've just tried yours - but it's missing important site parts, like "wp_content" for WP.

Thanks for the reference link. I will expand this to add some of those download options.

This application will rebuild any files under wp-content as long as they were archived and are being used on the page getting restored. Can you PM me the web archive url you tried to restore and what files you are expecting?

 

[NulledCode]

dupe

 

[welshman]

ah, that used to work so well

@cnick79, thank you! You have a great tool - but it's not there yet.

check this out: https://github.com/erlange/wbm-dl

this was working very well until a few weeks ago - maybe you will find some inspiration there.

it was enough to give the input as "domain.com" - and that tool was downloading every page with the correct structure.

I've just tried yours - but it's missing important site parts, like "wp_content" for WP.

 

[hideath]

surely interested

 

[NulledCode]

There's multiple ways to download files from the wayback machine. The default behavior of this application is to rebuild an entire website based on the snapshot url you specified. I first query the CDX server to get all of the snapshots for your url, this tells me all of the files that exist. Then I download the url you specified and extract all of the links (CSS/JS/Images/pages, etc) found on the page and if they have a snapshot I download it. This is a recursive process until everything has been restored. The end result of this process is a usable website with files you can upload straight to your web server.

I'm putting together some options that will allow you to:

* download exact url - no other files will be downloaded
* include files - only files matching this pattern will be downloaded
* exclude files - files matching this pattern will be excluded
* from timestamp - files archived from this date or earlier will be downloaded
* to timestamp - files archived before this date or later will be downloaded
* limit - limit number of files to download
* list - list all urls archived, does not download

The output directory will be something like this:

./example.com/20060715085250/index.html
./example.com/20051120005053/index.html
./example.com/20051122073441/about.html

Something to keep in mind is all pages on a website might not have a snapshot taken on the same day therefore pages will exist in different directories.

I think this will give you guys extreme flexibility in what you can download. You will be able to rebuild an entire website exactly as it appears on the Wayback Machine if your intent is to upload it a web server as is, or download snapshots of a website where you can then extract the content.

Stay tuned for updates...

 

[oiviivio]

There's multiple ways to download files from the wayback machine. The default behavior of this application is to rebuild an entire website based on the snapshot url you specified. I first query the https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server to get all of the snapshots for your url, this tells me all of the files that exist. Then I download the url you specified and extract all of the links (CSS/JS/Images/pages, etc) found on the page and if they have a snapshot I download it. This is a recursive process until everything has been restored. The end result of this process is a usable website with files you can upload straight to your web server.

I'm putting together some options that will allow you to:

* download exact url - no other files will be downloaded
* include files - only files matching this pattern will be downloaded
* exclude files - files matching this pattern will be excluded
* from timestamp - files archived from this date or earlier will be downloaded
* to timestamp - files archived before this date or later will be downloaded
* limit - limit number of files to download
* list - list all urls archived, does not download

The output directory will be something like this:

./example.com/20060715085250/index.html
./example.com/20051120005053/index.html
./example.com/20051122073441/about.html

Something to keep in mind is all pages on a website might not have a snapshot taken on the same day therefore pages will exist in different directories.

I think this will give you guys extreme flexibility in what you can download. You will be able to rebuild an entire website exactly as it appears on the Wayback Machine if your intent is to upload it a web server as is, or download snapshots of a website where you can then extract the content.

Stay tuned for updates...

Thanks for share. Wait for your updates.

 

[indianmojojojo]

Interested in this

 

[NulledCode]

I added additional capabilities that will allow you to download files from web.archive.org similar to some of the tools you guys mentioned above. This is different from the "restore" feature already included in the software because restoring will restore a website exactly "as-is" on web.archive.org. This new download feature will allow you to download files from any domain during any date range, as well as apply filters to include or exclude file types.

I made the library that does all of this available on npm if anyone wants to help contribute or create a cli utility for this have at it.

The Restorizor software you are beta testing is graphical UI wrapper on top of this library. I will soon update the software with a new version to take advantage of the new downloader options I created.

Stay Tuned.

 

[Shinichi Izumi]

i tried downloading a website but after 1.9s it says finished without having any files in the restore folder !

 

[NulledCode]

i tried downloading a website but after 1.9s it says finished without having any files in the restore folder !

What's the full web.archive.org link you are trying to restore? Either post it here or send it to me via PM and I will have a look.

 

[fcheshire]

Thanks for sharing. I will post my experience afterwards.

 

[NulledCode]

i tried downloading a website but after 1.9s it says finished without having any files in the restore folder !

I tried using the link your provided me and was not able to reproduce the error.

• Did you try running it again recently? Often times web.archive.org is dog slow and timeouts.
• You can also try restoring without the "/#!#" in your restore link.

 

[Shinichi Izumi]

I tried using the link your provided me and was not able to reproduce the error.

• Did you try running it again recently? Often times web.archive.org is dog slow and timeouts.
• You can also try restoring without the "/#!#" in your restore link.

yes , i have tried from my computer and from a 1gbps RDP , always having same problem and yes i removed those characters

 

[be9hop]

Downloaded and ran it on a site. Seems to work well.

 

[NulledCode]

yes , i have tried from my computer and from a 1gbps RDP , always having same problem and yes i removed those characters

What OS are you running this on?

Are you having this issue with all urls or just the one you provided?

 

[Shinichi Izumi]

What OS are you running this on?

Are you having this issue with all urls or just the one you provided?

am windows 10, 7 & Server manager 2016 and yeah it's just this exact url as i said on Pm this tool was & still giving a great help to me with this particular link it won't work

 

[GenesisOne]

Will give this a try.
THX op

 

[XenPai]

Please release Android version