Download entire sites from Wayback Machine

I have found two services that will download an archived website for you from Wayback Machine for a $15 fee. That's a decent price if you only have to do this once, but if you have to do it on a regular basis, it would be a better deal to figure out how to do that yourself.

So far I have tried HTTrack, with varied success. It works OK for small sites, although you have to remove some garbage code from the downloaded files manually. The problem with HTTrack is that for large sites, Wayback Machine doesn't always have complete snapshots for every single date. It crawls different pages on different dates, and when you download these files with HTTrack, you end up getting multiple partial snapshots of the same site, with lots of dupes or sometimes files with duplicate names or slightly different content, and it's virtually impossible to put it back together. An added drawback is that HTTrack doesn't strip Wayback Machine code (like the javascript) or metadata, which also creates extra work.

If you want a more technical explanation of why HTTrack is not fit for this job, you can check a discussion on SuperUser called "Trouble using wget or httrack to mirror archived website" (can't post links, sorry, but you can Google it).

So has anyone figured out a working way to download a complete version of a large site from Wayback Machine? I figured if anyone has figured it out, it would be here at BHW.

Have you read "Bring The Fresh' by Kelly Felix? In his course, he mentioned a service that copies sites over from archive.org to new domains. I think it's waybackdownloader.com.

I don't think you read even one sentence of my post...

I thought way back machine only archived a small number of pages. I never saw an archive that included the entire sites content.

Hawkster said:

I thought way back machine only archived a small number of pages. I never saw an archive that included the entire sites content.

Click to expand...

that's right.

The question is legit. I have the same issues. I copy big websites manually (hire freelancers from India and Philippine). Basically, they use HTTrack multiple times, going from the most recent results to the older ones, adding files without replacement. However, I think if you can get a nice snapshot, you should use the $15 services. The only reason I use freelancers is because I need the sites to be on certain engines (WP mostly) to be able to manage/maintain them using my scripts.

I built http://archivescraper.net to do just this. Also set up a monthly unlimited.

try flipping it after that to get more than that $15?

http://archivescraper.net - How does it actually work. I think it someone like this would work that would really help you to make $$$

i dont know about scraping the wayback machine, but if the site you want to scrape is still up and running you can use ubuntu and wget to easily download the site.

weblycos said:

http://archivescraper.net - How does it actually work. I think it someone like this would work that would really help you to make $$$

Click to expand...

You put in the url of the site you want scraped. Shows you the calendar, you select the day which one you want to scrape, shows you a iframe of the site to make sure its the right one. Then it goes into the queue and shows up in history when its done scraping.

I used this service to download an entire website for wayback machine (archiveorg) from waybackmachinedownloadercom
It contain a test if you need to download only the front page but I don't know why it sent the email of download link to spam folder(I emailed them to fix it).

Wayback is a godsend. Unfortunately, domain squatters like to do this really annoying thing, where they buy an old domain and stick a catch-all robots.txt on it, which prompts Wayback to hide all the archived content. So a lot of good content tends to be lost to the tides this way (sniff)

Hi buddy!
I also interested in getting these services.
Please describe me how to get this? Describe me in detail.

I use https://natural-networks.com/ great for what it says it does. I am not affiliated with it in any manner.

For waybackmachinedownloadercom , You put an archive url and your email. It scrap the html file, js , img and css(it also scrap "background-image" images and "import url" css files), it fix external links to be SEO freindly and then scrap internal link and do the same think with 10 deep. Infortunatly , it sent download link via email in spam folder(I emailed them about this).
But the html website that I get was perfect.

If you need a downloading service for cheap, have a look at my BST in my signature or visite websiterestore.com. You enter in a domain and a date the domain was last archived and we will restore every archived page linked to that domain at that date, including images, css, docs, videos, etc. We also rebuild URLs so the links work. We don't restore pages that resulted in a 302 on the wayback machine. Our restore process is unique and we aren't limited by how "deep" the links go.

seems that they updated their search function so now it takes a lot less to find older sites

I'm getting decent results with wayback-machine-downloader. If you have a VPS (or a Linux box), install Ruby 2.3 and then run:


It accepts date ranges, concurrent request limits. It dumps the website to disk deleting out the wayback navigation guff. Grabs pages, images, styles, javascript, and files. The code's on github hartator/wayback-machine-downloader

mofoparrot said:

I built http://archivescraper.net to do just this. Also set up a monthly unlimited.

Click to expand...

not working for me...

Some useful tips in the thread, been wanting to do this, just wasn't sure of which tool would be best. Will have to try them out.