How to scrape expired content from archive.org?

PeterPerseo · Jan 27, 2019

I have to scrape an expired website with thousands of pages, so some paid online tools I've found are too much expensive for that quantity of urls.

I know there is a free opensource script on GitHub, here https://github.com/hartator/wayback-machine-downloader but it requires some coding that I'm not able to do.

Does anyone know any free tool, or any bot made with that script above, good to download this type (and quantity!) of content (text,photos,html,etc)?

Thanks

PeterPerseo · Jan 30, 2019

No answer...

RayMayLaidPipeDay · Jan 30, 2019

I was wondering same thing, would appreciate if anyone had any input?

lynmongo · Jan 30, 2019

You can have a custom scraper bot made for you here, freelancer, upwork, etc

lorand · Jan 30, 2019

I used this https://github.com/hartator/wayback-machine-downloader
No coding needed, all instruction are there

graham25s · Jan 30, 2019

The hartator one is the best one i have tried, even better than the PHP one i coded lol, it keeps the site structure intact.

PeterPerseo · Jan 30, 2019

lorand said:
I used this https://github.com/hartator/wayback-machine-downloader
No coding needed, all instruction are there

I know it, if you read my OP I mentioned just that. Anyway it seems not easy, it need some coding

SocialManager · Jan 30, 2019

PeterPerseo said:
I know it, if you read my OP I mentioned just that. Anyway it seems not easy, it need some coding

I would say that you should hire someone to do the job for you. This is really a basic program/script and if you can't figure out how to use it, there is probably nothing out there that will help you do what you need. I am not meaning this to be insulting. I researched a bit last night for a similar thing, and basically EVERYONE recommends this github script. Lucky for me I found a backup of the site I needed on my old PC, but that isn't an option for you, so either learning this script or hiring someone else to do it is your best choice. Learning basic script programming is a great skill in your life too! I would guess someone would do this for under $10 USD.

Compazz · Jan 30, 2019

here you go https://www.blackhatworld.com/seo/domrecovery-resurrect-websites-from-wayback-machine.842968/

magicjones · Jan 30, 2019

I just did this on an Acer notebook running Windows 10. Real easy anyone can do it. Here's how I did it:

1. Download Linux Mint and mount to an 8 gig USB stick with Rufus
2. Turn on the notebook and immediately press F12 to be able to select the USB to boot from (if this doesn't work you may have to go into BIOS and disable safe mode).
3. Once it loads up, connect to the WiFi go to that github page for instructions
4. First you have to install Ruby so open a terminal window and type: sudo apt-get install ruby
5. Then install the Wayback Machine Downloader, type: gem install wayback_machine_downloader
6. Now just type: wayback_machine_downloader http://example.com

That's it. Most of the websites pages and files will be downloaded into a folder in your home directory. You can pop in another USB stick and transfer them there.

graham25s · Jan 30, 2019

Or you can download the Ruby installer: https://rubyinstaller.org/ for windows, which loads a command prompt with ruby.

magicjones · Jan 30, 2019

graham25s said:
Or you can download the Ruby installer: https://rubyinstaller.org/ for windows, which loads a command prompt with ruby.

Lol didn't realize it was that easy...

PeterPerseo · Jan 31, 2019

magicjones said:
.... Here's how I did it:
1. Download Linux Mint and mount to an 8 gig USB stick with Rufus
2. Turn on the notebook and immediately press F12 to be able to select the USB to boot from (if this doesn't work you may have to go into BIOS and disable safe mode).
3. Once it loads up, connect .....

WOW!! Thanks a lot, it's a fantastic help! I will try now!!

magicjones · Jan 31, 2019

PeterPerseo said:
WOW!! Thanks a lot, it's a fantastic help! I will try now!!

Sure no problem I think @graham25s has the much easier idea too you can skip a lot of those steps and just run the ruby commands on Windows with https://rubyinstaller.org/

kveldulv · Feb 1, 2019

This works well. It does sites up to 200 urls free, and their pricing for huge sites isn't that bad.
https://en.archivarix.com/

PeterPerseo · Feb 2, 2019

kveldulv said:
This works well. It does sites up to 200 urls free, and their pricing for huge sites isn't that bad.
https://en.archivarix.com/

Well, I tried it and the quote for the website I want to download was.... a couple of THOUSAND dollars

By the way I started to use the free script on GitHub, and I'm downloading the website since... a couple of days! It still has not finished!

jonsnow2019 · Feb 2, 2019

PeterPerseo said:
I'm downloading the website since... a couple of days! It still has not finished!

Next time use concurrency attribute to download more than 1 pages at a time, because by default is 1 page at a time.

magicjones · Feb 3, 2019

I just did it on Windows 10 with the Ruby installer. Downloaded the entire website, some parts broken but if you match the folder structure of the original you can get the old images to work still. You can import the pages to WordPress with this plugin https://wordpress.org/plugins/import-html-pages/

PeterPerseo · Feb 3, 2019

jonsnow2019 said:
Next time use concurrency attribute to download more than 1 pages at a time, because by default is 1 page at a time.

Right! I didn't know the site was soooo big! I will test it now, 'cos I had to stop the previous download (too large!).
Thanks!

magicjones said:
... some parts broken but if you match the folder structure of the original you can get the old images to work still. You can import the pages to WordPress with this plugin https://wordpress.org/plugins/import-html-pages/

Yes, the broken part is also my problem, I will see if it's possible to fix it.
Thanks for the tip of the WP plugin, I didn't know it!

mylastvacation · Feb 3, 2019

I wonder what's googles algorithm's take on websites that are resurrected this way?

How to scrape expired content from archive.org?

Senior Member

Senior Member

BANNED

BANNED

Regular Member

Power Member

Senior Member

Elite Member

BANNED

Power Member

Power Member

Power Member

Senior Member

Power Member

Regular Member

Senior Member

Regular Member

Power Member

Senior Member

Supreme Member

Main Menu

Marketplace

Making Money

BlackHat World