How can I clone website with 1 Million images?

[RatSeaExclusive]

I want to clone website and upload it on my own that has 1 million images with descriptions.
Descriptions need to be attached to a photo, like attached to database or something.

For example: https://www.freepik.com/photos/animals

I consider this: https://github.com/alexksikes/mass-scraping
or Puppeteer with Node.js
or possibly find a hidden api on website

How would I even start to make this happen? And what is the smartest way to do that?

 

[Ndiqi]

Get a high powered VPS and just ctrl + s the webpage?

 

[636173696e6f]

httrack will do it.

 

[RatSeaExclusive]

httrack will do it.

Are proxies needed?

Get a high powered VPS and just ctrl + s the webpage?

How can I do this from Ubuntu VPS command line?

EDIT: And what if images get updated overtime? How I can bring new images with descriptions at the time something new appears with the automated way?

 

[Ndiqi]

How can I do this from Ubuntu VPS command line?

EDIT: And what if images get updated overtime? How I can bring new images with descriptions at the time something new appears with the automated way?

That is beyond the scope of what you initially wanted to do, hence that requires more advanced steps.

You will need a cron jobs just for checking if content is updated from the origin site, you will need to setup a bunch of tools unrelated to the task of downloading the images with descriptions.

You can use either windows or Linux

 

[636173696e6f]

Are proxies needed?

It depends on each site individually and what you want to do the rules are the same for all scrapes:

You can use a single thread and use no proxies
More than 1 thread you will probably need proxies
Scraping 1 million images, you will probably want to use proxies

 

[uncutu]

since you want to keep an index of images that will be refreshed when the original host updates
youll likely need to spend some time in deep thought comparing and test all the made-for-you solutions that exist in this space.
proxies it varies. some (most) sites will rate-limit you after so many requests.