need solution to store website files in another server

i have a image video website and i create a crawler for it and i want to save the files in another server any solution fast upload etc ..
i have tried send files urls to storage server using zeromq with python and return new files url after they stored but it is not good and too bugy

You want Amazon S3 or Digitalocean Spaces.

NulledCode said:

You want Amazon S3 or Digitalocean Spaces.

Click to expand...

i think my question was not clear bcs i'm not looking for storage server but a way to store the files

painurek said:

i think my question was not clear bcs i'm not looking for storage server but a way to store the files

Click to expand...

S3 or Spaces (which is built on S3) is the way to store files, heck you could even install Minio to create your own S3 object store.

Your question is not clear, nor is what you are trying to do clear. If I understand it, you have a crawler downloading files on one server and you want to get them to your image/video website? If I had a crawler I would have the crawler save the images/videos to an S3 bucket which can then be served to your website or app, or send the path to the S3 object into your zeromq message to do whatever you need to do. You shouldn't have to change file URLs or move files around once you download them because you're using S3 as the way to store files.

If you need a way to manage files across servers you can have a look at rclone.

Also, instead of saving files to disk then having to read them only to move them, you should be using streams instead, because this allows you to download a file and save it somewhere in a memory buffer without needing to touch disk space.