- Jun 10, 2010
- 2,402
- 1,676
Have you ever wanted to download a file from the internet and upload it to S3? With this module, you can do just that! Use with Amazon S3, Digitalocean Spaces, or any other platform that utilizes S3.
I created a simple module that can transfer the contents of a URL to an S3 bucket. The beautify to this is, I am using streams so the data your are downloading is buffered into memory and streamed to your S3 bucket so you don't need to download the file locally, then upload to S3 because this is all done in memory. This is great for serverless functions!
File: s3.js. Make note of the required modules I am ussing as you will need to install them in your project.
Example Usage:
Create a new instance of the S3 class then call the "transfer" method to begin your download. By default, my code has a "cookies" parameter where I pass in the cookie string that a browser would set. If the endpoint your downloading from doesn't requires cookies then remove the cookies parameters from the code snippet above.
I created a simple module that can transfer the contents of a URL to an S3 bucket. The beautify to this is, I am using streams so the data your are downloading is buffered into memory and streamed to your S3 bucket so you don't need to download the file locally, then upload to S3 because this is all done in memory. This is great for serverless functions!
File: s3.js. Make note of the required modules I am ussing as you will need to install them in your project.
JavaScript:
const AWS = require("aws-sdk");
const axios = require("axios");
const { PassThrough } = require("stream");
module.exports = class S3 {
constructor(endpoint, accessKeyId, secretAccessKey, permissions) {
const spacesEndpoint = new AWS.Endpoint(endpoint);
this.permissions = permissions;
this.s3 = new AWS.S3({
endpoint: spacesEndpoint,
accessKeyId,
secretAccessKey,
});
}
uploadFromStream(fileResponse, fileName, bucket) {
const passThrough = new PassThrough();
const promise = this.s3
.upload({
Bucket: bucket,
Key: fileName,
ContentType: fileResponse.headers["content-type"],
ContentLength: fileResponse.headers["content-length"],
Body: passThrough,
ACL: this.permissions,
})
.promise();
return { passThrough, promise };
}
async downloadFile(fileUrl, cookies) {
return axios.get(fileUrl, {
timeout: 15000,
headers: {
Cookie: cookies,
},
responseType: "stream",
});
}
// Returns the location of file
async transfer(remote_url, file_name, bucket, cookies) {
const responseStream = await this.downloadFile(remote_url, cookies);
const { passThrough, promise } = this.uploadFromStream(
responseStream,
file_name,
bucket
);
responseStream.data.pipe(passThrough);
return promise
.then((result) => {
return result.Location;
})
.catch((e) => {
throw e;
});
}
async intercept(responseStream, file_name, bucket) {
const { passThrough, promise } = this.uploadFromStream(
responseStream,
file_name,
bucket
);
responseStream.pipe(passThrough);
return promise
.then((result) => {
return result.Location;
})
.catch((e) => {
throw e;
});
}
};Example Usage:
Create a new instance of the S3 class then call the "transfer" method to begin your download. By default, my code has a "cookies" parameter where I pass in the cookie string that a browser would set. If the endpoint your downloading from doesn't requires cookies then remove the cookies parameters from the code snippet above.
JavaScript:
const S3 = require("./s3");
// Use your S3 credentials for endpoint, api_key, and api_secret
// permissions: public-read | private
const s3 = new S3(endpoint, api_key, api_secret, permissions);
// downloadLink = A URL to a zip file, or something to download
// filename = The name of the file you want to use
// bucket = The S3 bucket to store your file
// cookies = This is a cookies string. Useful if you need to login and store cookies. Otherwise, remove.
const location = await s3.transfer(
downloadLink,
filename,
bucket,
cookies.map((ck) => ck.name + "=" + ck.value).join(";")
);
