[Journey] Saving Terabytes by Switching to AV1

[bunzee]

Hey guys,


First post here.

I’ve been putting this off for months.


I manage several projects that generate tons of video content.
The most recent part of my library is uploaded to a cost-effective S3 bucket, simply because I’ve run out of space elsewhere.

Over time, that bucket has ballooned to dozens of terabytes.

At first, it felt manageable.
But around three months ago, I noticed the S3 bill creeping up... and now it’s just too expensive to ignore.

Most of my business is at a standstill right now, so it’s the perfect time to finally tackle this.

---

I’ve known for a while that switching to AV1 could save 50–80% in space without losing visual quality.
I’ve already done a few VMAF tests (a tool that measures visual quality between videos) to confirm it.

But I never built the full workflow to process thousands of files... until now.

This journey is about getting started, staying consistent, and sharing progress along the way.

---

Right now, the files that cost me the most money are sitting in S3.
To kick things off, I’m assigning a dedicated PC to handle encoding.

I’ll use my RTX 3080 machine to run VMAF analysis, since it can’t encode AV1 (RTX 3000 only supports decoding, not encoding).
For actual AV1 transcoding, I’ve got two PCs with Intel Arc GPUs, which are great thanks to their hardware QSV support.

---

I also have to deal with a lot of .ts files.
Flexible, but messy.


Because .ts is a more permissive format, converting directly to AV1 can slightly alter the video duration.
That breaks frame alignment and makes VMAF comparisons unreliable.


To fix this, I first need to flatten the .ts files to H.264.
That gives me a clean and stable reference with consistent timing.


So I’m planning to store the original raw files in temporary cloud storage right after download, before any conversion or flattening.
That way, I avoid duplicating local space for no reason.

Next Steps​

• Finalize the VMAF Docker setup (with GPU acceleration)
• Set up temporary cloud storage for raw files
• Build the main Python script to manage the pipeline and log everything
• Scan and analyze duplicate files before processing

This isn’t a service. Just a personal project for now.
But it might help others who are also drowning in storage bills.

If you're in a similar spot, feel free to drop a reply.
I’ll post updates here as I go.


Let’s see how much space (and money) I can save with this.

 

[pala777]

Hey guys,


First post here.

I’ve been putting this off for months.


I manage several projects that generate tons of video content.
The most recent part of my library is uploaded to a cost-effective S3 bucket, simply because I’ve run out of space elsewhere.

Over time, that bucket has ballooned to dozens of terabytes.

At first, it felt manageable.
But around three months ago, I noticed the S3 bill creeping up... and now it’s just too expensive to ignore.

Most of my business is at a standstill right now, so it’s the perfect time to finally tackle this.

---

I’ve known for a while that switching to AV1 could save 50–80% in space without losing visual quality.
I’ve already done a few VMAF tests (a tool that measures visual quality between videos) to confirm it.

But I never built the full workflow to process thousands of files... until now.

This journey is about getting started, staying consistent, and sharing progress along the way.

---

Right now, the files that cost me the most money are sitting in S3.
To kick things off, I’m assigning a dedicated PC to handle encoding.

I’ll use my RTX 3080 machine to run VMAF analysis, since it can’t encode AV1 (RTX 3000 only supports decoding, not encoding).
For actual AV1 transcoding, I’ve got two PCs with Intel Arc GPUs, which are great thanks to their hardware QSV support.

---

I also have to deal with a lot of .ts files.
Flexible, but messy.


Because .ts is a more permissive format, converting directly to AV1 can slightly alter the video duration.
That breaks frame alignment and makes VMAF comparisons unreliable.


To fix this, I first need to flatten the .ts files to H.264.
That gives me a clean and stable reference with consistent timing.


So I’m planning to store the original raw files in temporary cloud storage right after download, before any conversion or flattening.
That way, I avoid duplicating local space for no reason.

Next Steps​

• Finalize the VMAF Docker setup (with GPU acceleration)
• Set up temporary cloud storage for raw files
• Build the main Python script to manage the pipeline and log everything
• Scan and analyze duplicate files before processing

This isn’t a service. Just a personal project for now.
But it might help others who are also drowning in storage bills.

If you're in a similar spot, feel free to drop a reply.
I’ll post updates here as I go.


Let’s see how much space (and money) I can save with this.

I wish you success in your story, I will follow this one as well, like every hopeful story.

 

[shijin]

You should add a deduplication script before uploading back to S3, saves tons.

 

[GenesisOne]

Kind of fishy, a 2015 account with just 1 post.
??????????????

 

[bunzee]

I wish you success in your story, I will follow this one as well, like every hopeful story.

thanks

You should add a deduplication script before uploading back to S3, saves tons.

Thanks for the deduplication tip, great idea.

Kind of fishy, a 2015 account with just 1 post.
??????????????

You’re right, so be careful.

Great initiative! Switching to AV1 is a smart move for storage savings without sacrificing quality. Your workflow with VMAF analysis, Intel Arc GPUs, and Docker sounds solid. Flattening .ts files first is a smart step. Looking forward to your updates—this could really help others dealing with massive video archives too.

I feel kind of alone in this situation, but if it helps others, that’s a bonus.

Update :

Tried getting the NVIDIA version of VMAF running, at first it just seemed tricky to install, but the more I dig, the more I think it’s just not ready yet.
Too new, too many things not working. I’ll probably fall back to CPU, even if it’s slower.

I’ll adapt the tests to keep them short but still representative.
I already had a script that compares only a portion of each video, but even that felt too long and that was after a lot of tweaking.

Now I’m focusing on setting up temporary cloud storage and finalizing the CPU-based VMAF workflow.
If it's really too slow, I might try spreading it across multiple PCs with multi-core setups to keep things moving.

 

[ElitePower]

Keep in mind that aws s3 is pretty expensive. We use cloudflare r2. It is compatible with s3 clients but charges per event for downloading/uploading and not bandwidth. It is also cheaper per gb for storage. So for the temporary storage you are now using for this project it might be a good choice.