bunzee
Newbie
- Dec 7, 2015
- 2
- 2
Hey guys,
First post here.
I’ve been putting this off for months.
I manage several projects that generate tons of video content.
The most recent part of my library is uploaded to a cost-effective S3 bucket, simply because I’ve run out of space elsewhere.
Over time, that bucket has ballooned to dozens of terabytes.
At first, it felt manageable.
But around three months ago, I noticed the S3 bill creeping up... and now it’s just too expensive to ignore.
Most of my business is at a standstill right now, so it’s the perfect time to finally tackle this.
I’ve known for a while that switching to AV1 could save 50–80% in space without losing visual quality.
I’ve already done a few VMAF tests (a tool that measures visual quality between videos) to confirm it.
But I never built the full workflow to process thousands of files... until now.
This journey is about getting started, staying consistent, and sharing progress along the way.
Right now, the files that cost me the most money are sitting in S3.
To kick things off, I’m assigning a dedicated PC to handle encoding.
I’ll use my RTX 3080 machine to run VMAF analysis, since it can’t encode AV1 (RTX 3000 only supports decoding, not encoding).
For actual AV1 transcoding, I’ve got two PCs with Intel Arc GPUs, which are great thanks to their hardware QSV support.
I also have to deal with a lot of .ts files.
Flexible, but messy.
Because .ts is a more permissive format, converting directly to AV1 can slightly alter the video duration.
That breaks frame alignment and makes VMAF comparisons unreliable.
To fix this, I first need to flatten the .ts files to H.264.
That gives me a clean and stable reference with consistent timing.
So I’m planning to store the original raw files in temporary cloud storage right after download, before any conversion or flattening.
That way, I avoid duplicating local space for no reason.
This isn’t a service. Just a personal project for now.
But it might help others who are also drowning in storage bills.
If you're in a similar spot, feel free to drop a reply.
I’ll post updates here as I go.
Let’s see how much space (and money) I can save with this.
First post here.
I’ve been putting this off for months.
I manage several projects that generate tons of video content.
The most recent part of my library is uploaded to a cost-effective S3 bucket, simply because I’ve run out of space elsewhere.
Over time, that bucket has ballooned to dozens of terabytes.
At first, it felt manageable.
But around three months ago, I noticed the S3 bill creeping up... and now it’s just too expensive to ignore.
Most of my business is at a standstill right now, so it’s the perfect time to finally tackle this.
I’ve known for a while that switching to AV1 could save 50–80% in space without losing visual quality.
I’ve already done a few VMAF tests (a tool that measures visual quality between videos) to confirm it.
But I never built the full workflow to process thousands of files... until now.
This journey is about getting started, staying consistent, and sharing progress along the way.
Right now, the files that cost me the most money are sitting in S3.
To kick things off, I’m assigning a dedicated PC to handle encoding.
I’ll use my RTX 3080 machine to run VMAF analysis, since it can’t encode AV1 (RTX 3000 only supports decoding, not encoding).
For actual AV1 transcoding, I’ve got two PCs with Intel Arc GPUs, which are great thanks to their hardware QSV support.
I also have to deal with a lot of .ts files.
Flexible, but messy.
Because .ts is a more permissive format, converting directly to AV1 can slightly alter the video duration.
That breaks frame alignment and makes VMAF comparisons unreliable.
To fix this, I first need to flatten the .ts files to H.264.
That gives me a clean and stable reference with consistent timing.
So I’m planning to store the original raw files in temporary cloud storage right after download, before any conversion or flattening.
That way, I avoid duplicating local space for no reason.
Next Steps
- Finalize the VMAF Docker setup (with GPU acceleration)
- Set up temporary cloud storage for raw files
- Build the main Python script to manage the pipeline and log everything
- Scan and analyze duplicate files before processing
This isn’t a service. Just a personal project for now.
But it might help others who are also drowning in storage bills.
If you're in a similar spot, feel free to drop a reply.
I’ll post updates here as I go.
Let’s see how much space (and money) I can save with this.
