scapper_0x
Newbie
- Dec 2, 2024
- 2
- 2
I'm facing a scalability issue with my current setup getting detected and blocked too quickly. I've built a programmatic downloader using yt-dlp that supports TikTok, YouTube, Facebook, and Instagram.
Here’s how my current system works:
Any insights, especially from those who've handled large-scale yt-dlp workloads, would be greatly appreciated.
Thanks in advance!
Here’s how my current system works:
- I run everything on a single Virtual Machine (via VirtualBox).
- Inside that VM, I run 7 Surfshark VPN containers, each exposing a SOCKS5 proxy.
- Due to Surfshark’s terms of service, I’m limited to using 7 concurrent VPN/proxy connections.
- Usage Proxies (e.g., 4 ports) – These are actively used for downloading, with each proxy handling one video at a time.
- Fallback Proxies (e.g., 3 ports) – These remain idle unless a Usage proxy gets flagged, throttled, or blocked. When that happens, the system automatically switches to a Fallback proxy to maintain continuity.
My Questions:
- Is there a better way to scale up this setup without hitting IP bans so quickly?
Any insights, especially from those who've handled large-scale yt-dlp workloads, would be greatly appreciated.
Thanks in advance!