Developer for Video Upload System to Streaming and Download Servers

Hello,


I’m looking for a developer to build a video upload system to streaming and download servers—a scalable service that ingests videos (MP4) and distributes them to ten external endpoints (five for “online” playback, five for downloads). Below are the core requirements:

---

1. Profile and Proof of Skills​

• Share a portfolio of projects demonstrating:
  
  • Kubernetes deployments (autoscaling, readiness/liveness probes, PodDisruptionBudgets).
  • CI/CD pipelines (GitHub Actions or GitLab CI) covering build, test, and deployment stages.
  • Infrastructure-as-Code (Terraform or Pulumi) provisioning a Kubernetes cluster, managed databases, Redis, RabbitMQ/Kafka, and S3/MinIO–compatible storage.
• Include brief documentation for each project (architecture diagrams and repository links with test credentials or instructions).

---

2. Back-End and Orchestration​

• Language/Framework:
  
  • Node.js + NestJS or Python 3.9+ + FastAPI.
  • Design a versioned REST API (e.g., /v1/movies).
  • Implement JWT-based authentication or integrate with Keycloak (OpenID Connect/OAuth2).
• Database:
  
  • PostgreSQL with proper indexing or partitioning (sharding/partitions by year or language).
  • Schema covering: users, movies/series, languages, quality levels, replication status, and external credentials.
• Caching and Queues:
  
  • Redis for caching TMDb metadata and distributed locks.
  • RabbitMQ or Apache Kafka to queue:
  • Video-upload tasks to all ten external servers.
  • Health checks and retry workflows (using circuit breakers and exponential back-off).
• Backup Storage:
  
  • MinIO or AWS S3–compatible storage for MP4 files, subtitle files, and torrent/magnet links.
  • Generate presigned URLs for server-side pulls whenever the external API supports it.

---

3. Upload Management and Fault Tolerance​

• Concurrent Workers:
  
  • One worker per external server, dynamically adjusted to keep your 1 Gbps link saturated.
• Network Tuning (TCP):
  
  • Set net.core.rmem_max/wmem_max to 16 MB and enable tcp_congestion_control=bbr.
• Server State Tracking:
  
  • Each endpoint is marked “Active” or “Degraded/Down.”
  • If two consecutive upload attempts fail (timeouts, 429 errors), mark as “Degraded” for 10 minutes before retrying.
  • Failed tasks are re-enqueued for the next cycle (medium priority).
• Traffic Prioritization (QoS):
  
  • High Priority: initial user-triggered uploads (reserve ~80 % of bandwidth).
  • Low Priority: daily reuploads and health checks (reserve ~20 %).
  • Enforce quotas via Linux tc to prevent low-priority tasks from starving primary uploads.

---

4. Infrastructure and DevOps​

• Containerization / Kubernetes:
  
  • Package each microservice in Docker.
  • Deploy to a Kubernetes cluster (GKE/EKS/AKS) with Horizontal Pod Autoscaling based on CPU, message-queue backlog, and network throughput.
  • Include readiness and liveness probes and PodDisruptionBudgets for zero-downtime rolling updates.
• Infrastructure as Code:
  
  • Use Terraform or Pulumi to provision:
    
    • Kubernetes cluster.
    • Managed PostgreSQL (with read replicas).
    • Redis instance.
    • RabbitMQ or Kafka cluster.
    • MinIO or S3 bucket.
    • VPC, subnets, firewalls, and load balancers.
• CI/CD:
  
  • Configure GitHub Actions or GitLab CI pipelines to run:
    
    • Code linters (ESLint, MyPy).
    • Unit and integration tests (e.g., pytest).
    • Automated deployment to staging and production, with rollback on health-check failures.
• Observability:
  
  • Prometheus + Grafana for application metrics (throughput, latency, error rates).
  • ELK Stack (Elasticsearch, Logstash, Kibana) or Grafana Loki for centralized log aggregation and search.
  • Alerting (Slack or email) on critical thresholds (e.g., > 5 % upload failures, endpoint down).

---

5. Admin Front-End (Optional)​

• React + TypeScript for a minimal admin panel where an administrator can:
  
  • View a list of uploaded videos with languages, quality levels, and replication status (green/yellow/red indicators).
  • Upload new MP4 files, subtitles, or torrent/magnet links.
  • Monitor server health and pending tasks in real time.

Please note that the technology stack is flexible—the most important requirement is a clean, efficient implementation that reliably fulfills all core functions. You don’t need to prove your expertise in a hundred different ways; simply provide a demonstrable portfolio of relevant work. We can discuss specific system details and architecture choices together during the interview.

.

Hello, I am really interested in your project.
spongejump

u know whats interesting, i made something for near-exact purposes, except for anime and i use mysql because postgre was being a bitch to implement lol.
u may want to look at a simpler way if ur making it only 1 user based, implement the use of transmission and folder-change conversion rather than the upload function.
it took me about 3 months to make, but to be honest, depending on whats more important to u (money VS time) u could do that urself with Cursor AI

.