Video upload and processing pipeline in HLD - System Design Guide

Problem Statement

Uploading and processing videos directly on a single server causes slow response times and frequent failures under high load. Large video files can overwhelm storage and CPU resources, leading to poor user experience and system crashes.

Solution

The system breaks video upload and processing into separate stages using asynchronous queues. Users upload videos to a scalable storage service, then processing tasks like transcoding and thumbnail generation run independently in worker services. This decouples upload from processing, enabling smooth scaling and fault tolerance.

Architecture

User

→Upload Server

↓

↓

Video Worker

(Processing)

↓

Metadata DB

This diagram shows the flow from user video upload through upload server to storage, then asynchronous processing via message queue and worker services, ending with metadata storage and content delivery.

Trade-offs

✓ Pros

→

Decouples upload and processing to improve system responsiveness and reliability.

→

Enables horizontal scaling of upload servers and processing workers independently.

→

Allows asynchronous retries and failure handling without blocking user uploads.

→

Supports large video files by offloading storage to scalable services.

✗ Cons

→

Increased system complexity due to multiple components and asynchronous flows.

→

Potential delays between upload completion and video availability due to processing time.

→

Requires careful design of message queue and worker scaling to avoid bottlenecks.

Use when handling large video files with high upload volume and complex processing needs, typically above hundreds of uploads per minute.

Avoid if video uploads are rare or processing is minimal, as the added infrastructure complexity may not justify benefits.

Real World Examples

YouTube

Separates video upload from transcoding and thumbnail generation to handle millions of uploads daily without blocking users.

Netflix

Uses asynchronous pipelines to transcode and package videos for multiple devices after upload to storage.

TikTok

Processes uploaded videos asynchronously to apply filters, compress, and prepare for fast delivery.

Alternatives

Synchronous processing

Processes video immediately during upload request, blocking user until completion.

Use when: Use only for very small videos or low upload volume where immediate availability is critical.

Edge processing

Processes videos closer to user devices or edge servers to reduce latency.

Use when: Use when low latency is critical and edge infrastructure is available.

Summary

Video upload and processing pipelines separate upload from processing to improve scalability and reliability.

Asynchronous queues and worker services handle heavy processing without blocking user uploads.

This design supports large files and high upload volumes while enabling fault tolerance and smooth scaling.