Bird
Raised Fist0
HLDsystem_design~12 mins

Video upload and processing pipeline in HLD - Architecture Diagram

Choose your learning style9 modes available
LearnWhyDeepArchPracticeChallengeDesignRecallScale
System Overview - Video upload and processing pipeline

This system allows users to upload videos which are then processed asynchronously to generate different formats and thumbnails. The key requirements are to handle large video files efficiently, process videos without blocking user requests, and deliver processed videos quickly.

Architecture Diagram
User
  |
  v
Load Balancer
  |
  v
API Gateway
  |
  v
Upload Service ---> Message Queue ---> Processing Service ---> Storage Service
                                      |                                   |
                                      v                                   v
                                  Notification Service                 CDN Cache
Components
User
client
Uploads videos and requests processed video playback
Load Balancer
load_balancer
Distributes incoming upload and playback requests evenly to API Gateway instances
API Gateway
api_gateway
Handles client requests, routes upload and playback requests to appropriate services
Upload Service
service
Receives video uploads, stores raw videos temporarily, and enqueues processing tasks
Message Queue
queue
Decouples upload from processing by holding video processing tasks asynchronously
Processing Service
service
Consumes tasks from queue, processes videos into various formats and thumbnails
Storage Service
storage
Stores processed videos and thumbnails for delivery
Notification Service
service
Sends notifications to users when processing is complete
CDN Cache
cache
Caches processed videos close to users for fast playback
Request Flow - 14 Hops
User → Load Balancer
Load Balancer → API Gateway
API Gateway → Upload Service
Upload Service → Message Queue
Message Queue → Processing Service
Processing Service → Storage Service
Processing Service → Notification Service
User → Load Balancer
Load Balancer → API Gateway
API Gateway → CDN Cache
CDN Cache → User
CDN Cache → Storage Service
Storage Service → CDN Cache
CDN Cache → User
Failure Scenario
Component Fails:Message Queue
Impact:Video processing tasks cannot be queued, causing processing delays and possible upload failures
Mitigation:Use a replicated queue system with failover; buffer uploads temporarily in Upload Service until queue recovers
Architecture Quiz - 3 Questions
Test your understanding
Which component ensures video processing happens asynchronously after upload?
AMessage Queue
BLoad Balancer
CCDN Cache
DAPI Gateway
Design Principle
This design uses asynchronous processing with a message queue to handle heavy video processing without blocking user uploads. Caching with a CDN improves playback speed and reduces load on storage. Load balancing and API gateway provide scalability and routing control.