Video upload and processing pipeline in HLD - System Design Exercise

Design: Video Upload and Processing Pipeline

Includes video upload, processing, storage, and serving. Excludes video playback UI and detailed CDN design.

Functional Requirements

FR1: Users can upload video files up to 5GB in size.

FR2: System should process videos to generate multiple resolutions (e.g., 1080p, 720p, 480p).

FR3: Support thumbnail extraction from videos.

FR4: Allow users to view upload progress in real-time.

FR5: Processed videos should be stored and served efficiently for playback.

FR6: System should handle 10,000 concurrent uploads.

FR7: Ensure video processing latency p99 < 5 minutes.

FR8: Provide retry mechanism for failed processing jobs.

Non-Functional Requirements

NFR1: System availability target: 99.9% uptime.

NFR2: Latency for upload API: p99 < 2 seconds.

NFR3: Storage must be scalable to petabytes.

NFR4: Processing should be horizontally scalable.

NFR5: Security: Only authenticated users can upload videos.

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

API Gateway or Load Balancer

Authentication Service

Upload Service with chunked upload support

Message Queue for processing jobs

Video Processing Workers

Object Storage (e.g., S3)

Metadata Database

Notification Service

Cache for upload progress

Design Patterns

Asynchronous processing with message queues

Event-driven architecture

Chunked file upload

Microservices for separation of concerns

Retry and dead-letter queue for failed jobs

Reference Architecture

API Gateway

Auth→Auth Service

↓

Upload Service

chunked upload→Object Storage

↓

Message Queue

→Video Processing Workers

↓

Object Storage

↓

Metadata DB

↓

Notification Service

Components

API Gateway

Nginx or AWS API Gateway

Route requests, enforce rate limits, and forward to services

Authentication Service

OAuth 2.0 / JWT

Authenticate users before allowing uploads

Upload Service

Node.js or Python microservice

Handle chunked uploads, validate files, store raw videos

Message Queue

RabbitMQ or AWS SQS

Queue video processing jobs asynchronously

Video Processing Workers

Docker containers running FFmpeg

Transcode videos to multiple resolutions and extract thumbnails

Object Storage

Amazon S3 or equivalent

Store raw and processed video files reliably and scalably

Metadata Database

PostgreSQL or DynamoDB

Store video metadata, processing status, user info

Notification Service

WebSocket server or Push Notification service

Notify clients about upload and processing progress

Cache

Redis

Store upload progress and quick status lookups

Request Flow

1. User authenticates via API Gateway and Auth Service.

2. User uploads video in chunks to Upload Service.

3. Upload Service stores chunks in Object Storage and updates progress in Cache.

4. Once upload completes, Upload Service sends a processing job message to Message Queue.

5. Video Processing Workers consume jobs, transcode videos, extract thumbnails, and store results in Object Storage.

6. Workers update Metadata Database with processing status and results.

7. Notification Service informs user about upload completion and processing progress.

8. User can access processed videos from Object Storage via CDN or streaming service.

Database Schema

Entities: - User: user_id (PK), name, email, auth_info - Video: video_id (PK), user_id (FK), original_file_path, upload_timestamp, status (uploaded, processing, completed, failed), metadata - VideoVariant: variant_id (PK), video_id (FK), resolution, file_path, size - Thumbnail: thumbnail_id (PK), video_id (FK), file_path Relationships: - User 1:N Video - Video 1:N VideoVariant - Video 1:1 Thumbnail

Scaling Discussion

Bottlenecks

Upload Service CPU and memory limits when handling many concurrent large uploads.

Message Queue saturation under high job volume.

Video Processing Workers CPU/GPU resource limits.

Object Storage throughput limits for large file reads/writes.

Metadata Database write contention with many status updates.

Solutions

Scale Upload Service horizontally behind load balancer; use chunked uploads to reduce memory pressure.

Use partitioned or sharded message queues; implement backpressure and rate limiting.

Auto-scale processing workers based on queue length; consider GPU acceleration for transcoding.

Use multi-region or multi-bucket Object Storage; enable CDN caching for reads.

Use database sharding or read replicas; optimize schema and indexing for write-heavy workloads.

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Emphasize asynchronous processing to handle large video files efficiently.

Discuss chunked upload to improve reliability and user experience.

Explain choice of scalable storage and processing components.

Highlight how to track and notify upload and processing progress.

Address failure handling and retry mechanisms.

Show awareness of bottlenecks and scaling strategies.