Bird
Raised Fist0
HLDsystem_design~25 mins

Video upload and processing pipeline in HLD - System Design Exercise

Choose your learning style10 modes available
LearnWhyDeepArchPracticeChallengeDesignRecallScale

Start learning this pattern below

Jump into concepts and practice - no test required

or
Recommended
Test this pattern10 questions across easy, medium, and hard to know if this pattern is strong
Design: Video Upload and Processing Pipeline
Includes video upload, processing, storage, and serving. Excludes video playback UI and detailed CDN design.
Functional Requirements
FR1: Users can upload video files up to 5GB in size.
FR2: System should process videos to generate multiple resolutions (e.g., 1080p, 720p, 480p).
FR3: Support thumbnail extraction from videos.
FR4: Allow users to view upload progress in real-time.
FR5: Processed videos should be stored and served efficiently for playback.
FR6: System should handle 10,000 concurrent uploads.
FR7: Ensure video processing latency p99 < 5 minutes.
FR8: Provide retry mechanism for failed processing jobs.
Non-Functional Requirements
NFR1: System availability target: 99.9% uptime.
NFR2: Latency for upload API: p99 < 2 seconds.
NFR3: Storage must be scalable to petabytes.
NFR4: Processing should be horizontally scalable.
NFR5: Security: Only authenticated users can upload videos.
Think Before You Design
Questions to Ask
❓ Question 1
❓ Question 2
❓ Question 3
❓ Question 4
❓ Question 5
❓ Question 6
Key Components
API Gateway or Load Balancer
Authentication Service
Upload Service with chunked upload support
Message Queue for processing jobs
Video Processing Workers
Object Storage (e.g., S3)
Metadata Database
Notification Service
Cache for upload progress
Design Patterns
Asynchronous processing with message queues
Event-driven architecture
Chunked file upload
Microservices for separation of concerns
Retry and dead-letter queue for failed jobs
Reference Architecture
API Gateway
Auth→Auth Service
↓
Upload Service
chunked upload→Object Storage
↓
Message Queue
→Video Processing Workers
↓
Object Storage
↓
Metadata DB
↓
Notification Service
Components
API Gateway
Nginx or AWS API Gateway
Route requests, enforce rate limits, and forward to services
Authentication Service
OAuth 2.0 / JWT
Authenticate users before allowing uploads
Upload Service
Node.js or Python microservice
Handle chunked uploads, validate files, store raw videos
Message Queue
RabbitMQ or AWS SQS
Queue video processing jobs asynchronously
Video Processing Workers
Docker containers running FFmpeg
Transcode videos to multiple resolutions and extract thumbnails
Object Storage
Amazon S3 or equivalent
Store raw and processed video files reliably and scalably
Metadata Database
PostgreSQL or DynamoDB
Store video metadata, processing status, user info
Notification Service
WebSocket server or Push Notification service
Notify clients about upload and processing progress
Cache
Redis
Store upload progress and quick status lookups
Request Flow
1. User authenticates via API Gateway and Auth Service.
2. User uploads video in chunks to Upload Service.
3. Upload Service stores chunks in Object Storage and updates progress in Cache.
4. Once upload completes, Upload Service sends a processing job message to Message Queue.
5. Video Processing Workers consume jobs, transcode videos, extract thumbnails, and store results in Object Storage.
6. Workers update Metadata Database with processing status and results.
7. Notification Service informs user about upload completion and processing progress.
8. User can access processed videos from Object Storage via CDN or streaming service.
Database Schema
Entities: - User: user_id (PK), name, email, auth_info - Video: video_id (PK), user_id (FK), original_file_path, upload_timestamp, status (uploaded, processing, completed, failed), metadata - VideoVariant: variant_id (PK), video_id (FK), resolution, file_path, size - Thumbnail: thumbnail_id (PK), video_id (FK), file_path Relationships: - User 1:N Video - Video 1:N VideoVariant - Video 1:1 Thumbnail
Scaling Discussion
Bottlenecks
Upload Service CPU and memory limits when handling many concurrent large uploads.
Message Queue saturation under high job volume.
Video Processing Workers CPU/GPU resource limits.
Object Storage throughput limits for large file reads/writes.
Metadata Database write contention with many status updates.
Solutions
Scale Upload Service horizontally behind load balancer; use chunked uploads to reduce memory pressure.
Use partitioned or sharded message queues; implement backpressure and rate limiting.
Auto-scale processing workers based on queue length; consider GPU acceleration for transcoding.
Use multi-region or multi-bucket Object Storage; enable CDN caching for reads.
Use database sharding or read replicas; optimize schema and indexing for write-heavy workloads.
Interview Tips
Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.
Emphasize asynchronous processing to handle large video files efficiently.
Discuss chunked upload to improve reliability and user experience.
Explain choice of scalable storage and processing components.
Highlight how to track and notify upload and processing progress.
Address failure handling and retry mechanisms.
Show awareness of bottlenecks and scaling strategies.

Practice

(1/5)
1. Which component in a video upload and processing pipeline is primarily responsible for converting raw uploaded videos into multiple formats suitable for playback?
easy
A. Content delivery network (CDN)
B. Video processing service
C. Metadata database
D. Upload service

Solution

  1. Step 1: Identify the role of each component

    The upload service handles receiving videos, the metadata database stores info, and CDN delivers content. The processing service converts videos.

  2. Step 2: Match the function to the question

    Converting raw videos into multiple formats is done by the video processing service to ensure compatibility.

  3. Final Answer:

    Video processing service -> Option B

  4. Quick Check:

    Conversion = Video processing service [OK]
Hint: Processing means converting video formats [OK]
Common Mistakes:
  • Confusing upload service with processing
  • Thinking CDN does video conversion
  • Assuming metadata database handles video files
2. Which of the following is the correct sequence of steps in a typical video upload and processing pipeline?
easy
A. Storage -> Upload service -> Video processing -> Metadata update
B. Video processing -> Upload service -> Metadata update -> Storage
C. Metadata update -> Upload service -> Storage -> Video processing
D. Upload service -> Video processing -> Storage -> Metadata update

Solution

  1. Step 1: Understand the logical flow

    Users first upload videos, then videos are processed, stored, and metadata is updated last.

  2. Step 2: Match the sequence to options

    Upload service -> Video processing -> Storage -> Metadata update correctly shows upload first, then processing, storage, and metadata update.

  3. Final Answer:

    Upload service -> Video processing -> Storage -> Metadata update -> Option D

  4. Quick Check:

    Upload first, then process, store, update metadata [OK]
Hint: Upload happens before processing and storage [OK]
Common Mistakes:
  • Starting with processing before upload
  • Updating metadata before storage
  • Mixing storage and upload order
3. Consider a video upload pipeline where the upload service places video metadata into a queue for processing. If the processing service crashes and stops consuming messages, what will happen to the queue and user experience?
medium
A. Upload service will reject new uploads immediately
B. Queue will empty quickly; users get instant processing
C. Queue will fill up, causing delays; users see slow processing
D. Metadata database will automatically process videos

Solution

  1. Step 1: Understand queue behavior when consumer stops

    If the processing service crashes, it stops consuming messages, so the queue fills up with unprocessed metadata.

  2. Step 2: Impact on user experience

    Since processing is delayed, users experience slow video availability or processing delays.

  3. Final Answer:

    Queue will fill up, causing delays; users see slow processing -> Option C

  4. Quick Check:

    Processing down -> queue fills -> delays [OK]
Hint: No consumer means queue backs up [OK]
Common Mistakes:
  • Assuming queue empties without consumer
  • Thinking upload service rejects uploads immediately
  • Believing metadata DB processes videos automatically
4. In a video processing pipeline, a developer notices that some videos fail to process and the system does not retry them. Which change will fix this issue?
medium
A. Implement a retry mechanism in the processing service for failed jobs
B. Remove the queue to speed up processing
C. Store videos only after processing completes
D. Disable metadata updates to avoid conflicts

Solution

  1. Step 1: Identify cause of failure handling

    Failures without retries mean the system lacks a retry mechanism for failed processing jobs.

  2. Step 2: Choose fix to handle failures

    Adding retries ensures failed jobs are re-attempted, improving reliability.

  3. Final Answer:

    Implement a retry mechanism in the processing service for failed jobs -> Option A

  4. Quick Check:

    Retries fix failed processing [OK]
Hint: Retries fix failed processing jobs [OK]
Common Mistakes:
  • Removing queue breaks asynchronous design
  • Storing before processing causes errors
  • Disabling metadata updates unrelated to retries
5. You need to design a scalable video upload and processing pipeline that supports millions of daily uploads with minimal user wait time. Which architectural choice best supports this goal?
hard
A. Use asynchronous upload service with message queues and distributed processing workers
B. Process videos synchronously during upload to ensure immediate availability
C. Store all videos on a single server to simplify management
D. Update metadata only after manual verification to ensure accuracy

Solution

  1. Step 1: Analyze scalability and user wait time needs

    Millions of uploads require asynchronous handling and distributed processing to avoid bottlenecks and reduce wait time.

  2. Step 2: Evaluate architectural options

    Asynchronous upload with queues and distributed workers allows parallel processing and smooth scaling.

  3. Final Answer:

    Use asynchronous upload service with message queues and distributed processing workers -> Option A

  4. Quick Check:

    Asynchronous + distributed = scalable and fast [OK]
Hint: Async + queues + distributed workers scale best [OK]
Common Mistakes:
  • Synchronous processing causes delays
  • Single server storage limits scalability
  • Manual metadata delays pipeline