Bird
Raised Fist0

Design YouTube - Upload, Transcoding Pipeline, CDN Delivery, View Count

HLDhardInterview freq: high
Choose your preparation mode4 modes available
🏗️
Design YouTube - Upload, Transcoding Pipeline, CDN Delivery, View Count
Design a scalable video upload, transcoding, CDN delivery, and view count system for YouTube-scale traffic
hardVideo and Media StreamingFreq: high
Prerequisites:DNSHTTP
⚙️
Functional Requirements
  • Users can upload videos of various formats and sizes.
  • System transcodes uploaded videos into multiple resolutions and formats (e.g., 144p to 4K, HLS/DASH).
  • Deliver videos globally with low latency via CDN.
  • Accurately count and update video view counts in near real-time.
  • Support video metadata storage and retrieval (title, description, thumbnails).
  • Allow video playback with adaptive bitrate streaming.
📊
Non-Functional Requirements
  • Support 2 billion daily active users (DAU).
  • Handle peak upload write QPS of ~925K (2B users × 40 uploads/day ÷ 86400 seconds).
  • 99.99% availability for video playback and metadata APIs.
  • Video playback P99 latency under 100ms after CDN cache hit.
  • View count updates visible within 1 minute of video play.
❓
Clarifying Questions to Ask
  • What is the average and max video size expected for uploads?
  • What video formats and resolutions must be supported for transcoding?
  • What is the expected retention period for videos and metadata?
  • How accurate and real-time must the view count be?
  • Is live streaming supported or only VOD (video on demand)?
  • What regions and CDN providers are targeted for delivery?
Out of Scope
✗ User authentication and authorization.
✗ Video recommendation and search ranking algorithms.
✗ Comments, likes, and social features.
✗ Monetization and ad serving.
📌
Assumptions
  • 2 billion daily active users (DAU).
  • Each user uploads 0.02 videos per day on average (40 videos per 2000 users).
  • Average video size after upload: 100 MB.
  • Each video generates 5 transcoded renditions on average.
⚡
QPS Calculation
write qps2B × 0.02 / 86400 ≈ 462,963 writes/sec (video uploads)
peak write qpsAssuming 2× peak factor: 925,926 writes/sec
read qpsAssuming 10 views per video per day: 2B × 0.02 × 10 / 86400 ≈ 4,629,630 reads/sec
💾
Storage Estimate
daily storage2B × 0.02 × 100MB = 4PB/day raw upload data
transcoded storage4PB × 5 renditions = 20PB/day
metadata storageAssuming 1KB metadata per video: 2B × 0.02 × 1KB = 40GB/day
retentionVideos retained for 1 year: 4PB × 365 = 1.46EB raw + 5× transcoded = 7.3EB total
with replication3× replication for durability: 7.3EB × 3 = 21.9EB total storage
🖥️
Server Estimate
upload serversAssuming 1 server handles 1000 uploads/sec: 925,926 / 1000 ≈ 926 upload servers
transcoding workersAssuming 1 transcoding worker handles 10 videos concurrently with avg 10 min per video: 462,963 / (6 videos/hour) ≈ 77,160 workers
cdn edge nodesThousands globally, scaled by regional demand; exact count depends on traffic distribution
🏗️
Architecture Diagram
Architecture Diagram
User
  |
Upload Service
  |
Kafka Upload Topic
  |
Transcoding Service
  |
Object Storage (e.g., GCS/S3)
  |
CDN Edge Nodes
  |
User Playback
  |
View Count Service
  |
NoSQL DB (e.g., Cassandra)
🧩
Components
Upload Service
Handles video upload requests, validates and stores raw video chunks temporarily
Why thisStateless HTTP servers behind load balancers scale horizontally to handle 925K QPS peak uploads
Why notUsing FTP or custom protocols would not scale globally and lack HTTP caching/CDN benefits
Kafka Upload Topic
Buffer and decouple upload ingestion from transcoding pipeline asynchronously
Why thisKafka supports high throughput (millions of messages/sec), partitioning for parallelism, and durability
Why notRabbitMQ lacks Kafka's partition scalability and throughput at 925K writes/sec
Transcoding Service
Consumes upload events, transcodes videos into multiple renditions and stores results
Why thisKubernetes-managed GPU-enabled workers scale horizontally; batch processing optimizes resource use
Why notServerless functions (e.g., AWS Lambda) cannot handle long-running GPU-intensive transcoding tasks
Object Storage (GCS/S3)
Stores raw and transcoded video files reliably with high durability and global availability
Why thisGCS/S3 provide virtually unlimited storage, automatic replication, and integration with CDN
Why notHDFS or self-managed storage clusters do not provide global availability and require complex ops
CDN Edge Nodes
Cache and deliver video content globally with low latency and high throughput
Why thisCDNs like Cloudflare or Akamai reduce origin load and provide <100ms P99 latency worldwide
Why notServing directly from origin storage causes high latency and bandwidth costs
View Count Service
Aggregates and updates video view counts in near real-time with eventual consistency
Why thisUses Redis Streams for ingestion and Cassandra for durable storage to handle millions of increments/sec
Why notRelational DBs like MySQL cannot handle write amplification and scale at 10M increments/sec
NoSQL DB (Cassandra)
Stores video metadata and view counts with high write throughput and linear scalability
Why thisCassandra's partitioning and replication support 925K writes/sec and multi-region deployments
Why notMongoDB's single primary node per shard limits write scalability at this scale
🔄
Data Flow
1
User → Upload Service
User uploads video via HTTP POST
Upload service validates and stores video chunks temporarily
2
Upload Service → Kafka Upload Topic
Upload service publishes video metadata and upload completion event
Kafka decouples upload from transcoding asynchronously
3
Kafka Upload Topic → Transcoding Service
Transcoding workers consume upload events and start transcoding
Multiple renditions generated in parallel using GPU workers
4
Transcoding Service → Object Storage
Store transcoded video files and update metadata
Object storage provides durable, scalable storage for all video renditions
5
User → CDN Edge Nodes
User requests video playback; CDN serves cached content or fetches from origin
CDN reduces latency and origin load
6
User Playback → View Count Service
Playback events sent to view count service for aggregation
View counts updated in near real-time with eventual consistency
7
View Count Service → NoSQL DB (Cassandra)
Persist aggregated view counts and metadata updates
NoSQL DB supports high write throughput and multi-region replication
🗄️
VideoMetadata
PK: video_id
video_id UUIDtitle textdescription textuploader_id UUIDupload_timestamp timestampthumbnail_url textduration intformats map<text, text>
🗄️
VideoViewCount
PK: video_id
video_id UUIDdate dateview_count counter
🔌
POST
UploadVideo
Request
{
  "user_id": "UUID",
  "video_file": "binary",
  "metadata": {
    "title": "string",
    "description": "string",
    "tags": [
      "string"
    ]
  }
}
Response
{
  "video_id": "UUID",
  "upload_status": "string"
}
Initiates video upload and returns video ID for tracking
🔌
GET
GetVideoMetadata
Request
{
  "video_id": "UUID"
}
Response
{
  "title": "string",
  "description": "string",
  "thumbnail_url": "string",
  "duration": "int",
  "formats": "map<string, string>"
}
Fetches metadata for a given video
🔌
GET
GetViewCount
Request
{
  "video_id": "UUID"
}
Response
{
  "total_views": "long",
  "last_updated": "timestamp"
}
Returns the current view count for the video

Practice

(1/5)
1. A company wants to hide internal client IP addresses from external servers while allowing clients to access the internet. Which proxy type should be used to achieve this?
easy
A. DNS proxy, because it resolves domain names for clients
B. Reverse proxy, because it hides the backend servers from clients
C. Load balancer, because it distributes requests evenly across servers
D. Forward proxy, because it hides client IPs from external servers

Solution

  1. Step 1: Identify proxy role in hiding client IPs

    A forward proxy acts on behalf of clients, masking their IP addresses from external servers.
  2. Step 2: Eliminate other options

    Reverse proxies hide backend servers, not clients; load balancers distribute load but don't hide clients; DNS proxies only resolve names.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Forward proxy hides client IPs from external servers [check]
Hint: Forward proxy hides clients; reverse proxy hides servers [check]
Common Mistakes:
2. A distributed system requires generating unique IDs across multiple services without coordination delays. Which component best ensures strictly increasing, collision-free IDs with minimal latency?
easy
A. Use timestamps combined with random bits generated locally
B. Use a UUID generator on each service independently
C. Use a distributed consensus algorithm like Raft for ID assignment
D. Use a centralized Ticket Server issuing sequential IDs

Solution

  1. Step 1: Identify uniqueness and ordering requirements

    UUIDs (A) are unique but not strictly increasing; timestamps with randomness (D) risk collisions under clock skew; consensus (C) adds coordination overhead and latency.
  2. Step 2: Recognize Ticket Server role

    The Ticket Server (B) centrally issues sequential IDs ensuring strict ordering and uniqueness with minimal latency due to lightweight allocation.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Centralized Ticket Server -> strict, collision-free IDs with low latency [check]
Hint: Central server issues ordered IDs, avoiding coordination delays [check]
Common Mistakes:
3. If the circuit breaker component is removed from a microservice architecture, what is the most likely failure sequence when the downstream service becomes slow or unresponsive?
medium
A. The system automatically queues requests and processes them later without blocking callers
B. The upstream service immediately fails requests without retries, causing data loss
C. The system switches to a backup service transparently without any delay
D. The upstream service blocks on calls until timeout, causing thread exhaustion and cascading failures

Solution

  1. Step 1: Understand circuit breaker removal impact

    Without circuit breaker, upstream calls block waiting for slow downstream responses.
  2. Step 2: Trace failure propagation

    Blocking calls exhaust thread pools, leading to cascading failures (B). Queuing (A) or immediate failover (D) require explicit mechanisms. Immediate fail without retries (C) is unlikely without circuit breaker logic.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Blocking calls cause thread exhaustion and cascade without circuit breaker [check]
Hint: No circuit breaker -> blocking calls exhaust threads -> cascade [OK]
Common Mistakes:
4. For a global social media platform with highly skewed user activity, which sharding strategy best balances load while minimizing cross-shard queries?
medium
A. Directory-based sharding to assign hot users to dedicated shards dynamically
B. Hash-based sharding to evenly distribute users regardless of activity
C. Range-based sharding to group users by geographic region
D. Random sharding to avoid hotspots by unpredictable distribution

Solution

  1. Step 1: Analyze skew and query patterns

    Skewed activity causes hotspots; static range or hash sharding can't adapt dynamically.
  2. Step 2: Evaluate strategies

    Directory-based sharding allows dynamic assignment of hot users to dedicated shards, balancing load and reducing cross-shard queries.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Directory-based sharding supports dynamic load balancing [check]
Hint: Directory sharding adapts dynamically to hotspots [check]
Common Mistakes:
5. For a chat application requiring low latency message delivery but tolerating occasional message loss, which architecture best balances latency and throughput?
medium
A. Use a batch processing system that aggregates messages every 5 seconds
B. Use a synchronous request-response system with guaranteed delivery acknowledgments
C. Use an asynchronous message queue with at-least-once delivery and eventual consistency
D. Use a distributed cache with write-through persistence for all messages

Solution

  1. Step 1: Analyze latency vs throughput needs

    Batch processing adds latency but can increase throughput and tolerate some message loss.
  2. Step 2: Evaluate architectures

    Synchronous (A) increases latency; asynchronous queue (B) reduces latency but may cause duplicates; distributed cache (D) lacks delivery guarantees; batch (C) balances throughput with acceptable latency for some use cases.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Batching trades latency for throughput, tolerating some loss [check]
Hint: Batching trades latency for throughput and tolerates loss [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard