Bird
Raised Fist0

Design YouTube - Upload, Transcoding Pipeline, CDN Delivery, View Count

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design YouTube - Upload, Transcoding Pipeline, CDN Delivery, View Count
45 min · senior
senior

You are designing a video streaming platform similar to YouTube with 50 million daily active users (DAU). The system must handle video uploads, transcoding into multiple formats, global content delivery via CDN, and accurate view count tracking. Videos must be encrypted at rest and in transit. The system should support offline video playback and maintain a P99 latency under 200ms for video start time. The platform must scale to handle peak upload rates and millions of concurrent viewers worldwide.

Constraints
  • All uploaded videos must be encrypted at rest using AES-256 and encrypted in transit with TLS 1.3.
  • The system must support offline video playback on mobile devices with DRM enforcement.
  • Video start time latency (P99) must be under 200ms globally.
  • The transcoding pipeline must handle 10,000 concurrent uploads with multiple output resolutions and formats.
  • View count updates must be accurate within 1 minute delay and resilient to concurrent increments.
📋
Guided Steps
1/5
Step 1
Designing the Video Upload Service
How would you design the video upload flow to handle large files securely and reliably at scale?
💡 Hint
Consider chunked uploads, resumability, and encryption during transit and at rest.
Key Decision
Use AWS S3 multipart upload with client-side encryption (CSE) using AWS KMS-managed AES-256 keys; upload metadata stored in DynamoDB for resumability and state tracking.
Interviewer Expectation
Mid-level: Names S3 multipart upload and encryption. Senior: Explains why client-side encryption with KMS is chosen over server-side to reduce trust surface and how DynamoDB partition keys are designed for high write throughput and resumability.
⚠️Common Wrong Answer
Using a monolithic HTTP upload endpoint without chunking leads to failures on network interruptions and poor scalability.
Step 2
Designing the Transcoding Pipeline
How would you architect the transcoding pipeline to process 10,000 concurrent uploads into multiple formats efficiently?
💡 Hint
Think about decoupling upload from processing, scalable task queues, and fault tolerance.
Key Decision
Use Apache Kafka as the ingestion queue with video upload events partitioned by video ID, and AWS Elastic Transcoder or FFmpeg workers autoscaled via Kubernetes consuming from Kafka partitions for parallel transcoding.
Interviewer Expectation
Mid-level: Mentions Kafka for decoupling. Senior: Explains Kafka partitioning by video ID ensures ordering and idempotent processing, and why RabbitMQ was rejected due to lower throughput at this scale.
⚠️Common Wrong Answer
Directly triggering transcoding synchronously after upload blocks upload threads and reduces system throughput.
Step 3
Designing CDN Delivery for Low Latency Playback
How would you design the video delivery system to achieve P99 start latency under 200ms globally?
💡 Hint
Consider CDN edge caching, adaptive bitrate streaming, and cache invalidation strategies.
Key Decision
Use CloudFront CDN with edge caching of HLS/DASH manifests and encrypted video segments stored in S3; implement signed URLs for DRM and cache keys based on video ID and bitrate to maximize cache hit ratio.
Interviewer Expectation
Mid-level: Names CDN and adaptive streaming. Senior: Explains signed URLs for secure access, cache key design to avoid cache fragmentation, and why Akamai was rejected due to cost constraints at startup scale.
⚠️Common Wrong Answer
Serving video directly from origin storage causes high latency and overloads backend during traffic spikes.
Step 4
Designing the View Count System
How would you design the view count tracking to be accurate within 1 minute delay and handle concurrent increments at scale?
💡 Hint
Think about eventual consistency, batching increments, and avoiding hot keys.
Key Decision
Use Redis Streams to collect view events per video, with a background worker batching increments every 30 seconds and updating view counts in Cassandra with video_id as partition key for linear scalability.
Interviewer Expectation
Mid-level: Mentions Redis or in-memory counters. Senior: Explains Redis Streams for ordered event collection, batching to reduce write amplification, and why single-node Redis counters were rejected due to hot key bottlenecks.
⚠️Common Wrong Answer
Incrementing view counts synchronously in a relational DB causes write contention and high latency.
Step 5
Ensuring Security and Offline Playback
How would you enforce encryption and DRM for offline playback on mobile devices?
💡 Hint
Consider key management, secure storage, and license delivery.
Key Decision
Use Widevine DRM integrated with encrypted HLS streams; keys managed via Google Cloud KMS with short-lived licenses delivered through a secure license server requiring user authentication.
Interviewer Expectation
Mid-level: Names DRM and encryption. Senior: Explains integration of Widevine with encrypted streams, key rotation policies, and why homegrown DRM was rejected due to security risks and compliance.
⚠️Common Wrong Answer
Storing decryption keys on client without DRM leads to piracy and violates content provider agreements.
🔌
API Design Challenge
Sketch 3 core API endpoints for the system: video upload initiation, video playback request, and view count update.
Hints
  • How does the client securely upload large video files in chunks?
  • How does the playback API ensure low latency and CDN cache hits?
  • How does the system handle concurrent view count increments without losing accuracy?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Secure, resumable video upload with client-side encryption and metadata tracking.
  • Decoupled, scalable transcoding pipeline using Kafka and autoscaled workers.
  • Global CDN delivery with adaptive streaming and signed URLs for security.
  • Accurate, scalable view count system using event streams and batch updates.
Good to Have - 30%
  • Offline playback support with DRM and secure license management.
  • Detailed API design covering upload, playback, and view count endpoints.
  • Consideration of failure modes and retry mechanisms in transcoding and delivery.
Bonus - 10%
  • Cost optimization strategies for CDN and transcoding resources.
  • Monitoring and alerting design for pipeline health and latency SLAs.
Anti-patterns to Avoid
Monolithic upload endpoints without chunking causing poor reliability and scalability, leading to frequent upload failures and inability to resume interrupted uploads.
Synchronous transcoding blocking upload threads leading to throughput collapse and increased upload latency during peak loads.
Serving video directly from origin storage causing high latency and backend overload during traffic spikes, resulting in poor user experience.
Synchronous view count increments in relational DB causing write contention and latency spikes, which degrade system responsiveness and scalability.
Storing DRM keys on client without DRM enforcement leads to piracy risks and violates content provider agreements, compromising content security.
✅
Model Answer
▼ Reveal

This design outlines a scalable video streaming platform supporting secure uploads, efficient transcoding, low-latency global delivery, accurate view counts, and secure offline playback with DRM enforcement.

Key Decisions
  • Choosing AWS S3 multipart upload with client-side encryption (AES-256 via AWS KMS) to ensure secure, resumable uploads and reduce trust surface.
  • Using Apache Kafka partitioned by video ID to guarantee ordered, idempotent transcoding tasks and enable horizontal scaling.
  • Autoscaling FFmpeg transcoding workers in Kubernetes to handle 10,000 concurrent uploads efficiently with fault tolerance.
  • Leveraging CloudFront CDN with signed URLs and cache keys based on video ID and bitrate to maximize cache hit ratio and meet P99 latency under 200ms.
  • Implementing Redis Streams to batch view count increments every 30 seconds, updating Cassandra with video_id partition key to avoid hot keys and ensure scalability.
  • Integrating Widevine DRM with encrypted HLS streams and managing keys via Google Cloud KMS with short-lived licenses to enforce secure offline playback.
  • AWS S3 multipart upload with client-side encryption using AWS KMS for secure, resumable uploads.
  • Apache Kafka with video ID partitioning for scalable, ordered transcoding task distribution.
  • Kubernetes-managed FFmpeg transcoding workers for elastic, fault-tolerant video processing.
  • CloudFront CDN with signed URLs and adaptive bitrate streaming for low-latency global delivery.
  • Redis Streams for collecting view events and batching updates to Cassandra for scalable, accurate view counts.
  • Widevine DRM integrated with encrypted HLS streams and Google Cloud KMS for secure offline playback.

Practice

(1/5)
1. In the typeahead system request flow, at step 4 the Ranking Service receives candidate suggestions from the Trie Cache. What does the Ranking Service do if the personalization data for the user is missing at this step?
easy
A. It returns an empty suggestion list immediately to avoid stale results
B. It blocks the request until personalization data is fetched synchronously
C. It falls back to global popularity-based ranking without personalization
D. It queries the Trie Cache again with a different prefix to compensate

Solution

  1. Step 1: Trace Ranking Service behavior at step 4

    Ranking Service combines candidate suggestions with personalization data; if missing, it must still return relevant results.

  2. Step 2: Evaluate fallback strategies

    Blocking for personalization data (C) causes latency; empty results (A) degrade UX; querying Trie again (D) is unrelated. Falling back to global popularity ranking ensures responsiveness.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Ranking Service fallback -> global popularity ranking [check]
Hint: Missing personalization -> fallback to global popularity ranking [check]
Common Mistakes:
2. In the vector database request flow, at Step 4 the ANN Search Worker queries the HNSW index with a new vector. What state does the worker observe if the index is currently rebuilding due to a recent bulk insert?
easy
A. The worker returns stale search results from the previous index snapshot
B. The worker blocks until the rebuild completes before returning results
C. The worker returns an empty result set indicating no neighbors found
D. The worker queries a fallback brute-force scan of all vectors

Solution

  1. Step 1: Understand HNSW index rebuild behavior

    HNSW index rebuilds asynchronously; during rebuild, queries use the last consistent snapshot.

  2. Step 2: Analyze worker behavior at Step 4

    The ANN Search Worker returns results from the previous stable index snapshot, so stale but consistent results are served.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    HNSW queries serve last snapshot during rebuild, avoiding blocking [check]
Hint: HNSW queries serve last snapshot during rebuild, no blocking [check]
Common Mistakes:
3. A live video streaming service requires minimal latency and can tolerate occasional packet loss. Which transport protocol is most appropriate?
easy
A. UDP, because it provides low latency without retransmission delays
B. SCTP, because it supports multi-streaming and multi-homing
C. TCP, because it guarantees packet delivery and ordering
D. HTTP over TLS, because it ensures secure and reliable delivery

Solution

  1. Step 1: Identify latency and loss tolerance requirements

    Live video streaming prioritizes low latency and can tolerate some packet loss, so retransmission delays are undesirable.

  2. Step 2: Match protocol characteristics

    UDP provides minimal overhead and no retransmission, fitting low latency needs. TCP guarantees delivery but adds delay due to retransmissions and congestion control.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    UDP suits low-latency, lossy streaming [check]
Hint: UDP trades reliability for speed in latency-sensitive apps [check]
Common Mistakes:
4. For Pastebin analytics tracking snippet views in real-time, which architecture best balances scalability and data freshness?
medium
A. Use a streaming platform like Kafka to collect events and process them asynchronously in batches
B. Store all view events directly in a relational database with immediate aggregation queries
C. Log view events to flat files and run daily MapReduce jobs for analytics
D. Send view events directly to a NoSQL store with eventual consistency for real-time queries

Solution

  1. Step 1: Analyze real-time analytics requirements

    High throughput and near real-time aggregation require decoupling ingestion and processing.

  2. Step 2: Evaluate architectures

    Kafka streaming (B) supports scalable event ingestion and asynchronous processing; relational DB (A) can't handle high write load; flat files (C) have high latency; NoSQL eventual consistency (D) may cause stale reads.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka decouples ingestion and processing for scalable real-time analytics [check]
Hint: Stream events asynchronously for scalable real-time analytics [check]
Common Mistakes:
5. For real-time message delivery to mobile clients with intermittent connectivity, which architecture best balances latency and resource usage?
medium
A. Use gRPC streaming connections from server to mobile clients
B. Use WebSocket connections with heartbeat and fallback to push notifications
C. Use long polling HTTP requests from clients to server
D. Use MQTT protocol with persistent sessions for mobile clients

Solution

  1. Step 1: Evaluate WebSocket vs gRPC streaming for mobile clients

    gRPC streaming is efficient but poorly supported on many mobile platforms; WebSocket is widely supported with heartbeat to detect disconnects.

  2. Step 2: Consider fallback mechanisms

    WebSocket with push notification fallback handles intermittent connectivity better than long polling or MQTT, which have higher overhead or complexity.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    WebSocket + push fallback balances latency and mobile constraints [check]
Hint: WebSocket + push fallback handles mobile connectivity best [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard