Bird
Raised Fist0

Design TikTok - Short Video Upload, For-You Feed Algorithm, Creator Stack

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design TikTok - Short Video Upload, For-You Feed Algorithm, Creator Stack
45 min · senior
senior

Design a system for a 50M DAU short-video platform similar to TikTok. Users upload short videos, consume a personalized For-You feed with sub-200ms P99 latency, and creators manage their content and analytics. The system must handle high write throughput, real-time feed ranking, and creator insights with strong data privacy and encryption.

Constraints
  • End-to-end encryption of user video uploads and metadata at rest and in transit
  • For-You feed P99 latency under 200ms for 50M DAU with average 40 videos/min user consumption
  • Offline video playback support with local caching on client devices
  • Creator analytics updated with near real-time data (within 1 minute delay)
  • Scalable to 50M DAU with peak upload throughput of 10K videos/sec
📋
Guided Steps
1/5
Step 1
Designing the Video Upload Pipeline
How would you design the video upload flow to handle 10K uploads/sec with encryption and durability?
💡 Hint
Consider chunked uploads, object storage, and encryption at rest and in transit.
Key Decision
Use AWS S3 with client-side AES-256 encryption for video storage, combined with Kafka for upload event ingestion to decouple upload from processing.
Interviewer Expectation
Mid: Names S3 and Kafka for storage and ingestion. Senior: Explains client-side encryption to meet end-to-end encryption, multipart upload for large files, and Kafka partitioning by user ID for ordered processing.
⚠️Common Wrong Answer
Using a relational DB like MySQL for storing videos leads to storage bottlenecks and poor scalability at 10K uploads/sec.
Step 2
Designing the For-You Feed Generation
How would you design the For-You feed to serve personalized videos with P99 latency under 200ms at 50M DAU?
💡 Hint
Think about real-time ranking, caching, and distributed serving layers.
Key Decision
Use Apache Flink for real-time stream processing of user interactions, Redis as a low-latency cache for precomputed ranked feeds, and Cassandra for storing user-video interaction history.
Interviewer Expectation
Mid: Mentions stream processing and caching. Senior: Explains Flink's event-time processing for freshness, Redis TTL-based cache invalidation, and Cassandra's wide-column model for efficient user interaction queries.
⚠️Common Wrong Answer
Relying solely on batch processing (e.g., Hadoop) causes feed freshness delays and violates 200ms P99 latency.
Step 3
Designing the Creator Analytics Stack
How would you design the analytics system to provide near real-time insights to creators with minimal delay?
💡 Hint
Consider event ingestion, aggregation, and query engines optimized for time-series data.
Key Decision
Use Kafka for event ingestion, Apache Druid for real-time OLAP analytics, and Presto for ad-hoc querying by creators.
Interviewer Expectation
Mid: Names Kafka and Druid for analytics. Senior: Explains Kafka topic partitioning by creator ID, Druid's segment granularity for sub-minute rollups, and Presto federated queries for flexible analytics.
⚠️Common Wrong Answer
Using a traditional RDBMS for analytics leads to slow aggregation and high query latency, failing near real-time requirements.
Step 4
Ensuring Offline Video Playback and Sync
How would you enable offline playback of videos and sync user state when back online?
💡 Hint
Consider client-side caching, TTL, and sync protocols for consistency.
Key Decision
Implement client-side local storage with encrypted video cache and use a sync protocol with Redis-based presence tracking and Kafka for event replay on reconnect.
Interviewer Expectation
Mid: Mentions local caching and sync. Senior: Explains Redis TTL for presence detection, Kafka consumer offsets for replaying missed events, and client-side encryption to maintain privacy offline.
⚠️Common Wrong Answer
Not encrypting cached videos on client risks data leakage; naive HTTP polling for sync causes excessive load.
Step 5
API Design for Core Endpoints
Sketch the API design for video upload, For-You feed fetch, and creator analytics retrieval.
💡 Hint
Focus on request/response structure, authentication, pagination, and error handling.
Key Decision
Design RESTful endpoints with JWT-based auth: POST /videos/upload with multipart encrypted payload, GET /feed/for-you with cursor-based pagination, GET /creator/analytics with time-range filters.
Interviewer Expectation
Mid: Defines basic REST endpoints. Senior: Details JWT scopes, multipart upload chunking, cursor pagination for feed, and analytics query parameters with rate limiting.
⚠️Common Wrong Answer
Using simple offset pagination for feed leads to inconsistent results under concurrent updates and high latency.
🔌
API Design Challenge
Sketch 3 core API endpoints for the system: video upload, fetching the For-You feed, and creator analytics retrieval.
Hints
  • How does the client securely upload videos and metadata?
  • How does the client request and paginate the personalized For-You feed?
  • How does the creator client retrieve aggregated analytics data efficiently?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of scalable object storage with encryption for video uploads
  • Real-time stream processing for feed ranking with latency under 200ms P99
  • Low-latency caching layer for feed serving
  • Near real-time analytics stack with OLAP engine
  • API design covering upload, feed fetch, and analytics retrieval
Good to Have - 30%
  • Client-side encryption and offline playback support
  • Partitioning strategies for Kafka and Cassandra to ensure ordered processing
  • Use of cursor-based pagination for feed consistency
  • Rate limiting and authentication details in API design
Bonus - 10%
  • Detailed failure mode analysis for upload pipeline and feed serving
  • Discussion of trade-offs between batch vs stream processing for feed freshness
  • Consideration of data privacy and compliance in design
Anti-patterns to Avoid
Storing videos in relational databases causes storage bottlenecks and poor scalability at 10K uploads/sec, leading to system failure under load.
Batch-only feed generation causes stale feeds and violates 200ms P99 latency, resulting in poor user experience.
Using offset pagination for feed leads to inconsistent user experience under concurrent updates due to shifting offsets.
Polling-based sync at 50M DAU results in 50M reconnects per minute just to check for messages, collapsing under load.
Not encrypting client-side cached videos risks data leakage and violates user privacy compliance.
Using traditional RDBMS for analytics causes slow aggregation and high query latency, failing near real-time requirements.
✅
Model Answer
▼ Reveal

This design addresses TikTok's core system requirements for short video upload, personalized For-You feed with low latency, and creator analytics with near real-time insights. It ensures scalability to 50M DAU, strong encryption for privacy, and offline playback support.

Key Decisions
  • Use AWS S3 with client-side AES-256 encryption for durable, scalable video storage meeting end-to-end encryption requirements.
  • Employ Kafka with user ID partitioning to ingest upload events, ensuring ordered processing and decoupling upload from downstream processing.
  • Leverage Apache Flink for real-time stream processing of user interactions to update feed rankings with event-time semantics for freshness.
  • Use Redis as a low-latency cache with TTL-based invalidation to serve personalized feeds within 200ms P99 latency.
  • Store user-video interaction history in Cassandra wide-column store optimized for efficient queries supporting feed personalization.
  • Implement Apache Druid for real-time OLAP analytics with sub-minute segment granularity to provide near real-time creator insights.
  • Use Presto for federated, ad-hoc querying enabling flexible analytics queries by creators.
  • Enable offline playback with client-side encrypted local video cache and sync protocol using Redis presence tracking and Kafka event replay on reconnect.
  • Design RESTful APIs with JWT-based authentication, multipart encrypted uploads, cursor-based pagination for feed consistency, and rate limiting for robustness.
  • AWS S3 with client-side AES-256 encryption for durable, scalable video storage meeting encryption constraints
  • Kafka with user ID partitioning for ordered, scalable upload event ingestion and decoupling upload from processing
  • Apache Flink for real-time stream processing enabling sub-second feed ranking updates at scale
  • Redis as a low-latency cache with TTL-based invalidation to meet 200ms P99 feed latency
  • Cassandra wide-column store for efficient user-video interaction history queries supporting feed personalization
  • Apache Druid for real-time OLAP analytics providing near real-time creator insights with sub-minute granularity
  • Presto for federated, ad-hoc querying of analytics data enabling flexible creator queries
  • Client-side encrypted local video cache with sync protocol using Redis presence and Kafka replay for offline support
  • RESTful API design with JWT authentication, multipart uploads, cursor pagination, and rate limiting for robustness

Practice

(1/5)
1. If the Kafka cluster buffering RTMP packets crashes and loses the latest messages, what exactly breaks in the live streaming pipeline?
medium
A. The CDN serves stale HLS segments but continues streaming without interruption
B. The RTMP ingest server retries sending lost packets automatically with no impact
C. The transcoder stalls waiting for missing packets, causing segment gaps and viewer buffering
D. The player switches to a backup stream seamlessly without user noticing

Solution

  1. Step 1: Understand Kafka's role

    Kafka buffers RTMP packets to ensure ordered, durable delivery to the transcoder.

  2. Step 2: Trace failure impact

    If Kafka loses messages, the transcoder stalls (A) due to missing packets, causing segment gaps. CDN (B) serves only what it has, so stale segments appear. RTMP ingest server (C) does not retry lost packets automatically; loss is permanent. Player (D) cannot switch streams without explicit failover.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Kafka loss causes permanent packet loss; RTMP ingest lacks automatic retry [check]
Hint: Kafka loss means lost packets; RTMP ingest has no auto-retry [check]
Common Mistakes:
2. For a recommendation engine requiring fast traversal of user-item relationships with frequent updates, which architecture is best suited?
medium
A. Use a key-value store with denormalized recommendation lists per user
B. Use a relational database with complex JOIN queries for recommendations
C. Use a batch-processing pipeline over a document store to precompute recommendations
D. Use an in-memory graph database with eventual consistency for fast traversals

Solution

  1. Step 1: Analyze requirement

    Fast traversal with frequent updates demands low-latency graph traversals and quick writes.

  2. Step 2: Evaluate architectures

    Relational DBs have high JOIN cost; batch pipelines add latency; key-value stores lack flexible traversal.

  3. Step 3: Choose architecture

    In-memory graph DBs with eventual consistency balance fast traversals and update throughput.

  4. Final Answer:

    Option D -> Option D

  5. Quick Check:

    In-memory graph DBs optimize traversal speed and handle frequent updates well [check]
Hint: In-memory graph DBs balance speed and updates best [check]
Common Mistakes:
3. Without the message broker (e.g., Kafka) in a high-throughput chat system, what exactly breaks in the message delivery flow?
medium
A. Messages are sent synchronously causing sender blocking and message loss on receiver failure
B. Alice's send operation blocks until Bob's server responds; if Bob's server crashes, the message is lost with no retry
C. Messages are encrypted incorrectly causing security breaches
D. The system automatically switches to batch mode causing high latency

Solution

  1. Step 1: Role of message broker

    Broker decouples sender and receiver, enabling asynchronous delivery and retries.

  2. Step 2: Consequence of missing broker

    Without it, messages are sent synchronously causing sender blocking; if receiver crashes, messages are lost; encryption (A) unaffected; blocking and loss (C) is exact failure; batch mode (D) not automatic.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Broker removal causes blocking sends and message loss on receiver failure [check]
Hint: No broker means synchronous sends block and lose messages [check]
Common Mistakes:
4. At 10x scale, which component in a push-based newsfeed system is most likely to fail first and why?
hard
A. The fan-out service, because pushing to millions of followers causes write amplification bottleneck
B. The database, because it cannot handle 10x writes per second
C. The cache layer, because it runs out of memory storing all user feeds
D. The web servers, because they cannot handle 10x concurrent HTTP connections

Solution

  1. Step 1: Analyze database scaling

    Databases can be sharded and scaled horizontally; not usually the first bottleneck.

  2. Step 2: Analyze fan-out service scaling

    Push-based fan-out causes write amplification proportional to follower count; at 10x scale, this bottleneck grows superlinearly and saturates first.

  3. Step 3: Analyze cache and web servers

    Caches can be scaled with eviction and sharding; web servers scale with load balancers and stateless design.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Fan-out write amplification causes first bottleneck at 10x scale [check]
Hint: Fan-out write amplification hits first at 10x scale [check]
Common Mistakes:
5. At 10x scale, which component is most likely to become the bottleneck first in the typeahead system, and why?
hard
A. The front-end API servers, because they cannot handle the increased number of concurrent connections
B. The personalization data store, because user profile reads increase linearly and cannot be sharded easily
C. The ranking service, because ranking computations grow exponentially with candidate suggestions
D. The Trie cache servers, because maintaining low-latency prefix lookups requires large in-memory state that is hard to shard further

Solution

  1. Step 1: Analyze component scalability

    Personalization stores can be sharded; ranking computations scale linearly; front-end servers can be horizontally scaled.

  2. Step 2: Identify Trie cache bottleneck

    Trie caches hold large in-memory prefix trees; sharding by prefix has limits due to prefix overlap and memory footprint, causing latency and memory bottlenecks.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Trie cache memory and shard limits -> first bottleneck at 10x scale [check]
Hint: Trie cache memory limits cause first bottleneck at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard