Bird
Raised Fist0

Design TikTok - Short Video Upload, For-You Feed Algorithm, Creator Stack

HLDhardInterview freq: high
Choose your preparation mode4 modes available
🏗️
Design TikTok - Short Video Upload, For-You Feed Algorithm, Creator Stack
Design a scalable short video platform with personalized feed and creator tools at 2B DAU scale
hardVideo and Media StreamingFreq: high
Prerequisites:System Design FundamentalsCAP TheoremDistributed SystemsData Modeling for Scale
⚙️
Functional Requirements
  • Users can upload short videos (up to 60 seconds) with metadata and tags
  • For-You feed delivers personalized video streams with sub-100ms P99 latency
  • Creator stack provides analytics, video management, and monetization tools
  • Real-time video transcoding and thumbnail generation
  • Support video likes, comments, shares, and follows
  • Push notifications for new content and creator updates
📊
Non-Functional Requirements
  • Support 2 billion daily active users (DAU)
  • Handle peak write throughput of ~1 million video uploads per minute
  • 99.99% system availability with multi-region failover
  • End-to-end latency for feed generation under 100ms P99
  • Data retention for videos and metadata for at least 1 year
❓
Clarifying Questions to Ask
  • What is the maximum video size and duration allowed per upload?
  • What is the expected average and peak number of videos watched per user per day?
  • How personalized should the For-You feed be? Is real-time feedback incorporated?
  • What are the SLAs for video upload processing and feed update latency?
  • Is the creator stack expected to support real-time analytics or batch processing?
  • Are there any geographic restrictions or data residency requirements?
Out of Scope
✗ Live streaming video features
✗ In-depth video editing tools within the app
✗ Ad targeting and bidding system
✗ Third-party integrations for external sharing
📌
Assumptions
  • 2 billion DAU globally
  • Average 40 video uploads per user per day (including re-uploads, edits)
  • Average video metadata size 100 KB (includes video pointers, tags, user info)
  • Average video file size 5 MB per upload (compressed short video)
⚡
QPS Calculation
write qps2B users × 40 uploads/day ÷ 86400 sec ≈ 925,926 writes/sec
peak write qpsAssuming 3× peak factor: 925,926 × 3 ≈ 2.78M writes/sec
read qpsAssuming 50 videos watched per user per day: 2B × 50 ÷ 86400 ≈ 1.16M reads/sec
💾
Storage Estimate
daily video storage2B × 40 × 5MB = 400TB/day raw video data
daily metadata storage2B × 40 × 100KB = 8TB/day metadata
retention 30 days(400TB + 8TB) × 30 = 12.24PB
with replication 3x12.24PB × 3 = 36.72PB total storage
🖥️
Server Estimate
upload serversAssuming 1 server handles 2000 uploads/sec, need 2.78M ÷ 2000 ≈ 1390 servers
feed generation serversAssuming 1 server handles 5000 feed requests/sec, need 1.16M ÷ 5000 ≈ 232 servers
🏗️
Architecture Diagram
Architecture Diagram
      +-------------+
      |  Mobile App |
      +------+------+
             | Upload Video + Metadata
             v
      +------+-------+
      | Upload Proxy |
      +------+-------+
             |
             v
      +------+-------+      +----------------+
      | Video Storage|<---->| Transcoding Svc|
      +------+-------+      +----------------+
             |
             v
      +------+-------+
      | Metadata DB  |<----------------+
      +------+-------+                 |
             |                        |
             v                        |
      +------+-------+        +-------+-------+
      | For-You Feed |<-------| User Profile  |
      |  Generator   |        +---------------+
      +------+-------+
             |
             v
      +------+-------+
      | Delivery CDN |
      +--------------+
🧩
Components
Upload Proxy
Handles video upload requests, performs authentication, rate limiting, and forwards to storage
Why thisNginx with Lua scripting for high concurrency and extensibility at edge
Why notAWS API Gateway rejected due to higher latency and cost at 2.78M writes/sec
Video Storage
Stores raw video files reliably with high throughput and durability
Why thisAmazon S3 with multi-AZ replication for 99.999999999% durability and elastic scaling
Why notHDFS rejected due to poor support for small file writes and high latency
Transcoding Service
Converts uploaded videos into multiple resolutions and generates thumbnails asynchronously
Why thisKubernetes-based microservices with FFmpeg for scalable, containerized transcoding
Why notServerless functions rejected due to cold start latency and limited runtime
Metadata DB
Stores video metadata, user info, and feed state for fast queries
Why thisApache Cassandra for linear write scalability and tunable consistency at 2.78M writes/sec
Why notPostgreSQL rejected due to complex manual sharding and write bottlenecks
For-You Feed Generator
Generates personalized video feed using ML models and user interaction data
Why thisApache Flink streaming jobs for real-time feed computation with low latency
Why notBatch Spark jobs rejected due to high latency and stale feed generation
User Profile Service
Manages user preferences, follows, and interaction history for personalization
Why thisRedis Cluster with TTL for fast read/write and ephemeral session data
Why notMemcached rejected due to lack of persistence and complex eviction policies
Delivery CDN
Delivers video content globally with low latency and high throughput
Why thisCloudflare CDN for edge caching and DDoS protection at global scale
Why notSelf-hosted CDN rejected due to operational complexity and slower global reach
🔄
Data Flow
1
Mobile App → Upload Proxy
User uploads video and metadata
Proxy authenticates and rate limits upload requests
2
Upload Proxy → Video Storage
Stores raw video file asynchronously
Ensures durability and availability with multi-AZ replication
3
Upload Proxy → Metadata DB
Stores video metadata and upload status
Metadata used for feed generation and search
4
Video Storage → Transcoding Service
Triggers transcoding jobs for multiple resolutions
Asynchronous processing to avoid upload latency
5
Transcoding Service → Metadata DB
Updates metadata with transcoding status and URLs
Feeds updated with playable video URLs
6
For-You Feed Generator → User Profile Service
Fetches user preferences and interaction history
Personalizes feed based on latest user data

Practice

(1/5)
1. A high-traffic e-commerce platform expects a sudden surge in user requests immediately after launch. Which component is best suited to pre-populate the cache with frequently accessed product data to avoid cold-start latency?
easy
A. Implement a cache warming service that preloads hot keys into the in-memory cache before launch
B. Use a CDN to cache static assets only, ignoring dynamic product data
C. Rely on lazy loading where cache entries are populated only on first user request
D. Use a write-through cache that updates cache only after database writes occur

Solution

  1. Step 1: Identify cache warming purpose

    Cache warming service proactively loads hot keys to avoid cold cache latency at launch.
  2. Step 2: Eliminate lazy and write-through options

    Lazy loading causes initial latency; write-through updates cache reactively, not proactively.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Preloading hot keys before launch avoids cold cache misses [check]
Hint: Preload hot keys before launch to avoid cold cache misses [OK]
Common Mistakes:
2. A company wants to hide internal client IP addresses from external servers while allowing clients to access the internet. Which proxy type should be used to achieve this?
easy
A. DNS proxy, because it resolves domain names for clients
B. Reverse proxy, because it hides the backend servers from clients
C. Load balancer, because it distributes requests evenly across servers
D. Forward proxy, because it hides client IPs from external servers

Solution

  1. Step 1: Identify proxy role in hiding client IPs

    A forward proxy acts on behalf of clients, masking their IP addresses from external servers.
  2. Step 2: Eliminate other options

    Reverse proxies hide backend servers, not clients; load balancers distribute load but don't hide clients; DNS proxies only resolve names.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Forward proxy hides client IPs from external servers [check]
Hint: Forward proxy hides clients; reverse proxy hides servers [check]
Common Mistakes:
3. A messaging system must handle 1 million writes per second and fan-out delivery to thousands of consumers without blocking producers. Which component is primarily responsible for decoupling producers from consumers to achieve this?
easy
A. Use a relational database with write-ahead logging to buffer messages
B. Use direct synchronous HTTP calls from producers to consumers
C. Use a centralized in-memory cache like Redis to store messages temporarily
D. Use a distributed message broker with partitioned logs and asynchronous consumers

Solution

  1. Step 1: Identify the decoupling mechanism

    Relational DBs with WAL (Use a relational database with write-ahead logging to buffer messages) are durable but cannot handle 1M writes/sec with fan-out efficiently; Redis (Use a centralized in-memory cache like Redis to store messages temporarily) is fast but lacks durability and persistence guarantees at this scale; direct HTTP calls (Use direct synchronous HTTP calls from producers to consumers) block producers and do not decouple.
  2. Step 2: Recognize distributed brokers' role

    Distributed message brokers with partitioned logs (Use a distributed message broker with partitioned logs and asynchronous consumers) allow asynchronous writes and reads, decoupling producers and consumers, enabling high throughput and scalable fan-out.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Partitioned broker logs decouple producers and consumers for high throughput [check]
Hint: Partitioned brokers decouple producers and consumers [check]
Common Mistakes:
4. If the Inverted Index Update Service crashes and stops updating document frequencies (DF), what is the exact failure observed in search relevance?
medium
A. The system automatically switches to BM25 scoring without DF dependency
B. Ranking uses stale DF causing relevance scores to drift and degrade over time
C. New documents are not indexed but existing DF remains accurate
D. Search queries fail with errors due to missing DF data

Solution

  1. Step 1: Understand DF update role

    Inverted Index Update Service maintains DF counts; if it crashes, DF stops updating and becomes stale.
  2. Step 2: Trace impact on ranking

    Ranking cannot rely on fresh DF; system falls back to BM25 variant that tolerates stale DF or uses approximations; queries do not error out; new documents may still be indexed but DF is outdated.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    System falls back to BM25 variant tolerating stale DF [check]
Hint: Without DF updates, system falls back to BM25 variant [check]
Common Mistakes:
5. At 10x scale, which MongoDB component is most likely to become the bottleneck first when using the bucket pattern for time-series data, and why?
hard
A. The application server, because it cannot generate bucket IDs fast enough for writes.
B. The MongoDB query router (mongos), because it must maintain metadata for all buckets and route writes precisely.
C. The shard servers, because bucket documents grow large and cause frequent document splits and index overhead.
D. The network layer, because increased bucket writes saturate bandwidth between shards.

Solution

  1. Step 1: Identify bucket pattern scaling challenges

    Bucket pattern creates many bucket documents requiring precise routing by mongos.
  2. Step 2: Mongos metadata overhead

    Mongos maintains routing metadata and must route each write to correct shard and bucket, increasing load at scale.
  3. Step 3: Compare other components

    Shard servers handle writes but document splits are less frequent; network saturates later; app server bottleneck unlikely.
  4. Final Answer:

    Option B -> Option B
  5. Quick Check:

    Mongos routing metadata load grows fastest at 10x scale [check]
Hint: Mongos routing metadata overload is first bottleneck at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard