Video recommendation system in HLD - Scalability & System Analysis

Scalability Analysis - Video recommendation system

Growth Table: Video Recommendation System

Scale	Users	Data Volume	Traffic	System Changes
Small	100 users	Few thousand videos, user profiles	Low QPS (~100 requests/sec)	Single app server, single DB instance, simple batch recommendations
Medium	10K users	Millions of videos, user interactions	Moderate QPS (~10K requests/sec)	Multiple app servers, DB read replicas, caching layer, offline model training
Large	1M users	Hundreds of millions of videos, rich user data	High QPS (~100K requests/sec)	Distributed databases, sharded user data, real-time streaming data pipelines, CDN for video delivery
Very Large	100M users	Billions of videos and interactions	Very high QPS (~10M requests/sec)	Multi-region deployment, advanced sharding, multi-level caching, AI model serving clusters, global CDN, data archiving

First Bottleneck

At small scale, the database is the first bottleneck because it must handle many read and write requests for user interactions and video metadata. As users grow, the recommendation model training and serving become bottlenecks due to heavy computation and data volume. At large scale, network bandwidth and data storage also become critical bottlenecks.

Scaling Solutions

• Database: Use read replicas to handle read traffic, connection pooling, and eventually shard user and video data by user ID or video category.
• Caching: Cache popular recommendations and video metadata using Redis or Memcached to reduce DB load.
• Application Servers: Horizontally scale app servers behind load balancers to handle increased request volume.
• Model Training and Serving: Use distributed computing frameworks for offline training and deploy models on dedicated serving clusters with GPU acceleration.
• Data Pipelines: Implement real-time streaming pipelines (e.g., Kafka) for user activity ingestion and feature updates.
• Content Delivery: Use a global CDN to serve video content efficiently and reduce latency.
• Storage: Use distributed object storage for videos and archive old data to cheaper storage tiers.

Back-of-Envelope Cost Analysis

• Requests per second: At 1M users, expect ~100K QPS for recommendations and video views.
• Storage: Videos require petabytes of storage; metadata and user data require terabytes to petabytes.
• Bandwidth: Video streaming consumes the most bandwidth; a 1 Gbps link can serve ~125 MB/s, so multiple CDN edge servers are needed.
• Compute: Model training requires GPU clusters; serving models requires CPU/GPU servers scaled horizontally.

Interview Tip

Start by clarifying scale and requirements. Discuss data volume, traffic patterns, and latency needs. Identify the first bottleneck and propose targeted solutions. Explain trade-offs between consistency, latency, and cost. Use real numbers to justify scaling steps. Show understanding of caching, sharding, and distributed systems.

Self Check

Your database handles 1000 QPS. Traffic grows 10x to 10,000 QPS. What do you do first?

Answer: Add read replicas and implement caching to reduce database load before considering sharding or more complex solutions.

Key Result

The database is the first bottleneck at low to medium scale; scaling requires caching, read replicas, and sharding, while at large scale, distributed model serving and CDN become critical.

Practice

1. What is the primary goal of a video recommendation system?

easy

A. To store all videos in a single database

B. To delete unpopular videos automatically

C. To compress videos for faster streaming

D. To personalize video content to increase user engagement

Solution

1. Step 1: Understand the purpose of recommendation systems

   Recommendation systems aim to suggest content that matches user preferences to keep them engaged.

2. Step 2: Identify the main goal in video platforms

   Personalizing video content helps users find videos they like, increasing engagement and satisfaction.

3. Final Answer:

   To personalize video content to increase user engagement -> Option D

4. Quick Check:

   Personalization = Engagement [OK]

Hint: Focus on user benefit and engagement goals [OK]

Common Mistakes:

• Confusing storage with recommendation
• Thinking compression is the main goal
• Assuming deletion is automatic

2. Which component is essential for real-time video recommendations?

easy

A. Real-time data streaming and processing

B. Offline video transcoding

C. Batch processing system only

D. Static video metadata storage

Solution

1. Step 1: Identify real-time needs in recommendations

   Real-time recommendations require processing fresh user interactions quickly.

2. Step 2: Match components to real-time processing

   Data streaming and processing systems handle live data to update recommendations instantly.

3. Final Answer:

   Real-time data streaming and processing -> Option A

4. Quick Check:

   Real-time = Streaming & Processing [OK]

Hint: Real-time means instant data handling, not batch [OK]

Common Mistakes:

• Choosing batch processing for real-time needs
• Confusing transcoding with recommendation
• Ignoring dynamic data updates

3. Consider a system where user watch history is updated every hour, but recommendations are generated in real-time. What is the likely output when a user watches a new video?

medium

A. The new video immediately influences recommendations

B. Recommendations update only after the next hourly batch

C. Recommendations never change after initial setup

D. The system crashes due to conflicting updates

Solution

1. Step 1: Understand real-time recommendation generation

   Real-time generation means recommendations can change instantly based on new data.

2. Step 2: Analyze watch history update frequency

   Even if watch history updates hourly, real-time components can use streaming data to update recommendations immediately.

3. Final Answer:

   The new video immediately influences recommendations -> Option A

4. Quick Check:

   Real-time generation = Immediate update [OK]

Hint: Real-time generation overrides batch delay [OK]

Common Mistakes:

• Assuming batch update controls recommendation timing
• Thinking system crashes on data conflict
• Believing recommendations are static

4. A video recommendation system is showing outdated videos despite recent user activity. What is the most likely cause?

medium

A. User profiles are deleted frequently

B. Real-time data pipeline is broken or delayed

C. Video metadata is missing thumbnails

D. Batch processing runs too frequently

Solution

1. Step 1: Identify the symptom - outdated recommendations

   Outdated videos suggest the system is not processing recent user actions timely.

2. Step 2: Check real-time data pipeline status

   If the real-time pipeline is broken or delayed, fresh user data won't update recommendations promptly.

3. Final Answer:

   Real-time data pipeline is broken or delayed -> Option B

4. Quick Check:

   Outdated = Pipeline delay [OK]

Hint: Outdated results often mean broken real-time updates [OK]

Common Mistakes:

• Blaming user profile deletion
• Confusing metadata issues with recommendation freshness
• Assuming batch runs cause outdated data

5. To scale a video recommendation system for millions of users, which design choice best balances freshness and computational cost?

hard

A. Generate recommendations on-demand without caching

B. Only use batch processing with daily updates

C. Use a hybrid approach combining batch model training with real-time feature updates

D. Store all user data in a single monolithic database

Solution

1. Step 1: Understand scalability challenges

   Millions of users require efficient computation and timely recommendations without overload.

2. Step 2: Evaluate design options for freshness and cost

   Hybrid systems use batch training for accuracy and real-time updates for freshness, balancing cost and performance.

3. Final Answer:

   Use a hybrid approach combining batch model training with real-time feature updates -> Option C

4. Quick Check:

   Hybrid approach = Freshness + Cost balance [OK]

Hint: Hybrid systems combine batch and real-time strengths [OK]

Common Mistakes:

• Relying only on batch causes stale data
• On-demand without caching is costly
• Monolithic DB limits scalability