Bird
Raised Fist0

Design an E-Learning Platform - Course, Video, Progress, Certificate at Scale

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design an E-Learning Platform - Course, Video, Progress, Certificate at Scale
Design a scalable e-learning platform handling courses, videos, user progress, and certificates for billions of users
hardSearch and DiscoveryFreq: high
Prerequisites:Inverted IndexElasticsearch
⚙️
Functional Requirements
  • Users can browse and search courses with rich metadata and full-text search.
  • Stream course videos with adaptive bitrate and low latency.
  • Track user progress per course and per video with real-time updates.
  • Issue certificates upon course completion with verifiable authenticity.
  • Support course creation and updates by instructors with versioning.
  • Provide personalized course recommendations based on user history.
📊
Non-Functional Requirements
  • Support 2 billion daily active users globally.
  • Handle peak write throughput of 925K progress updates per second.
  • Ensure 99.99% availability for course content and progress APIs.
  • Maintain 100ms P99 latency for read APIs (course info, progress).
  • Store and serve petabytes of video content with 99.9% durability.
❓
Clarifying Questions to Ask
  • What is the expected average video length and size per course?
  • Is progress tracking required to be strongly consistent or eventually consistent?
  • Are certificates digital only or do we need physical printing integration?
  • What is the expected read-to-write ratio for progress and course data?
  • Should the platform support offline video viewing and progress sync?
  • Is multi-language support and localization required for courses?
Out of Scope
✗ Payment and billing systems for course purchases.
✗ Live interactive classes or real-time chat features.
✗ Detailed analytics dashboards for instructors.
✗ Mobile app UI/UX design specifics.
📌
Assumptions
  • 2 billion DAU globally active users.
  • Each user generates 40 progress update messages per day (watching videos, quizzes).
  • Average progress update message size is 100 bytes.
  • Average course metadata size is 10 KB; average video size is 500 MB.
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 writes/sec (progress updates)
peak write qpsAssuming 3× peak factor → 2.78M writes/sec peak
read qpsAssuming 10 reads per user per day → 2B × 10 / 86400 ≈ 231,481 reads/sec
💾
Storage Estimate
daily storage925,926 writes/sec × 100 bytes × 86400 sec ≈ 8 TB/day
retention days30
total storage8 TB/day × 30 days = 240 TB raw
with replication240 TB × 3 (replication factor) = 720 TB total storage
🖥️
Server Estimate
progress write serversAssuming 10K writes/sec per server → 2.78M / 10K = 278 servers
video streaming serversAssuming 1 Gbps per server, 2B users streaming 1% concurrently at 2 Mbps → 20M × 2 Mbps = 40 Tbps → 40,000 servers
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |  User Clients  |
          +-------+--------+
                  |
          +-------v--------+
          |  API Gateway   |
          +-------+--------+
                  |
  +---------------+----------------+----------------+
  |               |                |                |
  |               |                |                |
+v+--+        +---v---+        +---v---+        +---v---+
| ES |        | Kafka |        | Cassandra |     | MinIO |
+----+        +-------+        +-----------+     +-------+
  |               |                |                |
  |               |                |                |
  |               |                |                |
  |               |                |                |
  +---------------+----------------+----------------+
                  |                |                |
          +-------v--------+  +----v-----+    +-----v-----+
          | Progress Svc   |  | Cert Svc |    | Video CDN |
          +----------------+  +----------+    +-----------+
🧩
Components
Elasticsearch
Full-text search and course metadata indexing
Why thisElasticsearch provides distributed inverted index with near real-time search and supports complex queries for course discovery
Why notRelational DB full-text search (e.g., PostgreSQL) does not scale to billions of documents and high QPS with low latency
Apache Kafka
Durable, scalable event streaming for progress updates and asynchronous processing
Why thisKafka supports high throughput (millions of messages/sec), partitioning, and retention needed for 2B users' progress events
Why notRabbitMQ lacks horizontal scalability and throughput at this scale; AWS SQS has higher latency and lower throughput
Apache Cassandra
Storing user progress and course metadata with high write throughput and availability
Why thisCassandra's wide-column model supports time-series progress data with tunable consistency and linear write scalability
Why notMySQL sharding is complex and does not handle 3M writes/sec peak; MongoDB has weaker consistency guarantees at scale
MinIO (S3-compatible object store)
Storing and serving course video files with high durability and scalability
Why thisMinIO provides S3 API compatibility, supports petabyte scale, and integrates with CDN for low latency video delivery
Why notHDFS is optimized for batch processing, not low-latency video serving; local file systems lack scalability and durability
Progress Service
API layer for ingesting and querying user progress data with strong consistency
Why thisCustom microservice using Cassandra for fast writes and reads with caching to meet 100ms P99 latency
Why notDirect client writes to DB cause schema coupling and lack of validation; serverless functions have cold start latency
Certificate Service
Generates and verifies course completion certificates with cryptographic signatures
Why thisDedicated service ensures certificate authenticity and supports revocation and audit logs
Why notEmbedding certificate logic in Progress Service mixes concerns and complicates scaling
Video CDN
Global content delivery network for adaptive bitrate video streaming
Why thisCDN caches video content close to users, reducing latency and bandwidth costs
Why notServing videos directly from origin storage causes high latency and bandwidth bottlenecks
API Gateway
Unified entry point for all client requests with authentication, rate limiting, and routing
Why thisAPI Gateway handles millions of concurrent connections and routes requests efficiently
Why notDirect client connections to backend services increase complexity and reduce security

Practice

(1/5)
1. A system must enforce a maximum number of API calls per user per minute to prevent abuse. Which component in an API Gateway architecture is best suited to handle this requirement?
easy
A. The API Gateway applies rate limiting before routing requests to backend services
B. The backend microservices implement rate limiting individually to distribute load
C. A centralized database tracks request counts and enforces limits synchronously
D. Clients are responsible for self-throttling their request rates

Solution

  1. Step 1: Identify rate limiting responsibility

    The API Gateway is the entry point and can efficiently enforce rate limits before requests reach backend services, preventing overload.

  2. Step 2: Eliminate other options

    The backend microservices implement rate limiting individually to distribute load spreads rate limiting, risking inconsistent enforcement and backend overload. A centralized database tracks request counts and enforces limits synchronously adds latency and complexity by synchronous DB checks. Clients are responsible for self-throttling their request rates trusts clients, which is insecure.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    API Gateway enforces rate limits upfront to protect backend [check]
Hint: Rate limiting belongs at the gateway entry point [check]
Common Mistakes:
2. In the request flow for delivering an HLS video segment, at step 4 the Delivery Worker checks the cache status of the requested segment. What happens if the segment is not found in the edge cache at this step?
easy
A. The Delivery Worker waits for the client to retry the request later
B. The Delivery Worker fetches the segment from the origin server and caches it locally before delivery
C. The Delivery Worker immediately returns a 404 error to the client
D. The Delivery Worker redirects the client to another edge node

Solution

  1. Step 1: Trace Delivery Worker decision at Step 4 -- cache miss handling

    On cache miss, the Delivery Worker does not immediately fetch from origin but queues the request and waits for the segment to be fetched asynchronously.

  2. Step 2: Eliminate other options

    Immediate 404 is incorrect as segments may be fetched; redirecting client is inefficient; immediate fetch blocks delivery worker, violating async design.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Delivery Worker waits for segment availability before responding [check]
Hint: Cache miss triggers async fetch; delivery waits, no immediate fetch [check]
Common Mistakes:
3. If the commit metadata database becomes unavailable, what is the immediate failure observed when a user tries to create a new branch?
medium
A. The system creates the branch but marks it as unverified until metadata is restored
B. The branch creation request fails immediately with an error indicating metadata service unavailability
C. The system queues the branch creation request and retries later without user notification
D. The branch is created locally on the client but not persisted to the server

Solution

  1. Step 1: Understand role of commit metadata DB

    Commit metadata DB stores branch pointers; without it, server cannot persist new branches.

  2. Step 2: Trace branch creation failure

    Client may create branch locally, but server cannot persist it, so branch is effectively lost or inconsistent.

  3. Step 3: Eliminate other options

    Queuing without notification hides failure; immediate error is possible but not typical if client allows local ops; marking unverified branches requires metadata DB access.

  4. Final Answer:

    Option D -> Option D

  5. Quick Check:

    Metadata DB down -> branch created locally but not persisted [check]
Hint: Metadata DB down -> local branch only, no server persistence [check]
Common Mistakes:
4. For real-time analytics requiring low-latency dashboards and complex ad-hoc queries, which architecture is best suited?
medium
A. Use only Kafka Streams for both aggregation and serving queries
B. Use Flink to write directly to a NoSQL key-value store for dashboards
C. Use batch ETL jobs to load data into a relational database for dashboards
D. Use Flink for streaming aggregation and an OLAP store for dashboard queries

Solution

  1. Step 1: Evaluate streaming aggregation and query needs

    Flink excels at complex streaming aggregations; OLAP stores support fast ad-hoc queries and low-latency dashboards.

  2. Step 2: Eliminate other options

    Kafka Streams lacks complex query capabilities; batch ETL adds latency unsuitable for real-time; NoSQL key-value stores lack rich query support for dashboards.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Flink + OLAP store balances streaming and query complexity [check]
Hint: Flink streams, OLAP queries for real-time + ad-hoc [check]
Common Mistakes:
5. If the cache invalidation mechanism for the Trie shards fails, what is the most likely failure observed in the typeahead suggestions?
medium
A. Users see outdated suggestions that do not reflect recent data updates
B. The system crashes due to memory overflow in the Trie cache
C. Suggestions become empty because the Trie cache returns null on failure
D. Personalization ranking fails but prefix matching remains accurate

Solution

  1. Step 1: Understand cache invalidation role

    Cache invalidation ensures Trie shards reflect recent data; failure means stale data remains served.

  2. Step 2: Analyze failure impact

    Memory overflow (B) or empty suggestions (C) are unrelated to invalidation failure; personalization ranking (D) is separate from cache invalidation.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Failed invalidation -> stale suggestions served [check]
Hint: Failed cache invalidation -> stale suggestions served [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard