Bird
Raised Fist0

Design a ChatGPT-Like System - LLM Serving, Context Window, Token Streaming

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design a ChatGPT-Like System - LLM Serving, Context Window, Token Streaming
Design a scalable, low-latency system for serving large language models with context window management and token streaming
hardAI and ML System DesignFreq: high
Prerequisites:LLM ServingToken Streaming
⚙️
Functional Requirements
  • Serve LLM inference requests with multi-turn conversation context support (up to 8K tokens).
  • Support real-time token streaming to clients as the model generates output tokens.
  • Maintain conversation state and context window management per user session.
  • Allow users to start new conversations and retrieve past conversation history.
  • Handle user authentication and rate limiting per user/session.
  • Support multi-modal input (text only for MVP, extensible later).
📊
Non-Functional Requirements
  • System must support 2 billion daily active users (DAU).
  • End-to-end latency for token streaming must be under 100ms P99.
  • System availability must be 99.99% (four nines) across all components.
  • Support peak write QPS of approximately 925K messages/sec (derived from DAU and usage).
  • Ensure data durability with replication and retention of conversation history for 30 days.
❓
Clarifying Questions to Ask
  • What is the maximum context window size (token limit) per conversation?
  • Are conversations single-user or multi-user (group chat) scenarios?
  • What is the expected average message size in tokens or bytes?
  • Is token streaming required to start immediately after generation or can it be batched?
  • What is the retention period for conversation history and logs?
  • Are there any SLAs for model update frequency or versioning?
Out of Scope
✗ Training or fine-tuning of the LLM models.
✗ Multi-modal input processing (images, audio) beyond text.
✗ Advanced personalization or user profiling beyond session context.
✗ Offline batch analytics or model evaluation pipelines.
📌
Assumptions
  • 2 billion DAU
  • Each user sends 40 messages per day on average
  • Average message size is 100 bytes
  • Each message generates approximately 50 tokens in response
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 messages/sec
peak write qpsAssuming 3× peak factor: 925,926 × 3 ≈ 2.78M messages/sec
read qpsAssuming 1.5 reads per write (fetching context/history): 925,926 × 1.5 ≈ 1.39M reads/sec
💾
Storage Estimate
daily storage gb(2B × 40 × 100 bytes) / (1024³) ≈ 7.45 TB/day
retention days30
total storage gb7.45 TB × 30 ≈ 223.5 TB
with replication223.5 TB × 3 (replication factor) ≈ 670.5 TB
🖥️
Server Estimate
llm inference serversAssuming each GPU server handles 100 QPS, need 2.78M / 100 = 27,800 GPU servers
context storage nodesAssuming each Cassandra node handles 10K writes/sec, need 2.78M / 10K = 278 nodes
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |   Client App   |
          +-------+--------+
                  | WebSocket / REST
                  v
          +-------+--------+
          |  API Gateway   |
          +-------+--------+
                  |
          +-------+--------+
          |  Auth Service  |
          +-------+--------+
                  |
          +-------+--------+
          | Conversation   |
          |   Service     |
          +-------+--------+
                  |
          +-------+--------+
          | Context Store  |
          |  (Cassandra)   |
          +-------+--------+
                  |
          +-------+--------+
          |  Message Queue |
          |   (Kafka)      |
          +-------+--------+
                  |
          +-------+--------+
          | LLM Inference  |
          |   Service      |
          +-------+--------+
                  |
          +-------+--------+
          | Token Streamer |
          |   (Redis Pub)  |
          +----------------+
🧩
Components
API Gateway
Entry point for client requests, handles routing, rate limiting, and load balancing
Why thisEnvoy proxy supports HTTP/2 and WebSocket for token streaming with low latency and high throughput
Why notNginx lacks native HTTP/2 full-duplex streaming support and less flexible dynamic routing
Auth Service
Validates user tokens, enforces rate limits and quotas per user/session
Why thisOAuth 2.0 with JWT tokens provides stateless, scalable authentication with minimal latency
Why notSession-based auth requires sticky sessions and stateful storage, which do not scale to 2B DAU
Conversation Service
Manages conversation metadata, session state, and context window logic
Why thisStateless microservice with Redis for ephemeral session state enables fast context window updates and eviction
Why notMonolithic service would create bottlenecks and reduce deployment agility
Context Store
Stores persistent conversation history and context tokens for retrieval and replay
Why thisApache Cassandra provides linear write scalability, tunable consistency, and time-series data model ideal for conversation logs
Why notPostgreSQL requires complex manual sharding at 2.78M writes/sec; B-tree writes don't scale linearly
Message Queue
Buffers and orders user messages and model responses asynchronously for LLM inference
Why thisApache Kafka supports high throughput, partitioned logs with exactly-once semantics and consumer groups for parallelism
Why notRabbitMQ has lower throughput and less efficient partitioning for 2.78M msg/sec scale
LLM Inference Service
Runs large language model inference on GPU clusters, consumes messages from Kafka
Why thisNVIDIA Triton Inference Server supports multi-model serving, batching, and GPU utilization optimization
Why notCustom inference code lacks scalability and batching optimizations needed at 27,800 GPU servers
Token Streamer
Streams generated tokens to clients in real-time via WebSocket
Why thisRedis Pub/Sub provides sub-millisecond publish/subscribe latency and horizontal scaling with clustering
Why notKafka has higher latency and is not optimized for real-time token streaming to clients
🔄
Data Flow
1
Client App → API Gateway
Send user message via WebSocket or REST
Supports real-time token streaming and request multiplexing
2
API Gateway → Auth Service
Validate user token and enforce rate limits
Reject unauthorized or rate-exceeded requests early
3
Auth Service → Conversation Service
Fetch and update conversation context window
Use Redis for fast ephemeral context state management
4
Conversation Service → Context Store
Persist user message and updated context
Write to Cassandra for durable storage and history
5
Conversation Service → Message Queue
Publish message event for LLM inference
Kafka partitions enable parallel inference processing
6
LLM Inference Service → Token Streamer
Stream generated tokens as they are produced
Redis Pub/Sub pushes tokens to client WebSocket in real-time
7
LLM Inference Service → Context Store
Persist generated tokens and updated context
Ensures conversation history is complete and consistent
8
Token Streamer → Client App
Deliver tokens incrementally over WebSocket
Enables low-latency streaming UI for user experience

Practice

(1/5)
1. In a distributed database, at Step 4 the coordinator node receives conflicting writes from two partitions during a network split. What state does the coordinator enter immediately after Step 4?
easy
A. Coordinator marks the writes as conflicting and delays commit until partition heals
B. Coordinator commits both writes immediately to maintain availability
C. Coordinator rejects all writes to maintain consistency during partition
D. Coordinator forwards writes to a third node to resolve conflicts synchronously

Solution

  1. Step 1: Trace coordinator behavior at Step 4

    The coordinator detects conflicting writes from partitions and cannot commit both safely.

  2. Step 2: Understand conflict resolution

    It marks writes as conflicting and delays commit until partitions heal to avoid inconsistency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Coordinator defers conflicting writes until partition resolution [check]
Hint: Coordinator delays conflicting writes until partition heals [check]
Common Mistakes:
2. A real-time leaderboard system must handle millions of score updates per second with low latency and provide instant rank queries. Which component is best suited to store and update scores efficiently while maintaining sorted order?
easy
A. A NoSQL document store like MongoDB for flexible schema
B. A relational database with indexed score columns
C. A distributed log like Kafka for storing score events
D. Redis Sorted Sets to maintain scores and ranks in-memory

Solution

  1. Step 1: Identify data structure for sorted order and fast updates

    Redis Sorted Sets provide O(log N) insertion and rank queries with in-memory speed, ideal for real-time leaderboards.

  2. Step 2: Eliminate other options based on latency and ordering

    Relational DBs have higher write latency and complex locking; Kafka stores events but doesn't maintain sorted order; MongoDB lacks efficient sorted queries at scale.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Redis Sorted Sets maintain sorted scores with low latency [check]
Hint: Sorted Sets keep scores sorted with fast rank queries [check]
Common Mistakes:
3. A distributed system needs to generate unique IDs for millions of events per second without coordination delays. Which component is best suited to generate these IDs efficiently?
easy
A. A decentralized ID generator using timestamp and node ID bits (e.g., Snowflake)
B. A centralized relational database with auto-increment primary keys
C. A UUID v4 generator relying on random number generation
D. A distributed cache like Redis with INCR commands for ID generation

Solution

  1. Step 1: Identify coordination bottlenecks in centralized DB

    Auto-increment keys require coordination, causing latency and contention at high scale.

  2. Step 2: Evaluate decentralized timestamp-based ID generators

    Snowflake-style IDs combine timestamp and node ID, enabling scalable, ordered unique IDs without coordination.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Decentralized timestamp + node ID -> scalable unique ID generation [check]
Hint: Decentralized timestamp + node ID avoids coordination delays [check]
Common Mistakes:
4. At 10x scale, which component in a consistent hashing system with virtual nodes is most likely to fail first and why?
hard
A. The physical nodes, because they cannot handle increased key storage and request load
B. The token metadata service, due to increased complexity managing many virtual nodes and frequent updates
C. The network, because consistent hashing increases network chatter exponentially with scale
D. The client-side hash function, as hashing large key sets becomes computationally infeasible

Solution

  1. Step 1: Identify scaling bottlenecks

    Physical nodes scale horizontally; network and hash function scale well with optimizations.

  2. Step 2: Analyze token metadata service

    Managing many virtual nodes and frequent node changes increases metadata complexity and update frequency, causing bottlenecks.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Token metadata service complexity grows fastest, causing first failure [check]
Hint: Token metadata complexity grows fastest at scale [check]
Common Mistakes:
5. At 10x scale, which component in a time-series database system is most likely to fail first and why?
hard
A. The storage nodes, because disk I/O bandwidth saturates with high write volume
B. The query engine, because complex time-range queries become exponentially slower
C. The ingestion pipeline's partition coordinator, due to increased partition management overhead
D. The metadata service, as it cannot handle the increased number of time-series namespaces

Solution

  1. Step 1: Identify bottlenecks at 10x scale

    High write volume increases partition count and complexity of partition coordination.

  2. Step 2: Evaluate component scalability

    Query engine (A) slows but can be scaled horizontally. Storage nodes (C) face I/O limits but can add disks. Metadata service (D) scales better with caching. Partition coordinator (B) faces coordination overhead and metadata contention, causing first failure.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Partition coordinator overhead grows fastest at scale [check]
Hint: Partition coordination overhead grows fastest at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard