Bird
Raised Fist0

Design a ChatGPT-Like System - LLM Serving, Context Window, Token Streaming

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/6
API Gateway

Receives user chat message

Accepts message over WebSocket from client

Protocol:WebSocket
BeforeUser has typed message but not sent
AfterMessage received and forwarded to Auth Service
Auth Service

Authenticates user

Validates user token and permissions

Protocol:REST API
BeforeMessage pending authentication
AfterUser authenticated; message allowed to proceed
Conversation Service

Fetches conversation context

Retrieves recent messages from Context Store

Protocol:Redis GET
BeforeNo context loaded for this conversation
AfterContext loaded and combined with new message
Message Queue

Queues request for LLM processing

Publishes combined input to Kafka topic

Protocol:Kafka Produce
BeforeNo pending LLM requests for this message
AfterRequest durably stored in queue for inference
LLM Inference Service

Consumes request and runs inference

Processes input and generates tokens

Protocol:Kafka Consume
BeforeRequest waiting in queue
AfterTokens generated and ready to stream
Token Streamer

Streams tokens to client

Sends tokens incrementally over gRPC streaming

Protocol:gRPC Stream
BeforeTokens generated but not sent
AfterClient receives tokens in real-time
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────┐
│ API Gateway   │
└───────┬───────┘
        │ WebSocket
        ▼
┌───────────────┐
│ Auth Service  │
└───────┬───────┘
        │ REST API
        ▼
┌───────────────────────┐
│ Conversation Service   │
│ (handles chat logic)  │
└───────┬───────────────┘
        │ Redis GET
        ▼
┌───────────────┐
│ Context Store │
└───────┬───────┘
        │ Kafka Produce
        ▼
┌───────────────┐
│ Message Queue │
└───────┬───────┘
        │ Kafka Consume
        ▼
┌───────────────────────┐
│ LLM Inference Service │
│ (runs model, generates│
│  tokens)              │
└───────┬───────────────┘
        │ gRPC Stream
        ▼
┌───────────────┐
│ Token Streamer│
│ (streams tokens│
│  to client)   │
└───────────────┘
Design Rationale

This architecture shows a vertical flow from client to token streaming for a ChatGPT-like system. The API Gateway accepts WebSocket messages, which are authenticated by the Auth Service. The Conversation Service fetches recent chat context from the Context Store cache to provide relevant input for the LLM. The combined input is queued in the Message Queue (Kafka) for asynchronous processing. The LLM Inference Service consumes requests from the queue, runs the model to generate tokens, and passes them to the Token Streamer. The Token Streamer uses gRPC streaming to send tokens back to the client incrementally, enabling real-time response. This design decouples components for scalability and supports context-aware, streaming LLM responses.

Primary Focus

✓

Practice

(1/5)
1. A system design interview requires you to explain how you handle conflicting requirements between latency and consistency. Which component or approach best helps you present this trade-off clearly to the interviewer?
easy
A. Use a decision matrix comparing latency and consistency impacts per component
B. Avoid discussing trade-offs and emphasize only the chosen technology
C. Describe only the eventual consistency model without latency details
D. Focus solely on the fastest technology without discussing consistency implications

Solution

  1. Step 1: Identify the need to clarify trade-offs

    Presenting trade-offs requires explicit comparison of conflicting requirements, such as latency vs consistency.

  2. Step 2: Use a decision matrix for clarity

    A decision matrix systematically compares options, showing pros and cons clearly, which helps interviewers understand your reasoning.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Decision matrix clarifies trade-offs explicitly [check]
Hint: Use decision matrices to clarify conflicting trade-offs [check]
Common Mistakes:
2. For real-time analytics requiring low-latency dashboards and complex ad-hoc queries, which architecture is best suited?
medium
A. Use only Kafka Streams for both aggregation and serving queries
B. Use Flink to write directly to a NoSQL key-value store for dashboards
C. Use batch ETL jobs to load data into a relational database for dashboards
D. Use Flink for streaming aggregation and an OLAP store for dashboard queries

Solution

  1. Step 1: Evaluate streaming aggregation and query needs

    Flink excels at complex streaming aggregations; OLAP stores support fast ad-hoc queries and low-latency dashboards.

  2. Step 2: Eliminate other options

    Kafka Streams lacks complex query capabilities; batch ETL adds latency unsuitable for real-time; NoSQL key-value stores lack rich query support for dashboards.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Flink + OLAP store balances streaming and query complexity [check]
Hint: Flink streams, OLAP queries for real-time + ad-hoc [check]
Common Mistakes:
3. Without the DAU Estimator component, what exactly breaks in the system's capacity planning process?
medium
A. Bandwidth estimation fails because DAU directly affects total data transfer calculations
B. The system cannot estimate peak QPS, causing under-provisioning of servers
C. Storage estimation becomes inaccurate because DAU is a key multiplier for daily data volume
D. User authentication fails because DAU tracks active sessions

Solution

  1. Step 1: Identify DAU Estimator role

    DAU Estimator provides daily active user count critical for bandwidth calculations as bandwidth depends on total active users.

  2. Step 2: Eliminate other options

    The system cannot estimate peak QPS, causing under-provisioning of servers is incorrect because peak QPS depends on instantaneous requests, not just DAU; Storage estimation becomes inaccurate because DAU is a key multiplier for daily data volume is partially true but storage depends more on total objects, not DAU alone; User authentication fails because DAU tracks active sessions is unrelated to authentication.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Bandwidth depends directly on DAU for total data transfer [check]
Hint: Bandwidth depends directly on DAU count [check]
Common Mistakes:
4. If the health check component is removed from a load balancer using Least Connections, what failure occurs?
medium
A. The load balancer continues to distribute requests evenly, but some servers may be down causing request failures
B. The load balancer switches to Round Robin automatically, losing connection count accuracy
C. The load balancer caches server states incorrectly, causing stale routing decisions
D. The load balancer sends requests to unhealthy servers, causing increased latency and failed connections without fallback

Solution

  1. Step 1: Role of health checks in Least Connections

    Health checks detect server availability; without them, the balancer cannot exclude unhealthy servers.

  2. Step 2: Behavior without health checks

    Least Connections relies on accurate connection counts and server health; without health checks, it cannot trust connection data and typically falls back to Round Robin.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Without health checks, Least Connections falls back to Round Robin losing load awareness [check]
Hint: No health checks -> Least Connections falls back to Round Robin [check]
Common Mistakes:
5. For a high-traffic API, which approach better captures user experience degradation under load: reporting average latency or reporting P95 latency, and why?
medium
A. P95 latency, because it highlights the slowest 5% of requests affecting users
B. Average latency, because it is easier to compute in real-time
C. Average latency, because it summarizes all requests evenly
D. P95 latency, because it ignores outliers and focuses on median performance

Solution

  1. Step 1: Understand average latency limitations

    Average latency can be skewed by many fast requests and hide slow request impact, thus missing user experience degradation.

  2. Step 2: Why P95 latency is better

    P95 latency captures the slowest 5% of requests, which often cause user frustration and indicate performance issues under load.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    P95 latency reveals slow requests impacting users better than average [check]
Hint: P95 reveals slow requests; average hides tail latency [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard