Bird
Raised Fist0

Design a ChatGPT-Like System - LLM Serving, Context Window, Token Streaming

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design a ChatGPT-Like System - LLM Serving, Context Window, Token Streaming
45 min · senior
senior

You are designing a ChatGPT-like conversational AI system for a startup with 50 million daily active users. The system must serve large language model (LLM) inference requests with a context window of up to 4,096 tokens, support real-time token streaming to clients, and maintain end-to-end encryption for user data. The system should achieve P99 latency under 200ms for token streaming and handle peak loads gracefully.

Constraints
  • End-to-end encryption of user prompts and responses in transit and at rest
  • Support context window management up to 4,096 tokens per conversation
  • Token-level streaming with P99 latency < 200ms
  • Scale to 50M DAU with peak concurrency of 1M simultaneous conversations
  • Graceful degradation under load without dropping active streams
📋
Guided Steps
1/5
Step 1
Designing the LLM Inference Serving Layer
How would you architect the LLM inference serving to handle 1M concurrent conversations with low latency?
💡 Hint
Consider GPU cluster orchestration and request routing based on conversation context.
Key Decision
Use NVIDIA Triton Inference Server with Kubernetes for autoscaling GPU pods; route requests by conversation_id to maintain context affinity and cache warm-up.
Interviewer Expectation
Mid-level: Names Triton and Kubernetes. Senior: Explains conversation_id affinity to reduce context loading overhead and GPU memory thrashing.
⚠️Common Wrong Answer
Using stateless HTTP load balancer without affinity causes repeated context reloads, increasing latency beyond 200ms P99.
Step 2
Context Window Management and Storage
How will you store and retrieve conversation history to support a 4,096-token context window efficiently?
💡 Hint
Think about append-optimized storage with fast range queries and TTL for inactive conversations.
Key Decision
Use Apache Cassandra with conversation_id as partition key for O(1) range scans of token sequences; TTL configured to expire inactive contexts after 7 days.
Interviewer Expectation
Mid-level: Mentions Cassandra for scalability. Senior: Details partition key choice and why sender_id or timestamp as partition keys cause cross-node queries and out-of-order retrieval.
⚠️Common Wrong Answer
Using relational DB with conversation_id as secondary index leads to high latency range scans and poor write throughput at scale.
Step 3
Implementing Token-Level Streaming to Clients
How would you implement real-time token streaming with P99 latency under 200ms to millions of clients?
💡 Hint
Consider WebSocket or HTTP/2 streaming and message brokers optimized for ordered delivery.
Key Decision
Use gRPC bidirectional streaming over HTTP/2 combined with Apache Kafka (partitioned by conversation_id) for ordered token delivery and backpressure handling.
Interviewer Expectation
Mid-level: Names WebSocket or gRPC streaming. Senior: Explains Kafka partitioning ensures token order and how backpressure prevents client overload.
⚠️Common Wrong Answer
Using HTTP polling causes excessive reconnects and latency spikes, collapsing under 50M DAU load.
Step 4
Ensuring End-to-End Encryption and Data Privacy
How will you guarantee end-to-end encryption for user data in transit and at rest without compromising performance?
💡 Hint
Think about TLS for transport and envelope encryption with key management for storage.
Key Decision
Use TLS 1.3 for all client-server communication; encrypt conversation data at rest with AWS KMS envelope encryption; keys scoped per user to isolate data access.
Interviewer Expectation
Mid-level: Mentions TLS and encryption at rest. Senior: Explains key scoping per user and how envelope encryption reduces latency compared to full disk encryption.
⚠️Common Wrong Answer
Relying only on TLS without encrypting data at rest risks data leakage if storage is compromised.
Step 5
Handling Load Spikes and Graceful Degradation
How would you design the system to gracefully degrade under sudden load spikes without dropping active token streams?
💡 Hint
Consider circuit breakers, rate limiting, and fallback mechanisms at the streaming layer.
Key Decision
Implement client-side rate limiting with token bucket algorithm; server-side circuit breakers in Triton pods; fallback to cached partial responses stored in Redis streams.
Interviewer Expectation
Mid-level: Mentions rate limiting. Senior: Explains token bucket parameters tuned for 1M concurrency and fallback cache reduces user-visible errors.
⚠️Common Wrong Answer
Dropping connections abruptly under load causes poor user experience and client retries exacerbate overload.
🔌
API Design Challenge
Sketch 3 core API endpoints for the ChatGPT-like system focusing on prompt submission, streaming token response, and context synchronization.
Hints
  • How does the client send a prompt and receive a streaming response?
  • How does the client synchronize conversation context across devices?
  • How does the client acknowledge receipt of tokens or handle partial failures?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of GPU inference serving with affinity to conversation_id to reduce context reload latency
  • Efficient context window storage with Cassandra partitioned by conversation_id for O(1) range scans
  • Token streaming implemented with gRPC streaming and Kafka partitioning for ordered delivery
  • End-to-end encryption with TLS and envelope encryption scoped per user
Good to Have - 30%
  • Fallback caching of partial responses in Redis Streams to improve availability
  • Rate limiting and circuit breakers to prevent overload and ensure graceful degradation
Bonus - 10%
  • Detailed API design covering prompt submission, streaming response, and context sync
  • Explanation of trade-offs between WebSocket and gRPC streaming protocols
Anti-patterns to Avoid
Stateless load balancer without affinity causes repeated context reloads, increasing latency beyond 200ms P99
Using relational databases for context storage leads to slow range queries and poor write throughput at scale
HTTP polling at 50M DAU causes 50 million reconnects per minute just to check for messages, collapsing the system under load
Encrypting only in transit without encrypting data at rest risks data leakage if storage is compromised
Dropping active streams abruptly under load causes poor user experience and retry storms that exacerbate overload
✅
Model Answer
▼ Reveal

The system uses NVIDIA Triton on Kubernetes for GPU-based LLM serving with conversation affinity, Apache Cassandra for context storage keyed by conversation_id, and gRPC streaming with Kafka for ordered token delivery. End-to-end encryption is ensured via TLS and AWS KMS envelope encryption. Load spikes are handled with rate limiting, circuit breakers, and Redis Streams fallback caching.

Key Decisions
  • Use conversation_id affinity in Triton to reduce context reload latency and GPU memory thrashing
  • Store conversation history in Cassandra partitioned by conversation_id for O(1) range scans and TTL-based expiration
  • Implement token streaming with gRPC bidirectional streaming and Kafka partitioning to guarantee token order and backpressure
  • Encrypt all client-server communication with TLS 1.3 and data at rest with AWS KMS envelope encryption scoped per user
  • Handle load spikes with client-side rate limiting, server-side circuit breakers, and fallback caching in Redis Streams to avoid dropping active streams
  • NVIDIA Triton Inference Server on Kubernetes for scalable GPU-based LLM serving with conversation_id affinity
  • Apache Cassandra with conversation_id partition key for efficient context window storage and retrieval
  • gRPC bidirectional streaming over HTTP/2 combined with Apache Kafka partitioned by conversation_id for ordered token streaming
  • TLS 1.3 for transport encryption and AWS KMS envelope encryption with per-user key scoping for data privacy
  • Redis Streams as fallback cache for partial token responses during overload or failures
  • Client-side token bucket rate limiting and server-side circuit breakers to handle load spikes gracefully

Practice

(1/5)
1. In the request flow, at step 5 the Intent Classifier component receives a user query. What happens if the confidence score is below the threshold at this step?
easy
A. The query is logged and a fallback notification is triggered asynchronously
B. The query is immediately sent to the human agent without further processing
C. The query is discarded and the user is asked to rephrase
D. The query is forwarded to the Fallback Handler component for alternative resolution

Solution

  1. Step 1: Trace Intent Classifier decision at Step 5

    When confidence is low, the system logs the query and triggers fallback notification asynchronously to avoid blocking.

  2. Step 2: Confirm fallback handling behavior

    Immediate human routing (A) blocks scalability. Forwarding directly (B) is plausible but synchronous. Discarding (C) harms UX. Logging and async notification (D) balances reliability and responsiveness.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Low confidence -> async fallback notification and logging [check]
Hint: Low confidence triggers async fallback notification [check]
Common Mistakes:
2. A web API receives sudden bursts of 10,000 requests per second but must process them at a steady rate of 1,000 requests per second to avoid backend overload. Which component best smooths this traffic to a steady rate without dropping requests?
easy
A. A leaky bucket queue that drains requests at a fixed rate
B. A token bucket that allows bursts up to a maximum token count
C. A fixed-size thread pool that rejects excess requests immediately
D. A simple FIFO queue without rate limiting

Solution

  1. Step 1: Identify smoothing mechanism

    The leaky bucket algorithm enforces a fixed output rate by draining requests steadily, smoothing bursts.

  2. Step 2: Eliminate alternatives

    A fixed-size thread pool that rejects excess requests immediately rejects excess requests immediately, not smoothing them. A token bucket that allows bursts up to a maximum token count (token bucket) allows bursts, not smoothing to steady rate. A simple FIFO queue without rate limiting has no rate control, so bursts pass through.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Leaky bucket drains at fixed rate, smoothing bursts [check]
Hint: Leaky bucket smooths bursts by fixed-rate draining [check]
Common Mistakes:
3. Given the need to quickly detect and rollback misconfigurations causing route leaks, which architecture is best suited for this scenario?
medium
A. Manual configuration changes with peer review before deployment
B. A centralized configuration management system with automated rollback triggers
C. A fully decentralized BGP control plane with no central oversight
D. A distributed consensus system that delays route announcements until quorum

Solution

  1. Step 1: Evaluate manual review trade-offs

    Manual peer review slows deployment but reduces risk of misconfiguration propagation.

  2. Step 2: Compare alternatives

    Decentralized control (A) lacks oversight; centralized rollback (B) is ideal but may introduce complexity; consensus delays (D) add latency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Manual review trades speed for safety in rollback scenarios [check]
Hint: Manual review trades speed for safety [check]
Common Mistakes:
4. If the sliding window counter's timestamp cleanup process fails, what is the most likely failure observed in rate limiting?
medium
A. Requests are allowed without limit because counters reset incorrectly
B. Fixed window counters reset too frequently causing bursty traffic spikes
C. Rate limiter blocks requests indefinitely due to stale timestamps inflating counts
D. Token bucket refills tokens too slowly causing underutilization

Solution

  1. Step 1: Understand sliding window cleanup role

    Cleanup removes timestamps outside the window; failure causes stale timestamps to accumulate.

  2. Step 2: Accumulated stale timestamps inflate request counts, causing the rate limiter to block legitimate requests indefinitely

    This leads to false positives in rate limiting, denying service unnecessarily.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Failed cleanup -> stale timestamps inflate counts -> requests blocked indefinitely [check]
Hint: No cleanup means stale timestamps block requests falsely [check]
Common Mistakes:
5. At 10x scale, which component is most likely to fail first due to the exact lookup requirement, and why?
hard
A. The backend database, because it cannot handle the increased exact lookup queries without Bloom filter filtering
B. The network layer, because increased traffic from exact lookups saturates bandwidth
C. The Bloom filter component, because its false positive rate grows linearly with scale, causing excessive backend queries
D. The cache layer, because it cannot store all keys for exact lookups at 10x scale

Solution

  1. Step 1: Understand Bloom filter scaling behavior

    Bloom filter false positive rate increases with scale unless filter size grows proportionally, causing more backend queries.

  2. Step 2: Analyze component failure likelihood

    Backend (A) fails eventually but Bloom filter inefficiency causes overload first. Network (C) and cache (D) are bottlenecks but secondary to Bloom filter false positive explosion.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Bloom filter false positives grow with scale, triggering backend overload first [check]
Hint: Bloom filter false positives grow with scale, overload backend first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard