Bird
Raised Fist0

Design an LLM-Powered Customer Support Bot - Intent Classification, Fallback

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design an LLM-Powered Customer Support Bot - Intent Classification, Fallback
Design a scalable, low-latency customer support chatbot using LLMs for intent classification and fallback handling
hardAI and ML System DesignFreq: medium
Prerequisites:System Design FundamentalsCAP TheoremDistributed SystemsMachine Learning Model Serving
⚙️
Functional Requirements
  • Receive and process customer messages in real-time
  • Classify user intent using a fine-tuned LLM model
  • Provide automated responses for recognized intents
  • Fallback to human agent or simpler rule-based bot if intent confidence is low
  • Log conversations and intents for analytics and model retraining
  • Support multi-channel input (web chat, mobile app, social media)
📊
Non-Functional Requirements
  • Handle 2 billion daily active users (DAU)
  • 99.99% system availability with failover
  • End-to-end latency under 100ms P99 for intent classification and response generation
  • Support 925K messages per second peak write throughput
  • Data retention for 90 days with GDPR compliance
❓
Clarifying Questions to Ask
  • What is the expected average message size and frequency per user?
  • What is the acceptable latency budget for fallback to human agents?
  • Are there any constraints on model size or hosting environment?
  • What channels (web, mobile, social) must be supported initially?
  • Is the fallback mechanism synchronous or asynchronous?
  • What are the security and privacy requirements for user data?
Out of Scope
✗ Designing the human agent interface
✗ Building the LLM training pipeline from scratch
✗ Multi-language translation support
✗ Voice-to-text transcription
📌
Assumptions
  • 2 billion DAU
  • 40 messages per user per day on average
  • 100 bytes average message size
  • Retention period of 90 days
  • Peak traffic is 3x average
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 msg/sec
peak write qps925,926 × 3 ≈ 2,777,778 msg/sec
read qpsAssuming 2 reads per write for context and fallback, 2 × 2,777,778 ≈ 5,555,556 read/sec
💾
Storage Estimate
daily storage gb(2,777,778 msg/sec × 100 bytes) × 86400 sec / (1024^3) ≈ 22.4 TB/day
retention storage gb22.4 TB/day × 90 days ≈ 2.02 PB
with replication2.02 PB × 3 (replication factor) ≈ 6.06 PB total
🖥️
Server Estimate
llm inference serversAssuming 50ms per inference, 2,777,778 peak QPS × 0.05s = 138,889 concurrent inferences; with 4 GPUs/server handling 50 concurrent inferences, need ~2,778 GPU servers
message ingestion serversKafka brokers and API servers to handle 2.7M msg/sec; with each Kafka broker handling ~100K msg/sec, need ~28 Kafka brokers; API servers with 1000 RPS each, need ~2,778 servers
🏗️
Architecture Diagram
Architecture Diagram
User
  |
API Gateway (Nginx + Envoy)
  |
Message Ingestion (Kafka Cluster)
  |
Intent Classification Service (LLM Inference Cluster)
  |
Response Generation & Fallback Handler
  |
Conversation Store (Cassandra Cluster)
  |
Monitoring & Analytics (Elasticsearch + Kibana)
  |
Human Agent Escalation Service
🧩
Components
API Gateway (Nginx + Envoy)
Handles incoming user requests, load balances, and performs TLS termination
Why thisNginx + Envoy provide high throughput, low latency, and advanced routing with observability
Why notAWS ALB or GCP Load Balancer are managed but less customizable and add latency at scale
Message Ingestion (Apache Kafka)
Durable, scalable message queue to buffer incoming messages and decouple ingestion from processing
Why thisKafka supports 10M+ messages/sec per cluster with partitioning and replication for fault tolerance
Why notRabbitMQ lacks horizontal scalability and throughput at 2.7M msg/sec; SQS has higher latency
Intent Classification Service (LLM Inference Cluster with NVIDIA Triton Inference Server)
Runs fine-tuned LLM models to classify user intents with low latency
Why thisTriton supports multi-model serving, GPU acceleration, batching, and dynamic batching for throughput and latency
Why notTensorFlow Serving lacks optimized GPU batching for large LLMs; custom REST APIs lack scalability
Conversation Store (Apache Cassandra)
Stores conversation history, intents, and fallback flags with high write throughput and availability
Why thisCassandra offers linear write scalability, tunable consistency, and multi-datacenter replication for 2.7M writes/sec
Why notPostgreSQL requires complex manual sharding at this scale; MongoDB has weaker write throughput guarantees
Response Generation & Fallback Handler
Generates automated responses or triggers fallback to human agents based on intent confidence
Why thisCustom microservice with business logic to integrate LLM output and fallback policies, scalable horizontally
Why notRule-based bots like Dialogflow lack flexibility and accuracy for fallback decisions at scale
Monitoring & Analytics (Elasticsearch + Kibana)
Aggregates logs, metrics, and conversation data for real-time monitoring and offline analytics
Why thisElasticsearch supports fast full-text search and Kibana provides rich dashboards for operational insights
Why notSplunk is costly at PB scale; Prometheus lacks full-text search capabilities
Human Agent Escalation Service
Routes fallback conversations to human agents with context and conversation history
Why thisCustom service integrated with CRM and chat platforms, supports asynchronous escalation and retries
Why notDirect LLM fallback without context risks poor user experience; third-party escalation tools lack integration
🔄
Data Flow
1
User → API Gateway
User sends message via chat interface
API Gateway handles TLS and load balancing
2
API Gateway → Kafka Cluster
Message published to Kafka topic for ingestion
Decouples ingestion from processing, buffers spikes
3
Kafka Cluster → Intent Classification Service
Consumes message, runs LLM inference to classify intent
Uses GPU-accelerated Triton servers with batching
4
Intent Classification Service → Response Generation & Fallback Handler
Sends intent and confidence score
Handler decides automated response or fallback
5
Response Generation & Fallback Handler → Conversation Store
Logs conversation, intent, and fallback flags
Ensures data persistence for analytics and retraining
6
Response Generation & Fallback Handler → User
Sends automated response if confidence is high
Low latency response generation
7
Response Generation & Fallback Handler → Human Agent Escalation Service
Triggers fallback if intent confidence is low
Routes conversation with context for human handling
8
Conversation Store → Monitoring & Analytics
Streams logs and metrics for monitoring
Supports real-time and offline analytics

Practice

(1/5)
1. A deployment system must ensure zero downtime during releases by switching traffic instantly between two identical environments. Which deployment strategy best fits this requirement?
easy
A. Rolling deployment updating servers one by one
B. Canary deployment with gradual traffic shifting to new version
C. Feature flags toggling new features on the same environment
D. Blue-Green deployment with instant switch of load balancer target

Solution

  1. Step 1: Identify zero downtime requirement

    Blue-Green deployment uses two identical environments and switches traffic instantly, ensuring zero downtime.

  2. Step 2: Eliminate other options

    Canary is gradual, not instant; Feature flags don't switch entire environments; Rolling updates cause partial downtime.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Blue-Green switches environments instantly for zero downtime [check]
Hint: Instant switch between identical environments is Blue-Green [OK]
Common Mistakes:
2. If the reconciliation service that compares debit and credit entries is temporarily down, what is the immediate failure mode observed in the digital wallet system?
medium
A. The system marks transactions as pending and delays final confirmation to users until reconciliation completes
B. New transactions are blocked until reconciliation service recovers to prevent ledger divergence
C. Transactions continue normally but inconsistencies accumulate silently until reconciliation resumes
D. Debit entries are committed immediately but credit entries are delayed, causing temporary imbalance

Solution

  1. Step 1: Understand reconciliation service role

    It ensures debit and credit entries match before confirming transactions to users.

  2. Step 2: Analyze failure impact

    Blocking all transactions (B) reduces availability unnecessarily; silent accumulation (A) risks unnoticed errors; committing debit only (D) breaks double-entry. Marking transactions pending (C) delays confirmation but preserves consistency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Pending status delays confirmation -> preserves ledger integrity [check]
Hint: Pending transactions delay confirmation, not silent failure [OK]
Common Mistakes:
3. For a distributed message queue requiring exactly-once delivery semantics, which architecture choice best balances complexity and correctness?
medium
A. Use at-least-once delivery with idempotent consumers to handle duplicates
B. Implement transactional writes in the broker with two-phase commit across producers and consumers
C. Use at-most-once delivery to avoid duplicates but risk message loss
D. Use a best-effort fire-and-forget model for maximum throughput

Solution

  1. Step 1: Analyze delivery semantics

    Exactly-once requires atomicity across producer and consumer; at-least-once (A) can cause duplicates; at-most-once (C) risks loss; fire-and-forget (D) ignores reliability.

  2. Step 2: Evaluate architecture complexity

    Two-phase commit (B) ensures atomic commit across components, enabling exactly-once semantics despite complexity; idempotent consumers (A) reduce duplicates but don't guarantee exactly-once.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Two-phase commit -> atomic exactly-once delivery despite complexity [check]
Hint: Two-phase commit enables atomic exactly-once delivery [check]
Common Mistakes:
4. For a video calling system targeting mobile clients with intermittent connectivity, which architecture best balances latency and reliability for media delivery?
medium
A. Use a multipoint control unit (MCU) media server that mixes all streams into one
B. Use a fully peer-to-peer WebRTC connection with no media server relay
C. Use a selective forwarding unit (SFU) media server to relay streams only when needed
D. Use HTTP Live Streaming (HLS) for media delivery to ensure reliability

Solution

  1. Step 1: Evaluate architectures for mobile intermittent connectivity

    Pure peer-to-peer (A) suffers from connectivity issues; MCU (C) adds latency and CPU overhead; HLS (D) is high latency and not real-time.

  2. Step 2: SFU (B) relays streams selectively, reducing latency and handling network variability well.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    SFU balances latency and reliability for mobile intermittent networks [check]
Hint: SFU relays selectively, best for mobile intermittent connectivity [check]
Common Mistakes:
5. At 10x scale, which component in Netflix's adaptive streaming pipeline is most likely to fail first and why?
hard
A. Personalization service because recommendation model retraining becomes too slow
B. Origin servers because storage I/O bandwidth saturates under high request volume
C. CDN edge servers because WebSocket connections for adaptive bitrate are stateful and cannot be easily sharded
D. Client devices because decoding multiple bitrate streams consumes excessive CPU

Solution

  1. Step 1: Identify stateful bottlenecks at scale

    Personalization retraining (Personalization service because recommendation model retraining becomes too slow) can be scaled horizontally with distributed training. Origin storage (Origin servers because storage I/O bandwidth saturates under high request volume) can be scaled with sharding and CDN caching. Client devices (Client devices because decoding multiple bitrate streams consumes excessive CPU) are outside server control.

  2. Step 2: Analyze CDN edge server scaling challenges

    WebSocket connections for adaptive streaming are stateful and bound to specific edge servers, limiting horizontal scaling and resharding, causing edge server overload at 10x scale.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Stateful WebSocket limits CDN edge server scalability [check]
Hint: Stateful WebSocket limits CDN edge scaling first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard