Bird
Raised Fist0

Design an LLM-Powered Customer Support Bot - Intent Classification, Fallback

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/7
API Gateway (Nginx + Envoy)

Receives customer message via WebSocket

Accepts incoming WebSocket message from client

Protocol:WebSocket
BeforeCustomer message is typed on client device, not yet sent
AfterMessage received and validated by API Gateway
Message Ingestion (Apache Kafka)

Publishes message to Kafka topic

Message durably stored in Kafka partition keyed by conversation ID

Protocol:Kafka
BeforeMessage is held transiently in API Gateway memory
AfterMessage durably committed to Kafka, ready for downstream processing
Intent Classification Service (LLM Inference Cluster with NVIDIA Triton Inference Server)

Consumes message and classifies intent

LLM model infers user intent and confidence score

Protocol:gRPC
BeforeMessage is available in Kafka topic, unprocessed
AfterIntent classification result produced with confidence metadata
Conversation Store (Apache Cassandra)

Stores message and classification result

Conversation history updated with new message and intent

Protocol:CQL
BeforeConversation record lacks latest message and intent
AfterConversation record updated with message and intent classification
Response Generation & Fallback Handler

Generates bot response or triggers fallback

If intent confidence low, escalates to human agent

Protocol:REST
BeforeIntent classification result stored, no response generated
AfterBot response sent or escalation initiated
Human Agent Escalation Service

Receives escalation request

Routes conversation to human agent interface

Protocol:REST
BeforeFallback triggered, no human agent assigned
AfterConversation assigned to human agent for manual handling
Monitoring & Analytics (Elasticsearch + Kibana)

Collects logs and metrics from all components

Aggregates telemetry for observability and alerting

Protocol:HTTP
BeforeLogs and metrics scattered across components
AfterCentralized monitoring data available for analysis
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────────────────┐
│ API Gateway                   │
│ (Nginx + Envoy)               │
└───────────────┬───────────────┘
                │ WebSocket
                ▼
┌───────────────────────────────┐
│ Message Ingestion             │
│ (Apache Kafka)                │
└───────────────┬───────────────┘
                │ Kafka
                ▼
┌─────────────────────────────────────────┐
│ Intent Classification Service            │
│ (LLM Inference Cluster with NVIDIA Triton)│
└───────────────┬─────────────────────────┘
                │ gRPC
                ▼
┌───────────────────────────────┐
│ Conversation Store            │
│ (Apache Cassandra)            │
└───────────────┬───────────────┘
                │ CQL
                ▼
┌───────────────────────────────┐
│ Response Generation & Fallback│
│ Handler                      │
└───────────────┬───────────────┘
                │ REST
                ▼
┌───────────────────────────────┐
│ Human Agent Escalation Service│
└───────────────┬───────────────┘
                │ REST
                ▼
┌───────────────────────────────┐
│ Monitoring & Analytics        │
│ (Elasticsearch + Kibana)      │
└───────────────────────────────┘
Design Rationale

This architecture shows a vertically stacked pipeline for an LLM-powered customer support bot. The API Gateway receives client messages over WebSocket and publishes them to Kafka for durable ingestion. The Intent Classification Service consumes messages from Kafka and uses NVIDIA Triton to run LLM inference, producing intent labels with confidence scores. Conversation Store (Cassandra) persists the message and classification results for stateful context. The Response Generation & Fallback Handler uses this data to generate bot replies or escalate low-confidence queries to human agents via the Escalation Service. Monitoring & Analytics collects logs and metrics from all components for observability. This design ensures durability, scalability, and fallback safety by separating concerns and using asynchronous messaging.

Primary Focus

✓

Practice

(1/5)
1. A recommendation system must handle 10,000 user rating updates per second and asynchronously update user embeddings without blocking user requests. Which component best handles this decoupling?
easy
A. Use a batch job that runs once per hour to update embeddings
B. Use an in-memory cache like Redis to store embeddings and update synchronously
C. Use a message queue like Kafka to buffer rating events for asynchronous processing
D. Use a relational database with synchronous writes for embedding updates

Solution

  1. Step 1: Identify decoupling need

    High write rate (10,000/sec) requires asynchronous processing to avoid blocking user requests.

  2. Step 2: Evaluate components

    Relational DB synchronous writes (A) block requests; Redis (B) lacks durability guarantees for event buffering; batch jobs (C) are too infrequent for real-time updates. Kafka (D) buffers events reliably and supports asynchronous consumption.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Kafka buffers high write throughput asynchronously [check]
Hint: Buffer high writes asynchronously with Kafka [check]
Common Mistakes:
2. A video calling system must establish peer-to-peer connections between clients behind NATs and firewalls. Which component is primarily responsible for enabling this connectivity?
easy
A. STUN/TURN servers that assist in NAT traversal and relay when direct connection fails
B. Media Server that relays all media streams to bypass NAT restrictions
C. Signaling Server that encodes and decodes media streams for transmission
D. WebRTC client library that manages user authentication and session persistence

Solution

  1. Step 1: Identify NAT traversal role

    STUN/TURN servers help clients discover public IPs and relay media if direct peer-to-peer fails.

  2. Step 2: Eliminate other components

    Media Server relays media but is not primarily for NAT traversal; Signaling Server handles session negotiation, not media encoding; WebRTC client manages connection but relies on STUN/TURN for NAT traversal.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    STUN/TURN enable NAT traversal, essential for peer connectivity [check]
Hint: STUN/TURN servers enable NAT traversal and relay fallback [check]
Common Mistakes:
3. A social media platform needs to store user posts with high write throughput and ensure efficient retrieval by user and timestamp. Which component best handles this workload?
easy
A. A wide-column store like Cassandra with partition keys on user ID
B. A relational database with normalized tables and joins
C. A document store like MongoDB with embedded user posts
D. An in-memory cache like Redis without persistence

Solution

  1. Step 1: Identify write throughput and query pattern

    The system requires high write throughput and efficient retrieval by user and timestamp, which suits wide-column stores that partition data by user ID.

  2. Step 2: Evaluate options against constraints

    Relational DBs struggle with high write throughput and joins; document stores lack efficient range queries by timestamp; Redis lacks durability and is not ideal for large persistent datasets.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Wide-column store partitions by user ID -> high write throughput and efficient user queries [check]
Hint: Partition by user ID for high write and read throughput [check]
Common Mistakes:
4. For a system requiring globally unique, roughly time-ordered IDs across multiple data centers, which approach best balances latency and uniqueness guarantees?
medium
A. Use a single centralized ID generator with synchronous replication to all data centers
B. Use decentralized Snowflake-like generators with loosely synchronized clocks and node IDs
C. Generate UUID v4 randomly at each data center without coordination
D. Use database auto-increment IDs partitioned by data center and combine with data center ID

Solution

  1. Step 1: Evaluate centralized generator with sync replication

    This causes high latency and single point of failure, unsuitable for geo-distributed low-latency needs.

  2. Step 2: Assess decentralized Snowflake approach

    Decentralized generators with node IDs and loosely synchronized clocks provide low latency and uniqueness with approximate ordering.

  3. Step 3: Consider UUID v4 and DB auto-increment

    UUID v4 lacks ordering; DB auto-increment with partitioning risks collisions and coordination overhead.

  4. Final Answer:

    Option B -> Option B

  5. Quick Check:

    Decentralized Snowflake -> low latency + uniqueness + approximate ordering [check]
Hint: Decentralized with node ID and clock sync balances latency and uniqueness [check]
Common Mistakes:
5. At 10x scale, which component in the News Feed system is most likely to fail first and why?
hard
A. The database, because it cannot handle the increased write volume for posts
B. The Delivery Workers, because they cannot process the exponentially larger fan-out queues fast enough
C. The cache layer, because cache invalidation traffic grows superlinearly with user count
D. The fan-out message queue (Kafka), because partition count limits prevent scaling beyond a threshold

Solution

  1. Step 1: Analyze bottlenecks at 10x scale

    Database (A) can be sharded; cache (C) scales horizontally; Kafka (D) can add partitions but has limits.

  2. Step 2: Delivery Workers face exponential fan-out growth

    Fan-out queues grow with followers; Delivery Workers must push updates to millions more followers, causing processing lag and backlog.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Delivery Workers bottleneck due to fan-out processing load [check]
Hint: Delivery Workers bottleneck on fan-out processing at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard