Bird
Raised Fist0

Design an LLM-Powered Customer Support Bot - Intent Classification, Fallback

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design an LLM-Powered Customer Support Bot - Intent Classification, Fallback
45 min · senior
senior

You are building a customer support chatbot for a startup with 50 million daily active users. The bot must classify user intents in real-time, handle fallback gracefully when intent classification confidence is low, and escalate to human agents if needed. The system must comply with end-to-end encryption requirements, maintain a P99 latency under 200ms for intent classification, and support 10,000 concurrent conversations per second. The bot should also support offline message delivery and ensure message ordering per conversation.

Constraints
  • End-to-end encryption of user messages and bot responses is mandatory.
  • P99 latency for intent classification must be under 200ms to ensure real-time interaction.
  • System must handle 10,000 concurrent conversations per second with ordered message delivery.
  • Fallback to human agents must be seamless with minimal delay.
  • Offline message delivery support for users with intermittent connectivity.
📋
Guided Steps
1/5
Step 1
Message Ingestion and Encryption
How will you securely ingest user messages ensuring end-to-end encryption and low latency?
💡 Hint
Consider client-side encryption and a messaging broker that supports high throughput and ordering.
Key Decision
Use AWS Kinesis Data Streams with client-side encryption libraries (e.g., libsodium) to ingest encrypted messages, ensuring ordered partitioning by conversation_id for message ordering and scalability.
Interviewer Expectation
Mid: Names Kinesis or Kafka for ingestion. Senior: Explains client-side encryption to meet end-to-end encryption and why conversation_id partition key ensures ordered processing per conversation.
⚠️Common Wrong Answer
Using HTTPS with server-side encryption only - breaks end-to-end encryption guarantees and exposes plaintext at ingestion.
Step 2
Intent Classification Architecture
How will you design the intent classification component to meet P99 < 200ms latency at scale?
💡 Hint
Consider model serving frameworks optimized for low latency and horizontal scaling with caching.
Key Decision
Deploy Hugging Face Transformers models using NVIDIA Triton Inference Server with GPU acceleration and Redis as a caching layer for repeated queries to meet P99 latency under 200ms.
Interviewer Expectation
Mid: Mentions model serving with GPUs. Senior: Details Triton’s batching and concurrency controls, Redis caching for hot intents, and fallback when cache misses occur.
⚠️Common Wrong Answer
Serving models on CPU-only servers without caching - leads to high latency and inability to meet P99 SLA.
Step 3
Fallback Handling and Human Escalation
How will you handle fallback when intent confidence is low and escalate to human agents efficiently?
💡 Hint
Think about event-driven workflows and integration with human agent queues.
Key Decision
Use Apache Kafka with conversation_id as partition key to stream fallback events to a human agent queue system like Zendesk or Salesforce Service Cloud via Kafka Connect, ensuring ordered escalation and traceability.
Interviewer Expectation
Mid: Names Kafka for event streaming. Senior: Explains partitioning by conversation_id to maintain order and exactly-once delivery semantics with Kafka Connect integration.
⚠️Common Wrong Answer
Polling a database for fallback flags - causes high latency and inconsistent escalation order.
Step 4
Offline Message Delivery and Sync
How will you support offline message delivery and ensure clients sync messages correctly upon reconnect?
💡 Hint
Consider a storage system optimized for time-series or conversation logs with efficient range queries.
Key Decision
Use Cassandra with partition key as conversation_id and clustering key as message_timestamp to store encrypted messages, enabling efficient cursor-based sync for offline clients.
Interviewer Expectation
Mid: Suggests a NoSQL store. Senior: Explains Cassandra’s LSM-tree write scalability and range queries for cursor-based sync, rejecting relational DBs due to write amplification.
⚠️Common Wrong Answer
Using relational databases for message storage - leads to write bottlenecks and poor offline sync performance.
Step 5
API Design for Client Interaction
Design 3 core API endpoints for sending messages, receiving intent classification results, and escalating fallback cases.
💡 Hint
Think about REST or gRPC endpoints with asynchronous patterns and secure authentication.
Key Decision
Expose RESTful endpoints: POST /messages (encrypted user message ingestion), GET /intents?conversation_id&cursor (poll classified intents), POST /fallback/escalate (trigger human escalation), all secured with OAuth 2.0 and TLS.
Interviewer Expectation
Mid: Defines basic endpoints. Senior: Details asynchronous polling with cursors, secure auth, and idempotency keys to handle retries.
⚠️Common Wrong Answer
Using synchronous blocking calls for intent results - causes client-side latency spikes and poor UX.
🔌
API Design Challenge
Sketch 3 core API endpoints for the customer support bot system that handle message sending, intent classification results retrieval, and fallback escalation.
Hints
  • How does the client send user messages securely to the bot?
  • How does the client receive classified intents and bot responses asynchronously?
  • How does the system escalate fallback cases to human agents and acknowledge resolution?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of client-side encryption to guarantee end-to-end encryption.
  • Partitioning by conversation_id in streaming and storage layers to maintain message order.
  • Low latency model serving with GPU acceleration and caching to meet P99 < 200ms.
  • Event-driven fallback escalation integrated with human agent platforms.
  • Offline message delivery support with efficient cursor-based sync.
Good to Have - 30%
  • OAuth 2.0 and TLS for API security.
  • Idempotency keys in APIs to handle retries gracefully.
  • Use of Kafka Connect for exactly-once integration with external systems.
Bonus - 10%
  • Detailed failure mode analysis for fallback escalation delays.
  • Discussion of scaling strategies for model serving under burst traffic.
Anti-patterns to Avoid
Server-side encryption only without client-side encryption - exposes plaintext at ingestion violating security constraints.
Polling a database for fallback flags - causes high latency and inconsistent escalation order.
Using relational databases for message storage - leads to write bottlenecks and poor offline sync performance.
Using synchronous blocking calls for intent results - causes client-side latency spikes and poor UX.
✅
Model Answer
▼ Reveal

This design outlines a secure, scalable, and low-latency customer support chatbot system that uses client-side encryption, streaming platforms, GPU-accelerated model serving, and event-driven fallback escalation to human agents. It supports offline message delivery with ordered message sync and exposes secure asynchronous APIs.

Key Decisions
  • Client-side encryption using libsodium to ensure end-to-end encryption and prevent plaintext exposure.
  • Partitioning streams and storage by conversation_id to maintain message order and scalability.
  • Using GPU-accelerated Triton Inference Server and Redis caching to meet P99 latency under 200ms.
  • Event-driven fallback escalation via Kafka with exactly-once semantics using Kafka Connect.
  • Offline message sync using Cassandra’s efficient range queries and LSM-tree write scalability.
  • API design with asynchronous endpoints, OAuth 2.0 authentication, TLS encryption, and idempotency keys for reliability.
  • AWS Kinesis Data Streams with client-side encryption ensures ordered, secure message ingestion at 10K conversations/sec.
  • NVIDIA Triton Inference Server accelerates Hugging Face Transformer models to meet P99 latency < 200ms.
  • Redis caching layer reduces repeated inference calls, improving throughput and latency consistency.
  • Apache Kafka with conversation_id partition key guarantees ordered fallback event streaming and integration with human agent platforms.
  • Cassandra with conversation_id partition key and message_timestamp clustering key supports efficient offline message sync and high write throughput.
  • RESTful APIs secured with OAuth 2.0 and TLS provide secure, asynchronous client interaction with idempotency for reliability.
  • Client-side encryption libraries (e.g., libsodium) maintain end-to-end encryption, preventing plaintext exposure at any server.
  • Kafka Connect enables seamless integration between fallback event streams and third-party human agent systems with exactly-once semantics.

Practice

(1/5)
1. A system must guarantee that user-uploaded files are stored durably and retrievable even after server crashes. Which component best addresses this non-functional requirement?
easy
A. In-memory cache like Redis for fast access
B. WebSocket server for real-time upload progress
C. Load balancer to distribute upload requests
D. Distributed persistent storage like Amazon S3

Solution

  1. Step 1: Identify durability requirement

    Durability requires persistent storage that survives crashes, ruling out volatile caches.

  2. Step 2: Match component to durability

    Amazon S3 provides distributed, durable storage with replication and fault tolerance, fitting the requirement.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Persistent storage ensures data survives crashes [check]
Hint: Durability needs persistent storage, not cache or load balancer [check]
Common Mistakes:
2. In a consistent hashing system, at Step 4 the request router receives a key and queries the hash ring to find the responsible shard. What happens if the shard mapped by the hash ring is currently down?
easy
A. The router retries the same shard until it recovers before forwarding the request
B. The router uses the next clockwise shard on the hash ring as a fallback for the key
C. The router returns an error immediately to the client indicating shard unavailability
D. The router hashes the key again with a different hash function to find another shard

Solution

  1. Step 1: Trace request router behavior on shard failure

    Retrying the same shard (The router retries the same shard until it recovers before forwarding the request) blocks requests; returning error (The router returns an error immediately to the client indicating shard unavailability) reduces availability; next clockwise shard (The router uses the next clockwise shard on the hash ring as a fallback for the key) is correct only if the ring is intact and shard failure is handled by virtual nodes.

  2. Step 2: Understand fallback hashing

    Rehashing with a different function (The router hashes the key again with a different hash function to find another shard) is a common fallback to find an alternate shard when the primary is down, ensuring availability without blocking or errors.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Fallback hashing finds alternate shard on failure [check]
Hint: Fallback hashing finds alternate shard if primary is down [OK]
Common Mistakes:
3. For running A/B tests on ranking models with minimal latency impact, which architecture is best suited?
medium
A. Use a feature flag system to route a percentage of requests to different model versions asynchronously
B. Run A/B tests by duplicating traffic at the load balancer and scoring both models synchronously
C. Deploy separate model servers and switch traffic at the DNS level for A/B testing
D. Run A/B tests by embedding model selection logic inside the Feature Store

Solution

  1. Step 1: Analyze A/B testing constraints

    Minimal latency impact and controlled traffic routing are key.

  2. Step 2: Evaluate architectures

    Duplicating traffic synchronously (A) doubles latency; DNS switching (C) is slow and coarse-grained; embedding model logic in Feature Store (D) mixes concerns and complicates feature retrieval. Feature flag routing (B) allows asynchronous, fine-grained control with minimal latency impact.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Feature flag routing -> low latency, flexible A/B testing [check]
Hint: Feature flags route traffic asynchronously for low-latency A/B tests [check]
Common Mistakes:
4. You must choose between a monolithic architecture and microservices for a system with rapidly evolving features and a small team. Which choice best justifies your design decision to the interviewer?
medium
A. Choose microservices because they always scale better regardless of team size
B. Choose monolith to reduce operational complexity and speed up feature delivery for a small team
C. Choose microservices to isolate failures even if it increases deployment overhead
D. Choose monolith because it requires no testing or CI/CD pipelines

Solution

  1. Step 1: Analyze team size and feature velocity constraints

    Small teams benefit from simpler architectures to reduce overhead and speed delivery.

  2. Step 2: Evaluate operational complexity of microservices

    Microservices add deployment and operational complexity, which can slow small teams.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Monolith reduces complexity, speeds delivery for small teams [check]
Hint: Small teams prefer monoliths for simplicity and speed [check]
Common Mistakes:
5. At 10x scale, which component of a distributed HyperLogLog-based unique user counting system is most likely to fail first, and why?
hard
A. The aggregator node, because it must merge many HyperLogLog sketches and becomes CPU-bound
B. The HyperLogLog register storage, because fixed-size registers cannot represent larger cardinalities accurately
C. The network bandwidth between servers, because merging large raw user ID sets saturates links
D. The user ID ingestion layer, because it must hash and update many HyperLogLog sketches concurrently

Solution

  1. Step 1: Analyze scaling impact on components

    Network bandwidth is low since only fixed-size sketches are transferred, not raw IDs; register size is fixed and does not fail but accuracy degrades slightly.

  2. Step 2: Identify bottleneck

    The aggregator merges many sketches concurrently, increasing CPU load and latency, making it the first to fail under 10x load.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Aggregator CPU-bound merging many sketches first bottleneck [check]
Hint: Aggregator CPU merges many sketches -> first bottleneck [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard