Bird
Raised Fist0

Design an AI Code Assistant - RAG Architecture, Code Indexing, Latency Budget

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design an AI Code Assistant - RAG Architecture, Code Indexing, Latency Budget
Design a scalable AI-powered code assistant using Retrieval-Augmented Generation (RAG) with efficient code indexing and strict latency constraints.
hardAI and ML System DesignFreq: medium
Prerequisites:System Design FundamentalsCAP TheoremDistributed DatabasesVector Search and EmbeddingsLatency and Throughput Optimization
⚙️
Functional Requirements
  • Accept natural language queries from developers about code usage, syntax, or best practices.
  • Retrieve relevant code snippets and documentation from a large indexed codebase using semantic search.
  • Generate context-aware code completions or explanations by combining retrieved code with LLM generation (RAG).
  • Support multi-language codebases (e.g., Python, Java, JavaScript) with language-specific indexing.
  • Allow incremental updates to the code index as repositories evolve.
  • Provide usage analytics and feedback loop for improving retrieval and generation quality.
📊
Non-Functional Requirements
  • Support 2 billion daily active users (DAU) with an average of 40 queries per user per day.
  • End-to-end query latency budget of 100ms P99 for retrieval plus generation combined.
  • System availability of 99.99% to ensure developer productivity.
  • Index storage must support petabyte scale with replication for durability.
  • Throughput must handle peak write QPS of 1.5 million and read QPS of 10 million.
❓
Clarifying Questions to Ask
  • What is the expected average and peak query rate per user and overall?
  • What is the maximum acceptable latency for retrieval and generation separately?
  • How frequently does the codebase update and how fast must the index reflect changes?
  • Are there any constraints on the size or format of code snippets to retrieve?
  • What languages and frameworks must be supported initially?
  • Is the system expected to support offline or batch query modes?
Out of Scope
✗ Training or fine-tuning the large language model itself.
✗ User authentication and authorization mechanisms.
✗ UI/UX design of the code assistant frontend.
✗ Support for non-code related queries or general chat.
📌
Assumptions
  • 2 billion DAU with 40 queries per user per day.
  • Average query payload size (request + response) is 100 bytes.
  • Codebase size is 10 petabytes with 30 days retention for incremental indexing.
  • Replication factor of 3 for durability and availability.
⚡
QPS Calculation
write qps2B users × 40 queries/day ÷ 86400 seconds ≈ 925,926 writes/sec
peak write qpsAssuming 2× peak factor: 925,926 × 2 ≈ 1,851,852 writes/sec
read qpsEach query triggers 1 read; assume 10M queries/sec peak for retrieval
💾
Storage Estimate
daily storage2B × 40 × 100 bytes = 8 TB/day raw data
retention storage8 TB/day × 30 days = 240 TB
with replication240 TB × 3 = 720 TB total storage
🖥️
Server Estimate
indexing serversAssuming 500 MB/sec indexing throughput per server: 92.6 MB/sec ÷ 500 MB/sec ≈ 1 server (plus overhead, use 5 for redundancy and spikes)
retrieval serversAssuming each retrieval server handles 10K QPS at 100ms latency: 10M QPS ÷ 10K = 1000 servers
🏗️
Architecture Diagram
Architecture Diagram
          +----------------------+
          |  Client (Developer)   |
          +----------+-----------+
                     | Query (NL)
                     v
          +----------+-----------+
          |  API Gateway / Proxy |
          +----------+-----------+
                     |
                     v
          +----------+-----------+
          |  Query Router / Load  |
          |      Balancer         |
          +----------+-----------+
           /                     \
          v                       v
 +--------+--------+     +--------+--------+
 |  Retrieval Svc   |     |  Generation Svc  |
 | (ElasticSearch + |     | (OpenAI GPT-4 /  |
 |  FAISS for vec)  |     |  Anthropic Claude)|
 +--------+--------+     +--------+--------+
          |                       |
          v                       |
 +--------+--------+              |
 |  Code Index DB   |<-------------+
 | (Cassandra +     |
 |  FAISS vectors)  |
 +------------------+
          |
          v
 +--------+--------+
 |  Indexing Worker |
 | (Kafka + Spark)  |
 +------------------+
🧩
Components
API Gateway / Proxy
Entry point for client queries, handles authentication, rate limiting, and routing.
Why thisEnvoy proxy supports high throughput, low latency, and integrates with service mesh for observability.
Why notNginx lacks native service mesh integration and advanced telemetry at scale.
Query Router / Load Balancer
Distributes incoming queries to retrieval and generation services based on load and latency budgets.
Why thisEnvoy’s Layer 7 routing supports weighted routing and retries, essential for latency-sensitive AI queries.
Why notHAProxy lacks native support for advanced routing policies and telemetry needed here.
Retrieval Service
Performs semantic search over code embeddings and metadata to fetch relevant code snippets.
Why thisFAISS vector search combined with Elasticsearch for metadata enables sub-50ms retrieval at 10M QPS scale.
Why notPure Elasticsearch vector search lacks FAISS’s optimized ANN algorithms and scalability.
Generation Service
Runs large language model inference (e.g., GPT-4) to generate code completions or explanations using retrieved context.
Why thisOpenAI GPT-4 or Anthropic Claude provide state-of-the-art generation with API support and latency SLAs.
Why notSelf-hosted LLMs like GPT-J have higher latency and require expensive GPU clusters to meet 100ms P99.
Code Index Database
Stores code snippets, embeddings, and metadata with fast read/write and horizontal scalability.
Why thisApache Cassandra offers linear write scalability, tunable consistency, and wide-column storage for code metadata.
Why notPostgreSQL requires complex manual sharding at 1.8M writes/sec; B-tree writes don’t scale linearly.
Indexing Worker
Processes codebase updates, generates embeddings, and updates the code index asynchronously.
Why thisApache Spark streaming jobs consume Kafka topics for scalable, fault-tolerant incremental indexing.
Why notStandalone batch jobs cause high latency in reflecting code changes; Kafka+Spark enables near real-time.
Kafka
Message queue for decoupling codebase update events and indexing workers to ensure reliable event delivery.
Why thisApache Kafka provides high-throughput, fault-tolerant, distributed messaging with strong ordering guarantees.
Why notRabbitMQ lacks Kafka’s scalability and partitioning features needed for massive event streams.

Practice

(1/5)
1. A customer support bot must handle 10,000 concurrent user queries with minimal latency and ensure fallback to human agents when intent classification confidence is low. Which component is best suited to manage the fallback routing reliably?
easy
A. A relational database storing fallback requests synchronously
B. An in-memory cache storing fallback intents temporarily
C. A REST API endpoint directly invoking human agent notification
D. A message queue system decoupling intent classification from fallback processing

Solution

  1. Step 1: Identify fallback routing requirements

    The fallback must be reliable and decoupled from real-time intent classification to avoid blocking user queries.

  2. Step 2: Evaluate component suitability

    A message queue decouples processing, enabling asynchronous fallback handling and retry mechanisms. Relational DB synchronous writes (A) cause latency. REST API direct calls (C) block the main flow. In-memory cache (D) lacks durability and reliability.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Message queue decouples and buffers fallback reliably [check]
Hint: Decouple fallback with queues for async reliability [check]
Common Mistakes:
2. In the message delivery trace, at Step 4 the Delivery Worker checks Bob's presence TTL in the presence service cache. What happens if Bob's presence TTL has expired at this step?
easy
A. Delivery Worker immediately marks message as delivered without retry
B. Delivery Worker retries presence check indefinitely until Bob reconnects
C. Delivery Worker forwards message to Bob's last known server regardless of presence
D. Delivery Worker queues message for offline storage and does not attempt real-time delivery

Solution

  1. Step 1: Trace Delivery Worker decision at Step 4 -- Bob's presence TTL

    If Bob's presence TTL expired, the system treats Bob as offline but still attempts delivery to his last known server to avoid message loss.

  2. Step 2: Eliminate other options

    Immediate delivery marking is incorrect as message may not be delivered; queuing for offline storage happens later; indefinite retries cause blocking; forwarding to last known server is the fallback.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Delivery Worker forwards to last known server if presence expired [check]
Hint: Expired presence TTL -> fallback to last known server delivery [check]
Common Mistakes:
3. A search system must handle millions of write requests per second while ensuring high availability and fault tolerance. Which Elasticsearch component primarily manages data distribution and fault tolerance across nodes?
easy
A. The Elasticsearch Ingest Node, because it preprocesses documents before indexing
B. The Shard, because it partitions data and replicates it across nodes for fault tolerance
C. The Coordinating Node, because it routes queries to the correct shards
D. The Master Node, because it handles cluster state and metadata management

Solution

  1. Step 1: Identify the role of shards

    Shards partition the index data and replicate it to provide fault tolerance and distribute write load.

  2. Step 2: Eliminate other components

    Ingest nodes preprocess data but don't manage distribution; coordinating nodes route queries but don't store data; master nodes manage cluster metadata but don't handle data distribution.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Shards partition and replicate data for fault tolerance [check]
Hint: Shards split and replicate data for fault tolerance [check]
Common Mistakes:
4. A web service receives highly variable traffic with some clients opening many simultaneous connections, while others open few. Which load balancing algorithm best distributes load evenly across servers?
easy
A. Round Robin, because it cycles through servers regardless of connection count
B. Weighted Round Robin, because it assigns requests based on server weights only
C. IP Hash, because it ensures requests from the same client always go to the same server
D. Least Connections, because it directs new requests to the server with the fewest active connections

Solution

  1. Step 1: Understand traffic variability

    Clients open varying numbers of connections, so servers can become unevenly loaded if requests are assigned blindly.

  2. Step 2: Evaluate load balancing algorithms

    Round Robin ignores current load, IP Hash pins clients to servers (may cause imbalance), Weighted Round Robin uses static weights ignoring dynamic load. Least Connections dynamically balances based on active connections.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Least Connections adapts to real-time load, balancing uneven client connections [check]
Hint: Least Connections adapts to active load, not just request count [check]
Common Mistakes:
5. If the Market Data Feed component crashes and stops publishing updates, what is the exact failure observed by external clients subscribing to real-time stock prices?
medium
A. Clients receive stale prices indefinitely with no indication of failure
B. Clients receive partial updates but with delayed timestamps causing confusion
C. Clients receive an error message and automatic reconnection attempts begin
D. Clients continue receiving cached data from the Matching Engine without updates

Solution

  1. Step 1: Understand Market Data Feed failure impact

    When Market Data Feed crashes, it stops pushing updates; clients detect connection loss and receive error events triggering reconnection.

  2. Step 2: Eliminate other options

    Clients receive stale prices indefinitely with no indication of failure is wrong because clients detect connection loss; Clients receive partial updates but with delayed timestamps causing confusion is wrong because partial updates don't occur without feed; Clients continue receiving cached data from the Matching Engine without updates is wrong because Matching Engine doesn't serve cached market data.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Feed crash -> clients get error + reconnect attempts [check]
Hint: Feed crash triggers client error and reconnect [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard