Bird
Raised Fist0

Design an AI Code Assistant - RAG Architecture, Code Indexing, Latency Budget

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design an AI Code Assistant - RAG Architecture, Code Indexing, Latency Budget
45 min · senior
senior

You are designing an AI-powered code assistant for a startup with 50 million daily active users (DAU). The system must provide relevant code completions and explanations by combining a large language model with a retrieval system over a massive indexed codebase. The system must meet a strict P99 latency budget of under 200ms for interactive user experience. All user queries and code data must be encrypted at rest and in transit. The system should support incremental code indexing and handle high query throughput with low tail latency.

Constraints
  • P99 end-to-end latency under 200ms for code completion requests
  • Encryption at rest and in transit for user queries and code data
  • Support incremental updates to the code index with minimal downtime
  • Handle 50M DAU with peak query load estimated at 1M QPS
  • Ensure high availability with 99.99% uptime SLA
📋
Guided Steps
1/6
Step 1
Designing the Retrieval Layer for Code Snippets
How would you design the code retrieval system to efficiently fetch relevant code snippets from a large codebase to augment the language model?
💡 Hint
Consider vector search engines optimized for semantic code search and incremental indexing capabilities.
Key Decision
Use Pinecone vector database for semantic code snippet retrieval with incremental indexing support and low-latency nearest neighbor search.
Interviewer Expectation
Mid-level: Names vector DB like Pinecone or FAISS. Senior: Explains Pinecone's managed service benefits, incremental indexing, and latency guarantees versus self-hosted FAISS.
⚠️Common Wrong Answer
Using a traditional relational database with full-text search - fails to scale semantic search and incurs high latency at 1M QPS.
Step 2
Choosing the Language Model Serving Architecture
What architecture and technology would you use to serve the large language model (LLM) for code generation within the 200ms P99 latency budget?
💡 Hint
Consider model size, batching, GPU inference, and caching strategies to meet strict latency requirements.
Key Decision
Deploy OpenAI's GPT-4 Turbo via managed API with edge caching of frequent completions and request batching to optimize GPU utilization and latency.
Interviewer Expectation
Mid-level: Mentions using OpenAI API or similar. Senior: Explains trade-offs of managed API vs self-hosted models, edge caching to reduce cold start latency, and batching impact on tail latency.
⚠️Common Wrong Answer
Self-hosting large LLM on CPU-only servers - leads to >500ms latency and inability to meet P99 SLA.
Step 3
Integrating Retrieval with Generation (RAG Pattern)
How do you architect the integration between the retrieval system and the language model to generate context-aware code completions?
💡 Hint
Think about how to combine retrieved snippets with prompt construction and how to handle failures in retrieval or generation.
Key Decision
Implement a RAG pipeline where retrieved code snippets from Pinecone are concatenated with user query to form LLM prompt; fallback to cached completions if retrieval fails.
Interviewer Expectation
Mid-level: Describes concatenation of retrieved docs with prompt. Senior: Details fallback mechanisms, prompt size limits, and latency impact of retrieval failures.
⚠️Common Wrong Answer
Calling LLM without retrieval - results in generic completions and poor relevance, increasing user dissatisfaction.
Step 4
Ensuring Data Security and Encryption
How would you ensure encryption of user queries and code data both at rest and in transit without violating latency constraints?
💡 Hint
Consider TLS for transit and envelope encryption or hardware security modules (HSM) for at-rest encryption with minimal overhead.
Key Decision
Use TLS 1.3 for all client-server communication and AWS KMS envelope encryption for data at rest in S3 and Pinecone, ensuring low-latency cryptographic operations.
Interviewer Expectation
Mid-level: Mentions TLS and encryption at rest. Senior: Explains envelope encryption benefits, key rotation, and latency impact mitigation strategies.
⚠️Common Wrong Answer
Encrypting data at application layer synchronously - adds significant latency and complexity, violating 200ms P99 SLA.
Step 5
Designing Incremental Code Indexing Pipeline
How would you design the pipeline to incrementally index new or updated code repositories into the retrieval system without downtime?
💡 Hint
Consider streaming data pipelines and zero-downtime index updates with versioned indexes or shadow indexing.
Key Decision
Use Apache Kafka for streaming code changes, process embeddings with a Spark job, and update Pinecone indexes via shadow indexing with atomic swap to avoid downtime.
Interviewer Expectation
Mid-level: Names Kafka and batch embedding jobs. Senior: Explains shadow indexing, atomic swap, and how to handle backpressure and consistency.
⚠️Common Wrong Answer
Rebuilding entire index offline and swapping - causes downtime and stale results during rebuild.
Step 6
API Design for Client Interaction
Sketch 3 core API endpoints for the AI Code Assistant and explain their roles.
💡 Hint
Think about endpoints for submitting queries, receiving completions, and sending feedback or corrections.
Key Decision
Define POST /query for submitting code snippets, GET /completion for streaming completions, and POST /feedback for user corrections, all secured with OAuth2 and rate-limited via Envoy proxy.
Interviewer Expectation
Mid-level: Lists endpoints with basic roles. Senior: Details streaming completions, idempotency, security via OAuth2, and rate limiting rationale.
⚠️Common Wrong Answer
Single monolithic endpoint without streaming or feedback - limits extensibility and degrades UX.
🔌
API Design Challenge
Sketch 3 core API endpoints for the AI Code Assistant: how clients send code queries, how they receive completions, and how the system acknowledges feedback or corrections.
Hints
  • How does the client submit a code snippet or query for completion?
  • How does the client receive streaming or batch completions with low latency?
  • How does the client send feedback or corrections to improve future suggestions?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of vector database (Pinecone) for semantic code retrieval
  • Managed LLM API with caching and batching to meet latency
  • RAG integration with fallback caching
  • TLS 1.3 and AWS KMS envelope encryption
  • Incremental indexing with Kafka and shadow indexing
  • OAuth2-secured, rate-limited API endpoints
Good to Have - 30%
    Bonus - 10%
      Anti-patterns to Avoid
      ✅
      Model Answer
      ▼ Reveal

      The AI Code Assistant is designed using a Retrieval-Augmented Generation (RAG) architecture combining Pinecone vector DB for semantic code snippet retrieval and OpenAI GPT-4 Turbo managed API for code generation. The system meets a strict 200ms P99 latency budget by leveraging edge caching, request batching, and low-latency vector search. Data security is ensured with TLS 1.3 in transit and AWS KMS envelope encryption at rest. Incremental code indexing is handled via a Kafka streaming pipeline, Spark embedding jobs, and Pinecone shadow indexing with atomic swaps to avoid downtime. The API exposes three core endpoints secured with OAuth2 and rate-limited via Envoy proxy to handle 1M QPS peak load.

      Key Decisions
      • Use Pinecone vector DB for low-latency semantic search with incremental indexing support
      • Deploy GPT-4 Turbo via managed API with edge caching and batching to meet 200ms P99 latency
      • Implement RAG pipeline concatenating retrieved snippets with user query and fallback to cached completions
      • Use TLS 1.3 for all client-server communication and AWS KMS envelope encryption for data at rest
      • Design incremental indexing pipeline with Kafka, Spark, and Pinecone shadow indexing with atomic swap
      • Expose POST /query, GET /completion (streaming), and POST /feedback endpoints secured with OAuth2 and rate-limited via Envoy
      • Pinecone vector database for semantic code snippet retrieval with incremental indexing support and <10ms query latency at 1M QPS
      • OpenAI GPT-4 Turbo managed API for LLM inference with edge caching and request batching to meet 200ms P99 latency
      • RAG pipeline combining Pinecone retrieval results with LLM prompt construction and fallback caching for resilience
      • TLS 1.3 for all data in transit and AWS KMS envelope encryption for data at rest to meet encryption requirements with minimal latency overhead
      • Apache Kafka streaming pipeline for incremental code indexing with Spark embedding jobs and Pinecone shadow indexing for zero downtime
      • Envoy proxy for OAuth2-secured, rate-limited API gateway exposing POST /query, GET /completion (streaming), and POST /feedback endpoints
      • Use of atomic index swaps in Pinecone to ensure consistency and availability during index updates
      • Edge caching layer (Redis) for popular completions to reduce load on LLM and retrieval systems

      Practice

      (1/5)
      1. In the News Feed system, at step 4 the Delivery Worker reads from the fan-out queue to push updates to followers. What happens if the follower's cache entry for the feed is expired at this step?
      easy
      A. The Delivery Worker fetches the latest feed snapshot from the database before pushing
      B. The Delivery Worker sends a stale cached feed to the follower
      C. The Delivery Worker retries the message later without updating the cache
      D. The Delivery Worker skips sending the update and marks it as delivered

      Solution

      1. Step 1: Trace Delivery Worker decision at Step 4 -- follower cache TTL expired

        If the cache is expired, the Delivery Worker must fetch fresh data to ensure the follower receives the latest feed snapshot.

      2. Step 2: Evaluate other options

        Skipping (A) causes missed updates; retrying without cache update (C) delays delivery; sending stale data (D) violates freshness guarantees.

      3. Final Answer:

        Option A -> Option A

      4. Quick Check:

        Expired cache triggers DB fetch for fresh feed [check]
      Hint: Expired cache triggers DB fetch before delivery [check]
      Common Mistakes:
      2. A web service receives highly variable traffic with some clients opening many simultaneous connections, while others open few. Which load balancing algorithm best distributes load evenly across servers?
      easy
      A. Round Robin, because it cycles through servers regardless of connection count
      B. Weighted Round Robin, because it assigns requests based on server weights only
      C. IP Hash, because it ensures requests from the same client always go to the same server
      D. Least Connections, because it directs new requests to the server with the fewest active connections

      Solution

      1. Step 1: Understand traffic variability

        Clients open varying numbers of connections, so servers can become unevenly loaded if requests are assigned blindly.

      2. Step 2: Evaluate load balancing algorithms

        Round Robin ignores current load, IP Hash pins clients to servers (may cause imbalance), Weighted Round Robin uses static weights ignoring dynamic load. Least Connections dynamically balances based on active connections.

      3. Final Answer:

        Option D -> Option D

      4. Quick Check:

        Least Connections adapts to real-time load, balancing uneven client connections [check]
      Hint: Least Connections adapts to active load, not just request count [check]
      Common Mistakes:
      3. In a system using a NoSQL key-value store with TTL for session data, at step 4 the Session Manager checks the TTL of a session key. What happens if the TTL has expired?
      easy
      A. The Session Manager queues the session for manual review
      B. The Session Manager treats the session as invalid and prompts re-authentication
      C. The Session Manager returns stale session data from a backup store
      D. The Session Manager extends the TTL automatically to keep the session alive

      Solution

      1. Step 1: Understand TTL behavior in NoSQL key-value stores

        TTL expiration means the key is deleted or inaccessible, so session data is invalid.

      2. Step 2: Trace Session Manager's response to expired TTL

        Session Manager cannot extend TTL automatically; queuing for manual review is a fallback for expired sessions.

      3. Final Answer:

        Option A -> Option A

      4. Quick Check:

        Expired TTL -> session queued for review [check]
      Hint: Expired TTL means session invalid, no auto extension [check]
      Common Mistakes:
      4. Without the seat lock expiration mechanism, what exact failure occurs during concurrent bookings?
      medium
      A. Users receive immediate errors due to deadlock detection in the database
      B. Seats remain locked indefinitely, causing availability to drop and bookings to stall
      C. Payment processing fails because seat status is inconsistent across services
      D. The system automatically cancels bookings after a fixed timeout regardless of payment

      Solution

      1. Step 1: Understand role of seat lock expiration

        Expiration releases seats if payment or booking is abandoned, preventing indefinite holds.

      2. Step 2: Trace failure without expiration

        Locks never expire, so seats remain reserved even if users abandon payment, reducing available seats and causing booking stalls.

      3. Final Answer:

        Option B -> Option B

      4. Quick Check:

        No TTL -> stale locks -> seats unavailable indefinitely [check]
      Hint: No expiration -> stale locks -> seats blocked forever [OK]
      Common Mistakes:
      5. At 10x scale, which component in a saga-based event-driven system is most likely to fail first due to stateful bottlenecks, and why?
      hard
      A. The saga coordinator, because it maintains in-memory state per saga and cannot shard easily
      B. The message broker, because partition count limits throughput and ordering guarantees
      C. The database, because increased writes cause lock contention on outbox tables
      D. The event relay service, because network bandwidth saturates before CPU

      Solution

      1. Step 1: Identify stateful bottleneck

        The saga coordinator holds per-saga state in memory, limiting horizontal scaling and sharding.

      2. Step 2: Compare other components

        Message brokers scale with partitions; DB scales with sharding; relay is stateless and network-bound.

      3. Final Answer:

        Option A -> Option A

      4. Quick Check:

        Saga coordinator's in-memory state limits scaling first [check]
      Hint: Stateful saga coordinator limits scaling before stateless components [check]
      Common Mistakes:
      Recommended

      Ready to prove it?

      Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

      10 questions~5 mineasy hard