Bird
Raised Fist0

Agent System Design - Tool Use, Planning, Memory Architecture

SYSTEM-DESIGNseniorInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Agent System Design - Tool Use, Planning, Memory Architecture
Designing intelligent agents with tool use, planning, and memory capabilities
system-designseniorFreq: highGenAI Systems
Prerequisites:LLM ServingRAG System Design
🎯
Problem Framing
Business Objective

Enhance user productivity by automating complex tasks using intelligent agents

ML Objective

Develop an agent that can plan, execute, and remember tasks using tool APIs

Constraints
  • Latency SLA: P99 < 100ms for tool execution
  • Data privacy regulations: GDPR compliance
  • Cold start: Limited initial user interaction data
  • Scalability: Support 10,000 concurrent users
  • Tool API variability: Different response formats and latencies
Clarifying Questions
  • What types of tools will the agent need to interact with?
  • How frequently do tool APIs change?
  • What is the expected user interaction pattern?
  • Are there any specific privacy concerns with user data?
  • What is the expected scale of deployment?
Out of Scope
✗ Developing new tool APIs
✗ Handling non-digital tasks
✗ Real-time collaboration features
📊
Offline Metrics
Task Completion Rate (TCR): Measures the percentage of tasks successfully completed by the agent
Planning Accuracy: Evaluates the correctness of the agent's planned actions
📈
Online Metrics
User Satisfaction Score: Collected via post-interaction surveys
Task Execution Latency: Average time taken to complete a task
⚖️
Trade-offs

High planning accuracy might increase latency, affecting user satisfaction if tasks take too long.

💼
Business Metric

Overall user productivity improvement measured by reduction in task completion time

🗄️
Data Sources
  • User Interaction Logs: JSON, real-time
  • Tool API Responses: JSON/XML, real-time
  • Historical Task Data: SQL, updated daily
🏷️
Data Properties
Labeling Strategy

Implicit feedback from user interactions and explicit labels from task outcomes

Data Volume

Approximately 500,000 interactions per day, 2TB/month

Freshness

Real-time for user interactions, T-1 day for historical data

⚠️
Data Challenges
  • Sparse labels for new tasks
  • Variability in tool API responses
  • Cold start with new users
  • Data privacy and compliance
🔧
Key Features
embedding
User Profile Embedding
Captures user preferences and history
numerical
Tool API Latency
Predicts execution time for planning
categorical
Task Context
Informs the agent about the task environment
numerical
Historical Success Rate
Guides decision-making based on past performance
embedding
Interaction Sequence
Models user-agent interaction patterns
⚡
Feature Engineering Notes
Real-time vs Batch

User Profile Embedding and Interaction Sequence need real-time computation; others can be pre-computed.

Feature Store

Required

🐍
Feature Snippet
Python
def compute_user_profile_embedding(user_id):
    user_data = fetch_user_data(user_id)
    return model.encode(user_data)
🤖
Model Selection
Baseline

Rule-based system using predefined task sequences

Proposed Model

Reinforcement Learning with Memory-Augmented Neural Networks

Handles dynamic environments and learns optimal task sequences while maintaining memory of past interactions.
✗
Alternatives Rejected
LSTM Networks: Rejected due to inability to handle complex planning tasks effectively
Transformer Models: Rejected due to high computational cost and latency issues
🏋️
Training Strategy

Hybrid approach with offline training on historical data and online fine-tuning with user interactions

🚀
Serving Infrastructure
LatencyP99 < 100ms end-to-end
ThroughputTarget 10,000 QPS
ArchitectureTwo-stage architecture with planning and execution stages to optimize latency and accuracy
CachingCache frequent tool API responses with a TTL of 5 minutes; invalidate on API updates
FallbackUse rule-based fallback for critical tasks if the model is unavailable
📡
Data Drift Signals
Change in tool API response formats
Shift in user interaction patterns
Increase in task failure rates
📉
Model Drift Signals
Decrease in Task Completion Rate
Increase in Task Execution Latency
Drop in User Satisfaction Score
🔔
Alerting & Retraining
Retraining Trigger

Task Completion Rate drops below 80%

Alerting Strategy

Page on-call engineer if latency exceeds 150ms; log all task failures; auto-recover by switching to fallback

🎯
Interview Angles
  • How does the system handle tool API variability?
  • What strategies are used to mitigate cold start problems?
  • How is user privacy ensured in data handling?
⚠️
Common Mistakes
Ignoring latency constraints in model selection
Overlooking the need for a robust fallback strategy
Failing to address data privacy and compliance
Neglecting to plan for tool API changes
📌
Key Decisions
  • Choosing a memory-augmented model for dynamic task environments
  • Implementing a two-stage serving architecture for efficiency
  • Designing a hybrid training strategy to balance offline and online learning

Practice

(1/5)
1. In a fine-tuning pipeline using RLHF, what is a likely failure mode if the reward model is not properly calibrated?
medium
A. The model will overfit to the training data.
B. The model will have increased latency during inference.
C. The model will produce outputs that are misaligned with human preferences.
D. The model will require more computational resources for training.

Solution

  1. Step 1: Identify the role of the reward model

    The reward model in RLHF is crucial for aligning model outputs with human preferences.

  2. Step 2: Eliminate wrong options

    A: Overfitting is not directly related to reward model calibration. C: Latency is unrelated to reward model calibration. D: Computational resources are not directly affected by reward model calibration.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Misalignment if reward model is off. [OK]
Hint: Misalignment if reward model is off. [OK]
Common Mistakes:
2. In a news feed system, what is a likely failure mode if the user feedback loop is not properly integrated?
medium
A. Faster page load times
B. Increased server downtime
C. Higher data storage costs
D. Stale recommendations

Solution

  1. Step 1: Understand the role of feedback loops

    Feedback loops are crucial for updating and refining recommendations based on user interactions.

  2. Step 2: Identify the impact of missing feedback

    Without proper integration, recommendations become stale as they don't adapt to user preferences.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Feedback loops prevent stale recommendations. [OK]
Hint: Feedback loops prevent stale recommendations. [OK]
Common Mistakes:
3. When choosing between FAISS and HNSW for a vector search task, which is more suitable for handling dynamic datasets with frequent updates?
medium
A. HNSW, because it provides better precision.
B. FAISS, because it supports dynamic updates efficiently.
C. FAISS, because it has better indexing speed.
D. HNSW, because it allows for efficient updates and deletions.

Solution

  1. Step 1: Evaluate update capabilities.

    HNSW supports dynamic updates efficiently.

  2. Step 2: Eliminate wrong options.

    FAISS is not optimized for frequent updates; indexing speed and precision are not the primary concerns here.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    HNSW handles dynamic updates well. [OK]
Hint: HNSW handles dynamic updates well. [OK]
Common Mistakes:
4. As the volume of queries increases tenfold, which component of an intent classification system is most likely to become a bottleneck?
hard
A. Training time
B. Data preprocessing pipeline
C. Model inference time
D. Storage for labeled data

Solution

  1. Step 1: Consider the impact of increased query volume.

    Increased queries primarily affect components involved in real-time processing.

  2. Step 2: Evaluate each component's scalability.

    Data preprocessing pipeline: Preprocessing can be parallelized and is not real-time. Training time: Training time is not affected by query volume. Storage for labeled data: Storage is a concern but not a real-time bottleneck. Model inference time: Model inference time directly impacts real-time query handling and is likely to become a bottleneck.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Inference time scales with query volume. [OK]
Hint: Inference time scales with query volume. [OK]
Common Mistakes:
5. As a RAG system scales to handle 10x more queries, which component is most likely to become a bottleneck first?
hard
A. The retriever model
B. The database storing the corpus
C. The encoder model
D. The generator model

Solution

  1. Step 1: Consider scaling impacts.

    As query volume increases, the database must handle more retrieval requests, potentially becoming a bottleneck.

  2. Step 2: Eliminate wrong options.

    Models scale with compute resources, but database I/O can become a limiting factor.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Database I/O limits scale. [OK]
Hint: Database I/O limits scale. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard