Bird
Raised Fist0

Agent System Design - Tool Use, Planning, Memory Architecture

SYSTEM-DESIGNseniorInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Tool Use vs Planning and Memory Architecture

In designing a GenAI system for dynamic task execution, would you prioritize tool use or planning and memory architecture? Justify your choice.

📊
Comparison
DimensionTool UsePlanning and Memory Architecture
Task AdaptabilityLowHigh
Execution EfficiencyHighMedium
ScalabilityMediumHigh
LatencyLowMedium
System ComplexityLowHigh
✅
Verdict
Default to tool use for environments with well-defined, repetitive tasks and high throughput needs. Switch to planning and memory architecture when task complexity and adaptability are paramount, especially in dynamic environments.
💡
Key Insight
Choose tool use for efficiency in stable environments and planning for adaptability in dynamic, complex task landscapes.
⚠️
Interview Trap
❌ Wrong: Always use tool use for efficiency.
This ignores the need for adaptability in dynamic environments where tasks are not predefined.
✅ Consider the task complexity and environment stability; use tool use for efficiency in stable environments and planning for adaptability in dynamic ones.
Tool Use
"Enables rapid execution of predefined tasks with high efficiency."
Weaknesses
  • Limited adaptability to new tasks not predefined.
  • High dependency on the availability and reliability of external tools.
  • Scalability issues when the number of tools increases significantly.
  • Potential latency due to tool invocation overhead.
Ideal When
  • When task types are well-defined and limited (< 50 types).
  • When external tools are reliable and have low latency (< 100ms).
  • When the system requires high throughput (> 1000 QPS).
  • When the environment is stable with infrequent changes.
Zapier uses tool use to automate workflows by integrating various apps and services efficiently.
Planning and Memory Architecture
"Provides flexibility and adaptability to handle novel and complex tasks."
Weaknesses
  • Higher computational overhead due to planning algorithms.
  • Increased complexity in system design and maintenance.
  • Potentially slower response times (P99 latency > 200ms).
  • Requires significant memory resources for storing and retrieving plans.
Ideal When
  • When tasks are complex and not predefined (> 100 types).
  • When adaptability to new tasks is crucial.
  • When the system can afford higher latency (P99 < 500ms).
  • When the environment is dynamic with frequent changes.
OpenAI's GPT-3 uses planning and memory architecture to generate coherent and contextually relevant text across diverse topics.
🐍
Tool Use - Example
▼ Show
Python
def execute_tool(task):
    tools = {'email': send_email, 'report': generate_report}
    if task in tools:
        return tools[task]()
    else:
        raise ValueError('Task not supported')
🐍
Planning and Memory Architecture - Example
▼ Show
Python
class Planner:
    def __init__(self):
        self.memory = {}
    def plan(self, task):
        if task not in self.memory:
            self.memory[task] = self.create_plan(task)
        return self.memory[task]
    def create_plan(self, task):
        # Complex planning logic here
        return f'Plan for {task}'
✅
Use Tool Use When
  • •When task types are well-defined and limited (< 50 types).
  • •When external tools are reliable and have low latency (< 100ms).
  • •When the system requires high throughput (> 1000 QPS).
✅
Use Planning and Memory Architecture When
  • •When tasks are complex and not predefined (> 100 types).
  • •When adaptability to new tasks is crucial.
  • •When the system can afford higher latency (P99 < 500ms).
↳
Nuances
  • Tool use can be limiting if the task landscape changes frequently, requiring constant updates.
  • Planning and memory architecture can be overkill for simple, repetitive tasks where tool use suffices.
  • Tool use may introduce bottlenecks if external tools have variable performance.
  • Planning systems can struggle with real-time constraints due to their inherent complexity.

Practice

(1/5)
1. For a learning to rank system, which metric is most appropriate to evaluate the quality of the ranking?
easy
A. Normalized Discounted Cumulative Gain (NDCG)
B. Mean Squared Error (MSE)
C. Precision
D. Recall

Solution

  1. Step 1: Identify the metric that evaluates ranking quality.

    NDCG is specifically designed to evaluate the quality of rankings.

  2. Step 2: Eliminate wrong options.

    MSE is for regression, precision and recall are for classification, not ranking.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    NDCG = ranking quality [OK]
Hint: NDCG = ranking quality [OK]
Common Mistakes:
2. In the context of intent classification, which of the following techniques is most suitable for handling imbalanced datasets?
easy
A. Apply SMOTE (Synthetic Minority Over-sampling Technique)
B. Use a simple logistic regression model
C. Use a linear SVM without any modifications
D. Ignore the imbalance and focus on feature engineering

Solution

  1. Step 1: Identify the challenge of imbalanced datasets.

    Imbalanced datasets can lead to biased models that favor the majority class.

  2. Step 2: Evaluate each option.

    Use a simple logistic regression model: Logistic regression without addressing imbalance will likely be biased. Use a linear SVM without any modifications: Linear SVM also suffers from imbalance without adjustments. Ignore the imbalance and focus on feature engineering: Ignoring imbalance is not a solution. Apply SMOTE (Synthetic Minority Over-sampling Technique): SMOTE is specifically designed to handle imbalance by generating synthetic samples.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    SMOTE for imbalance handling. [OK]
Hint: SMOTE is key for imbalanced datasets. [OK]
Common Mistakes:
3. A delivery forecasting model is consistently overestimating ETAs during peak traffic hours. What is the most likely failure mode?
medium
A. The model is not accounting for real-time traffic data
B. The model has too many parameters
C. The model is underfitting the data
D. The model is overfitting to the training data

Solution

  1. Step 1: Identify the cause of overestimation during peak hours

    Real-time traffic data is crucial for accurate ETA predictions.

  2. Step 2: Eliminate wrong options

    Underfitting (A) and overfitting (D) are general issues, not specific to peak hours. Too many parameters (C) would cause overfitting, not specific time errors.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Real-time data is key for peak accuracy. [OK]
Hint: Real-time data is key for peak accuracy. [OK]
Common Mistakes:
4. When implementing a Guardrails System for LLMs, should you prioritize precision over recall, and why?
medium
A. Yes, because precision ensures the system is efficient and fast.
B. No, because recall is more important to ensure all inappropriate content is caught.
C. Yes, to minimize false positives and avoid blocking legitimate content.
D. No, because precision and recall are equally important in all cases.

Solution

  1. Step 1: Understand the trade-off

    Prioritizing precision minimizes false positives, crucial for user experience.

  2. Step 2: Eliminate wrong options

    Recall is important but can lead to over-blocking, efficiency isn't the main concern, and equal importance isn't practical in all cases.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Prioritize precision to avoid blocking legitimate content. [OK]
Hint: Prioritize precision to avoid blocking legitimate content. [OK]
Common Mistakes:
5. As the number of queries increases tenfold, which component of a search ranking system is most likely to become a bottleneck first?
hard
A. Ranking computation
B. Document retrieval
C. Query parsing
D. Index updating

Solution

  1. Step 1: Consider the impact of scaling queries

    Ranking computation is resource-intensive and scales with query volume.

  2. Step 2: Evaluate other components

    Query parsing and document retrieval are less computationally intensive, and index updating is not query-dependent.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Ranking computation scales with query volume. [OK]
Hint: Ranking computation scales with query volume. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard