LangChain agents overview in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - LangChain agents overview

Which metric matters for LangChain agents and WHY

LangChain agents use AI models to decide actions based on inputs. The key metrics to check how well these agents perform are accuracy and task success rate. Accuracy tells us how often the agent picks the right action. Task success rate shows if the agent completes the user's goal correctly. These metrics matter because agents must understand instructions and respond properly to be helpful.

Confusion matrix for LangChain agent action selection

      | Predicted Action |
      |------------------|
      | Correct | Wrong  |
    -----------------------
    Actual |  TP    |  FN    |
    Action |  FP    |  TN    |
    

Here, TP means the agent chose the right action when it should. FP means it chose a wrong action mistakenly. FN means it missed the right action. TN means it correctly avoided wrong actions. Counting these helps calculate precision and recall for agent decisions.

Precision vs Recall tradeoff with LangChain agents

If an agent has high precision, it rarely picks wrong actions. This is good when wrong actions cause big problems, like sending wrong emails. But it might miss some correct actions (low recall).

If an agent has high recall, it tries to catch all correct actions, even if it sometimes picks wrong ones. This is good when missing any correct action is bad, like answering customer questions.

Choosing precision or recall depends on what matters more: avoiding mistakes or catching all correct actions.

Good vs Bad metric values for LangChain agents

• Good: Precision and recall above 85% means the agent picks right actions most times and rarely misses them.
• Bad: Precision or recall below 50% means the agent often picks wrong actions or misses many correct ones.
• Task success rate above 90% shows the agent completes user goals well.
• Low task success rate means the agent fails to help users effectively.

Common pitfalls in LangChain agent metrics

• Accuracy paradox: If most inputs need the same action, high accuracy can be misleading.
• Data leakage: Testing on data the agent saw during training inflates metrics falsely.
• Overfitting: Agent performs well on training tasks but poorly on new ones.
• Ignoring task success: Focusing only on action accuracy but not if the user goal was met.

Self-check question

Your LangChain agent has 98% accuracy but only 12% recall on critical actions. Is it good for production?

Answer: No. The agent frequently misses correct actions (low recall), so it fails to perform many needed tasks despite high accuracy. This means it is not reliable for real use.

Key Result

For LangChain agents, balancing precision and recall is key to ensure correct and complete action selection, with task success rate confirming overall usefulness.

Practice

1. What is the main purpose of LangChain agents in AI?

easy

A. To help AI decide which tools to use for a task

B. To store large amounts of data efficiently

C. To train AI models faster using GPUs

D. To create static reports from data

Solution

1. Step 1: Understand LangChain agents' role

   LangChain agents help AI decide actions by choosing tools or language models based on the task.

2. Step 2: Compare options with this role

   Only To help AI decide which tools to use for a task matches this purpose; others describe unrelated tasks.

3. Final Answer:

   To help AI decide which tools to use for a task -> Option A

4. Quick Check:

   Agent purpose = Decide tools [OK]

Hint: Agents decide actions and tools for AI tasks [OK]

Common Mistakes:

• Confusing agents with data storage systems
• Thinking agents speed up training
• Assuming agents create reports

2. Which of the following is the correct way to create a simple LangChain agent in Python?

easy

A. agent = Agent(llm, tools)

B. agent = Agent(llm=llm, tools=tools)

C. agent = Agent.create(llm, tools)

D. agent = create_agent(llm, tools)

Solution

1. Step 1: Recall LangChain agent creation syntax

   LangChain agents are created by calling Agent with named parameters like llm= and tools=.

2. Step 2: Check each option's syntax

   agent = Agent(llm=llm, tools=tools) uses named parameters correctly; others use incorrect or non-existent methods.

3. Final Answer:

   agent = Agent(llm=llm, tools=tools) -> Option B

4. Quick Check:

   Correct syntax uses named parameters [OK]

Hint: Use named parameters llm= and tools= to create agents [OK]

Common Mistakes:

• Omitting parameter names
• Using non-existent create methods
• Confusing function names

3. Given this code snippet, what will be the output?

from langchain.agents import Agent
llm = MockLLM(responses=["Answer 1"])
tools = [Tool(name="search", func=lambda x: "found info")]
agent = Agent(llm=llm, tools=tools)
result = agent.run("Find info about AI")
print(result)

medium

A. Error: Missing tool function

B. "found info"

C. "Answer 1"

D. "Find info about AI"

Solution

1. Step 1: Understand the MockLLM and tools setup

   The MockLLM is set to respond with "Answer 1" regardless of input; tools have a function but agent uses LLM response first.

2. Step 2: Analyze agent.run behavior

   Agent calls LLM which returns "Answer 1"; tools are available but not triggered to override LLM output.

3. Final Answer:

   "Answer 1" -> Option C

4. Quick Check:

   LLM response = "Answer 1" [OK]

Hint: MockLLM returns preset answer, tools don't override by default [OK]

Common Mistakes:

• Assuming tool output replaces LLM output
• Confusing input with output
• Expecting runtime errors without cause

4. What is wrong with this LangChain agent code?

from langchain.agents import Agent
llm = SomeLLM()
tools = [Tool(name="calc", func=calculate)]
agent = Agent(llm, tools)
result = agent.run("Calculate 2+2")
print(result)

medium

A. Tool function 'calculate' is undefined

B. LLM instance is not imported

C. Agent.run() requires extra arguments

D. Agent constructor missing named parameters

Solution

1. Step 1: Check Agent constructor usage

   Agent requires named parameters like llm= and tools=; code uses positional arguments incorrectly.

2. Step 2: Verify other parts

   Assuming 'calculate' is defined and LLM imported, the main error is constructor call.

3. Final Answer:

   Agent constructor missing named parameters -> Option D

4. Quick Check:

   Constructor needs llm= and tools= [OK]

Hint: Always use named parameters when creating Agent [OK]

Common Mistakes:

• Using positional arguments for Agent
• Assuming undefined functions cause error here
• Thinking run() needs extra args

5. You want to build a LangChain agent that uses both a calculator tool and a web search tool. Which approach best ensures the agent chooses the right tool based on the question?

hard

A. Provide both tools and use an agent type that decides tool usage automatically

B. Manually call each tool in sequence and combine results

C. Use only one tool at a time to avoid confusion

D. Train separate agents for each tool and merge outputs later

Solution

1. Step 1: Understand agent tool selection

   LangChain agents can automatically decide which tool to use when given multiple tools and an appropriate agent type.

2. Step 2: Evaluate options for flexibility and automation

   Provide both tools and use an agent type that decides tool usage automatically uses this automatic decision feature; others require manual or less efficient approaches.

3. Final Answer:

   Provide both tools and use an agent type that decides tool usage automatically -> Option A

4. Quick Check:

   Agent auto-selects tools = Provide both tools and use an agent type that decides tool usage automatically [OK]

Hint: Use agent types that pick tools automatically [OK]

Common Mistakes:

• Manually calling tools defeats agent purpose
• Using only one tool limits flexibility
• Training separate agents adds complexity