Cost tracking across runs in LangChain - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ component_behavior

intermediate

How does LangChain track API call costs across multiple runs?

Consider a LangChain application that uses an OpenAI model. How does LangChain keep track of the total cost of API calls when the application runs multiple times?

A LangChain uses environment variables to store cumulative cost data between runs.

B LangChain writes cost data to a local file after each run and reads it back on the next run.

C LangChain resets cost tracking after each run and does not accumulate costs automatically.

D LangChain stores cost data in a persistent memory object that accumulates costs across runs.

💡 Hint

Think about whether LangChain automatically saves cost data between separate program executions.

LangChain tracks costs during a single run in memory but does not persist cost data across separate runs automatically. To track costs across runs, you need to implement your own persistence.

❓ state_output

intermediate

What is the total cost after two runs with LangChain's built-in cost tracker?

You run a LangChain app twice. Each run makes 3 API calls costing $0.02 each. LangChain's cost tracker resets on each run. What is the total cost reported after the second run?

A $0.12

B $0.00

C $0.04

D $0.06

💡 Hint

Calculate cost per run and consider if costs accumulate across runs automatically.

Each run costs 3 calls × $0.02 = $0.06. Since LangChain resets cost tracking each run, after the second run the reported cost is $0.06, not cumulative $0.12.

🔧 Debug

advanced

Why does the LangChain cost tracker show zero after restarting the app?

You use LangChain's cost tracker to monitor API usage. After restarting your app, the cost tracker shows zero. What is the most likely reason?

A The cost tracker only tracks costs in memory and does not persist data between runs.

B The API key was invalidated, so no calls were made after restart.

C LangChain automatically clears cost data on restart for security reasons.

D The cost tracker requires manual reset after each run to show costs.

💡 Hint

Consider how LangChain stores cost data internally.

LangChain's cost tracker keeps data in memory only. Restarting the app clears memory, so cost data resets to zero.

🧠 Conceptual

advanced

How to implement persistent cost tracking across LangChain runs?

You want to track total API costs across multiple LangChain runs. Which approach best achieves this?

A Save cost data to a database or file after each run and load it at start.

B Rely on the API provider's dashboard to track costs externally.

C Store cost data in a global variable inside the LangChain library.

D Use LangChain's built-in persistent cost tracker that auto-saves data.

💡 Hint

Think about how to keep data between separate program executions.

LangChain does not persist cost data automatically. Saving cost info externally (database or file) and loading it on start is the correct way to track costs across runs.

📝 Syntax

expert

What error occurs when accessing cost data before any API calls in LangChain?

Given this snippet:
from langchain.callbacks import get_openai_callback with get_openai_callback() as cb: print(cb.total_cost)
What happens if no API calls are made inside the with block?

A Raises AttributeError because total_cost is undefined.

B It prints 0.0 without error.

C Raises RuntimeError because no API calls were tracked.

D Prints None because total_cost is not set.

💡 Hint

Consider default values of cost tracking attributes before usage.

The total_cost attribute initializes to 0.0 even if no API calls occur, so printing it outputs 0.0 without error.

Practice

1. What is the main purpose of using get_openai_callback() in LangChain?

easy

A. To connect LangChain with external databases

B. To speed up the execution of LangChain chains

C. To store the output of LangChain chains permanently

D. To track token usage and cost during LangChain runs

Solution

1. Step 1: Understand the role of get_openai_callback()

   This function is designed to monitor token usage and cost when running LangChain chains.

2. Step 2: Compare with other options

   The other options describe unrelated functionalities like speeding up execution, storing outputs, or database connections, which get_openai_callback() does not do.

3. Final Answer:

   To track token usage and cost during LangChain runs -> Option D

4. Quick Check:

   Cost tracking = A [OK]

Hint: Remember: callback tracks usage and cost only [OK]

Common Mistakes:

• Thinking it speeds up chain execution
• Confusing it with output storage
• Assuming it manages database connections

2. Which of the following is the correct way to use get_openai_callback() to track cost across multiple LangChain calls?

easy

A. with get_openai_callback() as cb:
chain.run(input1)
chain.run(input2)
print(cb.total_cost)

B. cb = get_openai_callback()
chain.run(input1)
chain.run(input2)
print(cb.total_cost)

C. chain.run(input1)
chain.run(input2)
cb = get_openai_callback()
print(cb.total_cost)

D. print(get_openai_callback().total_cost)
chain.run(input1)
chain.run(input2)

Solution

1. Step 1: Identify correct usage pattern

   The get_openai_callback() must be used as a context manager with a with block to track usage across multiple calls.

2. Step 2: Analyze options

   with get_openai_callback() as cb: chain.run(input1) chain.run(input2) print(cb.total_cost) correctly wraps calls inside the with block and accesses cb.total_cost after. The other options misuse the callback or access cost before running chains.

3. Final Answer:

   with get_openai_callback() as cb: chain.run(input1) chain.run(input2) print(cb.total_cost) -> Option A

4. Quick Check:

   Use with block for callback [OK]

Hint: Always use with to track cost across runs [OK]

Common Mistakes:

• Not using with block for callback
• Accessing cost before running chains
• Creating callback after chain runs

3. Given the code below, what will be printed?

with get_openai_callback() as cb:
    chain.run("Hello")
    chain.run("World")
print(cb.total_tokens)

medium

A. Zero, because tokens are not counted automatically

B. The total number of tokens used in both runs combined

C. The number of tokens used only in the last run

D. An error because total_tokens is not a valid attribute

Solution

1. Step 1: Understand token counting in callback

   Inside the with block, all calls to chain.run() accumulate token usage tracked by cb.

2. Step 2: Check what cb.total_tokens represents

   This attribute holds the total tokens used during the entire with block, so it sums tokens from both runs.

3. Final Answer:

   The total number of tokens used in both runs combined -> Option B

4. Quick Check:

   Total tokens = sum of all runs [OK]

Hint: Tokens add up inside the with block [OK]

Common Mistakes:

• Thinking it counts only last run tokens
• Assuming tokens are not tracked automatically
• Believing total_tokens is invalid

4. Identify the error in the following code snippet for tracking cost:

cb = get_openai_callback()
chain.run("Test")
print(cb.total_cost)

medium

A. The chain.run() method must be called before creating callback

B. The total_cost attribute does not exist on callback

C. Callback is not used as a context manager, so cost is not tracked

D. The print statement should be inside the callback block

Solution

1. Step 1: Check callback usage

   The callback must be used inside a with block to track usage properly. Here, it is created but not used as a context manager.

2. Step 2: Understand consequences

   Without the with block, the callback does not track token or cost usage, so total_cost will not reflect actual usage.

3. Final Answer:

   Callback is not used as a context manager, so cost is not tracked -> Option C

4. Quick Check:

   Use with for tracking [OK]

Hint: Always use with to activate callback tracking [OK]

Common Mistakes:

• Creating callback without with
• Assuming attributes exist without context
• Placing print outside tracking scope

5. You want to track token usage and cost for multiple LangChain runs, but also reset tracking between different user sessions. Which approach correctly achieves this?

hard

A. Use separate with get_openai_callback() as cb: blocks for each session

B. Create one callback outside all sessions and reuse it without resetting

C. Call get_openai_callback() once and manually reset cb.total_cost to zero

D. Track cost by printing cb.total_cost after each run without context manager

Solution

1. Step 1: Understand session-based tracking needs

   Each user session should have isolated cost tracking to avoid mixing token counts and costs.

2. Step 2: Evaluate approaches

   Using separate with blocks creates fresh callbacks per session, resetting counts automatically. Other options either reuse callbacks incorrectly or try manual resets which are unsupported.

3. Final Answer:

   Use separate with get_openai_callback() as cb: blocks for each session -> Option A

4. Quick Check:

   Separate with blocks reset tracking [OK]

Hint: Use new with block per session to reset cost [OK]

Common Mistakes:

• Reusing one callback for all sessions
• Trying to manually reset callback attributes
• Not using with blocks for tracking