Why Cost tracking across runs in LangChain? - Purpose & Use Cases

The Big Idea

What if you could see exactly how much each AI task costs without lifting a finger?

The Scenario

Imagine running multiple AI tasks one after another and trying to keep track of how much each run costs manually by checking logs or invoices.

The Problem

Manually tracking costs is slow, confusing, and easy to mess up. You might lose track of which run used what resources or how much you spent overall.

The Solution

Cost tracking across runs automatically records and summarizes expenses for each AI task, so you always know your spending without extra effort.

Before vs After

✗ Before

print('Check logs and add costs manually after each run')

✓ After

tracker = CostTracker()
tracker.record(run_id, cost)
print(tracker.summary())

What It Enables

This lets you focus on building AI features while effortlessly monitoring and controlling your spending over time.

Real Life Example

A developer runs multiple language model queries daily and uses cost tracking to see which queries are expensive and optimize usage accordingly.

Key Takeaways

Manual cost tracking is error-prone and tedious.

Automated cost tracking saves time and prevents mistakes.

You get clear insights into spending across all runs.

Practice

1. What is the main purpose of using get_openai_callback() in LangChain?

easy

A. To connect LangChain with external databases

B. To speed up the execution of LangChain chains

C. To store the output of LangChain chains permanently

D. To track token usage and cost during LangChain runs

Solution

1. Step 1: Understand the role of get_openai_callback()

   This function is designed to monitor token usage and cost when running LangChain chains.

2. Step 2: Compare with other options

   The other options describe unrelated functionalities like speeding up execution, storing outputs, or database connections, which get_openai_callback() does not do.

3. Final Answer:

   To track token usage and cost during LangChain runs -> Option D

4. Quick Check:

   Cost tracking = A [OK]

Hint: Remember: callback tracks usage and cost only [OK]

Common Mistakes:

• Thinking it speeds up chain execution
• Confusing it with output storage
• Assuming it manages database connections

2. Which of the following is the correct way to use get_openai_callback() to track cost across multiple LangChain calls?

easy

A. with get_openai_callback() as cb:
chain.run(input1)
chain.run(input2)
print(cb.total_cost)

B. cb = get_openai_callback()
chain.run(input1)
chain.run(input2)
print(cb.total_cost)

C. chain.run(input1)
chain.run(input2)
cb = get_openai_callback()
print(cb.total_cost)

D. print(get_openai_callback().total_cost)
chain.run(input1)
chain.run(input2)

Solution

1. Step 1: Identify correct usage pattern

   The get_openai_callback() must be used as a context manager with a with block to track usage across multiple calls.

2. Step 2: Analyze options

   with get_openai_callback() as cb: chain.run(input1) chain.run(input2) print(cb.total_cost) correctly wraps calls inside the with block and accesses cb.total_cost after. The other options misuse the callback or access cost before running chains.

3. Final Answer:

   with get_openai_callback() as cb: chain.run(input1) chain.run(input2) print(cb.total_cost) -> Option A

4. Quick Check:

   Use with block for callback [OK]

Hint: Always use with to track cost across runs [OK]

Common Mistakes:

• Not using with block for callback
• Accessing cost before running chains
• Creating callback after chain runs

3. Given the code below, what will be printed?

with get_openai_callback() as cb:
    chain.run("Hello")
    chain.run("World")
print(cb.total_tokens)

medium

A. Zero, because tokens are not counted automatically

B. The total number of tokens used in both runs combined

C. The number of tokens used only in the last run

D. An error because total_tokens is not a valid attribute

Solution

1. Step 1: Understand token counting in callback

   Inside the with block, all calls to chain.run() accumulate token usage tracked by cb.

2. Step 2: Check what cb.total_tokens represents

   This attribute holds the total tokens used during the entire with block, so it sums tokens from both runs.

3. Final Answer:

   The total number of tokens used in both runs combined -> Option B

4. Quick Check:

   Total tokens = sum of all runs [OK]

Hint: Tokens add up inside the with block [OK]

Common Mistakes:

• Thinking it counts only last run tokens
• Assuming tokens are not tracked automatically
• Believing total_tokens is invalid

4. Identify the error in the following code snippet for tracking cost:

cb = get_openai_callback()
chain.run("Test")
print(cb.total_cost)

medium

A. The chain.run() method must be called before creating callback

B. The total_cost attribute does not exist on callback

C. Callback is not used as a context manager, so cost is not tracked

D. The print statement should be inside the callback block

Solution

1. Step 1: Check callback usage

   The callback must be used inside a with block to track usage properly. Here, it is created but not used as a context manager.

2. Step 2: Understand consequences

   Without the with block, the callback does not track token or cost usage, so total_cost will not reflect actual usage.

3. Final Answer:

   Callback is not used as a context manager, so cost is not tracked -> Option C

4. Quick Check:

   Use with for tracking [OK]

Hint: Always use with to activate callback tracking [OK]

Common Mistakes:

• Creating callback without with
• Assuming attributes exist without context
• Placing print outside tracking scope

5. You want to track token usage and cost for multiple LangChain runs, but also reset tracking between different user sessions. Which approach correctly achieves this?

hard

A. Use separate with get_openai_callback() as cb: blocks for each session

B. Create one callback outside all sessions and reuse it without resetting

C. Call get_openai_callback() once and manually reset cb.total_cost to zero

D. Track cost by printing cb.total_cost after each run without context manager

Solution

1. Step 1: Understand session-based tracking needs

   Each user session should have isolated cost tracking to avoid mixing token counts and costs.

2. Step 2: Evaluate approaches

   Using separate with blocks creates fresh callbacks per session, resetting counts automatically. Other options either reuse callbacks incorrectly or try manual resets which are unsupported.

3. Final Answer:

   Use separate with get_openai_callback() as cb: blocks for each session -> Option A

4. Quick Check:

   Separate with blocks reset tracking [OK]

Hint: Use new with block per session to reset cost [OK]

Common Mistakes:

• Reusing one callback for all sessions
• Trying to manually reset callback attributes
• Not using with blocks for tracking