Cost tracking across runs in LangChain - Performance & Optimization

Performance: Cost tracking across runs

MEDIUM IMPACT

This concept affects the responsiveness and resource usage during multiple executions of language model tasks, impacting user wait times and backend load.

Tracking API usage cost separately for each run without aggregation

LangChain

total_cost = 0
for run in runs:
    run_cost = sum(call.price for call in run.api_calls)
    total_cost += run_cost
print(f"Total cost across runs: {total_cost}")

Aggregates costs cumulatively, reducing redundant calculations and enabling faster cost reporting.

📈 Performance Gainimproves INP by reducing repeated summations and consolidating cost tracking

Tracking API usage cost separately for each run without aggregation

LangChain

for run in runs:
    cost = 0
    for call in run.api_calls:
        cost += call.price
    print(f"Run cost: {cost}")

Calculates cost independently for each run, causing repeated summations and no cumulative insight.

📉 Performance Costblocks interaction responsiveness due to repeated cost calculations per run

Performance Comparison

Pattern	DOM Operations	Reflows	Paint Cost	Verdict
Independent cost calculation per run	Minimal	0	0	[!] OK
Aggregated cost tracking across runs	Minimal	0	0	[OK] Good

Rendering Pipeline

Cost tracking logic runs in the backend or client script and affects how quickly cost data is available for display, influencing interaction responsiveness.

→Script Execution

→Data Processing

→UI Update

⚠️ BottleneckData Processing when recalculating costs repeatedly

Core Web Vital Affected

INP

This concept affects the responsiveness and resource usage during multiple executions of language model tasks, impacting user wait times and backend load.

Optimization Tips

1Avoid recalculating costs independently for each run to reduce processing overhead.

2Aggregate cost data incrementally to improve interaction responsiveness.

3Use DevTools Performance panel to monitor scripting time during cost tracking.

Performance Quiz - 3 Questions

What is the main performance benefit of aggregating cost tracking across multiple runs?

A Increases DOM nodes for better visualization

B Reduces repeated calculations improving interaction responsiveness

C Triggers more reflows to update UI faster

D Blocks rendering to ensure accurate cost display

DevTools: Performance

How to check: Record a session while running multiple runs and tracking costs; look for scripting time spikes during cost calculations.

What to look for: Lower scripting time and smoother interaction responsiveness indicate efficient cost tracking.

Practice

1. What is the main purpose of using get_openai_callback() in LangChain?

easy

A. To connect LangChain with external databases

B. To speed up the execution of LangChain chains

C. To store the output of LangChain chains permanently

D. To track token usage and cost during LangChain runs

Solution

1. Step 1: Understand the role of get_openai_callback()

   This function is designed to monitor token usage and cost when running LangChain chains.

2. Step 2: Compare with other options

   The other options describe unrelated functionalities like speeding up execution, storing outputs, or database connections, which get_openai_callback() does not do.

3. Final Answer:

   To track token usage and cost during LangChain runs -> Option D

4. Quick Check:

   Cost tracking = A [OK]

Hint: Remember: callback tracks usage and cost only [OK]

Common Mistakes:

• Thinking it speeds up chain execution
• Confusing it with output storage
• Assuming it manages database connections

2. Which of the following is the correct way to use get_openai_callback() to track cost across multiple LangChain calls?

easy

A. with get_openai_callback() as cb:
chain.run(input1)
chain.run(input2)
print(cb.total_cost)

B. cb = get_openai_callback()
chain.run(input1)
chain.run(input2)
print(cb.total_cost)

C. chain.run(input1)
chain.run(input2)
cb = get_openai_callback()
print(cb.total_cost)

D. print(get_openai_callback().total_cost)
chain.run(input1)
chain.run(input2)

Solution

1. Step 1: Identify correct usage pattern

   The get_openai_callback() must be used as a context manager with a with block to track usage across multiple calls.

2. Step 2: Analyze options

   with get_openai_callback() as cb: chain.run(input1) chain.run(input2) print(cb.total_cost) correctly wraps calls inside the with block and accesses cb.total_cost after. The other options misuse the callback or access cost before running chains.

3. Final Answer:

   with get_openai_callback() as cb: chain.run(input1) chain.run(input2) print(cb.total_cost) -> Option A

4. Quick Check:

   Use with block for callback [OK]

Hint: Always use with to track cost across runs [OK]

Common Mistakes:

• Not using with block for callback
• Accessing cost before running chains
• Creating callback after chain runs

3. Given the code below, what will be printed?

with get_openai_callback() as cb:
    chain.run("Hello")
    chain.run("World")
print(cb.total_tokens)

medium

A. Zero, because tokens are not counted automatically

B. The total number of tokens used in both runs combined

C. The number of tokens used only in the last run

D. An error because total_tokens is not a valid attribute

Solution

1. Step 1: Understand token counting in callback

   Inside the with block, all calls to chain.run() accumulate token usage tracked by cb.

2. Step 2: Check what cb.total_tokens represents

   This attribute holds the total tokens used during the entire with block, so it sums tokens from both runs.

3. Final Answer:

   The total number of tokens used in both runs combined -> Option B

4. Quick Check:

   Total tokens = sum of all runs [OK]

Hint: Tokens add up inside the with block [OK]

Common Mistakes:

• Thinking it counts only last run tokens
• Assuming tokens are not tracked automatically
• Believing total_tokens is invalid

4. Identify the error in the following code snippet for tracking cost:

cb = get_openai_callback()
chain.run("Test")
print(cb.total_cost)

medium

A. The chain.run() method must be called before creating callback

B. The total_cost attribute does not exist on callback

C. Callback is not used as a context manager, so cost is not tracked

D. The print statement should be inside the callback block

Solution

1. Step 1: Check callback usage

   The callback must be used inside a with block to track usage properly. Here, it is created but not used as a context manager.

2. Step 2: Understand consequences

   Without the with block, the callback does not track token or cost usage, so total_cost will not reflect actual usage.

3. Final Answer:

   Callback is not used as a context manager, so cost is not tracked -> Option C

4. Quick Check:

   Use with for tracking [OK]

Hint: Always use with to activate callback tracking [OK]

Common Mistakes:

• Creating callback without with
• Assuming attributes exist without context
• Placing print outside tracking scope

5. You want to track token usage and cost for multiple LangChain runs, but also reset tracking between different user sessions. Which approach correctly achieves this?

hard

A. Use separate with get_openai_callback() as cb: blocks for each session

B. Create one callback outside all sessions and reuse it without resetting

C. Call get_openai_callback() once and manually reset cb.total_cost to zero

D. Track cost by printing cb.total_cost after each run without context manager

Solution

1. Step 1: Understand session-based tracking needs

   Each user session should have isolated cost tracking to avoid mixing token counts and costs.

2. Step 2: Evaluate approaches

   Using separate with blocks creates fresh callbacks per session, resetting counts automatically. Other options either reuse callbacks incorrectly or try manual resets which are unsupported.

3. Final Answer:

   Use separate with get_openai_callback() as cb: blocks for each session -> Option A

4. Quick Check:

   Separate with blocks reset tracking [OK]

Hint: Use new with block per session to reset cost [OK]

Common Mistakes:

• Reusing one callback for all sessions
• Trying to manually reset callback attributes
• Not using with blocks for tracking