Cost tracking across runs in LangChain - Mini Project: Build & Apply

Cost Tracking Across Runs with Langchain

📖 Scenario: You are building a simple Langchain application that calls an AI model multiple times. You want to keep track of the total cost spent on these calls across different runs of your program.This helps you understand how much you are spending and manage your budget better.

🎯 Goal: Create a Langchain program that tracks the total cost of AI calls across multiple runs by saving and loading the cost from a file.You will set up the initial cost data, configure the file path, update the cost after each call, and save the total cost back to the file.

📋 What You'll Learn

Create a variable to hold the total cost loaded from a file

Create a variable for the file path to store the cost

Update the total cost after a simulated AI call

Save the updated total cost back to the file

💡 Why This Matters

🌍 Real World

Tracking costs helps developers manage their spending on AI services and avoid surprises in billing.

💼 Career

Many jobs require monitoring API usage costs, especially when working with cloud AI services like Langchain.

1

Set up initial total cost variable

Create a variable called total_cost and set it to 0.0 to represent the starting cost before any AI calls.

LangChain

# Your code here

Hint

Think of total_cost as your wallet balance starting at zero.

2

Add file path configuration for cost storage

Create a variable called cost_file_path and set it to the string 'cost_data.txt' to store the total cost between runs.

LangChain

total_cost = 0.0
# Your code here

Hint

This file will act like a notebook where you write down your spending.

3

Update total cost after an AI call

Add code to simulate an AI call cost of 0.05 and add it to total_cost. Use ai_call_cost = 0.05 and then update total_cost by adding ai_call_cost.

LangChain

total_cost = 0.0
cost_file_path = 'cost_data.txt'
# Your code here

Hint

Think of ai_call_cost as the price of one coffee you just bought, adding to your total spending.

4

Save the updated total cost to the file

Add code to open the file at cost_file_path in write mode and save the string version of total_cost to it. Use with open(cost_file_path, 'w') as f: and f.write(str(total_cost)).

LangChain

total_cost = 0.0
cost_file_path = 'cost_data.txt'
ai_call_cost = 0.05
total_cost += ai_call_cost
# Your code here

Hint

Saving your spending in a file is like writing it down in your diary.

Practice

1. What is the main purpose of using get_openai_callback() in LangChain?

easy

A. To connect LangChain with external databases

B. To speed up the execution of LangChain chains

C. To store the output of LangChain chains permanently

D. To track token usage and cost during LangChain runs

Solution

1. Step 1: Understand the role of get_openai_callback()

   This function is designed to monitor token usage and cost when running LangChain chains.

2. Step 2: Compare with other options

   The other options describe unrelated functionalities like speeding up execution, storing outputs, or database connections, which get_openai_callback() does not do.

3. Final Answer:

   To track token usage and cost during LangChain runs -> Option D

4. Quick Check:

   Cost tracking = A [OK]

Hint: Remember: callback tracks usage and cost only [OK]

Common Mistakes:

• Thinking it speeds up chain execution
• Confusing it with output storage
• Assuming it manages database connections

2. Which of the following is the correct way to use get_openai_callback() to track cost across multiple LangChain calls?

easy

A. with get_openai_callback() as cb:
chain.run(input1)
chain.run(input2)
print(cb.total_cost)

B. cb = get_openai_callback()
chain.run(input1)
chain.run(input2)
print(cb.total_cost)

C. chain.run(input1)
chain.run(input2)
cb = get_openai_callback()
print(cb.total_cost)

D. print(get_openai_callback().total_cost)
chain.run(input1)
chain.run(input2)

Solution

1. Step 1: Identify correct usage pattern

   The get_openai_callback() must be used as a context manager with a with block to track usage across multiple calls.

2. Step 2: Analyze options

   with get_openai_callback() as cb: chain.run(input1) chain.run(input2) print(cb.total_cost) correctly wraps calls inside the with block and accesses cb.total_cost after. The other options misuse the callback or access cost before running chains.

3. Final Answer:

   with get_openai_callback() as cb: chain.run(input1) chain.run(input2) print(cb.total_cost) -> Option A

4. Quick Check:

   Use with block for callback [OK]

Hint: Always use with to track cost across runs [OK]

Common Mistakes:

• Not using with block for callback
• Accessing cost before running chains
• Creating callback after chain runs

3. Given the code below, what will be printed?

with get_openai_callback() as cb:
    chain.run("Hello")
    chain.run("World")
print(cb.total_tokens)

medium

A. Zero, because tokens are not counted automatically

B. The total number of tokens used in both runs combined

C. The number of tokens used only in the last run

D. An error because total_tokens is not a valid attribute

Solution

1. Step 1: Understand token counting in callback

   Inside the with block, all calls to chain.run() accumulate token usage tracked by cb.

2. Step 2: Check what cb.total_tokens represents

   This attribute holds the total tokens used during the entire with block, so it sums tokens from both runs.

3. Final Answer:

   The total number of tokens used in both runs combined -> Option B

4. Quick Check:

   Total tokens = sum of all runs [OK]

Hint: Tokens add up inside the with block [OK]

Common Mistakes:

• Thinking it counts only last run tokens
• Assuming tokens are not tracked automatically
• Believing total_tokens is invalid

4. Identify the error in the following code snippet for tracking cost:

cb = get_openai_callback()
chain.run("Test")
print(cb.total_cost)

medium

A. The chain.run() method must be called before creating callback

B. The total_cost attribute does not exist on callback

C. Callback is not used as a context manager, so cost is not tracked

D. The print statement should be inside the callback block

Solution

1. Step 1: Check callback usage

   The callback must be used inside a with block to track usage properly. Here, it is created but not used as a context manager.

2. Step 2: Understand consequences

   Without the with block, the callback does not track token or cost usage, so total_cost will not reflect actual usage.

3. Final Answer:

   Callback is not used as a context manager, so cost is not tracked -> Option C

4. Quick Check:

   Use with for tracking [OK]

Hint: Always use with to activate callback tracking [OK]

Common Mistakes:

• Creating callback without with
• Assuming attributes exist without context
• Placing print outside tracking scope

5. You want to track token usage and cost for multiple LangChain runs, but also reset tracking between different user sessions. Which approach correctly achieves this?

hard

A. Use separate with get_openai_callback() as cb: blocks for each session

B. Create one callback outside all sessions and reuse it without resetting

C. Call get_openai_callback() once and manually reset cb.total_cost to zero

D. Track cost by printing cb.total_cost after each run without context manager

Solution

1. Step 1: Understand session-based tracking needs

   Each user session should have isolated cost tracking to avoid mixing token counts and costs.

2. Step 2: Evaluate approaches

   Using separate with blocks creates fresh callbacks per session, resetting counts automatically. Other options either reuse callbacks incorrectly or try manual resets which are unsupported.

3. Final Answer:

   Use separate with get_openai_callback() as cb: blocks for each session -> Option A

4. Quick Check:

   Separate with blocks reset tracking [OK]

Hint: Use new with block per session to reset cost [OK]

Common Mistakes:

• Reusing one callback for all sessions
• Trying to manually reset callback attributes
• Not using with blocks for tracking