Cost tracking across runs in LangChain - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to import the class used for tracking costs in LangChain.

LangChain

from langchain.callbacks import [1]

Drag options to blanks, or click blank then click option'

AMemoryBuffer

BCallbackManager

CCostTracker

DRunLogger

Attempts:

3 left

💡 Hint

Common Mistakes

Importing unrelated classes like CallbackManager or MemoryBuffer.

Using a class name that does not exist in langchain.callbacks.

The CostTracker class is used to track costs across runs in LangChain.

2fill in blank

medium

Complete the code to create a new cost tracker instance.

LangChain

tracker = [1]()

Drag options to blanks, or click blank then click option'

ARunManager

BCallbackHandler

CMemoryBuffer

DCostTracker

Attempts:

3 left

💡 Hint

Common Mistakes

Using unrelated classes like RunManager or CallbackHandler.

Forgetting to instantiate the class.

To track costs, you create an instance of CostTracker.

3fill in blank

hard

Fix the error in the code to add the cost tracker to the callback manager.

LangChain

callback_manager = CallbackManager(callbacks=[[1]])

Drag options to blanks, or click blank then click option'

Atracker

BCostTracker

CCallbackHandler

Drun_manager

Attempts:

3 left

💡 Hint

Common Mistakes

Passing the class name instead of the instance.

Passing unrelated variables.

You must pass the instance tracker to the callbacks list, not the class name.

4fill in blank

hard

Fill both blanks to update the cost tracker after a run and then print the total cost.

LangChain

tracker.[1](run)
print(tracker.[2])

Drag options to blanks, or click blank then click option'

Aon_run_end

Btotal_cost

Cstart_run

Dcost

Attempts:

3 left

💡 Hint

Common Mistakes

Using a method that does not exist like 'start_run'.

Trying to print a method instead of a property.

Use on_run_end to update the tracker after a run, and total_cost to get the current total cost.

5fill in blank

hard

Fill all three blanks to reset the cost tracker, run a new process, and then print the updated cost.

LangChain

tracker.[1]()
result = process.run(input)
tracker.[2](result)
print(tracker.[3])

Drag options to blanks, or click blank then click option'

Areset

Bon_run_end

Ctotal_cost

Dstart

Attempts:

3 left

💡 Hint

Common Mistakes

Forgetting to reset before running.

Using wrong method names for updating or printing cost.

First, reset the tracker with reset(). Then update it with on_run_end() after the run. Finally, print the total_cost.

Practice

1. What is the main purpose of using get_openai_callback() in LangChain?

easy

A. To connect LangChain with external databases

B. To speed up the execution of LangChain chains

C. To store the output of LangChain chains permanently

D. To track token usage and cost during LangChain runs

Solution

1. Step 1: Understand the role of get_openai_callback()

   This function is designed to monitor token usage and cost when running LangChain chains.

2. Step 2: Compare with other options

   The other options describe unrelated functionalities like speeding up execution, storing outputs, or database connections, which get_openai_callback() does not do.

3. Final Answer:

   To track token usage and cost during LangChain runs -> Option D

4. Quick Check:

   Cost tracking = A [OK]

Hint: Remember: callback tracks usage and cost only [OK]

Common Mistakes:

• Thinking it speeds up chain execution
• Confusing it with output storage
• Assuming it manages database connections

2. Which of the following is the correct way to use get_openai_callback() to track cost across multiple LangChain calls?

easy

A. with get_openai_callback() as cb:
chain.run(input1)
chain.run(input2)
print(cb.total_cost)

B. cb = get_openai_callback()
chain.run(input1)
chain.run(input2)
print(cb.total_cost)

C. chain.run(input1)
chain.run(input2)
cb = get_openai_callback()
print(cb.total_cost)

D. print(get_openai_callback().total_cost)
chain.run(input1)
chain.run(input2)

Solution

1. Step 1: Identify correct usage pattern

   The get_openai_callback() must be used as a context manager with a with block to track usage across multiple calls.

2. Step 2: Analyze options

   with get_openai_callback() as cb: chain.run(input1) chain.run(input2) print(cb.total_cost) correctly wraps calls inside the with block and accesses cb.total_cost after. The other options misuse the callback or access cost before running chains.

3. Final Answer:

   with get_openai_callback() as cb: chain.run(input1) chain.run(input2) print(cb.total_cost) -> Option A

4. Quick Check:

   Use with block for callback [OK]

Hint: Always use with to track cost across runs [OK]

Common Mistakes:

• Not using with block for callback
• Accessing cost before running chains
• Creating callback after chain runs

3. Given the code below, what will be printed?

with get_openai_callback() as cb:
    chain.run("Hello")
    chain.run("World")
print(cb.total_tokens)

medium

A. Zero, because tokens are not counted automatically

B. The total number of tokens used in both runs combined

C. The number of tokens used only in the last run

D. An error because total_tokens is not a valid attribute

Solution

1. Step 1: Understand token counting in callback

   Inside the with block, all calls to chain.run() accumulate token usage tracked by cb.

2. Step 2: Check what cb.total_tokens represents

   This attribute holds the total tokens used during the entire with block, so it sums tokens from both runs.

3. Final Answer:

   The total number of tokens used in both runs combined -> Option B

4. Quick Check:

   Total tokens = sum of all runs [OK]

Hint: Tokens add up inside the with block [OK]

Common Mistakes:

• Thinking it counts only last run tokens
• Assuming tokens are not tracked automatically
• Believing total_tokens is invalid

4. Identify the error in the following code snippet for tracking cost:

cb = get_openai_callback()
chain.run("Test")
print(cb.total_cost)

medium

A. The chain.run() method must be called before creating callback

B. The total_cost attribute does not exist on callback

C. Callback is not used as a context manager, so cost is not tracked

D. The print statement should be inside the callback block

Solution

1. Step 1: Check callback usage

   The callback must be used inside a with block to track usage properly. Here, it is created but not used as a context manager.

2. Step 2: Understand consequences

   Without the with block, the callback does not track token or cost usage, so total_cost will not reflect actual usage.

3. Final Answer:

   Callback is not used as a context manager, so cost is not tracked -> Option C

4. Quick Check:

   Use with for tracking [OK]

Hint: Always use with to activate callback tracking [OK]

Common Mistakes:

• Creating callback without with
• Assuming attributes exist without context
• Placing print outside tracking scope

5. You want to track token usage and cost for multiple LangChain runs, but also reset tracking between different user sessions. Which approach correctly achieves this?

hard

A. Use separate with get_openai_callback() as cb: blocks for each session

B. Create one callback outside all sessions and reuse it without resetting

C. Call get_openai_callback() once and manually reset cb.total_cost to zero

D. Track cost by printing cb.total_cost after each run without context manager

Solution

1. Step 1: Understand session-based tracking needs

   Each user session should have isolated cost tracking to avoid mixing token counts and costs.

2. Step 2: Evaluate approaches

   Using separate with blocks creates fresh callbacks per session, resetting counts automatically. Other options either reuse callbacks incorrectly or try manual resets which are unsupported.

3. Final Answer:

   Use separate with get_openai_callback() as cb: blocks for each session -> Option A

4. Quick Check:

   Separate with blocks reset tracking [OK]

Hint: Use new with block per session to reset cost [OK]

Common Mistakes:

• Reusing one callback for all sessions
• Trying to manually reset callback attributes
• Not using with blocks for tracking