Bird
Raised Fist0
Agentic AIml~8 mins

Token usage and cost tracking in Agentic AI - Model Metrics & Evaluation

Choose your learning style10 modes available
LearnWhyDeepModelTryChallengeExperimentRecallMetrics

Start learning this pattern below

Jump into concepts and practice - no test required

or
Recommended
Test this pattern10 questions across easy, medium, and hard to know if this pattern is strong
Metrics & Evaluation - Token usage and cost tracking
Which metric matters for Token usage and cost tracking and WHY

When working with AI models that use tokens, the key metrics are token count and cost per token. Token count measures how many pieces of text the model processes. Cost per token tells us how much each token costs to use. Tracking these helps us control expenses and optimize usage. For example, if a chatbot uses too many tokens, the cost can grow quickly. So, knowing token usage helps keep the project affordable and efficient.

Confusion matrix or equivalent visualization

Token usage does not use a confusion matrix like classification tasks. Instead, we track token counts in categories such as:

    +-------------------+------------+
    | Token Type        | Count      |
    +-------------------+------------+
    | Prompt tokens     | 1,200      |
    | Completion tokens | 800        |
    | Total tokens      | 2,000      |
    +-------------------+------------+

This table helps visualize how tokens are split between input (prompt) and output (completion), which affects cost.

Precision vs Recall tradeoff (or equivalent) with concrete examples

In token usage, the tradeoff is between model performance and cost. Using more tokens can improve answers but costs more money. Using fewer tokens saves money but may reduce quality.

Example: A chatbot that answers questions with long, detailed replies uses many tokens (high cost). If you limit tokens, replies are shorter and cheaper but might miss details.

Balancing token usage means finding the sweet spot where answers are good enough without overspending.

What "good" vs "bad" metric values look like for this use case

Good token usage: Total tokens per request are low enough to keep costs manageable, while still delivering useful responses. For example, 500-1000 tokens per interaction with clear answers.

Bad token usage: Excessive tokens per request (e.g., 5000+ tokens) causing high costs without much improvement in response quality. Or very low tokens that make answers incomplete or confusing.

Metrics pitfalls
  • Ignoring token split: Not separating prompt and completion tokens can hide where costs come from.
  • Overlooking hidden tokens: Some systems add tokens for system messages or formatting, increasing cost unexpectedly.
  • Not tracking usage over time: Costs can spike if token usage grows unnoticed.
  • Assuming more tokens always mean better results: Sometimes shorter prompts with fewer tokens work just as well.
Self-check question

Your AI model uses 10,000 tokens per request and costs $0.02 per 1,000 tokens. You want to reduce costs but keep good answers. What should you do?

Answer: Try reducing tokens per request by shortening prompts or limiting completion length. Monitor if answer quality stays acceptable. This balances cost and performance.

Key Result
Tracking token counts and cost per token helps balance AI model performance with budget control.

Practice

(1/5)
1. What is a token in the context of AI language models?
easy
A. A programming language used for AI
B. A type of AI model architecture
C. A small piece of text like a word or part of a word
D. A hardware component for AI processing

Solution

  1. Step 1: Understand the definition of a token

    A token is a small piece of text that AI models use to understand language. It can be a word or part of a word.

  2. Step 2: Differentiate tokens from other AI terms

    Tokens are not models, programming languages, or hardware. They are units of text input.

  3. Final Answer:

    A small piece of text like a word or part of a word -> Option C

  4. Quick Check:

    Token = small text piece [OK]
Hint: Tokens are text pieces, not models or hardware [OK]
Common Mistakes:
  • Confusing tokens with AI models
  • Thinking tokens are programming languages
  • Assuming tokens are hardware parts
2. Which of the following is the correct way to track token usage in a Python script using an AI API?
easy
A. tokens_used = response['total_tokens']
B. tokens_used = response['usage']['total_tokens']
C. tokens_used = response['usage']['tokens']
D. tokens_used = response['token_count']

Solution

  1. Step 1: Identify the correct key for token usage in API response

    Most AI APIs return token usage under response['usage']['total_tokens'].

  2. Step 2: Compare options with common API response structure

    Only tokens_used = response['usage']['total_tokens'] matches the standard nested key for total tokens used.

  3. Final Answer:

    tokens_used = response['usage']['total_tokens'] -> Option B

  4. Quick Check:

    Correct key path = response['usage']['total_tokens'] [OK]
Hint: Look for nested 'usage' then 'total_tokens' key [OK]
Common Mistakes:
  • Using wrong key names like 'token_count'
  • Missing nested 'usage' dictionary
  • Assuming flat keys for token counts
3. Given the following code snippet, what will be printed? 
response = {'usage': {'prompt_tokens': 50, 'completion_tokens': 30, 'total_tokens': 80}}
print(response['usage']['total_tokens'])
medium
A. 80
B. 30
C. 50
D. Error

Solution

  1. Step 1: Access the 'total_tokens' key in the nested dictionary

    The code accesses response['usage']['total_tokens'], which is 80.

  2. Step 2: Confirm the print output

    Printing this value outputs 80 without error.

  3. Final Answer:

    80 -> Option A

  4. Quick Check:

    response['usage']['total_tokens'] = 80 [OK]
Hint: Check nested dictionary keys carefully [OK]
Common Mistakes:
  • Confusing prompt_tokens with total_tokens
  • Mixing completion_tokens with total_tokens
  • Assuming code causes error
4. You wrote this code to track token usage but get a KeyError: 
tokens = response['usage']['token_total']
print(tokens)
What is the likely cause?
medium
A. The 'usage' key is missing in the response
B. The response variable is not defined
C. The print statement syntax is incorrect
D. The key 'token_total' does not exist in the response dictionary

Solution

  1. Step 1: Check the key names in the response dictionary

    The correct key is 'total_tokens', not 'token_total'.

  2. Step 2: Understand KeyError cause

    Using a wrong key name causes KeyError because that key does not exist.

  3. Final Answer:

    The key 'token_total' does not exist in the response dictionary -> Option D

  4. Quick Check:

    Wrong key name causes KeyError [OK]
Hint: Verify exact key names in API response [OK]
Common Mistakes:
  • Assuming similar key names exist
  • Ignoring case sensitivity in keys
  • Thinking print syntax causes error
5. You want to estimate the cost of an AI request. The model charges $0.002 per 1000 tokens. If your request uses 2500 tokens, what is the total cost?
hard
A. $0.005
B. $0.0025
C. $0.05
D. $0.0005

Solution

  1. Step 1: Calculate cost per token

    Cost per token = $0.002 / 1000 = $0.000002 per token.

  2. Step 2: Multiply by number of tokens used

    Total cost = 2500 tokens * $0.000002 = $0.005.

  3. Final Answer:

    $0.005 -> Option A

  4. Quick Check:

    2500 tokens x $0.002/1000 = $0.005 [OK]
Hint: Multiply tokens by cost per token (divide by 1000 first) [OK]
Common Mistakes:
  • Multiplying by 0.002 directly without dividing by 1000
  • Using wrong token count
  • Confusing decimals in cost calculation