Token usage and cost tracking in Agentic Ai - Model Metrics & Evaluation

Metrics & Evaluation - Token usage and cost tracking

Which metric matters for Token usage and cost tracking and WHY

When working with AI models that use tokens, the key metrics are token count and cost per token. Token count measures how many pieces of text the model processes. Cost per token tells us how much each token costs to use. Tracking these helps us control expenses and optimize usage. For example, if a chatbot uses too many tokens, the cost can grow quickly. So, knowing token usage helps keep the project affordable and efficient.

Confusion matrix or equivalent visualization

Token usage does not use a confusion matrix like classification tasks. Instead, we track token counts in categories such as:

    +-------------------+------------+
    | Token Type        | Count      |
    +-------------------+------------+
    | Prompt tokens     | 1,200      |
    | Completion tokens | 800        |
    | Total tokens      | 2,000      |
    +-------------------+------------+
    

This table helps visualize how tokens are split between input (prompt) and output (completion), which affects cost.

Precision vs Recall tradeoff (or equivalent) with concrete examples

In token usage, the tradeoff is between model performance and cost. Using more tokens can improve answers but costs more money. Using fewer tokens saves money but may reduce quality.

Example: A chatbot that answers questions with long, detailed replies uses many tokens (high cost). If you limit tokens, replies are shorter and cheaper but might miss details.

Balancing token usage means finding the sweet spot where answers are good enough without overspending.

What "good" vs "bad" metric values look like for this use case

Good token usage: Total tokens per request are low enough to keep costs manageable, while still delivering useful responses. For example, 500-1000 tokens per interaction with clear answers.

Bad token usage: Excessive tokens per request (e.g., 5000+ tokens) causing high costs without much improvement in response quality. Or very low tokens that make answers incomplete or confusing.

Metrics pitfalls

• Ignoring token split: Not separating prompt and completion tokens can hide where costs come from.
• Overlooking hidden tokens: Some systems add tokens for system messages or formatting, increasing cost unexpectedly.
• Not tracking usage over time: Costs can spike if token usage grows unnoticed.
• Assuming more tokens always mean better results: Sometimes shorter prompts with fewer tokens work just as well.

Self-check question

Your AI model uses 10,000 tokens per request and costs $0.02 per 1,000 tokens. You want to reduce costs but keep good answers. What should you do?

Answer: Try reducing tokens per request by shortening prompts or limiting completion length. Monitor if answer quality stays acceptable. This balances cost and performance.

Key Result

Tracking token counts and cost per token helps balance AI model performance with budget control.