Latency and cost benchmarking in Agentic AI - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Metrics

intermediate

Understanding latency measurement units

You run a latency benchmark on an AI model and get a result of 250 ms. What does this number represent?

A The size of the model in megabytes

B The total cost in dollars to run the model for one hour

C The number of inputs the model can process per second

D The time it takes for the model to process one input from start to finish

💡 Hint

Latency is about time, not cost or size.

Latency measures how long the model takes to respond to a single input, usually in milliseconds.

❓ Model Choice

intermediate

Choosing a model for low cost and moderate latency

You want to deploy an AI model that balances low cost and moderate latency for a chatbot. Which model type is best?

A A very large transformer model with billions of parameters

B A small transformer model optimized for fast inference

C A rule-based system with no machine learning

D A deep convolutional neural network designed for image tasks

💡 Hint

Smaller models usually cost less and run faster.

Small transformer models are designed to run efficiently with reasonable latency and cost, suitable for chatbots.

❓ Predict Output

advanced

Calculating average latency from benchmark data

What is the output of this Python code that calculates average latency in milliseconds?

Agentic AI

latencies = [120, 150, 130, 160, 140]
avg_latency = sum(latencies) / len(latencies)
print(f"Average latency: {avg_latency} ms")

A Average latency: 700 ms

B Average latency: 150 ms

C Average latency: 140.0 ms

D SyntaxError

💡 Hint

Sum all values and divide by count.

The sum is 700 and there are 5 values, so average is 700/5 = 140.0 ms.

🔧 Debug

advanced

Identifying the cause of high cost in benchmarking

You benchmarked two AI models and found one costs 10x more to run despite similar latency. What is the most likely cause?

A The expensive model uses more compute resources per request

B The cheaper model has higher latency

C The benchmarking code has a syntax error

D The expensive model is smaller in size

💡 Hint

Cost depends on compute usage, not just latency.

Higher cost usually comes from more compute power or memory used per request, even if latency is similar.

🧠 Conceptual

expert

Interpreting latency and cost trade-offs in AI deployment

Which statement best explains why reducing latency might increase cost in AI model deployment?

A Using more powerful hardware to reduce latency usually increases operational cost

B Reducing latency always reduces cost because the model runs faster

C Latency and cost are unrelated metrics in AI deployment

D Increasing latency reduces cost because it uses more resources

💡 Hint

Think about hardware and resource usage.

Faster hardware or more parallel processing reduces latency but costs more to operate.

Practice

1. What does latency measure when benchmarking an AI model?

easy

A. The cost to train the model

B. The amount of memory the model uses

C. The accuracy of the model's predictions

D. The time it takes for the model to respond

Solution

1. Step 1: Understand latency in AI benchmarking

   Latency refers to how long a model takes to give an answer after receiving input.

2. Step 2: Differentiate latency from other metrics

   Memory usage, accuracy, and training cost are different metrics; latency is about response time.

3. Final Answer:

   The time it takes for the model to respond -> Option D

4. Quick Check:

   Latency = response time [OK]

Hint: Latency means response speed, not memory or cost [OK]

Common Mistakes:

• Confusing latency with accuracy
• Thinking latency measures memory use
• Mixing latency with training cost

2. Which Python code snippet correctly measures latency of a model's prediction function model.predict()?

easy

A. start = time.time(); model.predict(); end = time.time(); latency = end - start

B. latency = model.predict().time()

C. latency = time.predict(model)

D. latency = model.time() - predict.time()

Solution

1. Step 1: Identify correct timing method in Python

   Using time.time() before and after calling model.predict() measures elapsed time correctly.

2. Step 2: Check incorrect options for syntax errors

   Options A, B, and D use invalid method calls or wrong order, so they won't work.

3. Final Answer:

   start = time.time(); model.predict(); end = time.time(); latency = end - start -> Option A

4. Quick Check:

   Use time.time() before and after call [OK]

Hint: Use time.time() before and after prediction call [OK]

Common Mistakes:

• Calling non-existent methods like predict.time()
• Subtracting wrong attributes
• Not capturing time before and after prediction

3. Given this code measuring latency and cost, what is the printed output?

import time

start = time.time()
model_response = model.predict(input_data)
end = time.time()
latency = end - start
cost = latency * 0.05  # cost per second
print(round(latency, 2), round(cost, 3))

If model.predict() takes 0.24 seconds, what prints?

medium

A. 0.24 0.012

B. 0.24 0.12

C. 0.24 0.0012

D. 0.24 0.024

Solution

1. Step 1: Calculate latency and cost

   Latency is 0.24 seconds. Cost = latency * 0.05 = 0.24 * 0.05 = 0.012.

2. Step 2: Round values as printed

   Latency rounded to 2 decimals is 0.24. Cost rounded to 3 decimals is 0.012.

3. Final Answer:

   0.24 0.012 -> Option A

4. Quick Check:

   Cost = latency * 0.05 = 0.012 [OK]

Hint: Multiply latency by cost rate, then round [OK]

Common Mistakes:

• Multiplying cost by 10 or 100 by mistake
• Rounding cost incorrectly
• Confusing latency and cost values

4. This code tries to measure latency but gives wrong results. What is the bug?

import time
start = time.time()
model.predict(input_data)
latency = time.time() - start
print('Latency:', latency)

medium

A. The model.predict call is missing parentheses

B. The code does not import the model

C. Latency is measured correctly; no bug

D. Latency should be measured before calling model.predict

Solution

1. Step 1: Check timing logic

   The code records time before and after model.predict(input_data), then subtracts to get latency.

2. Step 2: Verify correctness of measurement

   This is the correct way to measure latency; parentheses are present and timing is after call.

3. Final Answer:

   Latency is measured correctly; no bug -> Option C

4. Quick Check:

   Start time before, end time after call [OK]

Hint: Latency = end time minus start time around call [OK]

Common Mistakes:

• Measuring time before call only
• Forgetting parentheses on function call
• Measuring latency after print statement

5. You want to compare two AI models for latency and cost. Model A takes 0.3 seconds per prediction and costs $0.04 per second. Model B takes 0.25 seconds but costs $0.06 per second. Which model is cheaper per prediction and which is faster?

hard

A. Model A is cheaper and faster; Model B is slower and more expensive

B. Model A is cheaper and slower; Model B is faster and more expensive

C. Model B is cheaper and slower; Model A is faster and more expensive

D. Model B is cheaper and faster; Model A is slower and more expensive

Solution

1. Step 1: Calculate cost per prediction for each model

   Model A cost = 0.3 * 0.04 = $0.012; Model B cost = 0.25 * 0.06 = $0.015.

2. Step 2: Compare latency and cost

   Model A is cheaper ($0.012 < $0.015) but slower (0.3s > 0.25s). Model B is faster but more expensive.

3. Final Answer:

   Model A is cheaper and slower; Model B is faster and more expensive -> Option B

4. Quick Check:

   Cost = latency * rate; compare values [OK]

Hint: Multiply latency by cost rate to compare total cost [OK]

Common Mistakes:

• Ignoring cost per second rate
• Mixing up which model is faster
• Calculating cost incorrectly