Why serving architecture affects latency and cost in MLOps - See It in Action

Why Serving Architecture Affects Latency and Cost

📖 Scenario: You work in a team that deploys machine learning models to serve predictions to users. Your team wants to understand how different serving architectures impact the speed of responses (latency) and the money spent (cost).Imagine you have two ways to serve a model: one that handles requests one by one (simple server), and another that batches requests together to save resources.

🎯 Goal: Build a simple Python simulation that models request handling in two serving architectures. You will create data for requests, configure batch size, apply logic to simulate processing time, and output the average latency and estimated cost for each architecture.

📋 What You'll Learn

Create a list of exactly 10 request processing times in milliseconds

Add a configuration variable called batch_size with value 3

Write code to calculate average latency for simple and batch serving

Print the average latency and estimated cost for both architectures

💡 Why This Matters

🌍 Real World

In real machine learning deployments, choosing how to serve models affects how fast users get predictions and how much cloud resources cost.

💼 Career

Understanding serving architectures helps DevOps and MLOps engineers optimize performance and budget in production systems.

1

Create request processing times list

Create a list called request_times with these exact values in milliseconds: [120, 150, 100, 130, 110, 140, 115, 125, 135, 105]

MLOps

# Create the request_times list with given values
# Your code here

Need a hint?

Use square brackets and separate numbers with commas exactly as shown.

2

Add batch size configuration

Add a variable called batch_size and set it to 3

MLOps

request_times = [120, 150, 100, 130, 110, 140, 115, 125, 135, 105]
# Set batch_size to 3
# Your code here

Need a hint?

Just assign the number 3 to the variable named batch_size.

3

Calculate average latency for simple and batch serving

Write code to calculate simple_avg_latency as the average of all request_times, and batch_avg_latency by grouping request_times into batches of size batch_size, taking the maximum time in each batch as batch processing time, then averaging these batch times.

MLOps

request_times = [120, 150, 100, 130, 110, 140, 115, 125, 135, 105]
batch_size = 3
# Calculate simple_avg_latency and batch_avg_latency
# Your code here

Need a hint?

Use list slicing and max() to find batch times, then average them.

4

Print average latency and estimated cost

Print the average latency for simple serving as Simple Avg Latency: X ms and for batch serving as Batch Avg Latency: Y ms. Then print estimated cost assuming simple serving costs $0.10 per request and batch serving costs $0.25 per batch, formatted as Simple Cost: $Z and Batch Cost: $W. Use two decimal places for costs.

MLOps

request_times = [120, 150, 100, 130, 110, 140, 115, 125, 135, 105]
batch_size = 3
simple_avg_latency = sum(request_times) / len(request_times)
batches = [request_times[i:i+batch_size] for i in range(0, len(request_times), batch_size)]
batch_times = [max(batch) for batch in batches]
batch_avg_latency = sum(batch_times) / len(batch_times)
# Print average latencies and costs
# Your code here

Need a hint?

Calculate costs by multiplying per-request or per-batch rates. Use f-strings to format output.