GPU vs CPU inference tradeoffs in MLOps - Hands-On Comparison

GPU vs CPU Inference Tradeoffs

📖 Scenario: You work in a company that deploys machine learning models. You want to understand how using a GPU or a CPU affects the speed of running predictions (inference) on a model. This helps decide which hardware to use for your app.

🎯 Goal: Build a simple Python script that simulates inference times on CPU and GPU, compares them, and prints which hardware is faster for the given batch size.

📋 What You'll Learn

Create a dictionary with exact inference times (in milliseconds) for CPU and GPU for batch sizes 1, 10, and 100.

Add a variable to select the batch size to test.

Write code to pick the inference time for the selected batch size and hardware.

Print the inference times and which hardware is faster.

💡 Why This Matters

🌍 Real World

In real machine learning deployments, choosing between CPU and GPU for inference affects cost, speed, and user experience. This project helps understand those tradeoffs.

💼 Career

DevOps and MLOps engineers often decide hardware for model serving. Knowing how to compare inference times helps optimize resources and performance.

1

Create inference times dictionary

Create a dictionary called inference_times with keys 'CPU' and 'GPU'. Each key maps to another dictionary with batch sizes 1, 10, and 100 as keys and these exact values (in milliseconds):
CPU: {1: 50, 10: 400, 100: 3500}
GPU: {1: 30, 10: 100, 100: 800}

MLOps

# Create the inference_times dictionary with CPU and GPU batch times
# Your code here

Need a hint?

Use nested dictionaries. The outer keys are 'CPU' and 'GPU'. The inner keys are batch sizes 1, 10, and 100 with given values.

2

Set batch size variable

Create a variable called batch_size and set it to 10.

MLOps

inference_times = {
    'CPU': {1: 50, 10: 400, 100: 3500},
    'GPU': {1: 30, 10: 100, 100: 800}
}
# Set the batch_size variable
# Your code here

Need a hint?

Just assign the number 10 to the variable named batch_size.

3

Select inference times for batch size

Create two variables called cpu_time and gpu_time. Set cpu_time to the CPU inference time for batch_size from inference_times. Set gpu_time to the GPU inference time for batch_size from inference_times.

MLOps

inference_times = {
    'CPU': {1: 50, 10: 400, 100: 3500},
    'GPU': {1: 30, 10: 100, 100: 800}
}
batch_size = 10
# Get cpu_time and gpu_time from inference_times for batch_size
# Your code here

Need a hint?

Use dictionary access with keys 'CPU' and 'GPU' and then the batch_size variable.

4

Print inference times and faster hardware

Print the CPU and GPU inference times in milliseconds using print. Then print which hardware is faster for the selected batch_size. Use this exact format:
"CPU time: X ms"
"GPU time: Y ms"
"Faster hardware: Z"
where X and Y are the times and Z is either CPU or GPU.

MLOps

inference_times = {
    'CPU': {1: 50, 10: 400, 100: 3500},
    'GPU': {1: 30, 10: 100, 100: 800}
}
batch_size = 10
cpu_time = inference_times['CPU'][batch_size]
gpu_time = inference_times['GPU'][batch_size]
# Print CPU and GPU times and which is faster
# Your code here

Need a hint?

Use print statements with f-strings. Compare cpu_time and gpu_time to find the faster hardware.