GPU vs CPU inference tradeoffs in MLOps - Hands-On Comparison

GPU vs CPU Inference Tradeoffs

📖 Scenario: You work in a company that deploys machine learning models. You want to understand how using a GPU or a CPU affects the speed of running predictions (inference) on a model. This helps decide which hardware to use for your app.

🎯 Goal: Build a simple Python script that simulates inference times on CPU and GPU, compares them, and prints which hardware is faster for the given batch size.

📋 What You'll Learn

Create a dictionary with exact inference times (in milliseconds) for CPU and GPU for batch sizes 1, 10, and 100.

Add a variable to select the batch size to test.

Write code to pick the inference time for the selected batch size and hardware.

Print the inference times and which hardware is faster.

💡 Why This Matters

🌍 Real World

In real machine learning deployments, choosing between CPU and GPU for inference affects cost, speed, and user experience. This project helps understand those tradeoffs.

💼 Career

DevOps and MLOps engineers often decide hardware for model serving. Knowing how to compare inference times helps optimize resources and performance.

1

Create inference times dictionary

Create a dictionary called inference_times with keys 'CPU' and 'GPU'. Each key maps to another dictionary with batch sizes 1, 10, and 100 as keys and these exact values (in milliseconds):
CPU: {1: 50, 10: 400, 100: 3500}
GPU: {1: 30, 10: 100, 100: 800}

MLOps

# Create the inference_times dictionary with CPU and GPU batch times
# Your code here

Hint

Use nested dictionaries. The outer keys are 'CPU' and 'GPU'. The inner keys are batch sizes 1, 10, and 100 with given values.

2

Set batch size variable

Create a variable called batch_size and set it to 10.

MLOps

inference_times = {
    'CPU': {1: 50, 10: 400, 100: 3500},
    'GPU': {1: 30, 10: 100, 100: 800}
}
# Set the batch_size variable
# Your code here

Hint

Just assign the number 10 to the variable named batch_size.

3

Select inference times for batch size

Create two variables called cpu_time and gpu_time. Set cpu_time to the CPU inference time for batch_size from inference_times. Set gpu_time to the GPU inference time for batch_size from inference_times.

MLOps

inference_times = {
    'CPU': {1: 50, 10: 400, 100: 3500},
    'GPU': {1: 30, 10: 100, 100: 800}
}
batch_size = 10
# Get cpu_time and gpu_time from inference_times for batch_size
# Your code here

Hint

Use dictionary access with keys 'CPU' and 'GPU' and then the batch_size variable.

4

Print inference times and faster hardware

Print the CPU and GPU inference times in milliseconds using print. Then print which hardware is faster for the selected batch_size. Use this exact format:
"CPU time: X ms"
"GPU time: Y ms"
"Faster hardware: Z"
where X and Y are the times and Z is either CPU or GPU.

MLOps

inference_times = {
    'CPU': {1: 50, 10: 400, 100: 3500},
    'GPU': {1: 30, 10: 100, 100: 800}
}
batch_size = 10
cpu_time = inference_times['CPU'][batch_size]
gpu_time = inference_times['GPU'][batch_size]
# Print CPU and GPU times and which is faster
# Your code here

Hint

Use print statements with f-strings. Compare cpu_time and gpu_time to find the faster hardware.

Practice

1. Which of the following is a main advantage of using a GPU over a CPU for machine learning inference?

easy

A. Lower power consumption for small tasks

B. Cheaper hardware cost

C. Better performance on single-threaded tasks

D. Faster processing for large batches of data

Solution

1. Step 1: Understand GPU design for parallelism

   GPUs have many cores designed to handle many operations at once, making them faster for large data batches.

2. Step 2: Compare CPU and GPU strengths

   CPUs are better for single-threaded or small tasks, but GPUs excel at parallel processing, speeding up large inference jobs.

3. Final Answer:

   Faster processing for large batches of data -> Option D

4. Quick Check:

   GPU parallelism = Faster large batch inference [OK]

Hint: GPUs excel at many tasks at once, CPUs at few tasks fast [OK]

Common Mistakes:

• Thinking GPUs always use less power
• Assuming CPUs are cheaper for large-scale inference
• Confusing single-threaded speed with parallel speed

2. Which command correctly runs a TensorFlow model inference on CPU only, ignoring GPUs?

easy

A. CUDA_VISIBLE_DEVICES=0 python inference.py

B. CUDA_VISIBLE_DEVICES='' python inference.py

C. CUDA_VISIBLE_DEVICES=-1 python inference.py

D. CUDA_VISIBLE_DEVICES=all python inference.py

Solution

1. Step 1: Understand CUDA_VISIBLE_DEVICES usage

   Setting CUDA_VISIBLE_DEVICES to an empty string disables GPU visibility, forcing CPU usage.

2. Step 2: Check each option's effect

   CUDA_VISIBLE_DEVICES='' python inference.py disables GPUs correctly; others either select GPUs or use invalid values.

3. Final Answer:

   CUDA_VISIBLE_DEVICES='' python inference.py -> Option B

4. Quick Check:

   Empty CUDA_VISIBLE_DEVICES disables GPU [OK]

Hint: Empty CUDA_VISIBLE_DEVICES means no GPU used [OK]

Common Mistakes:

• Using 0 disables only GPU 0, not all GPUs
• Using -1 is invalid for CUDA_VISIBLE_DEVICES
• Assuming 'all' enables all GPUs but not disables

3. Given this Python snippet for inference timing:

import time
start = time.time()
# Run model inference here
end = time.time()
print(round(end - start, 2))

If GPU inference takes 0.05 seconds and CPU inference takes 0.5 seconds, what will be printed when running on CPU?

medium

A. 0.05

B. 50.0

C. 0.5

D. 5.0

Solution

1. Step 1: Understand timing code output

   The code prints the elapsed time rounded to 2 decimals, so it shows seconds taken.

2. Step 2: Match CPU inference time to output

   CPU inference takes 0.5 seconds, so the printed output is 0.5.

3. Final Answer:

   0.5 -> Option C

4. Quick Check:

   CPU time = 0.5 seconds printed [OK]

Hint: Printed time matches actual elapsed seconds rounded [OK]

Common Mistakes:

• Confusing milliseconds with seconds
• Choosing GPU time instead of CPU time
• Misreading rounding precision

4. You run inference on a GPU but notice it is slower than CPU. Which fix is most likely to improve GPU inference speed?

medium

A. Increase batch size to better use GPU parallelism

B. Reduce batch size to avoid GPU overload

C. Disable GPU and force CPU usage

D. Use single-threaded CPU mode

Solution

1. Step 1: Identify GPU performance factors

   GPUs perform best with larger batch sizes to utilize many cores efficiently.

2. Step 2: Evaluate options for improving GPU speed

   Increasing batch size improves GPU throughput; reducing batch size or disabling GPU lowers performance.

3. Final Answer:

   Increase batch size to better use GPU parallelism -> Option A

4. Quick Check:

   GPU speed improves with larger batches [OK]

Hint: Bigger batches = better GPU use [OK]

Common Mistakes:

• Thinking smaller batches speed up GPU
• Disabling GPU to fix GPU slowness
• Using single-thread CPU instead of GPU

5. You have a small model and low input volume but a tight budget. Which inference setup is best to minimize cost while maintaining reasonable speed?

hard

A. Use CPU inference with small batch sizes

B. Use GPU inference with large batch sizes

C. Use GPU inference with small batch sizes

D. Use CPU inference with large batch sizes

Solution

1. Step 1: Analyze model size and input volume impact

   Small models and low input do not benefit much from GPU parallelism, so GPU cost is less justified.

2. Step 2: Consider budget and batch size tradeoffs

   CPU inference with small batches reduces cost and matches low volume needs without GPU overhead.

3. Final Answer:

   Use CPU inference with small batch sizes -> Option A

4. Quick Check:

   Small model + low volume + budget = CPU small batch [OK]

Hint: Small model + low volume = CPU for cost savings [OK]

Common Mistakes:

• Choosing GPU despite low volume and budget
• Using large batches on CPU causing delays
• Ignoring cost when selecting GPU