GPU vs CPU inference tradeoffs in MLOps - Practice Questions

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding latency differences between GPU and CPU inference

Which statement best explains why GPU inference can have higher latency for small batch sizes compared to CPU inference?

A GPUs have higher clock speeds than CPUs, causing delays in small tasks.

B CPUs use more cores than GPUs, making them faster for all batch sizes.

C GPUs require data transfer overhead and kernel launch time, which dominate at small batch sizes.

D GPUs cannot run inference on neural networks with small input sizes.

💡 Hint

Think about the extra steps GPUs need before starting computation.

GPUs need to transfer data from CPU memory and launch compute kernels. For small batches, this overhead can be larger than the actual compute time, causing higher latency than CPUs.

💻 Command Output

intermediate

Comparing CPU and GPU memory usage during inference

Given the command outputs below showing memory usage during inference, which output corresponds to GPU inference?

MLOps

CPU memory usage: 2.5 GB
GPU memory usage: 6.8 GB

A CPU memory usage: 2.5 GB

B GPU memory usage: 2.5 GB

C CPU memory usage: 6.8 GB

D GPU memory usage: 6.8 GB

💡 Hint

GPUs usually allocate more dedicated memory for model weights and activations.

GPU inference typically uses more dedicated memory on the GPU device, often higher than CPU RAM usage for the same model.

🔀 Workflow

advanced

Optimizing inference deployment for mixed CPU/GPU environments

You have a service that must handle both low-latency single requests and high-throughput batch requests. Which deployment strategy best balances GPU and CPU usage?

A Use CPU only to avoid GPU overhead and simplify deployment.

B Use CPU for single requests and GPU for batch requests to optimize latency and throughput.

C Route all requests to GPU to maximize throughput, ignoring latency.

D Use GPU only for single requests and CPU for batch requests.

💡 Hint

Consider the strengths of CPU and GPU for different request sizes.

CPUs handle small, low-latency requests efficiently without GPU overhead. GPUs excel at large batches for throughput. Combining both optimizes overall performance.

❓ Troubleshoot

advanced

Diagnosing slow GPU inference despite high GPU utilization

Your GPU inference shows high GPU utilization but slow overall response time. What is the most likely cause?

A Data transfer between CPU and GPU is a bottleneck causing delays.

B The model is too small to benefit from GPU acceleration.

C GPU drivers are outdated causing incorrect utilization reporting.

D The batch size is too large causing GPU memory overflow.

💡 Hint

High GPU usage but slow response often means waiting on data movement.

Even with high GPU usage, slow response can happen if data transfer from CPU to GPU is slow, causing the GPU to wait and delaying inference.

✅ Best Practice

expert

Choosing hardware for cost-effective inference at scale

You manage a cloud deployment for ML inference with fluctuating demand. Which approach best balances cost and performance?

A Use CPU instances for low demand and scale GPU instances only when batch sizes increase.

B Always use GPU instances to maximize speed regardless of cost.

C Use only CPU instances to minimize complexity and cost.

D Use GPU instances for all requests to simplify autoscaling.

💡 Hint

Think about matching hardware to workload patterns to save money.

Using CPUs for low demand avoids expensive GPU costs. Scaling GPUs only when batch sizes grow balances cost and performance effectively.

Practice

1. Which of the following is a main advantage of using a GPU over a CPU for machine learning inference?

easy

A. Lower power consumption for small tasks

B. Cheaper hardware cost

C. Better performance on single-threaded tasks

D. Faster processing for large batches of data

Solution

1. Step 1: Understand GPU design for parallelism

   GPUs have many cores designed to handle many operations at once, making them faster for large data batches.

2. Step 2: Compare CPU and GPU strengths

   CPUs are better for single-threaded or small tasks, but GPUs excel at parallel processing, speeding up large inference jobs.

3. Final Answer:

   Faster processing for large batches of data -> Option D

4. Quick Check:

   GPU parallelism = Faster large batch inference [OK]

Hint: GPUs excel at many tasks at once, CPUs at few tasks fast [OK]

Common Mistakes:

• Thinking GPUs always use less power
• Assuming CPUs are cheaper for large-scale inference
• Confusing single-threaded speed with parallel speed

2. Which command correctly runs a TensorFlow model inference on CPU only, ignoring GPUs?

easy

A. CUDA_VISIBLE_DEVICES=0 python inference.py

B. CUDA_VISIBLE_DEVICES='' python inference.py

C. CUDA_VISIBLE_DEVICES=-1 python inference.py

D. CUDA_VISIBLE_DEVICES=all python inference.py

Solution

1. Step 1: Understand CUDA_VISIBLE_DEVICES usage

   Setting CUDA_VISIBLE_DEVICES to an empty string disables GPU visibility, forcing CPU usage.

2. Step 2: Check each option's effect

   CUDA_VISIBLE_DEVICES='' python inference.py disables GPUs correctly; others either select GPUs or use invalid values.

3. Final Answer:

   CUDA_VISIBLE_DEVICES='' python inference.py -> Option B

4. Quick Check:

   Empty CUDA_VISIBLE_DEVICES disables GPU [OK]

Hint: Empty CUDA_VISIBLE_DEVICES means no GPU used [OK]

Common Mistakes:

• Using 0 disables only GPU 0, not all GPUs
• Using -1 is invalid for CUDA_VISIBLE_DEVICES
• Assuming 'all' enables all GPUs but not disables

3. Given this Python snippet for inference timing:

import time
start = time.time()
# Run model inference here
end = time.time()
print(round(end - start, 2))

If GPU inference takes 0.05 seconds and CPU inference takes 0.5 seconds, what will be printed when running on CPU?

medium

A. 0.05

B. 50.0

C. 0.5

D. 5.0

Solution

1. Step 1: Understand timing code output

   The code prints the elapsed time rounded to 2 decimals, so it shows seconds taken.

2. Step 2: Match CPU inference time to output

   CPU inference takes 0.5 seconds, so the printed output is 0.5.

3. Final Answer:

   0.5 -> Option C

4. Quick Check:

   CPU time = 0.5 seconds printed [OK]

Hint: Printed time matches actual elapsed seconds rounded [OK]

Common Mistakes:

• Confusing milliseconds with seconds
• Choosing GPU time instead of CPU time
• Misreading rounding precision

4. You run inference on a GPU but notice it is slower than CPU. Which fix is most likely to improve GPU inference speed?

medium

A. Increase batch size to better use GPU parallelism

B. Reduce batch size to avoid GPU overload

C. Disable GPU and force CPU usage

D. Use single-threaded CPU mode

Solution

1. Step 1: Identify GPU performance factors

   GPUs perform best with larger batch sizes to utilize many cores efficiently.

2. Step 2: Evaluate options for improving GPU speed

   Increasing batch size improves GPU throughput; reducing batch size or disabling GPU lowers performance.

3. Final Answer:

   Increase batch size to better use GPU parallelism -> Option A

4. Quick Check:

   GPU speed improves with larger batches [OK]

Hint: Bigger batches = better GPU use [OK]

Common Mistakes:

• Thinking smaller batches speed up GPU
• Disabling GPU to fix GPU slowness
• Using single-thread CPU instead of GPU

5. You have a small model and low input volume but a tight budget. Which inference setup is best to minimize cost while maintaining reasonable speed?

hard

A. Use CPU inference with small batch sizes

B. Use GPU inference with large batch sizes

C. Use GPU inference with small batch sizes

D. Use CPU inference with large batch sizes

Solution

1. Step 1: Analyze model size and input volume impact

   Small models and low input do not benefit much from GPU parallelism, so GPU cost is less justified.

2. Step 2: Consider budget and batch size tradeoffs

   CPU inference with small batches reduces cost and matches low volume needs without GPU overhead.

3. Final Answer:

   Use CPU inference with small batch sizes -> Option A

4. Quick Check:

   Small model + low volume + budget = CPU small batch [OK]

Hint: Small model + low volume = CPU for cost savings [OK]

Common Mistakes:

• Choosing GPU despite low volume and budget
• Using large batches on CPU causing delays
• Ignoring cost when selecting GPU