GPU vs CPU inference tradeoffs in MLOps - Practice Questions

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding latency differences between GPU and CPU inference

Which statement best explains why GPU inference can have higher latency for small batch sizes compared to CPU inference?

A GPUs have higher clock speeds than CPUs, causing delays in small tasks.

B CPUs use more cores than GPUs, making them faster for all batch sizes.

C GPUs require data transfer overhead and kernel launch time, which dominate at small batch sizes.

D GPUs cannot run inference on neural networks with small input sizes.

💡 Hint

Think about the extra steps GPUs need before starting computation.

GPUs need to transfer data from CPU memory and launch compute kernels. For small batches, this overhead can be larger than the actual compute time, causing higher latency than CPUs.

💻 Command Output

intermediate

Comparing CPU and GPU memory usage during inference

Given the command outputs below showing memory usage during inference, which output corresponds to GPU inference?

MLOps

CPU memory usage: 2.5 GB
GPU memory usage: 6.8 GB

A CPU memory usage: 2.5 GB

B GPU memory usage: 2.5 GB

C CPU memory usage: 6.8 GB

D GPU memory usage: 6.8 GB

💡 Hint

GPUs usually allocate more dedicated memory for model weights and activations.

GPU inference typically uses more dedicated memory on the GPU device, often higher than CPU RAM usage for the same model.

🔀 Workflow

advanced

Optimizing inference deployment for mixed CPU/GPU environments

You have a service that must handle both low-latency single requests and high-throughput batch requests. Which deployment strategy best balances GPU and CPU usage?

A Use CPU only to avoid GPU overhead and simplify deployment.

B Use CPU for single requests and GPU for batch requests to optimize latency and throughput.

C Route all requests to GPU to maximize throughput, ignoring latency.

D Use GPU only for single requests and CPU for batch requests.

💡 Hint

Consider the strengths of CPU and GPU for different request sizes.

CPUs handle small, low-latency requests efficiently without GPU overhead. GPUs excel at large batches for throughput. Combining both optimizes overall performance.

❓ Troubleshoot

advanced

Diagnosing slow GPU inference despite high GPU utilization

Your GPU inference shows high GPU utilization but slow overall response time. What is the most likely cause?

A Data transfer between CPU and GPU is a bottleneck causing delays.

B The model is too small to benefit from GPU acceleration.

C GPU drivers are outdated causing incorrect utilization reporting.

D The batch size is too large causing GPU memory overflow.

💡 Hint

High GPU usage but slow response often means waiting on data movement.

Even with high GPU usage, slow response can happen if data transfer from CPU to GPU is slow, causing the GPU to wait and delaying inference.

✅ Best Practice

expert

Choosing hardware for cost-effective inference at scale

You manage a cloud deployment for ML inference with fluctuating demand. Which approach best balances cost and performance?

A Use CPU instances for low demand and scale GPU instances only when batch sizes increase.

B Always use GPU instances to maximize speed regardless of cost.

C Use only CPU instances to minimize complexity and cost.

D Use GPU instances for all requests to simplify autoscaling.

💡 Hint

Think about matching hardware to workload patterns to save money.

Using CPUs for low demand avoids expensive GPU costs. Scaling GPUs only when batch sizes grow balances cost and performance effectively.