Bird
Raised Fist0
MLOpsdevops~5 mins

GPU support in containers in MLOps - Commands & Configuration

Choose your learning style10 modes available
LearnWhyDeepVisualTryChallengeProjectRecallTime

Start learning this pattern below

Jump into concepts and practice - no test required

or
Recommended
Test this pattern10 questions across easy, medium, and hard to know if this pattern is strong
Introduction
Sometimes, machine learning tasks need powerful graphics cards called GPUs to run faster. Running these tasks inside containers can be tricky because the container needs to use the GPU hardware on the computer. GPU support in containers solves this by letting containers access GPUs safely and easily.
When you want to train a machine learning model inside a container and need faster processing using a GPU.
When you want to run deep learning inference in a container that requires GPU acceleration.
When you want to share GPU resources between multiple containerized applications without conflicts.
When you want to package your ML app with all dependencies and GPU support for easy deployment.
When you want to test GPU-enabled ML code in a consistent environment across different machines.
Config File - Dockerfile
Dockerfile
FROM nvidia/cuda:12.1.1-runtime-ubuntu22.04

RUN apt-get update && apt-get install -y python3 python3-pip

COPY requirements.txt /app/requirements.txt
WORKDIR /app
RUN pip3 install -r requirements.txt

COPY . /app

CMD ["python3", "train.py"]

This Dockerfile uses an official NVIDIA CUDA base image that includes GPU drivers and libraries needed for GPU tasks.

It installs Python and pip, then copies and installs Python dependencies from requirements.txt.

The application code is copied into the container, and the default command runs the training script.

Commands
This command builds the Docker image named 'gpu-ml-app' from the Dockerfile in the current directory. It packages the app with GPU support libraries.
Terminal
docker build -t gpu-ml-app .
Expected OutputExpected
[+] Building 12.3s (10/10) FINISHED => [internal] load build definition from Dockerfile 0.1s => [internal] load .dockerignore 0.0s => [internal] load metadata for nvidia/cuda:12.1.1-runtime-ubuntu22.04 1.2s => [1/7] FROM nvidia/cuda:12.1.1-runtime-ubuntu22.04 0.0s => [2/7] RUN apt-get update && apt-get install -y python3 python3-pip 8.5s => [3/7] COPY requirements.txt /app/requirements.txt 0.0s => [4/7] WORKDIR /app 0.0s => [5/7] RUN pip3 install -r requirements.txt 2.3s => [6/7] COPY . /app 0.1s => [7/7] CMD ["python3", "train.py"] 0.0s => exporting to image 0.1s => writing image sha256:abc123def456... 0.0s => naming to docker.io/library/gpu-ml-app 0.0s
This command runs the container with access to all GPUs on the host. The '--gpus all' flag enables GPU support inside the container.
Terminal
docker run --gpus all --rm gpu-ml-app
Expected OutputExpected
Epoch 1/10 Loss: 0.45 Epoch 2/10 Loss: 0.38 Training complete.
→
--gpus all - Allows the container to access all GPUs on the host machine.
→
--rm - Automatically removes the container after it stops to keep the system clean.
This command shows the status of NVIDIA GPUs on the host machine, including usage and running processes. It helps verify that GPUs are available.
Terminal
nvidia-smi
Expected OutputExpected
Tue Jun 20 12:00:00 2024 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.85.12 Driver Version: 525.85.12 CUDA Version: 12.1 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA RTX 3090 On | 00000000:01:00.0 Off | N/A | | 30% 45C P8 20W / 350W | 500MiB / 24576MiB | 10% Default | +-------------------------------+----------------------+----------------------+
Key Concept

If you remember nothing else from this pattern, remember: use the '--gpus' flag with Docker and a CUDA base image to enable GPU access inside containers.

Common Mistakes
Not using the '--gpus' flag when running the container.
Without this flag, the container cannot access the GPU hardware, so GPU-accelerated code will fail or run on CPU only.
Always add '--gpus all' or specify GPUs explicitly when running GPU-dependent containers.
Using a base image without CUDA or NVIDIA drivers.
The container will lack necessary GPU libraries, causing errors when trying to use the GPU.
Use official NVIDIA CUDA base images that include GPU drivers and libraries.
Running 'nvidia-smi' inside a container without GPU support enabled.
'nvidia-smi' will fail or show no GPUs because the container cannot see the GPU hardware.
Run containers with '--gpus' flag and verify GPU availability on the host with 'nvidia-smi'.
Summary
Build a Docker image using an NVIDIA CUDA base image to include GPU support libraries.
Run the container with the '--gpus all' flag to enable GPU access inside the container.
Use 'nvidia-smi' on the host to check GPU status and confirm availability.

Practice

(1/5)
1. What is the main purpose of enabling GPU support in containers?
easy
A. To reduce the container's memory usage
B. To increase the container's disk space
C. To enable network access inside the container
D. To allow containers to use the host's GPU for faster computing

Solution

  1. Step 1: Understand GPU role in containers

    GPUs speed up computing tasks by handling parallel processing efficiently.

  2. Step 2: Identify GPU support purpose

    Enabling GPU support allows containers to access the host's GPU hardware for faster computation.

  3. Final Answer:

    To allow containers to use the host's GPU for faster computing -> Option D

  4. Quick Check:

    GPU support = faster computing [OK]
Hint: GPU support means using host GPU inside container [OK]
Common Mistakes:
  • Confusing GPU support with disk or memory changes
  • Thinking GPU enables network access
  • Assuming GPU support reduces container size
2. Which Docker command flag is used to enable GPU support when running a container?
easy
A. --gpus
B. --enable-gpu
C. --gpu-access
D. --use-gpu

Solution

  1. Step 1: Recall Docker GPU flag syntax

    The official Docker flag to enable GPU support is --gpus.

  2. Step 2: Verify other options

    Options like --enable-gpu, --gpu-access, and --use-gpu are incorrect or do not exist.

  3. Final Answer:

    --gpus -> Option A

  4. Quick Check:

    Docker GPU flag = --gpus [OK]
Hint: Docker GPU flag is exactly --gpus [OK]
Common Mistakes:
  • Using incorrect flag names like --enable-gpu
  • Confusing GPU flag with network or volume flags
  • Omitting the flag entirely
3. What will be the output of the command docker run --gpus all nvidia/cuda:11.0-base nvidia-smi if the host has a compatible NVIDIA GPU and drivers installed?
medium
A. Displays the NVIDIA GPU status and driver information
B. Shows an error: 'nvidia-smi command not found'
C. Runs the container but shows no GPU information
D. Fails with 'GPU not accessible' error

Solution

  1. Step 1: Understand the command purpose

    The command runs a container with full GPU access and executes nvidia-smi to show GPU info.

  2. Step 2: Check host requirements

    If the host has compatible NVIDIA GPU and drivers, nvidia-smi runs successfully inside the container.

  3. Final Answer:

    Displays the NVIDIA GPU status and driver information -> Option A

  4. Quick Check:

    Host GPU + drivers + --gpus = nvidia-smi output [OK]
Hint: If host GPU ready, nvidia-smi shows GPU info inside container [OK]
Common Mistakes:
  • Assuming nvidia-smi is missing inside official CUDA image
  • Ignoring host driver requirements
  • Expecting GPU info without --gpus flag
4. You run docker run --gpus all nvidia/cuda:11.0-base nvidia-smi but get the error: 'docker: Error response from daemon: could not select device driver'. What is the most likely cause?
medium
A. The container command syntax is incorrect
B. The Docker image does not support GPUs
C. The NVIDIA Container Toolkit is not installed on the host
D. The host has no internet connection

Solution

  1. Step 1: Analyze the error message

    The error indicates Docker cannot find a GPU device driver to assign to the container.

  2. Step 2: Identify missing component

    This usually happens if the NVIDIA Container Toolkit (nvidia-docker2) is not installed or configured on the host.

  3. Final Answer:

    The NVIDIA Container Toolkit is not installed on the host -> Option C

  4. Quick Check:

    Missing NVIDIA toolkit = device driver error [OK]
Hint: Device driver error means NVIDIA Container Toolkit missing [OK]
Common Mistakes:
  • Blaming Docker image for GPU support
  • Assuming syntax error causes this message
  • Thinking internet is required for this error
5. You want to run a container with access to only GPUs 0 and 1 on a host with 4 GPUs. Which Docker run command correctly limits GPU access?
hard
A. docker run --gpus 2 nvidia/cuda:11.0-base nvidia-smi
B. docker run --gpus 'device=0,1' nvidia/cuda:11.0-base nvidia-smi
C. docker run --gpus all nvidia/cuda:11.0-base nvidia-smi
D. docker run --gpus 'count=2' nvidia/cuda:11.0-base nvidia-smi

Solution

  1. Step 1: Understand GPU selection syntax

    To limit to specific GPUs 0 and 1, Docker uses the --gpus 'device=0,1' syntax to specify GPU IDs.

  2. Step 2: Evaluate options

    docker run --gpus 2 nvidia/cuda:11.0-base nvidia-smi requests any 2 GPUs but does not specify GPUs 0 and 1. docker run --gpus 'count=2' nvidia/cuda:11.0-base nvidia-smi uses invalid syntax count=2. docker run --gpus all nvidia/cuda:11.0-base nvidia-smi uses all GPUs.

  3. Final Answer:

    docker run --gpus 'device=0,1' nvidia/cuda:11.0-base nvidia-smi -> Option B

  4. Quick Check:

    Specify GPUs by device IDs with --gpus 'device=...' [OK]
Hint: Use --gpus 'device=0,1' to pick specific GPUs [OK]
Common Mistakes:
  • Using --gpus 2 without device IDs
  • Using invalid syntax like count=2
  • Assuming --gpus all limits GPUs