Docker for ML workloads in MLOps - Deep Dive

Overview - Docker for ML workloads

What is it?

Docker is a tool that packages software and its environment into a container. For machine learning (ML) workloads, Docker helps bundle the ML code, libraries, and dependencies so they run the same everywhere. This means you can train or deploy ML models without worrying about differences in computers or servers. It makes ML projects more reliable and easier to share.

Why it matters

Without Docker, ML projects often break when moved between computers because of missing or different software versions. This causes wasted time fixing environment issues instead of focusing on the ML itself. Docker solves this by creating a consistent, isolated space for ML workloads, making collaboration smoother and deployment faster. It helps teams deliver ML models to users reliably and repeatedly.

Where it fits

Before learning Docker for ML, you should understand basic ML workflows and how software dependencies work. After mastering Docker, you can explore Kubernetes for scaling ML workloads or CI/CD pipelines to automate ML model training and deployment.

Mental Model

Core Idea

Docker creates a portable, isolated box that holds your ML code and all its needs, so it runs the same anywhere.

Think of it like...

Imagine packing everything you need for a camping trip—tent, food, clothes—into a single backpack. No matter where you go, you have all essentials ready. Docker is like that backpack for ML projects.

┌─────────────────────────────┐
│        Host Machine         │
│ ┌─────────────────────────┐ │
│ │       Docker Engine      │ │
│ │ ┌─────────────────────┐ │ │
│ │ │   ML Container      │ │ │
│ │ │ ┌───────────────┐  │ │ │
│ │ │ │ ML Code &     │  │ │ │
│ │ │ │ Dependencies  │  │ │ │
│ │ │ └───────────────┘  │ │ │
│ │ └─────────────────────┘ │ │
│ └─────────────────────────┘ │
└─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

What is Docker and Containers

Concept: Introduce Docker as a tool that uses containers to package software and its environment.

Docker is software that creates containers. Containers are like small, lightweight boxes that hold your program and everything it needs to run. This means the program works the same on any computer with Docker installed. Unlike virtual machines, containers share the computer's system but keep the program isolated.

Result

Learners understand that Docker packages software and dependencies into containers for consistent execution.

Understanding containers as isolated, lightweight environments is key to grasping how Docker solves environment problems.

2

Foundation

Why ML Workloads Need Containers

Concept: Explain the challenges ML projects face with dependencies and environments.

ML projects use many libraries like TensorFlow or PyTorch, each with specific versions. Different computers might have different versions or missing libraries, causing errors. Containers bundle all these libraries with the ML code, so the project runs without errors anywhere.

Result

Learners see the problem of inconsistent ML environments and how containers prevent it.

Knowing the complexity of ML dependencies clarifies why containerization is essential for ML workloads.

3

Intermediate

Building a Docker Image for ML

🤔Before reading on: do you think a Docker image includes just your ML code or also the libraries and system tools? Commit to your answer.

Concept: Teach how to write a Dockerfile to create an image that includes ML code and dependencies.

A Dockerfile is a text file with instructions to build a Docker image. For ML, it starts from a base image like 'python:3.9', installs ML libraries, copies your code, and sets the command to run your ML script. Example: FROM python:3.9 RUN pip install tensorflow numpy COPY . /app WORKDIR /app CMD ["python", "train.py"]

Result

Learners can create Docker images that package ML code and dependencies.

Knowing how to build images lets you control exactly what your ML environment contains, ensuring consistency.

4

Intermediate

Running and Managing ML Containers

🤔Before reading on: do you think running a container changes your computer’s main system or keeps it separate? Commit to your answer.

Concept: Show how to run containers and manage their lifecycle without affecting the host system.

Use 'docker run' to start a container from your ML image. The container runs isolated from your main system, so it won’t change your computer’s files or settings. You can stop, restart, or remove containers easily. Example: docker run --rm my-ml-image The '--rm' flag removes the container after it stops.

Result

Learners can run ML workloads safely in containers and manage them.

Understanding container isolation prevents accidental changes to your main system and helps manage ML experiments cleanly.

5

Intermediate

Sharing ML Environments with Docker Hub

Concept: Explain how to share Docker images via public or private registries.

Docker Hub is a service to store and share Docker images. After building your ML image, you can upload it to Docker Hub. Others can download and run the exact same environment. Commands: docker tag my-ml-image username/my-ml-image:tag docker push username/my-ml-image:tag This makes collaboration easy and reproducible.

Result

Learners can share ML environments with teammates or deploy on servers.

Knowing how to share images enables teamwork and consistent deployment across different machines.

6

Advanced

Optimizing Docker Images for ML Workloads

🤔Before reading on: do you think smaller Docker images run faster or just save disk space? Commit to your answer.

Concept: Teach techniques to reduce image size and improve performance for ML containers.

Large images slow down transfer and startup. Use multi-stage builds to separate build tools from runtime. Choose slim base images like 'python:3.9-slim'. Cache dependencies to avoid reinstalling. Example snippet: FROM python:3.9-slim RUN pip install --no-cache-dir tensorflow numpy COPY . /app Smaller images start faster and use less storage.

Result

Learners create efficient ML Docker images that save time and resources.

Understanding image optimization improves ML workflow speed and resource use, critical for production.

7

Expert

Handling GPU Support in ML Containers

🤔Before reading on: do you think Docker containers can use your computer’s GPU by default? Commit to your answer.

Concept: Explain how to enable GPU access inside Docker containers for ML training acceleration.

By default, containers cannot use GPUs. You need NVIDIA’s container toolkit and drivers installed on the host. Run containers with special flags: docker run --gpus all my-ml-image This allows ML code inside the container to use GPUs for faster training. The container must have GPU libraries like CUDA installed.

Result

Learners can run GPU-accelerated ML workloads inside Docker containers.

Knowing how to enable GPU access bridges container isolation with hardware acceleration, essential for real ML workloads.

Under the Hood

Docker uses OS-level virtualization to create containers. It shares the host OS kernel but isolates processes, file systems, and network using namespaces and control groups (cgroups). This isolation ensures containers run independently without interfering with each other or the host. Docker images are layered filesystems built from instructions in Dockerfiles. When running a container, Docker combines these layers into a single view and starts the process inside the isolated environment.

Why designed this way?

Docker was designed to be lightweight and fast compared to full virtual machines. Using OS-level features avoids the overhead of running separate OS instances. Layered images allow reuse of common parts, saving space and speeding up builds. This design balances isolation with performance, making it ideal for packaging complex ML environments that need consistency without heavy resource use.

Host OS Kernel
┌─────────────────────────────┐
│ Docker Engine               │
│ ┌───────────────┐           │
│ │ Namespaces    │           │
│ │ & cgroups     │           │
│ └───────────────┘           │
│ ┌───────────────┐           │
│ │ Container 1   │           │
│ │ (ML workload) │           │
│ └───────────────┘           │
│ ┌───────────────┐           │
│ │ Container 2   │           │
│ │ (Other app)   │           │
│ └───────────────┘           │
└─────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think Docker containers are full virtual machines? Commit yes or no.

Common Belief: Docker containers are just like virtual machines with their own full operating system.

Reality: Docker containers share the host OS kernel and isolate only processes and filesystems, making them much lighter than virtual machines.

Why it matters: Thinking containers are heavy like VMs leads to overestimating resource needs and misunderstanding container startup speed.

Quick: Can Docker containers automatically use your GPU without extra setup? Commit yes or no.

Common Belief: Docker containers can use GPUs by default just like the host system.

Reality: Containers need special drivers and runtime setup to access GPUs; otherwise, they cannot use GPU hardware.

Why it matters: Assuming GPU works by default causes wasted time debugging ML training performance issues.

Quick: Does sharing a Docker image guarantee your ML code will run identically everywhere? Commit yes or no.

Common Belief: If I share a Docker image, my ML code will always run exactly the same on any machine.

Reality: While Docker ensures environment consistency, differences in hardware (like GPUs) or external data sources can still cause variations.

Why it matters: Overreliance on Docker images alone can lead to overlooked issues in production related to hardware or data.

Quick: Is a smaller Docker image always faster to run? Commit yes or no.

Common Belief: Smaller Docker images always start and run faster than larger ones.

Reality: Smaller images reduce download and startup time, but runtime speed depends on the ML code and hardware, not image size alone.

Why it matters: Focusing only on image size can distract from optimizing actual ML workload performance.

Expert Zone

1

Docker image layers cache can cause stale dependencies if not managed carefully, leading to hard-to-debug ML bugs.

2

Using multi-stage builds not only reduces image size but also improves security by excluding build tools from the final image.

3

GPU support requires matching host driver versions with container CUDA libraries; mismatches cause silent failures or crashes.

When NOT to use

Docker is not ideal when ultra-low latency or direct hardware access is required beyond GPU support. In such cases, bare-metal deployment or specialized orchestration like Kubernetes with device plugins is better. Also, for very simple ML scripts without complex dependencies, virtual environments might suffice.

Production Patterns

In production, ML teams use Docker images combined with CI/CD pipelines to automate training and deployment. Images are versioned and stored in private registries. GPU-enabled containers run on cloud or on-prem clusters managed by Kubernetes. Monitoring and logging are integrated to track ML model performance and resource use.

Connections

Virtual Machines

Docker containers are a lightweight alternative to virtual machines.

Understanding the difference helps choose the right isolation tool for ML workloads balancing performance and security.

Continuous Integration/Continuous Deployment (CI/CD)

Docker images are often built and tested automatically in CI/CD pipelines for ML projects.

Knowing Docker enables smoother automation of ML model updates and deployment.

Supply Chain Packaging

Docker containers bundle all parts needed to run ML code, similar to how supply chains package and deliver goods reliably.

Recognizing this connection highlights the importance of packaging completeness and consistency in both software and physical goods delivery.

Common Pitfalls

#1 Not specifying exact library versions in Dockerfile causes inconsistent ML environments.

Wrong approach:

RUN pip install tensorflow numpy

Correct approach:

RUN pip install tensorflow==2.12.0 numpy==1.24.3

Root cause:

Assuming latest versions are always compatible leads to unexpected breaks when dependencies update.

#2 Running containers without cleaning up leads to many stopped containers consuming disk space.

Wrong approach:

docker run my-ml-image

Correct approach:

docker run --rm my-ml-image

Root cause:

Not using '--rm' flag or manual cleanup causes clutter and wasted storage.

#3 Trying to use GPU inside container without installing NVIDIA container toolkit causes errors.

Wrong approach:

docker run --gpus all my-ml-image (without toolkit installed)

Correct approach:

Install NVIDIA container toolkit on host, then run: docker run --gpus all my-ml-image

Root cause:

Assuming GPU access works out-of-the-box ignores necessary host setup.

Key Takeaways

Docker containers package ML code and all dependencies into a portable, consistent environment.

Containers share the host OS kernel but isolate processes and files, making them lightweight compared to virtual machines.

Building Docker images with exact dependency versions ensures reproducible ML environments.

GPU support in Docker requires special setup beyond just running containers.

Optimizing Docker images and managing containers properly improves ML workflow efficiency and reliability.