Kubernetes for ML workloads in MLOps - Commands & Configuration

Introduction

Kubernetes helps run machine learning tasks reliably by managing computing resources and scaling automatically. It solves the problem of running ML training or inference jobs on many machines without manual setup.

When you want to train a machine learning model on multiple servers to speed up the process.

When you need to deploy a trained ML model as a service that can handle many user requests.

When your ML workload requires automatic restarting if a training job fails.

When you want to run batch ML jobs that start and stop without manual intervention.

When you want to share GPU resources among different ML tasks efficiently.

Config File - ml-job.yaml

ml-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: ml-training-job
spec:
  template:
    spec:
      containers:
      - name: trainer
        image: tensorflow/tensorflow:2.12.0
        command: ["python", "train.py"]
        resources:
          limits:
            nvidia.com/gpu: 1
      restartPolicy: Never
  backoffLimit: 4

This YAML file defines a Kubernetes Job to run a machine learning training task.

apiVersion and kind specify this is a batch job.

metadata.name names the job.

spec.template.spec.containers defines the container image and command to run the training script.

resources.limits requests one GPU for the container.

restartPolicy: Never means the pod won't restart automatically if it fails, but Kubernetes will retry the job up to backoffLimit times.

Commands

This command creates the ML training job in Kubernetes using the configuration file. It tells Kubernetes to start the job with the specified container and resources.

Terminal

kubectl apply -f ml-job.yaml

Expected Output

job.batch/ml-training-job created

This command lists all batch jobs running or completed in the Kubernetes cluster to check the status of the ML training job.

Terminal

kubectl get jobs

Expected Output

NAME              COMPLETIONS   DURATION   AGE

ml-training-job   0/1           10s        15s

This command lists the pods created by the ML training job to see if the training container is running or completed.

Terminal

kubectl get pods -l job-name=ml-training-job

Expected Output

NAME                       READY   STATUS      RESTARTS   AGE

ml-training-job-abc123      1/1     Running     0          20s

→

-l job-name=ml-training-job - Filter pods by the job name label

This command shows the output logs of the ML training container to monitor training progress or debug errors.

Terminal

kubectl logs ml-training-job-abc123

Expected Output

Epoch 1/10

loss: 0.45 - accuracy: 0.85

Epoch 2/10

loss: 0.30 - accuracy: 0.90

Training complete.

Key Concept

If you remember nothing else from this pattern, remember: Kubernetes Jobs let you run ML training tasks reliably with automatic retries and resource management.

Common Mistakes

Not specifying restartPolicy: Never in the job pod spec

Without restartPolicy: Never, the pod may restart endlessly on failure, causing unexpected resource use.

Always set restartPolicy: Never for batch jobs to let Kubernetes handle retries at the job level.

Forgetting to request GPU resources in the container spec

The training job will run without GPU acceleration, making training slower or failing if GPU is required.

Add resource limits like nvidia.com/gpu: 1 to request GPU access for the container.

Not checking pod logs to monitor training progress

You miss important feedback on training status or errors, making debugging harder.

Use kubectl logs on the job pod to see real-time output from the training script.

Summary

Create a Kubernetes Job YAML file to define the ML training task with container image, command, and resource requests.

Use kubectl apply to start the job and kubectl get jobs to check its status.

List pods created by the job and view their logs to monitor training progress and troubleshoot.

Practice

1. What is the primary Kubernetes resource used to run a one-time ML training task?

easy

A. Job

B. Deployment

C. Service

D. ConfigMap

Solution

1. Step 1: Understand Kubernetes resource types

   Jobs are designed to run tasks that complete once, like ML training.

2. Step 2: Match resource to ML training task

   Since training is a one-time batch task, Job is the correct resource.

3. Final Answer:

   Job -> Option A

4. Quick Check:

   One-time ML training = Job [OK]

Hint: Use Job for one-time tasks like training [OK]

Common Mistakes:

• Choosing Deployment which is for long-running services
• Confusing Service with workload resource
• Using ConfigMap which stores config data only

2. Which of the following is the correct YAML snippet to request 2 GPUs in a Kubernetes pod spec?

easy

A. resources:
requests:
cpu: 2

B. resources:
limits:
memory: 2Gi

C. resources:
limits:
nvidia.com/gpu: 2

D. resources:
requests:
gpu: 2

Solution

1. Step 1: Identify GPU resource naming in Kubernetes

   GPUs are requested using the vendor-specific resource name like nvidia.com/gpu.

2. Step 2: Check correct YAML structure for limits

   GPUs are usually set under limits, not requests, with the correct key.

3. Final Answer:

   resources: limits: nvidia.com/gpu: 2 -> Option C

4. Quick Check:

   GPU request uses nvidia.com/gpu under limits [OK]

Hint: GPU requests use 'limits' with 'nvidia.com/gpu' key [OK]

Common Mistakes:

• Using 'gpu' instead of 'nvidia.com/gpu'
• Placing GPU under requests instead of limits
• Confusing CPU or memory keys with GPU

3. Given this Kubernetes Job YAML snippet, what will happen when applied?

apiVersion: batch/v1
kind: Job
metadata:
  name: ml-train
spec:
  template:
    spec:
      containers:
      - name: trainer
        image: ml-image:latest
        command: ["python", "train.py"]
      restartPolicy: Never
  backoffLimit: 3

medium

A. The Job runs the training once and retries up to 3 times on failure

B. The Job runs continuously without stopping

C. The Job will fail immediately due to missing restartPolicy

D. The Job creates a Deployment instead of a batch task

Solution

1. Step 1: Understand Job behavior with backoffLimit

   The backoffLimit sets how many retries happen on failure before Job stops.

2. Step 2: Check restartPolicy and command

   restartPolicy: Never means pods won't restart automatically; Job controller retries pods.

3. Final Answer:

   The Job runs the training once and retries up to 3 times on failure -> Option A

4. Quick Check:

   Job with backoffLimit retries 3 times [OK]

Hint: backoffLimit controls retry count for Job failures [OK]

Common Mistakes:

• Thinking Job runs continuously like Deployment
• Assuming restartPolicy: Never causes immediate failure
• Confusing Job with Deployment resource

4. You deployed an ML model with a Deployment but the pods keep restarting. Which is the most likely cause?

medium

A. The ConfigMap is not mounted

B. The Deployment spec is missing replicas field

C. The Service is not exposing the Deployment

D. The container image is missing or incorrect

Solution

1. Step 1: Analyze pod restart reasons

   Pods restarting often means container crashes, commonly due to bad image or command.

2. Step 2: Check other options relevance

   Missing replicas defaults to 1, Service exposure doesn't cause restarts, ConfigMap missing causes config errors but not always restarts.

3. Final Answer:

   The container image is missing or incorrect -> Option D

4. Quick Check:

   Pod restarts usually mean bad container image [OK]

Hint: Pod restarts often mean container image or command error [OK]

Common Mistakes:

• Assuming missing replicas causes restarts
• Confusing Service exposure with pod health
• Thinking ConfigMap absence always crashes pods

5. You want to deploy an ML model serving system that automatically scales based on CPU usage. Which Kubernetes resource and feature combination is best?

hard

A. DaemonSet to run one pod per node

B. Deployment with Horizontal Pod Autoscaler (HPA)

C. StatefulSet with persistent volumes

D. Job with backoffLimit set to 5

Solution

1. Step 1: Identify resource for long-running model serving

   Deployment manages long-running pods and supports updates.

2. Step 2: Choose scaling feature for CPU-based autoscaling

   Horizontal Pod Autoscaler (HPA) automatically adjusts pod count based on CPU usage.

3. Final Answer:

   Deployment with Horizontal Pod Autoscaler (HPA) -> Option B

4. Quick Check:

   Use Deployment + HPA for scalable model serving [OK]

Hint: Use Deployment + HPA for auto-scaling model serving [OK]

Common Mistakes:

• Using Job which is for batch tasks, not serving
• Choosing StatefulSet which is for stateful apps
• DaemonSet runs pods on all nodes, not for scaling