Kubernetes for ML workloads in MLOps - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Kubernetes Pod Scheduling for ML Jobs

Which Kubernetes feature ensures that ML training pods are scheduled on nodes with GPUs?

A Node Affinity with GPU label selectors

B Horizontal Pod Autoscaler

C ConfigMap volume mounts

D Pod Security Policies

💡 Hint

Think about how Kubernetes selects nodes based on hardware capabilities.

Node Affinity allows pods to be scheduled on nodes that have specific labels, such as those indicating GPU availability. This is essential for ML workloads requiring GPUs.

💻 Command Output

intermediate

Output of kubectl describe on ML Training Pod

What is the output of kubectl describe pod ml-train-pod if the pod is pending due to insufficient GPU resources?

MLOps

kubectl describe pod ml-train-pod

A

Events:
  Type     Normal  Reason    Age   From               Message
  ----     ------  ------    ----  ----               -------
  Normal   Scheduled  1m    default-scheduler  Successfully assigned ml-train-pod to node-1

B

Status: Running
Containers:
  ml-container:
    State: Running
    Ready: True

C Error from server (NotFound): pods "ml-train-pod" not found

D

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  2m    default-scheduler  0/3 nodes are available: 3 Insufficient nvidia.com/gpu.

💡 Hint

Look for messages about resource availability in the pod events.

The pod is pending because no nodes have enough GPUs available. The scheduler logs this as a FailedScheduling event with a message about insufficient GPUs.

❓ Configuration

advanced

Configuring Persistent Storage for ML Data in Kubernetes

Which YAML snippet correctly defines a PersistentVolumeClaim (PVC) for 50Gi of storage with ReadWriteOnce access mode suitable for ML training data?

A

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ml-data-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    limits:
      storage: 50Gi

B

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ml-data-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    limits:
      storage: 50Gi

C

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ml-data-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

D

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ml-data-pvc
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 50Gi

💡 Hint

Check the correct field for storage size and access mode for single node write access.

The correct PVC uses requests for storage size and ReadWriteOnce for access mode, allowing one node to write to the volume.

🔀 Workflow

advanced

Steps to Deploy a Distributed ML Training Job on Kubernetes

What is the correct order of steps to deploy a distributed ML training job using Kubernetes?

A 3,1,2,4

B 1,3,2,4

C 1,2,3,4

D 2,1,3,4

💡 Hint

Think about building the image before pushing and defining manifests before applying.

First build the Docker image, then push it to a registry, next define manifests referencing the image, and finally apply manifests to run the job.

❓ Troubleshoot

expert

Diagnosing ML Pod CrashLoopBackOff due to GPU Driver Issues

An ML training pod repeatedly crashes with CrashLoopBackOff. Logs show Failed to initialize GPU device. What is the most likely cause?

A The node lacks the NVIDIA GPU device plugin or drivers are not installed properly.

B The pod's container image is missing the ML training code.

C The PersistentVolumeClaim is not bound to a PersistentVolume.

D The pod's CPU resource requests exceed node capacity.

💡 Hint

Consider hardware and driver compatibility for GPU access.

GPU initialization failures usually indicate missing or misconfigured GPU drivers or device plugins on the node, causing the pod to crash.