Distributed training basics in MLOps - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Parameter Server Role

In a distributed training setup using a parameter server architecture, what is the primary role of the parameter server?

A It manages the hardware resources like GPUs and CPUs.

B It performs data preprocessing before sending batches to workers.

C It stores and updates the global model parameters shared among workers.

D It handles user authentication and access control for the training cluster.

💡 Hint

Think about where the model weights are kept and updated during training.

The parameter server holds the global model parameters and updates them as workers send gradients. It acts as a central point for synchronizing model state.

💻 Command Output

intermediate

Output of Distributed Training Node Status Command

What is the expected output of the command kubectl get pods -l app=distributed-train if three training pods are running successfully?

MLOps

kubectl get pods -l app=distributed-train

A

NAME                 READY   STATUS    RESTARTS   AGE
train-worker-0       0/1     Pending   0          10m
train-worker-1       0/1     Pending   0          10m
train-worker-2       0/1     Pending   0          10m

B

NAME                 READY   STATUS    RESTARTS   AGE
train-worker-0       1/1     Running   0          10m
train-worker-1       1/1     Running   0          10m
train-worker-2       1/1     Running   0          10m

C Error from server (NotFound): pods "train-worker-0" not found

D No resources found in default namespace.

💡 Hint

Look for pods with status 'Running' and readiness '1/1'.

The command lists pods with label app=distributed-train. If three pods are running successfully, each will show 1/1 READY and Running STATUS.

❓ Configuration

advanced

Correct Distributed Training Configuration Snippet

Which configuration snippet correctly sets up a distributed training job using Horovod with 4 worker replicas in Kubernetes?

A

apiVersion: batch/v1
kind: Job
metadata:
  name: horovod-job
spec:
  parallelism: 4
  template:
    spec:
      containers:
      - name: horovod
        image: horovod/horovod:latest
        command: ["horovodrun", "-np", "4", "python", "train.py"]
      restartPolicy: Never

B

apiVersion: batch/v1
kind: Job
metadata:
  name: horovod-job
spec:
  replicas: 4
  template:
    spec:
      containers:
      - name: horovod
        image: horovod/horovod:latest
        command: ["horovodrun", "-np", "4", "python", "train.py"]
      restartPolicy: Always

C

apiVersion: apps/v1
kind: Deployment
metadata:
  name: horovod-deployment
spec:
  replicas: 4
  template:
    spec:
      containers:
      - name: horovod
        image: horovod/horovod:latest
        command: ["horovodrun", "-np", "1", "python", "train.py"]
      restartPolicy: Always

D

apiVersion: batch/v1
kind: Job
metadata:
  name: horovod-job
spec:
  parallelism: 1
  template:
    spec:
      containers:
      - name: horovod
        image: horovod/horovod:latest
        command: ["horovodrun", "-np", "4", "python", "train.py"]
      restartPolicy: Never

💡 Hint

Check the correct field for parallelism in a Kubernetes Job and the restart policy.

For a Kubernetes Job, parallelism controls how many pods run in parallel. The command must request 4 processes. Restart policy should be Never for batch jobs.

❓ Troubleshoot

advanced

Diagnosing Worker Node Failure in Distributed Training

During distributed training, one worker node repeatedly crashes with an error related to 'connection refused' when trying to reach the parameter server. What is the most likely cause?

A The parameter server is not running or not reachable on the expected network address.

B The worker node has insufficient GPU memory to run the training batch.

C The worker node's disk is full, preventing checkpoint saving.

D The training script has a syntax error causing the crash.

💡 Hint

Connection refused usually means network or service availability issues.

'Connection refused' means the worker tried to connect but no service was listening at the address. This usually means the parameter server is down or network settings block access.

🔀 Workflow

expert

Correct Sequence for Distributed Training Job Deployment

Arrange the steps in the correct order to deploy a distributed training job on a Kubernetes cluster using a containerized training image.

A 3,1,2,4

B 2,1,3,4

C 1,3,2,4

D 1,2,3,4

💡 Hint

Think about the logical order from building to running and monitoring.

You must first build the image, then push it to a registry, then create the Kubernetes resources that use the image, and finally monitor the job.

Practice

1. What is the main purpose of distributed training in machine learning?

easy

A. To avoid using GPUs during training

B. To split the training workload across multiple machines or GPUs

C. To increase the learning rate automatically

D. To reduce the size of the training dataset

Solution

1. Step 1: Understand distributed training goal

   Distributed training is designed to share the training task among several machines or GPUs to speed up the process.

2. Step 2: Analyze options

   Only To split the training workload across multiple machines or GPUs correctly describes this purpose. Options A, B, and C do not relate to workload distribution.

3. Final Answer:

   To split the training workload across multiple machines or GPUs -> Option B

4. Quick Check:

   Distributed training = workload split [OK]

Hint: Distributed training means sharing work across machines [OK]

Common Mistakes:

• Thinking distributed training reduces dataset size
• Confusing distributed training with hyperparameter tuning
• Believing distributed training avoids GPU use

2. Which of the following is the correct way to initialize a process group for distributed training in PyTorch?

easy

A. torch.distributed.init_process_group(backend='nccl', rank=0, world_size=1)

B. torch.init_process_group(backend='nccl', rank=0, world_size=1)

C. torch.distributed.start_process_group(backend='nccl', rank=0, world_size=1)

D. torch.distributed.init_group(backend='nccl', rank=0, world_size=1)

Solution

1. Step 1: Identify correct function name

   The correct function to initialize communication is torch.distributed.init_process_group.

2. Step 2: Check syntax correctness

   torch.distributed.init_process_group(backend='nccl', rank=0, world_size=1) uses the correct module and function name with proper parameters. Other options use incorrect function names or modules.

3. Final Answer:

   torch.distributed.init_process_group(backend='nccl', rank=0, world_size=1) -> Option A

4. Quick Check:

   Correct init function = torch.distributed.init_process_group(backend='nccl', rank=0, world_size=1) [OK]

Hint: Use torch.distributed.init_process_group to start communication [OK]

Common Mistakes:

• Using wrong function names like start_process_group
• Calling init_process_group from wrong module
• Misspelling function or module names

3. Given the following code snippet for distributed training setup, what is the output of print(rank, world_size)?

import torch.distributed as dist
rank = 2
world_size = 4
print(rank, world_size)

medium

A. 4 2

B. Error: rank and world_size undefined

C. 0 1

D. 2 4

Solution

1. Step 1: Analyze variable assignments

   Variables rank and world_size are assigned values 2 and 4 respectively before the print statement.

2. Step 2: Understand print output

   Printing rank and world_size will output '2 4' exactly as assigned.

3. Final Answer:

   2 4 -> Option D

4. Quick Check:

   Print rank, world_size = 2 4 [OK]

Hint: Print variables as assigned to see output [OK]

Common Mistakes:

• Confusing rank with world_size order
• Assuming variables are undefined
• Expecting automatic values without assignment

4. You wrote this code to initialize distributed training but get an error:

import torch.distributed as dist
dist.init_process_group(backend='nccl', rank=0)

What is missing that causes the error?

medium

A. The rank parameter should be a string

B. The backend parameter is incorrect

C. The world_size parameter is missing

D. The import statement is wrong

Solution

1. Step 1: Check init_process_group parameters

   The function requires both rank and world_size parameters to know the total number of processes.

2. Step 2: Identify missing parameter

   The code misses world_size, which causes the error.

3. Final Answer:

   The world_size parameter is missing -> Option C

4. Quick Check:

   Missing world_size causes error [OK]

Hint: Always provide world_size with rank in init_process_group [OK]

Common Mistakes:

• Omitting world_size parameter
• Using wrong backend names
• Passing rank as string instead of int

5. In a distributed training setup with 4 GPUs, you want each process to know its rank and the total number of processes. Which code snippet correctly sets this up and prints the rank and world size?

hard

A. import torch.distributed as dist
dist.init_process_group(backend='nccl')
rank = dist.get_rank()
world_size = dist.get_world_size()
print(rank, world_size)

B. import torch.distributed as dist
world_size = 4
rank = dist.get_rank()
dist.init_process_group(backend='nccl', rank=rank, world_size=world_size)
print(rank, world_size)

C. import torch.distributed as dist
rank = 0
world_size = 4
dist.init_process_group(backend='nccl', rank=rank, world_size=world_size)
print(rank, world_size)

D. import torch.distributed as dist
rank = dist.get_rank()
world_size = dist.get_world_size()
dist.init_process_group(backend='nccl', rank=rank, world_size=world_size)
print(rank, world_size)

Solution

1. Step 1: Understand correct initialization order

   dist.init_process_group must be called before calling dist.get_rank() or dist.get_world_size() to initialize communication.

2. Step 2: Analyze each option

   import torch.distributed as dist dist.init_process_group(backend='nccl') rank = dist.get_rank() world_size = dist.get_world_size() print(rank, world_size) initializes the process group first, then gets rank and world size, then prints them. Other options either get rank before initialization or pass rank manually without initialization.

3. Final Answer:

   import torch.distributed as dist dist.init_process_group(backend='nccl') rank = dist.get_rank() world_size = dist.get_world_size() print(rank, world_size) -> Option A

4. Quick Check:

   Init first, then get rank/world_size = import torch.distributed as dist dist.init_process_group(backend='nccl') rank = dist.get_rank() world_size = dist.get_world_size() print(rank, world_size) [OK]

Hint: Initialize before getting rank and world size [OK]

Common Mistakes:

• Calling get_rank before init_process_group
• Passing rank manually without init
• Not calling init_process_group at all