Compute resource management in MLOps - Mini Project: Build & Apply

Compute Resource Management

📖 Scenario: You are managing a small machine learning project that needs to allocate compute resources efficiently. You want to track the available compute nodes and their capacities to decide where to run your training jobs.

🎯 Goal: Build a simple Python program that stores compute nodes and their CPU capacities, sets a minimum CPU threshold, filters nodes that meet this threshold, and prints the eligible nodes.

📋 What You'll Learn

Create a dictionary called compute_nodes with exact node names and CPU counts

Create a variable called min_cpu with the minimum CPU threshold

Use a dictionary comprehension called eligible_nodes to filter nodes with CPUs >= min_cpu

Print the eligible_nodes dictionary

💡 Why This Matters

🌍 Real World

Managing compute resources is essential in machine learning projects to allocate jobs efficiently and avoid overloading nodes.

💼 Career

DevOps and MLOps engineers often write scripts to monitor and manage compute resources to optimize performance and cost.

1

Create the compute nodes dictionary

Create a dictionary called compute_nodes with these exact entries: 'node1': 8, 'node2': 4, 'node3': 16, 'node4': 2

MLOps

# Create the compute_nodes dictionary with exact node names and CPU counts
# Your code here

Hint

Use curly braces {} to create a dictionary with keys as node names and values as CPU counts.

2

Set the minimum CPU threshold

Create a variable called min_cpu and set it to the integer 6

MLOps

compute_nodes = {'node1': 8, 'node2': 4, 'node3': 16, 'node4': 2}
# Create min_cpu variable and set it to 6
# Your code here

Hint

Just assign the number 6 to the variable min_cpu.

3

Filter eligible nodes with enough CPUs

Use a dictionary comprehension called eligible_nodes to include only nodes from compute_nodes where the CPU count is greater than or equal to min_cpu

MLOps

compute_nodes = {'node1': 8, 'node2': 4, 'node3': 16, 'node4': 2}
min_cpu = 6
# Create eligible_nodes dictionary comprehension filtering nodes with CPUs >= min_cpu
# Your code here

Hint

Use {node: cpu for node, cpu in compute_nodes.items() if cpu >= min_cpu} to filter the dictionary.

4

Print the eligible nodes

Write a print statement to display the eligible_nodes dictionary

MLOps

compute_nodes = {'node1': 8, 'node2': 4, 'node3': 16, 'node4': 2}
min_cpu = 6
eligible_nodes = {node: cpu for node, cpu in compute_nodes.items() if cpu >= min_cpu}
# Print the eligible_nodes dictionary
# Your code here

Hint

Use print(eligible_nodes) to show the filtered nodes.

Practice

1. What is the main purpose of compute resource management in MLOps?

easy

A. To write machine learning model code

B. To store data permanently on disk

C. To create user interfaces for ML applications

D. To control CPU, memory, and GPU usage for efficient job execution

Solution

1. Step 1: Understand resource management role

   Compute resource management controls hardware resources like CPU, memory, and GPU.

2. Step 2: Identify its purpose in MLOps

   It ensures jobs run efficiently and avoid crashes by managing these resources.

3. Final Answer:

   To control CPU, memory, and GPU usage for efficient job execution -> Option D

4. Quick Check:

   Resource management = control CPU, memory, GPU [OK]

Hint: Think about what hardware resources need managing [OK]

Common Mistakes:

• Confusing resource management with coding tasks
• Thinking it manages data storage only
• Assuming it builds user interfaces

2. Which command correctly allocates GPU resources for a job in Kubernetes?

easy

A. kubectl run job --gpu=2

B. kubectl run job --requests=nvidia.com/gpu=2

C. kubectl run job --memory=2Gi

D. kubectl run job --cpu=2

Solution

1. Step 1: Recall Kubernetes resource request syntax

   Kubernetes uses resource requests like --requests=nvidia.com/gpu=2 to allocate GPUs.

2. Step 2: Match correct GPU allocation command

   kubectl run job --requests=nvidia.com/gpu=2 uses the correct syntax for GPU requests in Kubernetes.

3. Final Answer:

   kubectl run job --requests=nvidia.com/gpu=2 -> Option B

4. Quick Check:

   GPU allocation uses --requests=nvidia.com/gpu [OK]

Hint: Look for --requests with nvidia.com/gpu key [OK]

Common Mistakes:

• Using --gpu directly (not valid syntax)
• Confusing memory or CPU flags with GPU
• Missing the resource request keyword

3. Given this Kubernetes pod spec snippet, what is the CPU limit set for the container?

resources:
  limits:
    cpu: "4"
  requests:
    cpu: "2"

medium

A. 4 CPUs

B. 6 CPUs

C. No CPU limit set

D. 2 CPUs

Solution

1. Step 1: Identify CPU limit in pod spec

   The 'limits' section sets the maximum CPU usage, here cpu: "4" means 4 CPUs.

2. Step 2: Understand difference between requests and limits

   Requests are minimum guaranteed (2 CPUs), limits are max allowed (4 CPUs).

3. Final Answer:

   4 CPUs -> Option A

4. Quick Check:

   CPU limit = 4 CPUs [OK]

Hint: Limits set max CPU, requests set minimum [OK]

Common Mistakes:

• Confusing requests with limits
• Ignoring quotes around CPU values
• Assuming no limit means unlimited

4. You see this error when submitting a job: Insufficient cpu resources. What is the most likely cause?

medium

A. The job is missing GPU allocation

B. The job has no CPU requests set

C. The job requests more CPU than available on the cluster

D. The job memory limit is too high

Solution

1. Step 1: Interpret the error message

   'Insufficient cpu resources' means requested CPU exceeds cluster capacity.

2. Step 2: Identify cause from options

   The job requests more CPU than available on the cluster matches the error cause: job requests more CPU than available.

3. Final Answer:

   The job requests more CPU than available on the cluster -> Option C

4. Quick Check:

   Insufficient CPU = request > available [OK]

Hint: Error means requested CPU > cluster CPU [OK]

Common Mistakes:

• Assuming missing CPU requests cause this error
• Confusing CPU and GPU errors
• Blaming memory limits for CPU shortage

5. You want to run multiple ML training jobs on a GPU cluster. Which strategy best manages GPU resources to avoid conflicts?

hard

A. Allocate GPUs explicitly per job and release after completion

B. Run all jobs without GPU limits and share GPUs freely

C. Assign CPU limits only and ignore GPU allocation

D. Use only CPU resources to avoid GPU conflicts

Solution

1. Step 1: Understand GPU resource management needs

   Explicit allocation prevents multiple jobs from using the same GPU simultaneously.

2. Step 2: Evaluate options for best practice

   Allocate GPUs explicitly per job and release after completion correctly allocates and releases GPUs per job to avoid conflicts.

3. Final Answer:

   Allocate GPUs explicitly per job and release after completion -> Option A

4. Quick Check:

   Explicit GPU allocation avoids conflicts [OK]

Hint: Always allocate and release GPUs per job [OK]

Common Mistakes:

• Ignoring GPU allocation causing conflicts
• Assuming CPU limits control GPU usage
• Avoiding GPUs when cluster has them