Compute resource management in MLOps - Commands & Configuration

Introduction

When running machine learning tasks, your computer needs enough power like CPU or GPU to finish jobs fast and without errors. Compute resource management helps you control and use these resources smartly so your tasks run smoothly without wasting power or crashing.

When training a machine learning model that needs a GPU to speed up calculations.

When running multiple experiments on the same server and you want to share resources fairly.

When you want to limit how much CPU or memory a training job can use to avoid slowing down other tasks.

When you need to track which resources your model training used to optimize future runs.

When deploying models and you want to assign specific hardware to each model for better performance.

Commands

This command starts running your ML project with MLflow and assigns it to an experiment named 'compute_resource_test' to organize runs.

Terminal

mlflow run . --experiment-name compute_resource_test

Expected Output

2024/06/01 12:00:00 INFO mlflow.projects: === Run (ID '123abc') launched ===

2024/06/01 12:00:01 INFO mlflow.projects: === Run (ID '123abc') succeeded ===

→

--experiment-name - Assigns the run to a named experiment for tracking.

This command runs the ML project and tells it to use GPU resources if available by passing the parameter 'use_gpu=true'.

Terminal

mlflow run . -P use_gpu=true

Expected Output

2024/06/01 12:05:00 INFO mlflow.projects: === Run (ID '456def') launched ===

2024/06/01 12:05:01 INFO mlflow.projects: Using GPU for training

2024/06/01 12:10:00 INFO mlflow.projects: === Run (ID '456def') succeeded ===

→

-P - Passes parameters to the ML project to control resource usage.

This command shows the current GPU usage and processes using the GPU to help you monitor compute resources.

Terminal

nvidia-smi

Expected Output

Tue Jun  1 12:10:00 2024       

+-----------------------------------------------------------------------------+

| NVIDIA-SMI 525.60.13    Driver Version: 525.60.13    CUDA Version: 12.0     |

|-------------------------------+----------------------+----------------------+

| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |

| Fan  Temp  Perf  Pwr:Usage/Cap| Memory-Usage | GPU-Util  Compute M. |

|                               |                      |               MIG M. |

|===============================+======================+======================|

|   0  Tesla T4            Off  | 00000000:00:1E.0 Off |                    0 |

| N/A   55C    P0    30W /  70W |   500MiB / 15079MiB |     20%      Default |

+-------------------------------+----------------------+----------------------+



+-----------------------------------------------------------------------------+

| Processes:                                                                  |

|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |

|        ID   ID                                                   Usage      |

|=============================================================================|

|    0   N/A  N/A      1234      C   python                          480MiB   |

+-----------------------------------------------------------------------------+

This command runs the ML project limiting it to use at most 4 CPU cores and 8192 MB of memory to avoid overloading the system.

Terminal

mlflow run . -P max_cpu=4 -P max_memory=8192

Expected Output

2024/06/01 12:15:00 INFO mlflow.projects: === Run (ID '789ghi') launched ===

2024/06/01 12:15:01 INFO mlflow.projects: Limiting CPU to 4 cores and memory to 8192 MB

2024/06/01 12:20:00 INFO mlflow.projects: === Run (ID '789ghi') succeeded ===

→

-P - Passes resource limits as parameters to the ML project.

Key Concept

If you remember nothing else from this pattern, remember: controlling compute resources ensures your ML tasks run efficiently without crashing or slowing down other work.

Common Mistakes

Not specifying resource limits when running heavy ML jobs.

This can cause your computer to freeze or slow down because the job uses too much CPU or memory.

Always pass parameters to limit CPU and memory usage when running ML projects on shared machines.

Assuming GPU will be used without enabling it explicitly.

Your ML job might run on CPU only, making it slower if GPU is available but not requested.

Use parameters like 'use_gpu=true' to tell your ML project to use GPU resources.

Ignoring GPU usage monitoring.

You might overload the GPU or not notice if your job is actually using it.

Run 'nvidia-smi' regularly to check GPU usage and adjust your jobs accordingly.

Summary

Use 'mlflow run' with parameters to control CPU, memory, and GPU usage for ML tasks.

Check GPU usage with 'nvidia-smi' to monitor resource consumption.

Setting resource limits prevents system overload and ensures fair sharing on shared machines.

Practice

1. What is the main purpose of compute resource management in MLOps?

easy

A. To write machine learning model code

B. To store data permanently on disk

C. To create user interfaces for ML applications

D. To control CPU, memory, and GPU usage for efficient job execution

Solution

1. Step 1: Understand resource management role

   Compute resource management controls hardware resources like CPU, memory, and GPU.

2. Step 2: Identify its purpose in MLOps

   It ensures jobs run efficiently and avoid crashes by managing these resources.

3. Final Answer:

   To control CPU, memory, and GPU usage for efficient job execution -> Option D

4. Quick Check:

   Resource management = control CPU, memory, GPU [OK]

Hint: Think about what hardware resources need managing [OK]

Common Mistakes:

• Confusing resource management with coding tasks
• Thinking it manages data storage only
• Assuming it builds user interfaces

2. Which command correctly allocates GPU resources for a job in Kubernetes?

easy

A. kubectl run job --gpu=2

B. kubectl run job --requests=nvidia.com/gpu=2

C. kubectl run job --memory=2Gi

D. kubectl run job --cpu=2

Solution

1. Step 1: Recall Kubernetes resource request syntax

   Kubernetes uses resource requests like --requests=nvidia.com/gpu=2 to allocate GPUs.

2. Step 2: Match correct GPU allocation command

   kubectl run job --requests=nvidia.com/gpu=2 uses the correct syntax for GPU requests in Kubernetes.

3. Final Answer:

   kubectl run job --requests=nvidia.com/gpu=2 -> Option B

4. Quick Check:

   GPU allocation uses --requests=nvidia.com/gpu [OK]

Hint: Look for --requests with nvidia.com/gpu key [OK]

Common Mistakes:

• Using --gpu directly (not valid syntax)
• Confusing memory or CPU flags with GPU
• Missing the resource request keyword

3. Given this Kubernetes pod spec snippet, what is the CPU limit set for the container?

resources:
  limits:
    cpu: "4"
  requests:
    cpu: "2"

medium

A. 4 CPUs

B. 6 CPUs

C. No CPU limit set

D. 2 CPUs

Solution

1. Step 1: Identify CPU limit in pod spec

   The 'limits' section sets the maximum CPU usage, here cpu: "4" means 4 CPUs.

2. Step 2: Understand difference between requests and limits

   Requests are minimum guaranteed (2 CPUs), limits are max allowed (4 CPUs).

3. Final Answer:

   4 CPUs -> Option A

4. Quick Check:

   CPU limit = 4 CPUs [OK]

Hint: Limits set max CPU, requests set minimum [OK]

Common Mistakes:

• Confusing requests with limits
• Ignoring quotes around CPU values
• Assuming no limit means unlimited

4. You see this error when submitting a job: Insufficient cpu resources. What is the most likely cause?

medium

A. The job is missing GPU allocation

B. The job has no CPU requests set

C. The job requests more CPU than available on the cluster

D. The job memory limit is too high

Solution

1. Step 1: Interpret the error message

   'Insufficient cpu resources' means requested CPU exceeds cluster capacity.

2. Step 2: Identify cause from options

   The job requests more CPU than available on the cluster matches the error cause: job requests more CPU than available.

3. Final Answer:

   The job requests more CPU than available on the cluster -> Option C

4. Quick Check:

   Insufficient CPU = request > available [OK]

Hint: Error means requested CPU > cluster CPU [OK]

Common Mistakes:

• Assuming missing CPU requests cause this error
• Confusing CPU and GPU errors
• Blaming memory limits for CPU shortage

5. You want to run multiple ML training jobs on a GPU cluster. Which strategy best manages GPU resources to avoid conflicts?

hard

A. Allocate GPUs explicitly per job and release after completion

B. Run all jobs without GPU limits and share GPUs freely

C. Assign CPU limits only and ignore GPU allocation

D. Use only CPU resources to avoid GPU conflicts

Solution

1. Step 1: Understand GPU resource management needs

   Explicit allocation prevents multiple jobs from using the same GPU simultaneously.

2. Step 2: Evaluate options for best practice

   Allocate GPUs explicitly per job and release after completion correctly allocates and releases GPUs per job to avoid conflicts.

3. Final Answer:

   Allocate GPUs explicitly per job and release after completion -> Option A

4. Quick Check:

   Explicit GPU allocation avoids conflicts [OK]

Hint: Always allocate and release GPUs per job [OK]

Common Mistakes:

• Ignoring GPU allocation causing conflicts
• Assuming CPU limits control GPU usage
• Avoiding GPUs when cluster has them