Organizing experiments with tags and notes in MLOps - Commands & Configuration

Introduction

When you run many machine learning experiments, it can be hard to remember what each one did. Tags and notes help you label and describe experiments so you can find and compare them easily later.

When you want to quickly find experiments that used a specific dataset or model type.

When you want to add a short description to explain what changed in an experiment.

When you want to group experiments by project phase, like 'baseline' or 'tuning'.

When you want to share experiment details with teammates clearly.

When you want to track which experiments gave the best results for a metric.

Commands

This command runs an MLflow project in the current directory with a parameter alpha set to 0.5. It starts a new experiment run.

Terminal

mlflow run . -P alpha=0.5

Expected Output

2024/06/01 12:00:00 INFO mlflow.projects: === Run (ID 123abc) started ===

2024/06/01 12:00:05 INFO mlflow.projects: === Run (ID 123abc) succeeded ===

→

-P - Set a parameter value for the run

This runs the project with the same parameter but adds an experiment name, a run name, and tags to organize the run.

Terminal

mlflow run . -P alpha=0.5 --experiment-name 'my-experiment' --run-name 'alpha-0.5-run' --tags phase=baseline,model=linear

Expected Output

2024/06/01 12:05:00 INFO mlflow.projects: === Run (ID 456def) started ===

2024/06/01 12:05:10 INFO mlflow.projects: === Run (ID 456def) succeeded ===

→

--experiment-name - Assign the run to a named experiment

→

--run-name - Give a friendly name to the run

→

--tags - Add key-value tags to the run for organization

Starts the MLflow tracking UI so you can see your experiments, runs, tags, and notes in a web browser.

Terminal

mlflow ui

Expected Output

2024/06/01 12:10:00 INFO mlflow.server: Starting MLflow UI at http://127.0.0.1:5000

Adds a note or description to the run with ID 456def to explain what this run tested.

Terminal

mlflow runs update --run-id 456def --description 'Tuned alpha to 0.5 for baseline linear model'

Expected Output

Run 456def updated successfully

Key Concept

If you remember nothing else from this pattern, remember: tags and notes let you label and describe experiments so you can find and understand them later.

Code Example

MLOps

import mlflow

with mlflow.start_run(run_name='alpha-0.5-run', tags={'phase': 'baseline', 'model': 'linear'}):
    mlflow.log_param('alpha', 0.5)
    mlflow.log_metric('accuracy', 0.82)
    mlflow.set_tag('dataset', 'census')
    mlflow.set_tag('note', 'Tuned alpha to 0.5 for baseline linear model')

print('Experiment run logged with tags and notes')

Common Mistakes

Not using tags or notes when running experiments

Without tags or notes, it becomes hard to remember what each experiment did or to find specific runs later.

Always add meaningful tags and notes when you start or update an experiment run.

Using unclear or inconsistent tag names

Inconsistent tags make it difficult to group or filter experiments effectively.

Use clear, consistent tag keys and values that everyone on your team understands.

Summary

Run experiments with mlflow run and add tags to label important details.

Use mlflow runs update to add notes or descriptions to explain your runs.

Start mlflow ui to view and filter experiments by tags and notes in a browser.

Practice

1. What is the main purpose of using tags in organizing machine learning experiments?

easy

A. To delete old experiments automatically

B. To run experiments faster

C. To increase the accuracy of models

D. To label experiments with key details for easy searching

Solution

1. Step 1: Understand the role of tags

   Tags are labels that help identify and group experiments by important details.

2. Step 2: Connect tags to experiment organization

   Using tags makes it easier to search and filter experiments later.

3. Final Answer:

   To label experiments with key details for easy searching -> Option D

4. Quick Check:

   Tags = Labels for searching [OK]

Hint: Tags help find experiments quickly by labeling details [OK]

Common Mistakes:

• Thinking tags speed up experiment runs
• Confusing tags with deleting experiments
• Believing tags improve model accuracy

2. Which of the following is the correct way to add a tag named version with value 1.0 to an experiment using a CLI command?

easy

A. mlflow experiments set-tag 123 version 1.0

B. mlflow experiments tag --id 123 version 1.0

C. mlflow add-tag experiment 123 version=1.0

D. mlflow experiments add-tag --experiment-id 123 version=1.0

Solution

1. Step 1: Recall correct MLflow CLI syntax for tagging

   The correct command uses mlflow experiments set-tag followed by experiment ID, key, and value (space-separated).

2. Step 2: Compare options to syntax

   mlflow experiments set-tag 123 version 1.0 matches the correct syntax exactly.

3. Final Answer:

   mlflow experiments set-tag 123 version 1.0 -> Option A

4. Quick Check:

   Correct CLI syntax = mlflow experiments set-tag 123 version 1.0 [OK]

Hint: Use 'mlflow experiments set-tag' with ID key value [OK]

Common Mistakes:

• Using 'add-tag' instead of 'set-tag'
• Using '=' between key and value
• Incorrect command order or missing flags

3. Given this Python snippet using MLflow to log tags and notes:

import mlflow
with mlflow.start_run():
    mlflow.set_tag("model", "resnet")
    mlflow.set_tag("dataset", "cifar10")
    mlflow.set_tag("version", "2.1")
    mlflow.set_tag("note", "Improved accuracy")
    mlflow.log_metric("accuracy", 0.92)

What tags will be visible when you list this experiment's tags?

medium

A. {"dataset": "cifar10", "version": "2.1"}

B. {"model": "resnet", "accuracy": "0.92", "note": "Improved accuracy"}

C. {"model": "resnet", "dataset": "cifar10", "version": "2.1", "note": "Improved accuracy"}

D. {"accuracy": 0.92, "note": "Improved accuracy"}

Solution

1. Step 1: Identify tags set in the code

   The code sets tags: model, dataset, version, and note with given values.

2. Step 2: Understand metrics vs tags

   Accuracy is logged as a metric, not a tag, so it won't appear in tags list.

3. Final Answer:

   {"model": "resnet", "dataset": "cifar10", "version": "2.1", "note": "Improved accuracy"} -> Option C

4. Quick Check:

   Tags = model, dataset, version, note [OK]

Hint: Tags are set by set_tag; metrics are separate [OK]

Common Mistakes:

• Confusing metrics with tags
• Missing some tags in the output
• Including metrics as tags

4. You tried to add a note to an experiment using this code:

mlflow.set_tag("note", "Test run")

But the note does not appear in the experiment UI. What is the likely problem?

medium

A. You forgot to start an MLflow run context before setting the tag

B. The tag key "note" is reserved and cannot be used

C. The note must be logged as a metric, not a tag

D. MLflow does not support notes in experiments

Solution

1. Step 1: Check MLflow run context requirement

   Tags must be set inside an active MLflow run context to be saved.

2. Step 2: Identify missing run context

   The code snippet lacks with mlflow.start_run():, so the tag is not recorded.

3. Final Answer:

   You forgot to start an MLflow run context before setting the tag -> Option A

4. Quick Check:

   Run context needed for tags [OK]

Hint: Always start a run before setting tags or logging [OK]

Common Mistakes:

• Assuming tags work outside run context
• Thinking 'note' is a reserved key
• Confusing tags with metrics

5. You want to organize experiments by multiple tags and add detailed notes. Which approach best helps you find experiments where the model is 'xgboost', version is '3.0', and notes mention 'early stopping'?

hard

A. Only use a single tag combining all info like 'xgboost_3.0_early stopping' to simplify filtering

B. Tag experiments with keys 'model'='xgboost' and 'version'='3.0', and include 'early stopping' in a 'note' tag; then filter by these tags and search notes text

C. Log 'early stopping' as a metric and filter experiments by metric value

D. Add notes as comments in code but do not tag; rely on manual search

Solution

1. Step 1: Use multiple tags for structured filtering

   Tagging 'model' and 'version' separately allows filtering by each key-value pair.

2. Step 2: Add detailed notes in a 'note' tag for text search

   Including 'early stopping' in notes helps find experiments by keyword search.

3. Step 3: Avoid combining all info in one tag or misusing metrics

   Single combined tags or metrics for notes reduce search flexibility and clarity.

4. Final Answer:

   Tag experiments with keys 'model'='xgboost' and 'version'='3.0', and include 'early stopping' in a 'note' tag; then filter by these tags and search notes text -> Option B

5. Quick Check:

   Use multiple tags + notes for flexible search [OK]

Hint: Use separate tags plus notes for detailed, flexible filtering [OK]

Common Mistakes:

• Combining all info in one tag
• Logging notes as metrics
• Not tagging and relying on manual search