Comparing experiment runs in MLOps - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

Comparing experiment runs with MLflow CLI

You run the command mlflow runs compare --run-ids 123 456 to compare two experiment runs. What output should you expect?

A A table showing parameter, metric, and tag differences between runs 123 and 456.

B A list of all experiment runs in the current experiment.

C A JSON dump of all artifacts stored in runs 123 and 456.

D An error stating that the run IDs must be numeric values only.

💡 Hint

Think about what 'compare' means in the context of experiment runs.

The mlflow runs compare command outputs a table highlighting differences in parameters, metrics, and tags between the specified runs.

🧠 Conceptual

intermediate

Understanding experiment run comparison metrics

When comparing two experiment runs, which metric difference is most useful to determine model improvement?

A Difference in the number of artifacts logged.

B Difference in training time between the runs.

C Difference in the experiment run IDs.

D Difference in validation accuracy or loss metric.

💡 Hint

Focus on metrics that reflect model performance.

Validation accuracy or loss directly measures how well the model performs on unseen data, making it the key metric for improvement.

🔀 Workflow

advanced

Steps to compare experiment runs programmatically

Which sequence correctly describes how to compare two experiment runs using the MLflow Python API?

A 1,3,2,4

B 2,1,3,4

C 1,2,3,4

D 3,1,2,4

💡 Hint

Think about the logical order of API usage.

You first import and create a client, then retrieve runs, extract data, and finally compare.

❓ Troubleshoot

advanced

Troubleshooting missing run comparison output

You run mlflow runs compare --run-ids 101 102 but see no differences reported, even though you expect some. What is the most likely cause?

A The run IDs are from different experiments and cannot be compared.

B Both runs have identical parameters, metrics, and tags.

C The MLflow tracking server is down and cannot fetch run data.

D The command requires an additional flag to show differences.

💡 Hint

Consider what it means if no differences appear.

If no differences show, it usually means the runs have identical parameters, metrics, and tags.

✅ Best Practice

expert

Best practice for comparing multiple experiment runs

What is the best practice when comparing multiple experiment runs to identify the best performing model?

A Use automated scripts to extract and aggregate key metrics across runs for analysis.

B Only compare runs with the same run ID prefix to reduce complexity.

C Ignore parameter differences and focus only on artifact sizes.

D Compare runs pairwise manually using CLI commands for each pair.

💡 Hint

Think about scalability and automation.

Automated scripts help efficiently aggregate and analyze metrics across many runs, enabling better decision-making.

Practice

1.

What is the main purpose of comparing experiment runs in MLOps?

easy

A. To identify which model performs best by reviewing their results side by side

B. To delete old experiment runs to save space

C. To create new experiment runs automatically

D. To change the code of the model during training

Solution

1. Step 1: Understand experiment runs

   Experiment runs record model training results and metrics.

2. Step 2: Purpose of comparing runs

   Comparing runs helps see which model version performs better by looking at their results side by side.

3. Final Answer:

   To identify which model performs best by reviewing their results side by side -> Option A

4. Quick Check:

   Comparing runs = find best model [OK]

Hint: Comparing runs means checking results to pick the best model [OK]

Common Mistakes:

• Thinking comparing runs deletes data
• Confusing comparing with creating runs
• Believing comparing changes model code

2.

Which command syntax correctly compares two experiment runs with IDs run1 and run2 under experiment exp123?

mlflow experiments compare-runs --experiment-id exp123 --run-ids run1 run2

easy

A. mlflow compare runs --experiment exp123 --ids run1,run2

B. mlflow experiments compare-runs --experiment-id exp123 --run-ids run1 run2

C. mlflow compare-runs --experiment exp123 --run-ids run1 run2

D. mlflow experiments compare --experiment-id exp123 --runs run1 run2

Solution

1. Step 1: Check official command format

   The correct MLflow command uses 'mlflow experiments compare-runs' with '--experiment-id' and '--run-ids' flags.

2. Step 2: Match options to syntax

   mlflow experiments compare-runs --experiment-id exp123 --run-ids run1 run2 matches the correct syntax exactly with proper flags and parameters.

3. Final Answer:

   mlflow experiments compare-runs --experiment-id exp123 --run-ids run1 run2 -> Option B

4. Quick Check:

   Correct command syntax = mlflow experiments compare-runs --experiment-id exp123 --run-ids run1 run2 [OK]

Hint: Use 'mlflow experiments compare-runs' with correct flags [OK]

Common Mistakes:

• Using wrong flags like --runs instead of --run-ids
• Mixing command order or names
• Separating run IDs with commas instead of spaces

3.

Given two runs with metrics:
run1: accuracy=0.85, loss=0.35
run2: accuracy=0.88, loss=0.40
Which run is better if accuracy is the main metric?

medium

A. run1 because it has higher accuracy

B. run1 because it has lower loss

C. run2 because it has higher accuracy

D. run2 because it has lower loss

Solution

1. Step 1: Identify main metric

   The question states accuracy is the main metric to compare runs.

2. Step 2: Compare accuracy values

   run1 accuracy = 0.85, run2 accuracy = 0.88. Higher accuracy is better.

3. Final Answer:

   run2 because it has higher accuracy -> Option C

4. Quick Check:

   Main metric accuracy = higher is better [OK]

Hint: Focus on main metric value to pick best run [OK]

Common Mistakes:

• Choosing run with lower loss when accuracy is main metric
• Confusing higher and lower metric values
• Ignoring stated main metric

4.

What is wrong with this command to compare runs?
mlflow experiments compare-runs --experiment-id exp123 --run-ids run1,run2

medium

A. Command should be 'mlflow compare-runs' without 'experiments'

B. Experiment ID flag should be --experiment, not --experiment-id

C. Run IDs must be specified with --runs, not --run-ids

D. Run IDs should be separated by spaces, not commas

Solution

1. Step 1: Check run IDs format

   MLflow expects run IDs separated by spaces, not commas.

2. Step 2: Verify other flags

   --experiment-id and --run-ids are correct flags; command includes 'experiments' correctly.

3. Final Answer:

   Run IDs should be separated by spaces, not commas -> Option D

4. Quick Check:

   Run IDs separated by spaces [OK]

Hint: Separate run IDs with spaces, not commas [OK]

Common Mistakes:

• Using commas between run IDs
• Changing correct flags incorrectly
• Removing 'experiments' from command

5.

You want to compare three runs but only focus on the f1_score metric. Which command correctly filters to show only this metric?

mlflow experiments compare-runs --experiment-id exp456 --run-ids runA runB runC --metric-keys f1_score

hard

A. mlflow experiments compare-runs --experiment-id exp456 --run-ids runA runB runC --metric-keys f1_score

B. mlflow experiments compare-runs --experiment-id exp456 --run-ids runA runB runC --metrics f1_score

C. mlflow experiments compare-runs --experiment-id exp456 --run-ids runA runB runC --filter f1_score

D. mlflow experiments compare-runs --experiment-id exp456 --run-ids runA runB runC --metric-filter f1_score

Solution

1. Step 1: Identify correct flag for metric filtering

   The correct flag to filter metrics in MLflow compare-runs is '--metric-keys'.

2. Step 2: Match command with options

   mlflow experiments compare-runs --experiment-id exp456 --run-ids runA runB runC --metric-keys f1_score uses '--metric-keys' correctly with the metric name 'f1_score'.

3. Final Answer:

   mlflow experiments compare-runs --experiment-id exp456 --run-ids runA runB runC --metric-keys f1_score -> Option A

4. Quick Check:

   Use --metric-keys to focus on specific metric [OK]

Hint: Use --metric-keys flag to show only chosen metric [OK]

Common Mistakes:

• Using wrong flag like --metrics or --filter
• Misspelling flag names
• Omitting metric filter when needed