Performance metric tracking in MLOps - Step-by-Step Execution

Process Flow - Performance metric tracking

Start Training Model

↓

Calculate Metrics

↓

Log Metrics to Tracking System

↓

Visualize Metrics

↓

Evaluate Model Performance

↓

Decide: Improve or Deploy?

No→End

Yes↓

Adjust Model or Data

↩Back to Start Training Model

This flow shows how model training metrics are calculated, logged, visualized, and used to decide next steps.

Execution Sample

MLOps

metrics = {'accuracy': 0.85, 'loss': 0.35}
log_metrics(metrics)
visualize_metrics()
if metrics['accuracy'] > 0.8:
    deploy_model()

This code calculates metrics, logs them, visualizes, and deploys if accuracy is good.

Process Table

Step	Action	Metrics State	System State	Output/Result
1	Calculate metrics after training	{'accuracy': 0.85, 'loss': 0.35}	Metrics ready	Metrics dictionary created
2	Log metrics to tracking system	{'accuracy': 0.85, 'loss': 0.35}	Metrics logged	Metrics stored in tracking DB
3	Visualize metrics	{'accuracy': 0.85, 'loss': 0.35}	Metrics visualized	Graphs displayed on dashboard
4	Evaluate if accuracy > 0.8	{'accuracy': 0.85, 'loss': 0.35}	Decision point	Condition True
5	Deploy model	{'accuracy': 0.85, 'loss': 0.35}	Model deployed	Model available for use
6	End process	{'accuracy': 0.85, 'loss': 0.35}	Process complete	No further action

💡 Accuracy condition met, model deployed, process ends

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
metrics	{}	{'accuracy': 0.85, 'loss': 0.35}	{'accuracy': 0.85, 'loss': 0.35}	{'accuracy': 0.85, 'loss': 0.35}	{'accuracy': 0.85, 'loss': 0.35}	{'accuracy': 0.85, 'loss': 0.35}
system_state	Idle	Metrics ready	Metrics logged	Metrics visualized	Decision point	Model deployed

Key Moments - 2 Insights

Why do we log metrics before visualizing them?

Logging stores metrics persistently so visualization tools can access accurate data, as shown in step 2 and 3 of the execution_table.

What happens if the accuracy is below 0.8?

The condition in step 4 would be False, so deployment would not happen and the process would loop back to improve the model.

Visual Quiz - 3 Questions

Look at the execution_table, what is the system state after logging metrics?

A Metrics ready

B Metrics logged

C Metrics visualized

D Decision point

💡 Hint

Check the 'System State' column at step 2 in the execution_table.

At which step does the model get deployed?

A Step 5

B Step 3

C Step 4

D Step 6

💡 Hint

Look for the 'Action' column mentioning deployment in the execution_table.

If accuracy was 0.75 instead of 0.85, what would happen at step 4?

A Metrics would not be logged

B Condition would be True and model deploys

C Condition would be False and deployment skipped

D Visualization would fail

💡 Hint

Refer to the 'Evaluate if accuracy > 0.8' condition in step 4 of the execution_table.

Concept Snapshot

Performance Metric Tracking:
- Calculate metrics after model training
- Log metrics to a tracking system
- Visualize metrics on dashboards
- Use metrics to decide deployment
- Loop back if performance is insufficient

Full Transcript

Performance metric tracking in MLOps involves calculating key metrics like accuracy and loss after training a model. These metrics are then logged into a tracking system to keep a record. Visualization tools read these logged metrics to display graphs and charts for easy understanding. Based on the metrics, a decision is made whether to deploy the model or improve it further. If the accuracy is above a threshold, the model is deployed; otherwise, the process repeats with adjustments. This flow ensures continuous monitoring and improvement of model performance.

Practice

1.

What is the main purpose of performance metric tracking in MLOps?

easy

A. To manage user access to the model

B. To store raw training data

C. To measure how well a machine learning model performs

D. To create new machine learning models automatically

Solution

1. Step 1: Understand the role of performance metrics

   Performance metrics are used to evaluate the quality and effectiveness of a machine learning model.

2. Step 2: Identify the main goal of tracking these metrics

   Tracking helps to see how well the model works over time and in different conditions.

3. Final Answer:

   To measure how well a machine learning model performs -> Option C

4. Quick Check:

   Performance metric tracking = measure model quality [OK]

Hint: Metrics track model quality, not data or access [OK]

Common Mistakes:

• Confusing metrics with data storage
• Thinking metrics create models
• Mixing metrics with user management

2.

Which of the following is the correct way to log a metric named accuracy with value 0.95 using a typical MLOps tracking tool?

easy

A. log_metric(name="accuracy", value=0.95)

B. log_metric(accuracy=0.95)

C. log_metric("accuracy", "0.95")

D. log_metric(value=0.95)

Solution

1. Step 1: Identify required parameters for logging

   Logging a metric usually requires a name and a numeric value.

2. Step 2: Check syntax correctness

   log_metric(name="accuracy", value=0.95) uses named parameters with correct types: name as string and value as number.

3. Final Answer:

   log_metric(name="accuracy", value=0.95) -> Option A

4. Quick Check:

   Correct syntax needs name and numeric value [OK]

Hint: Always provide metric name and numeric value explicitly [OK]

Common Mistakes:

• Passing metric name as a keyword argument
• Using string instead of numeric value
• Omitting the metric name

3.

Given the following code snippet for metric logging, what will be the output or effect?

metrics = {}

# Log accuracy at step 1
metrics[1] = 0.85

# Log accuracy at step 2
metrics[2] = 0.90

print(metrics[2])

medium

A. 0.85

B. 0.90

C. KeyError

D. None

Solution

1. Step 1: Understand the dictionary assignments

   metrics[1] is set to 0.85, metrics[2] is set to 0.90.

2. Step 2: Identify the printed value

   print(metrics[2]) outputs the value stored at key 2, which is 0.90.

3. Final Answer:

   0.90 -> Option B

4. Quick Check:

   metrics[2] = 0.90 so print outputs 0.90 [OK]

Hint: Print the value at the requested key in the dictionary [OK]

Common Mistakes:

• Confusing keys 1 and 2
• Expecting error due to missing key
• Assuming default None output

4.

Identify the error in this metric logging code snippet:

def log_metric(name, value):
    print(f"Metric {name}: {value}")

log_metric("loss")

medium

A. Metric name should be numeric

B. Incorrect function definition syntax

C. Using print instead of return

D. Missing value argument when calling log_metric

Solution

1. Step 1: Check function parameters and call

   The function requires two arguments: name and value.

2. Step 2: Identify the call mistake

   The call provides only one argument ("loss"), missing the value argument.

3. Final Answer:

   Missing value argument when calling log_metric -> Option D

4. Quick Check:

   Function call missing required argument [OK]

Hint: Match function call arguments to definition exactly [OK]

Common Mistakes:

• Ignoring missing arguments
• Thinking print must be replaced
• Assuming metric name can be number

5.

You want to track multiple metrics (accuracy, loss) over training steps and compare models. Which approach best supports this in an MLOps system?

1. Log each metric with its name, value, and step number.
2. Store metrics in a structured format like a table or database.
3. Use the stored metrics to generate comparison reports.

Which option describes the best practice?

hard

A. Log metrics with name, value, and step; store structured; generate reports

B. Log metrics without step info and store as plain text files

C. Only log final metric values after training completes

D. Store metrics randomly without names or steps

Solution

1. Step 1: Understand metric logging needs

   Logging with name, value, and step allows tracking progress over time.

2. Step 2: Importance of structured storage and reporting

   Structured storage enables easy querying and comparison; reports help analyze results.

3. Final Answer:

   Log metrics with name, value, and step; store structured; generate reports -> Option A

4. Quick Check:

   Complete tracking needs structured logging and reporting [OK]

Hint: Include step info and use structured storage for comparisons [OK]

Common Mistakes:

• Ignoring step numbers in logs
• Storing metrics unstructured
• Logging only final values