Platform observability and SLAs in MLOps - Commands & Configuration

Introduction

When you run machine learning models in production, you need to watch how well the system works and make sure it meets promised performance levels. Platform observability helps you see inside the system, and SLAs set clear goals for uptime and response times.

When you want to track if your ML model is running without errors in production

When you need to know if your prediction service is responding quickly enough for users

When you want to get alerts if the system is down or behaving badly

When you want to share clear performance promises with your team or customers

When you want to improve your ML system by understanding its behavior over time

Commands

Start the MLflow tracking server to collect metrics and logs from your ML models. This helps observe model performance and system health.

Terminal

mlflow server --host 0.0.0.0 --port 5000

Expected Output

2024/06/01 12:00:00 INFO mlflow.server: Starting MLflow tracking server at http://0.0.0.0:5000

→

--host - Bind the server to all network interfaces so it can be accessed remotely

→

--port - Set the port number where the server listens

Run your ML project which logs metrics and parameters to the MLflow server for observability.

Terminal

mlflow run .

Expected Output

2024/06/01 12:01:00 INFO mlflow.projects: Running ML project

2024/06/01 12:01:10 INFO mlflow.projects: Run completed successfully

Fetch metrics for a specific MLflow run to check model performance and system behavior.

Terminal

curl -X POST http://localhost:5000/api/2.0/preview/mlflow/metrics/get -d '{"run_id": "12345"}'

Expected Output

{"metrics": [{"key": "accuracy", "value": 0.92, "timestamp": 1685610000}]}

Create a simple alert rule to notify if prediction latency exceeds 200 milliseconds, helping maintain SLA.

Terminal

echo 'alert: high_latency
condition: prediction_latency > 200ms
action: send_email' > alert_rule.yaml

Expected Output

No output (command runs silently)

Apply the alert rule to the MLflow monitoring system to enforce SLA conditions and get notified on issues.

Terminal

mlflow alerts apply -f alert_rule.yaml

Expected Output

Alert rule 'high_latency' applied successfully

Key Concept

If you remember nothing else from this pattern, remember: observability means collecting clear data about your ML system so you can meet and prove your SLAs.

Common Mistakes

Not starting the MLflow server before running the ML project

Metrics and logs have nowhere to go, so you lose observability data

Always start the MLflow tracking server first to collect data

Ignoring alert rules and not setting thresholds for key metrics

You won't get notified when the system breaks SLA, causing downtime or bad user experience

Define clear alert rules for important metrics like latency and error rates

Fetching metrics without specifying the correct run ID

You get no data or wrong data, making it hard to understand system health

Always use the exact run ID from your MLflow runs when querying metrics

Summary

Start the MLflow tracking server to collect metrics and logs from your ML models.

Run your ML project to log performance data for observability.

Fetch metrics using MLflow API to check if your system meets SLAs.

Create and apply alert rules to get notified when SLAs are violated.

Practice

1. What is the main purpose of platform observability in MLOps?

easy

A. To monitor and understand system performance in real time

B. To set legal contracts with users

C. To deploy machine learning models automatically

D. To store large amounts of data efficiently

Solution

1. Step 1: Understand observability concept

   Observability means seeing how the system behaves and performs live.

2. Step 2: Match purpose with options

   Only To monitor and understand system performance in real time talks about monitoring and understanding performance in real time.

3. Final Answer:

   To monitor and understand system performance in real time -> Option A

4. Quick Check:

   Observability = Real-time performance monitoring [OK]

Hint: Observability = watching system health live [OK]

Common Mistakes:

• Confusing observability with deployment
• Thinking observability sets contracts
• Mixing observability with data storage

2. Which of the following is the correct way to define an SLA uptime of 99.9% in a YAML configuration?

easy

A. sla:
uptime: '99.9%'

B. sla:
uptime: 99.9

C. sla:
uptime: 0.999

D. sla:
uptime: '99,9%'

Solution

1. Step 1: Understand SLA uptime format

   SLA uptime is usually expressed as a percentage string like '99.9%'.

2. Step 2: Check YAML syntax and value correctness

   sla: uptime: '99.9%' uses correct YAML syntax and proper string format with percent sign.

3. Final Answer:

   sla:\n uptime: '99.9%' -> Option A

4. Quick Check:

   Correct SLA uptime format = '99.9%' string [OK]

Hint: Use string with percent sign for SLA uptime [OK]

Common Mistakes:

• Using number without percent sign
• Using decimal instead of percentage
• Using comma instead of dot in percentage

3. Given this monitoring alert rule snippet:

if error_rate > 0.05:
  alert('High error rate')
else:
  alert('Error rate normal')

What will be the alert message if error_rate is 0.03?

medium

A. No alert

B. High error rate

C. Error rate normal

D. Syntax error

Solution

1. Step 1: Evaluate the condition with error_rate = 0.03

   0.03 is less than 0.05, so the condition error_rate > 0.05 is false.

2. Step 2: Determine which alert triggers

   Since condition is false, the else branch runs, triggering alert('Error rate normal').

3. Final Answer:

   Error rate normal -> Option C

4. Quick Check:

   0.03 < 0.05 triggers else alert [OK]

Hint: Check if error_rate exceeds threshold [OK]

Common Mistakes:

• Confusing greater than with less than
• Assuming no alert triggers
• Thinking code has syntax error

4. You have this SLA configuration:

sla:
  uptime: '99.95%'
  response_time_ms: 200

But your monitoring shows frequent alerts for response time exceeding 200ms. What is the most likely cause?

medium

A. The uptime percentage is incorrect

B. The SLA response_time_ms is set too low for actual system performance

C. The SLA syntax is invalid YAML

D. The monitoring tool is not running

Solution

1. Step 1: Analyze SLA and alert mismatch

   The SLA sets response_time_ms to 200ms, but alerts show it often exceeds this.

2. Step 2: Identify cause of frequent alerts

   This means the system often responds slower than 200ms, so SLA is too strict or system needs improvement.

3. Final Answer:

   The SLA response_time_ms is set too low for actual system performance -> Option B

4. Quick Check:

   Strict SLA causes frequent alerts [OK]

Hint: Check if SLA limits match real system speed [OK]

Common Mistakes:

• Blaming uptime for response time alerts
• Assuming YAML syntax error without checking
• Ignoring monitoring tool status

5. You want to combine observability metrics and SLA checks to alert only when uptime drops below 99.9% and error rate exceeds 1%. Which pseudo-code correctly implements this?

hard

A. if uptime >= 99.9 and error_rate >= 0.01:
alert('SLA breach')

B. if uptime > 99.9 or error_rate < 0.01:
alert('SLA breach')

C. if uptime <= 99.9 and error_rate <= 0.01:
alert('SLA breach')

D. if uptime < 99.9 and error_rate > 0.01:
alert('SLA breach')

Solution

1. Step 1: Understand SLA breach conditions

   SLA breach means uptime is less than 99.9% AND error rate is greater than 1% (0.01).

2. Step 2: Match condition logic with options

   if uptime < 99.9 and error_rate > 0.01: alert('SLA breach') uses < for uptime and > for error rate combined with AND, matching the requirement exactly.

3. Final Answer:

   if uptime < 99.9 and error_rate > 0.01:\n alert('SLA breach') -> Option D

4. Quick Check:

   Use AND with correct inequalities for SLA breach [OK]

Hint: Use AND with uptime < 99.9 and error_rate > 0.01 [OK]

Common Mistakes:

• Using OR instead of AND
• Reversing inequality signs
• Alerting on normal conditions