Canary releases for model updates in MLOps - Mini Project: Build & Apply

Canary releases for model updates

📖 Scenario: You work in a team that manages machine learning models deployed in production. Your team wants to update the model safely by releasing the new version to a small group of users first. This is called a canary release. It helps catch problems early without affecting all users.

🎯 Goal: You will write a simple Python script that simulates a canary release. The script will have a list of users and two model versions. It will assign a small percentage of users to the new model (canary) and the rest to the old model (stable). Finally, it will print which user gets which model version.

📋 What You'll Learn

Create a list of exactly 10 user IDs as strings from 'user1' to 'user10'.

Create a variable called canary_percentage set to 20 to represent 20%.

Write a loop that assigns the first 20% of users to model version 'v2' (canary) and the rest to 'v1' (stable).

Print each user ID with their assigned model version.

💡 Why This Matters

🌍 Real World

Canary releases are used in real machine learning deployments to reduce risk when updating models. By exposing only a small group to the new model, teams can monitor performance and catch issues early.

💼 Career

Understanding canary releases is important for MLOps engineers and DevOps professionals who manage safe and reliable model updates in production systems.

1

Create the user list

Create a list called users with these exact string values: 'user1', 'user2', 'user3', 'user4', 'user5', 'user6', 'user7', 'user8', 'user9', 'user10'.

MLOps

# Create the list of users
# Your code here

Hint

Use square brackets [] to create a list and put each user ID as a string inside quotes.

2

Set the canary percentage

Create an integer variable called canary_percentage and set it to 20 to represent 20%.

MLOps

users = ['user1', 'user2', 'user3', 'user4', 'user5', 'user6', 'user7', 'user8', 'user9', 'user10']
# Set the canary percentage
# Your code here

Hint

Use = to assign the value 20 to the variable canary_percentage.

3

Assign users to model versions

Write a for loop using the variable index to go through range(len(users)). Inside the loop, assign model version 'v2' to users with index less than len(users) * canary_percentage // 100, else assign 'v1'. Store these assignments in a dictionary called user_model with user IDs as keys and model versions as values.

MLOps

users = ['user1', 'user2', 'user3', 'user4', 'user5', 'user6', 'user7', 'user8', 'user9', 'user10']
canary_percentage = 20
# Assign model versions to users
# Your code here

Hint

Use integer division // to calculate the number of users for the canary group.

4

Print user assignments

Write a for loop using variables user and model to iterate over user_model.items(). Inside the loop, print the string "User {user} is assigned to model {model}" using an f-string.

MLOps

users = ['user1', 'user2', 'user3', 'user4', 'user5', 'user6', 'user7', 'user8', 'user9', 'user10']
canary_percentage = 20
user_model = {}
for index in range(len(users)):
    if index < len(users) * canary_percentage // 100:
        user_model[users[index]] = 'v2'
    else:
        user_model[users[index]] = 'v1'
# Print user assignments
# Your code here

Hint

Use for user, model in user_model.items(): and an f-string inside print() to show the assignments.

Practice

1. What is the main purpose of a canary release when updating machine learning models?

easy

A. To train the model faster using more data

B. To immediately replace the old model with the new one for all users

C. To test the new model on a small group of users before full deployment

D. To reduce the size of the model for faster inference

Solution

1. Step 1: Understand canary release concept

   Canary releases deploy a new model to a small subset of users first to test its performance safely.

2. Step 2: Compare options

   Only To test the new model on a small group of users before full deployment describes testing on a small group before full rollout, which is the main purpose.

3. Final Answer:

   To test the new model on a small group of users before full deployment -> Option C

4. Quick Check:

   Canary release = small group test [OK]

Hint: Canary means small test group before full rollout [OK]

Common Mistakes:

• Thinking canary releases replace models immediately
• Confusing canary with model training speed
• Assuming canary reduces model size

2. Which of the following is the correct way to specify 10% traffic to a new model version in a deployment configuration?

easy

A. "traffic_split": {"new_model": 10, "old_model": 90}

B. "traffic_split": {"new_model": 0.1, "old_model": 0.9}

C. "traffic_split": {"new_model": "10%", "old_model": "90%"}

D. "traffic_split": {"new_model": 1, "old_model": 9}

Solution

1. Step 1: Understand traffic split format

   Traffic splits are usually specified as fractions summing to 1.0, representing percentages as decimals.

2. Step 2: Evaluate options

   "traffic_split": {"new_model": 0.1, "old_model": 0.9} uses decimal fractions (0.1 and 0.9) correctly. "traffic_split": {"new_model": 10, "old_model": 90} uses integers but not fractions. "traffic_split": {"new_model": "10%", "old_model": "90%"} uses strings with percent signs, which is invalid syntax. "traffic_split": {"new_model": 1, "old_model": 9} sums to 10, not 1.

3. Final Answer:

   "traffic_split": {"new_model": 0.1, "old_model": 0.9} -> Option B

4. Quick Check:

   Traffic split decimals sum to 1 [OK]

Hint: Use decimals summing to 1 for traffic percentages [OK]

Common Mistakes:

• Using integers instead of decimals for traffic split
• Including percent signs in values
• Traffic splits not summing to 1

3. Given this simplified code snippet for routing traffic in a canary release:

def route_request(user_id):
    if user_id % 10 == 0:
        return "new_model"
    else:
        return "old_model"

print(route_request(20))
print(route_request(23))

What will be the output?

medium

A. new_model\nold_model

B. old_model\nnew_model

C. new_model\nnew_model

D. old_model\nold_model

Solution

1. Step 1: Analyze routing logic

   The function sends users with user_id divisible by 10 to the new model, others to old model.

2. Step 2: Evaluate given user_ids

   For user_id 20: 20 % 10 == 0, so returns "new_model". For user_id 23: 23 % 10 == 3, so returns "old_model".

3. Final Answer:

   new_model old_model -> Option A

4. Quick Check:

   Divisible by 10 = new_model [OK]

Hint: Check modulo condition for routing [OK]

Common Mistakes:

• Misunderstanding modulo operator
• Swapping outputs for user IDs
• Assuming all users get new model

4. You deployed a canary release but noticed the new model is receiving 100% of traffic instead of 10%. Which fix will correct this issue?

medium

A. Change traffic split from {"new_model": 1, "old_model": 0} to {"new_model": 0.1, "old_model": 0.9}

B. Increase the new model traffic to 50% to balance load

C. Restart the deployment without changing traffic split

D. Remove the old model from deployment

Solution

1. Step 1: Identify traffic split error

   Current split {"new_model": 1, "old_model": 0} sends all traffic to new model, causing 100% traffic.

2. Step 2: Correct traffic split values

   Setting split to {"new_model": 0.1, "old_model": 0.9} correctly routes 10% traffic to new model and 90% to old model.

3. Final Answer:

   Change traffic split from {"new_model": 1, "old_model": 0} to {"new_model": 0.1, "old_model": 0.9} -> Option A

4. Quick Check:

   Traffic split controls user percentage [OK]

Hint: Check traffic split decimals sum to 1 [OK]

Common Mistakes:

• Restarting without fixing traffic split
• Increasing new model traffic without reason
• Removing old model prematurely

5. You want to safely update a model with a canary release. The new model shows better accuracy but higher latency. What is the best approach to decide whether to proceed with full rollout?

hard

A. Deploy new model only to internal users without monitoring

B. Ignore latency since accuracy is more important; rollout immediately

C. Increase traffic to new model to 100% to gather more data quickly

D. Monitor both accuracy and latency metrics during canary; rollback if latency impact is unacceptable

Solution

1. Step 1: Understand trade-offs in canary release

   Canary releases test new model performance including accuracy and latency to ensure overall user experience.

2. Step 2: Choose monitoring and rollback strategy

   Monitoring both metrics allows informed decision; rollback if latency harms user experience despite accuracy gains.

3. Final Answer:

   Monitor both accuracy and latency metrics during canary; rollback if latency impact is unacceptable -> Option D

4. Quick Check:

   Balance metrics and rollback if needed [OK]

Hint: Watch all key metrics before full rollout [OK]

Common Mistakes:

• Ignoring latency impact
• Rushing full rollout without monitoring
• Skipping rollback plans