Canary releases for model updates in MLOps - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to deploy a new model version using a canary release.

MLOps

deploy_model(version='v2', traffic_split=[1])

Drag options to blanks, or click blank then click option'

A10

B100

C0

D50

Attempts:

3 left

💡 Hint

Common Mistakes

Setting traffic to 100% immediately, which is not a canary release.

In canary releases, a small percentage of traffic (like 10%) is routed to the new model version first.

2fill in blank

medium

Complete the code to monitor the canary model's performance metric.

MLOps

monitor_metric(model_version='v2', metric='latency', threshold=[1])

Drag options to blanks, or click blank then click option'

A0.5

B5.0

C1.0

D10.0

Attempts:

3 left

💡 Hint

Common Mistakes

Setting threshold too high, missing early performance problems.

The threshold is set to 1.0 seconds latency to detect performance issues early.

3fill in blank

hard

Fix the error in the canary release rollout command.

MLOps

rollout_canary(model='v2', traffic=[1], duration='1h')

Drag options to blanks, or click blank then click option'

A'10%'

B10

C0.1

Dten

Attempts:

3 left

💡 Hint

Common Mistakes

Using string values for traffic percentage.

The traffic parameter expects an integer percentage without quotes.

4fill in blank

hard

Fill both blanks to create a dictionary for traffic routing in canary release.

MLOps

traffic_split = {'stable': [1], 'canary': [2]

Drag options to blanks, or click blank then click option'

A90

B10

C50

D100

Attempts:

3 left

💡 Hint

Common Mistakes

Assigning equal traffic to stable and canary.

In canary releases, most traffic goes to stable (90%) and a small part to canary (10%).

5fill in blank

hard

Fill all three blanks to define a canary release function with parameters for version, traffic, and duration.

MLOps

def canary_release(version=[1], traffic=[2], duration=[3]):
    print(f"Deploying {version} with {traffic}% traffic for {duration}.")

Drag options to blanks, or click blank then click option'

A'v2'

B10

C'1h'

D'v1'

Attempts:

3 left

💡 Hint

Common Mistakes

Using wrong data types for parameters.

The function deploys version 'v2' with 10% traffic for 1 hour duration.

Practice

1. What is the main purpose of a canary release when updating machine learning models?

easy

A. To train the model faster using more data

B. To immediately replace the old model with the new one for all users

C. To test the new model on a small group of users before full deployment

D. To reduce the size of the model for faster inference

Solution

1. Step 1: Understand canary release concept

   Canary releases deploy a new model to a small subset of users first to test its performance safely.

2. Step 2: Compare options

   Only To test the new model on a small group of users before full deployment describes testing on a small group before full rollout, which is the main purpose.

3. Final Answer:

   To test the new model on a small group of users before full deployment -> Option C

4. Quick Check:

   Canary release = small group test [OK]

Hint: Canary means small test group before full rollout [OK]

Common Mistakes:

• Thinking canary releases replace models immediately
• Confusing canary with model training speed
• Assuming canary reduces model size

2. Which of the following is the correct way to specify 10% traffic to a new model version in a deployment configuration?

easy

A. "traffic_split": {"new_model": 10, "old_model": 90}

B. "traffic_split": {"new_model": 0.1, "old_model": 0.9}

C. "traffic_split": {"new_model": "10%", "old_model": "90%"}

D. "traffic_split": {"new_model": 1, "old_model": 9}

Solution

1. Step 1: Understand traffic split format

   Traffic splits are usually specified as fractions summing to 1.0, representing percentages as decimals.

2. Step 2: Evaluate options

   "traffic_split": {"new_model": 0.1, "old_model": 0.9} uses decimal fractions (0.1 and 0.9) correctly. "traffic_split": {"new_model": 10, "old_model": 90} uses integers but not fractions. "traffic_split": {"new_model": "10%", "old_model": "90%"} uses strings with percent signs, which is invalid syntax. "traffic_split": {"new_model": 1, "old_model": 9} sums to 10, not 1.

3. Final Answer:

   "traffic_split": {"new_model": 0.1, "old_model": 0.9} -> Option B

4. Quick Check:

   Traffic split decimals sum to 1 [OK]

Hint: Use decimals summing to 1 for traffic percentages [OK]

Common Mistakes:

• Using integers instead of decimals for traffic split
• Including percent signs in values
• Traffic splits not summing to 1

3. Given this simplified code snippet for routing traffic in a canary release:

def route_request(user_id):
    if user_id % 10 == 0:
        return "new_model"
    else:
        return "old_model"

print(route_request(20))
print(route_request(23))

What will be the output?

medium

A. new_model\nold_model

B. old_model\nnew_model

C. new_model\nnew_model

D. old_model\nold_model

Solution

1. Step 1: Analyze routing logic

   The function sends users with user_id divisible by 10 to the new model, others to old model.

2. Step 2: Evaluate given user_ids

   For user_id 20: 20 % 10 == 0, so returns "new_model". For user_id 23: 23 % 10 == 3, so returns "old_model".

3. Final Answer:

   new_model old_model -> Option A

4. Quick Check:

   Divisible by 10 = new_model [OK]

Hint: Check modulo condition for routing [OK]

Common Mistakes:

• Misunderstanding modulo operator
• Swapping outputs for user IDs
• Assuming all users get new model

4. You deployed a canary release but noticed the new model is receiving 100% of traffic instead of 10%. Which fix will correct this issue?

medium

A. Change traffic split from {"new_model": 1, "old_model": 0} to {"new_model": 0.1, "old_model": 0.9}

B. Increase the new model traffic to 50% to balance load

C. Restart the deployment without changing traffic split

D. Remove the old model from deployment

Solution

1. Step 1: Identify traffic split error

   Current split {"new_model": 1, "old_model": 0} sends all traffic to new model, causing 100% traffic.

2. Step 2: Correct traffic split values

   Setting split to {"new_model": 0.1, "old_model": 0.9} correctly routes 10% traffic to new model and 90% to old model.

3. Final Answer:

   Change traffic split from {"new_model": 1, "old_model": 0} to {"new_model": 0.1, "old_model": 0.9} -> Option A

4. Quick Check:

   Traffic split controls user percentage [OK]

Hint: Check traffic split decimals sum to 1 [OK]

Common Mistakes:

• Restarting without fixing traffic split
• Increasing new model traffic without reason
• Removing old model prematurely

5. You want to safely update a model with a canary release. The new model shows better accuracy but higher latency. What is the best approach to decide whether to proceed with full rollout?

hard

A. Deploy new model only to internal users without monitoring

B. Ignore latency since accuracy is more important; rollout immediately

C. Increase traffic to new model to 100% to gather more data quickly

D. Monitor both accuracy and latency metrics during canary; rollback if latency impact is unacceptable

Solution

1. Step 1: Understand trade-offs in canary release

   Canary releases test new model performance including accuracy and latency to ensure overall user experience.

2. Step 2: Choose monitoring and rollback strategy

   Monitoring both metrics allows informed decision; rollback if latency harms user experience despite accuracy gains.

3. Final Answer:

   Monitor both accuracy and latency metrics during canary; rollback if latency impact is unacceptable -> Option D

4. Quick Check:

   Balance metrics and rollback if needed [OK]

Hint: Watch all key metrics before full rollout [OK]

Common Mistakes:

• Ignoring latency impact
• Rushing full rollout without monitoring
• Skipping rollback plans