Automated model validation before promotion in MLOps - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to run model validation using a CI pipeline step.

MLOps

steps:
  - name: Validate Model
    run: python validate_model.py --model [1]

Drag options to blanks, or click blank then click option'

Alatest_model.pkl

Btrain.py

Cmodel.pkl

Ddeploy.sh

Attempts:

3 left

💡 Hint

Common Mistakes

Using a script file instead of the model file.

Using a deployment script instead of the model.

The validation script requires the model file, which is typically named model.pkl.

2fill in blank

medium

Complete the code to check if model accuracy meets the threshold before promotion.

MLOps

if accuracy [1] 0.85:
    print('Model passed validation')
else:
    print('Model failed validation')

Drag options to blanks, or click blank then click option'

A!=

B<

C==

D>=

Attempts:

3 left

💡 Hint

Common Mistakes

Using less than operator which rejects good models.

Using equality which is too strict.

The model should pass validation if accuracy is greater than or equal to 0.85.

3fill in blank

hard

Fix the error in the validation script to correctly load the model file.

MLOps

with open('[1]', 'rb') as f:
    model = pickle.load(f)

Drag options to blanks, or click blank then click option'

Amodel.pkl

Bmodel.json

Cmodel.txt

Dmodel.py

Attempts:

3 left

💡 Hint

Common Mistakes

Using a text or JSON file which causes load errors.

Using a Python script file instead of a model file.

The model file is a binary pickle file, so it should be model.pkl.

4fill in blank

hard

Fill both blanks to create a dictionary comprehension that filters models with accuracy above 0.9.

MLOps

high_accuracy_models = {model[1]: acc for model, acc in results.items() if acc [2] 0.9}

Drag options to blanks, or click blank then click option'

A.upper()

B>

C>=

D.lower()

Attempts:

3 left

💡 Hint

Common Mistakes

Using lowercase instead of uppercase for model names.

Using >= instead of > which includes 0.9 exactly.

We convert model names to uppercase and select those with accuracy greater than 0.9.

5fill in blank

hard

Fill all three blanks to create a dictionary of models with accuracy above 0.8 and keys as lowercase model names.

MLOps

validated_models = { [1]: [2] for [3] in results.items() if [2] > 0.8}

Drag options to blanks, or click blank then click option'

Amodel.lower()

Bacc

C(model, acc)

Daccuracy

Attempts:

3 left

💡 Hint

Common Mistakes

Incorrect iteration variable like 'model' instead of '(model, acc)'.

Using wrong variable names for iteration or values.

The dictionary keys are lowercase model names, values are accuracies, iterating over model, acc pairs.

Practice

1. What is the main purpose of automated model validation before promotion in MLOps?

easy

A. To check if the model meets quality standards before deployment

B. To speed up the training process of the model

C. To manually review the model code for errors

D. To collect more data for training the model

Solution

1. Step 1: Understand the goal of validation

   Automated model validation is designed to ensure the model performs well and meets quality standards before it is used in production.

2. Step 2: Differentiate from other tasks

   Speeding training, manual code review, or data collection are separate tasks not directly related to validation before promotion.

3. Final Answer:

   To check if the model meets quality standards before deployment -> Option A

4. Quick Check:

   Validation ensures quality before deployment = D [OK]

Hint: Validation means checking quality before use [OK]

Common Mistakes:

• Confusing validation with training speed
• Thinking validation is manual code review
• Mixing validation with data collection

2. Which of the following is a correct way to automate model validation in a CI/CD pipeline?

easy

A. Run a script that tests model accuracy and returns pass/fail status

B. Manually check model predictions after deployment

C. Skip validation to save time during deployment

D. Only validate the model after it is in production

Solution

1. Step 1: Identify automation in CI/CD

   Automation requires scripts or tools that run tests automatically and give clear pass/fail results.

2. Step 2: Eliminate manual or delayed checks

   Manual checks or skipping validation do not fit automation principles and risk bad models in production.

3. Final Answer:

   Run a script that tests model accuracy and returns pass/fail status -> Option A

4. Quick Check:

   Automated validation uses scripts with pass/fail output = C [OK]

Hint: Automation means scripts with pass/fail results [OK]

Common Mistakes:

• Choosing manual checks as automation
• Skipping validation to save time
• Validating only after deployment

3. Given this Python snippet in a validation script:

accuracy = 0.82
threshold = 0.80
if accuracy >= threshold:
    print('PASS')
else:
    print('FAIL')

What will be the output?

medium

A. FAIL

B. PASS

C. SyntaxError

D. No output

Solution

1. Step 1: Compare accuracy with threshold

   The accuracy is 0.82, which is greater than or equal to the threshold 0.80.

2. Step 2: Determine the printed output

   Since 0.82 >= 0.80 is true, the script prints 'PASS'.

3. Final Answer:

   PASS -> Option B

4. Quick Check:

   0.82 >= 0.80 means PASS [OK]

Hint: Check if accuracy meets or exceeds threshold [OK]

Common Mistakes:

• Confusing greater than with less than
• Thinking 0.82 is less than 0.80
• Assuming syntax error due to >= symbol

4. A validation script uses this code:

if model_accuracy > threshold
    print('PASS')
else:
    print('FAIL')

What is the error and how to fix it?

medium

A. Wrong comparison operator; replace > with <

B. Incorrect variable name; change model_accuracy to accuracy

C. Indentation error; remove indentation before print

D. Missing colon after if condition; add ':' after threshold

Solution

1. Step 1: Identify syntax error in if statement

   The if statement is missing a colon ':' at the end of the condition line.

2. Step 2: Correct the syntax

   Add a colon ':' after 'threshold' to fix the syntax error.

3. Final Answer:

   Missing colon after if condition; add ':' after threshold -> Option D

4. Quick Check:

   if statements need ':' at end = A [OK]

Hint: if statements always end with ':' [OK]

Common Mistakes:

• Ignoring missing colon causing syntax error
• Changing variable names unnecessarily
• Misunderstanding indentation rules

5. You want to automate model validation to check multiple metrics before promotion. Which approach is best?

hard

A. Manually review metrics and decide promotion later

B. Promote the model if any one metric passes the threshold

C. Write a script that checks all metrics and returns 'PASS' only if all meet thresholds

D. Ignore metrics and promote based on training completion

Solution

1. Step 1: Understand multi-metric validation

   For reliable validation, all important metrics should meet their thresholds before promotion.

2. Step 2: Choose automation that enforces all checks

   A script that returns 'PASS' only if all metrics pass ensures no weak model is promoted.

3. Final Answer:

   Write a script that checks all metrics and returns 'PASS' only if all meet thresholds -> Option C

4. Quick Check:

   All metrics must pass for promotion = A [OK]

Hint: All metrics must meet thresholds to pass [OK]

Common Mistakes:

• Promoting if only one metric passes
• Relying on manual review instead of automation
• Ignoring metrics and promoting anyway