Bias detection and fairness metrics in MLOps - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Demographic Parity

Which statement best describes the concept of demographic parity in bias detection?

A The model has equal false positive rates across all groups.

B The model's accuracy is the same for every subgroup in the data.

C The model's predictions are independent of the input features.

D The model predicts positive outcomes equally across all groups regardless of actual outcomes.

💡 Hint

Think about fairness in terms of prediction rates, not errors or accuracy.

Demographic parity means the model predicts positive outcomes at the same rate for all groups, ignoring true labels.

💻 Command Output

intermediate

Output of Fairness Metric Calculation

Given a confusion matrix for two groups, what is the output of calculating equal opportunity difference?

MLOps

Group A: TP=40, FN=10; Group B: TP=30, FN=20
Equal Opportunity Difference = TPR_GroupA - TPR_GroupB
TPR = TP / (TP + FN)

A 0.25

B 0.1

C 0.2

D 0.3

💡 Hint

Calculate TPR for each group first, then subtract.

TPR_GroupA = 40 / (40 + 10) = 0.8; TPR_GroupB = 30 / (30 + 20) = 0.6; Difference = 0.8 - 0.6 = 0.2

🔀 Workflow

advanced

Bias Detection Workflow in MLOps Pipeline

Which step correctly fits into a bias detection workflow in an MLOps pipeline?

A Collect data, train model, evaluate bias metrics, then deploy if acceptable.

B Deploy the model immediately after training without bias checks.

C Only evaluate bias metrics after deployment to save time.

D Skip bias evaluation if accuracy is above 90%.

💡 Hint

Bias detection should happen before deployment to prevent unfair models in production.

Evaluating bias metrics before deployment ensures fairness and compliance in production.

❓ Troubleshoot

advanced

Troubleshooting Unexpected Bias Metric Results

You observe that your fairness metric shows zero bias, but manual inspection reveals unfair treatment of a subgroup. What is the most likely cause?

A The fairness metric used does not capture the type of bias present.

B The model is perfectly fair and manual inspection is incorrect.

C The dataset is too large to detect bias accurately.

D The bias metric calculation has a syntax error causing zero output.

💡 Hint

Different fairness metrics capture different bias aspects.

Some metrics focus on specific fairness definitions; if the metric doesn't align with the bias type, it may miss it.

✅ Best Practice

expert

Best Practice for Continuous Bias Monitoring

What is the best practice for integrating bias detection into a continuous MLOps deployment pipeline?

A Run bias detection only during initial model training and ignore after deployment.

B Automate bias metric calculations on new data and trigger alerts if thresholds are exceeded.

C Disable bias detection to improve deployment speed.

D Manually review bias metrics quarterly without automation.

💡 Hint

Continuous monitoring requires automation and alerting.

Automating bias detection on new data ensures ongoing fairness and quick response to issues.

Practice

1. What is the main purpose of bias detection in machine learning models?

easy

A. To improve the speed of model training

B. To find unfair treatment or discrimination in model predictions

C. To increase the size of the training dataset

D. To reduce the cost of cloud computing resources

Solution

1. Step 1: Understand bias detection context

   Bias detection focuses on identifying unfair or unequal treatment of different groups by a model.

2. Step 2: Compare options to purpose

   Only To find unfair treatment or discrimination in model predictions correctly describes bias detection as finding unfair treatment in predictions.

3. Final Answer:

   To find unfair treatment or discrimination in model predictions -> Option B

4. Quick Check:

   Bias detection = find unfair treatment [OK]

Hint: Bias detection finds unfairness in model results [OK]

Common Mistakes:

• Confusing bias detection with model speed optimization
• Thinking bias detection changes dataset size
• Mixing bias detection with cost reduction

2. Which of the following is a correct way to calculate demographic parity difference in Python?

easy

A. dp_diff = abs(rate_group1 - rate_group2)

B. dp_diff = rate_group1 + rate_group2

C. dp_diff = rate_group1 * rate_group2

D. dp_diff = rate_group1 / rate_group2

Solution

1. Step 1: Understand demographic parity difference formula

   It is the absolute difference between positive outcome rates of two groups.

2. Step 2: Match formula to options

   dp_diff = abs(rate_group1 - rate_group2) correctly uses absolute difference, others use incorrect operations.

3. Final Answer:

   dp_diff = abs(rate_group1 - rate_group2) -> Option A

4. Quick Check:

   Demographic parity difference = absolute difference [OK]

Hint: Use absolute difference for parity difference [OK]

Common Mistakes:

• Using addition or multiplication instead of difference
• Forgetting to take absolute value
• Dividing rates which is not standard

3. Given the following Python code snippet, what is the output?

group1_positive_rate = 0.7
group2_positive_rate = 0.5
dp_diff = abs(group1_positive_rate - group2_positive_rate)
print(round(dp_diff, 2))

medium

A. 0.2

B. 1.20

C. 0.12

D. 0.35

Solution

1. Step 1: Calculate difference between rates

   0.7 - 0.5 = 0.2

2. Step 2: Apply absolute and rounding

   Absolute value is 0.2, rounded to 2 decimals is 0.2

3. Final Answer:

   0.2 -> Option A

4. Quick Check:

   abs(0.7 - 0.5) = 0.2 [OK]

Hint: Subtract and round absolute difference [OK]

Common Mistakes:

• Mixing up subtraction order
• Not rounding output
• Confusing decimal places

4. You wrote this code to calculate equal opportunity difference but it gives wrong results:

tpr_group1 = 0.8
tpr_group2 = 0.6
equal_opp_diff = tpr_group1 - tpr_group2
print(equal_opp_diff)

What is the likely issue?

medium

A. You need to add the true positive rates, not subtract

B. You should multiply the true positive rates instead of subtracting

C. You forgot to take the absolute value of the difference

D. The variable names are incorrect

Solution

1. Step 1: Understand equal opportunity difference metric

   It measures the absolute difference between true positive rates of groups.

2. Step 2: Check code calculation

   Code subtracts but does not take absolute value, so negative results possible.

3. Final Answer:

   You forgot to take the absolute value of the difference -> Option C

4. Quick Check:

   Equal opportunity difference = absolute difference [OK]

Hint: Always use absolute difference for fairness metrics [OK]

Common Mistakes:

• Ignoring absolute value leads to negative results
• Using addition or multiplication wrongly
• Assuming variable names cause errors

5. You want to ensure fairness in a loan approval model. The model predicts positive outcomes for two groups with rates 0.65 and 0.55. Which fairness metric and threshold would best detect bias if you want less than 10% difference between groups?

hard

A. Use accuracy score and check if difference < 0.1

B. Use recall difference and check if difference > 0.2

C. Use precision difference and check if difference > 0.1

D. Use demographic parity difference and check if difference < 0.1

Solution

1. Step 1: Identify appropriate fairness metric

   Demographic parity difference measures difference in positive prediction rates between groups.

2. Step 2: Apply threshold for bias detection

   Checking if difference is less than 0.1 (10%) ensures fairness within acceptable limits.

3. Final Answer:

   Use demographic parity difference and check if difference < 0.1 -> Option D

4. Quick Check:

   Demographic parity difference < 0.1 = fairness [OK]

Hint: Demographic parity difference < threshold detects bias [OK]

Common Mistakes:

• Using accuracy instead of fairness metrics
• Checking for difference greater than threshold incorrectly
• Confusing precision or recall with demographic parity