Fairness metrics in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Equal Opportunity in Fairness Metrics

Which statement best describes the Equal Opportunity fairness metric in machine learning?

A It requires that the overall accuracy is equal across different groups.

B It requires that the false positive rates are equal across different groups.

C It requires that the true positive rates are equal across different groups.

D It requires that the predicted positive rates are equal across different groups.

💡 Hint

Think about which error type Equal Opportunity focuses on balancing.

Equal Opportunity demands that the model correctly identifies positive cases equally well across groups, meaning the true positive rates should be the same.

❓ Predict Output

intermediate

Output of Fairness Metric Calculation

Given the following confusion matrices for two groups, what is the difference in false positive rates (FPR) between Group A and Group B?

Group A: TP=40, FP=10, TN=50, FN=5
Group B: TP=30, FP=20, TN=40, FN=10

A 0.17

B 0.20

C 0.10

D 0.25

💡 Hint

FPR = FP / (FP + TN). Calculate for each group and subtract.

Group A FPR = 10 / (10 + 50) = 0.1667; Group B FPR = 20 / (20 + 40) = 0.3333; Difference = 0.3333 - 0.1667 = 0.1666 ≈ 0.17.

❓ Model Choice

advanced

Choosing a Model to Optimize Demographic Parity

You want to build a classifier that satisfies Demographic Parity. Which model choice below is most likely to help achieve this?

A A model trained with a fairness constraint that equalizes predicted positive rates across groups.

B A model trained to minimize false negative rates only.

C A model trained to maximize overall accuracy without constraints.

D A model trained with a loss function ignoring group membership.

💡 Hint

Demographic Parity focuses on equalizing positive predictions, not just accuracy.

Demographic Parity requires the predicted positive rates to be equal across groups, so training with a constraint targeting this is best.

❓ Metrics

advanced

Interpreting Disparate Impact Metric

What does a Disparate Impact value of 0.6 indicate about a model's fairness?

A The model's false positive rate is 60% higher for the privileged group.

B The model favors the unprivileged group by predicting positives more often.

C The model has 60% accuracy on the unprivileged group.

D The model predicts positive outcomes for the unprivileged group at 60% the rate of the privileged group.

💡 Hint

Disparate Impact compares positive prediction rates between groups.

A Disparate Impact of 0.6 means the unprivileged group receives positive predictions at 60% the rate of the privileged group, indicating potential bias.

🔧 Debug

expert

Debugging Fairness Metric Calculation Code

What error will this Python code raise when calculating the statistical parity difference?

def stat_parity_difference(y_pred, group):
    pos_rate_priv = sum(y_pred[group == 0]) / len(y_pred[group == 0])
    pos_rate_unpriv = sum(y_pred[group == 1]) / len(y_pred[group == 1])
    return pos_rate_unpriv - pos_rate_priv

# y_pred = [1, 0, 1, 1], group = [0, 1, 0]

ML Python

def stat_parity_difference(y_pred, group):
    pos_rate_priv = sum(y_pred[group == 0]) / len(y_pred[group == 0])
    pos_rate_unpriv = sum(y_pred[group == 1]) / len(y_pred[group == 1])
    return pos_rate_unpriv - pos_rate_priv

# y_pred = [1, 0, 1, 1], group = [0, 1, 0]

A No error; the function returns a float value.

B TypeError because boolean indexing is not supported on lists.

C IndexError due to mismatched lengths of y_pred and group arrays.

D ZeroDivisionError because one group has zero members.

💡 Hint

Check the data types and operations used for indexing.

Boolean indexing like y_pred[group == 0] works on numpy arrays, but y_pred and group are lists here, so this raises a TypeError.