Fairness metrics in ML Python - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of fairness metrics in machine learning?

Fairness metrics help us check if a model treats different groups of people equally, avoiding bias and unfair treatment.

beginner

Explain 'Demographic Parity' as a fairness metric.

Demographic Parity means the model should give positive outcomes equally across different groups, regardless of their group membership.

intermediate

What does 'Equalized Odds' measure in fairness?

Equalized Odds checks if the model has equal true positive rates and false positive rates for all groups, ensuring fairness in errors and correct predictions.

intermediate

Define 'Predictive Parity' in fairness metrics.

Predictive Parity means the model's positive predictions should be equally accurate for all groups, so the chance that a positive prediction is correct is the same.

advanced

Why can it be difficult to satisfy all fairness metrics at once?

Because some fairness metrics can conflict with each other, improving one might make another worse. This is called the fairness trade-off.

Which fairness metric requires equal positive prediction rates across groups?

A Demographic Parity

B Equalized Odds

C Predictive Parity

D Calibration

Demographic Parity means the model should give positive outcomes at the same rate for all groups.

Equalized Odds ensures equal rates of which of the following across groups?

A Positive prediction values

B Overall accuracy

C True positive and false positive rates

D Negative prediction values

Equalized Odds requires equal true positive rates and false positive rates for all groups.

Predictive Parity focuses on equalizing what aspect of model predictions?

A The accuracy of positive predictions

B The proportion of positive predictions

C The false negative rate

D The overall error rate

Predictive Parity means the chance that a positive prediction is correct should be equal across groups.

What is a common challenge when applying multiple fairness metrics simultaneously?

A They always improve model accuracy

B They require no data preprocessing

C They guarantee fairness without tuning

D They can conflict and cause trade-offs

Fairness metrics can conflict, so improving one may worsen another, creating trade-offs.

Which fairness metric is concerned with equalizing error rates between groups?

A Demographic Parity

B Equalized Odds

C Predictive Parity

D Statistical Parity

Equalized Odds focuses on equalizing true positive and false positive error rates across groups.

Describe three common fairness metrics used to evaluate machine learning models and explain what each one measures.

Think about how each metric checks fairness from a different angle: outcomes, errors, or prediction accuracy.

Explain why it might be impossible to satisfy all fairness metrics at the same time in a machine learning model.

Consider how improving fairness for one group or metric can affect others negatively.