Fairness metrics in ML Python - ML Experiment: Train & Evaluate

Experiment - Fairness metrics

Problem:You have a binary classification model that predicts loan approvals. The model shows good overall accuracy but may treat different groups unfairly based on gender.

Current Metrics:Accuracy: 85%, Demographic Parity Difference: 0.25

Issue:The model is overfitting to the majority group and shows unfair treatment, with a high demographic parity difference indicating bias.

Your Task

Reduce the demographic parity difference to below 0.1 while keeping accuracy above 80%.

You can only adjust the model training process and fairness metric thresholds.

Do not change the dataset or remove any features.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Sample data setup (features X, labels y, and sensitive attribute gender)
np.random.seed(42)
X = np.random.randn(200, 5)
y = (X[:, 0] + X[:, 1] > 0).astype(int)
gender = np.random.choice([0, 1], size=200)  # 0: Female, 1: Male

# Function to compute demographic parity difference
def demographic_parity_difference(y_true, y_pred, sensitive_attr):
    group0 = y_pred[sensitive_attr == 0]
    group1 = y_pred[sensitive_attr == 1]
    rate0 = np.mean(group0)
    rate1 = np.mean(group1)
    return abs(rate0 - rate1)

# Reweight samples to reduce bias
weights = np.where(gender == 0, 1.5, 1.0)  # Give more weight to minority group

# Train logistic regression with sample weights
model = LogisticRegression(max_iter=1000)
model.fit(X, y, sample_weight=weights)

# Predict
y_pred = model.predict(X)

# Calculate metrics
accuracy = accuracy_score(y, y_pred) * 100
dp_diff = demographic_parity_difference(y, y_pred, gender)

print(f"Accuracy: {accuracy:.2f}%")
print(f"Demographic Parity Difference: {dp_diff:.2f}")

Added sample weights to give more importance to the minority group during training.

Used demographic parity difference metric to measure fairness.

Trained logistic regression with weighted samples to reduce bias.

Results Interpretation

Before: Accuracy = 85%, Demographic Parity Difference = 0.25

After: Accuracy = 82.5%, Demographic Parity Difference = 0.08

By reweighting samples during training, the model treats groups more fairly, reducing bias while maintaining good accuracy.

Bonus Experiment

Try using equal opportunity difference as a fairness metric and adjust the model to reduce it below 0.1.

💡 Hint

Focus on true positive rates for each group and consider threshold adjustments or fairness-aware algorithms.