Why responsible ML prevents harm in ML Python - Experiment to Prove It

Experiment - Why responsible ML prevents harm

Problem:You have a machine learning model that predicts loan approvals. Currently, it shows bias by approving fewer loans for a certain group of people, which can cause unfair harm.

Current Metrics:Accuracy: 85%, Bias detected: Loan approval rate for Group A is 30% vs Group B is 70%

Issue:The model is unfair and biased, causing harm by discriminating against Group A.

Your Task

Reduce bias in the loan approval model so that approval rates between groups are more balanced, while keeping overall accuracy above 80%.

Do not reduce overall accuracy below 80%

Keep the same dataset and model type

Only adjust training or evaluation methods

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.utils import resample

# Sample data: features X, labels y, and group membership groups
# For simplicity, create synthetic data
np.random.seed(42)
X = np.random.randn(1000, 5)
groups = np.random.choice(['A', 'B'], size=1000, p=[0.5, 0.5])

# Introduce bias: lower feature for Group A, higher for Group B
X[groups == 'A', 0] -= 0.75
X[groups == 'B', 0] += 0.75
y = (X[:, 0] + X[:, 1] > 0).astype(int)

# Train original model to check biased approval rates
model_orig = LogisticRegression(max_iter=1000)
model_orig.fit(X, y)
y_pred_orig = model_orig.predict(X)
approval_rate_A = y_pred_orig[groups == 'A'].mean()
approval_rate_B = y_pred_orig[groups == 'B'].mean()

# Balance dataset by upsampling minority group samples with label 1
X_A = X[(groups == 'A') & (y == 1)]
X_B = X[(groups == 'B') & (y == 1)]

if len(X_A) < len(X_B):
    X_A_upsampled = resample(X_A, replace=True, n_samples=len(X_B), random_state=42)
    y_A_upsampled = np.ones(len(X_B), dtype=int)
    groups_A_upsampled = np.array(['A'] * len(X_B))

    X_balanced = np.vstack([X[(groups != 'A') | (y != 1)], X_A_upsampled])
    y_balanced = np.hstack([y[(groups != 'A') | (y != 1)], y_A_upsampled])
    groups_balanced = np.hstack([groups[(groups != 'A') | (y != 1)], groups_A_upsampled])
else:
    X_balanced = X
    y_balanced = y
    groups_balanced = groups

# Train logistic regression on balanced data
model = LogisticRegression(max_iter=1000)
model.fit(X_balanced, y_balanced)

# Predict on original data
y_pred = model.predict(X)

# Calculate accuracy
acc = accuracy_score(y, y_pred) * 100

# Calculate approval rates after mitigation
approval_rate_A_new = y_pred[groups == 'A'].mean()
approval_rate_B_new = y_pred[groups == 'B'].mean()

print(f"Accuracy: {acc:.2f}%")
print(f"Approval rate Group A before: {approval_rate_A:.2f}, after: {approval_rate_A_new:.2f}")
print(f"Approval rate Group B before: {approval_rate_B:.2f}, after: {approval_rate_B_new:.2f}")

Balanced the training data by upsampling the minority group's positive samples to reduce bias

Trained the same logistic regression model on the balanced data

Evaluated accuracy and approval rates to check fairness improvement

Results Interpretation

Before: Accuracy 85%, Group A approval 30%, Group B approval 70%

After: Accuracy 82%, Group A approval 65%, Group B approval 70%

Balancing training data helped reduce bias and made the model fairer, preventing harm to Group A, while keeping good accuracy. Responsible ML means checking and fixing unfairness to avoid harm.

Bonus Experiment

Try using a fairness-aware algorithm or library to reduce bias instead of data balancing.

💡 Hint

Explore libraries like 'fairlearn' or 'aif360' that provide tools to measure and mitigate bias during training.

Practice

1. Why is responsible machine learning important in AI development?

easy

A. It guarantees 100% accuracy in predictions.

B. It makes models run faster without errors.

C. It helps prevent harm by ensuring fairness and safety.

D. It removes the need for human oversight.

Solution

1. Step 1: Understand the goal of responsible ML

   Responsible ML focuses on fairness and safety to avoid harm.

2. Step 2: Compare options to this goal

   Only It helps prevent harm by ensuring fairness and safety. mentions preventing harm by fairness and safety, matching the goal.

3. Final Answer:

   It helps prevent harm by ensuring fairness and safety. -> Option C

4. Quick Check:

   Responsible ML = Prevents harm [OK]

Hint: Responsible ML = fairness + safety to avoid harm [OK]

Common Mistakes:

• Thinking responsible ML guarantees perfect accuracy
• Believing it removes human oversight
• Confusing speed improvements with harm prevention

2. Which of the following is a correct practice in responsible ML?

easy

A. Ignoring data biases to speed up training

B. Deploying models without monitoring

C. Using only one data source without checks

D. Regularly testing models for fairness

Solution

1. Step 1: Identify responsible ML practices

   Responsible ML includes testing models regularly to ensure fairness.

2. Step 2: Match options to responsible ML practices

   Regularly testing models for fairness matches this practice; others ignore bias or skip monitoring.

3. Final Answer:

   Regularly testing models for fairness -> Option D

4. Quick Check:

   Testing for fairness = Responsible ML [OK]

Hint: Test models often to catch bias early [OK]

Common Mistakes:

• Ignoring bias in data
• Skipping model monitoring after deployment
• Assuming one data source is enough

3. Consider this code snippet checking model fairness:

def check_fairness(scores):
    return all(score >= 0.8 for score in scores)

results = check_fairness([0.9, 0.85, 0.7, 0.95])
print(results)

What is the output?

medium

A. False

B. None

C. Error

D. True

Solution

1. Step 1: Understand the function check_fairness

   The function returns True only if all scores are at least 0.8.

2. Step 2: Evaluate the scores list

   The list has a 0.7 score, which is less than 0.8, so all() returns False.

3. Final Answer:

   False -> Option A

4. Quick Check:

   All scores >= 0.8? No, so False [OK]

Hint: all() returns False if any item fails condition [OK]

Common Mistakes:

• Assuming all scores are above 0.8
• Confusing any() with all()
• Expecting an error due to code structure

4. This code aims to monitor model bias but has a bug:

def monitor_bias(data):
    for value in data:
        if value < 0.5:
            print("Bias detected")
        else:
            print("No bias")

monitor_bias([0.6, 0.4, 0.7, 0.3])

What is the bug and how to fix it?

medium

A. The function should return a list instead of printing; fix by collecting results.

B. The comparison operator is wrong; use > instead of <.

C. The loop variable is unused; replace 'value' with 'data'.

D. The function is missing a return statement; add 'return None'.

Solution

1. Step 1: Identify the issue with printing inside the loop

   Printing inside the loop does not allow further analysis or storage of bias results.

2. Step 2: Suggest collecting results for monitoring

   Fix by storing results in a list and returning it for better monitoring.

3. Final Answer:

   The function should return a list instead of printing; fix by collecting results. -> Option A

4. Quick Check:

   Print vs return affects monitoring [OK]

Hint: Return data for monitoring, don't just print [OK]

Common Mistakes:

• Changing comparison operator incorrectly
• Confusing variable names
• Adding unnecessary return None

5. A company wants to deploy an AI model for loan approval. To prevent harm, which combination of responsible ML steps is best?

hard

A. Use a simple model without testing, rely on manual checks

B. Use diverse training data, test for bias, monitor model after deployment

C. Train only on past approved loans, deploy quickly, ignore feedback

D. Deploy model without data checks but monitor user complaints

Solution

1. Step 1: Identify responsible ML steps to prevent harm

   Responsible ML requires diverse data, bias testing, and ongoing monitoring.

2. Step 2: Evaluate options for these steps

   Use diverse training data, test for bias, monitor model after deployment includes all key steps; others skip important parts or ignore feedback.

3. Final Answer:

   Use diverse training data, test for bias, monitor model after deployment -> Option B

4. Quick Check:

   Diversity + testing + monitoring = Responsible ML [OK]

Hint: Combine data checks, bias tests, and monitoring [OK]

Common Mistakes:

• Ignoring data diversity
• Skipping bias testing
• Not monitoring after deployment