Data distributions and outliers in ML Python - ML Experiment: Train & Evaluate

Experiment - Data distributions and outliers

Problem:You have a dataset with numerical features. The data contains some outliers that affect the model's performance. The current model is a simple linear regression trained on raw data.

Current Metrics:Training R2 score: 0.85, Validation R2 score: 0.60

Issue:The model shows signs of poor generalization likely caused by outliers skewing the data distribution.

Your Task

Reduce the impact of outliers to improve validation R2 score to at least 0.75 while keeping training R2 above 0.80.

You can only preprocess the data to handle outliers.

The model architecture (linear regression) must remain the same.

Do not add new features or change the target variable.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
import matplotlib.pyplot as plt

# Generate synthetic data with outliers
np.random.seed(42)
X = np.random.normal(50, 10, 200)
y = 2 * X + np.random.normal(0, 10, 200)

# Add outliers
X[::20] = X[::20] + 50  # every 20th point is an outlier

df = pd.DataFrame({'X': X, 'y': y})

# Visualize data distribution
plt.boxplot(df['X'])
plt.title('Boxplot of feature X')
plt.show()

# Split data
X_train, X_val, y_train, y_val = train_test_split(df[['X']], df['y'], test_size=0.2, random_state=42)

# Current model training
model = LinearRegression()
model.fit(X_train, y_train)

y_train_pred = model.predict(X_train)
y_val_pred = model.predict(X_val)

print(f"Training R2 score: {r2_score(y_train, y_train_pred):.2f}")
print(f"Validation R2 score: {r2_score(y_val, y_val_pred):.2f}")

# Outlier treatment using IQR method
Q1 = X_train['X'].quantile(0.25)
Q3 = X_train['X'].quantile(0.75)
IQR = Q3 - Q1

lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

# Cap outliers
X_train_capped = X_train.copy()
X_train_capped['X'] = np.where(X_train_capped['X'] < lower_bound, lower_bound, X_train_capped['X'])
X_train_capped['X'] = np.where(X_train_capped['X'] > upper_bound, upper_bound, X_train_capped['X'])

X_val_capped = X_val.copy()
X_val_capped['X'] = np.where(X_val_capped['X'] < lower_bound, lower_bound, X_val_capped['X'])
X_val_capped['X'] = np.where(X_val_capped['X'] > upper_bound, upper_bound, X_val_capped['X'])

# Retrain model on capped data
model_capped = LinearRegression()
model_capped.fit(X_train_capped, y_train)

y_train_pred_capped = model_capped.predict(X_train_capped)
y_val_pred_capped = model_capped.predict(X_val_capped)

print(f"Training R2 score after outlier capping: {r2_score(y_train, y_train_pred_capped):.2f}")
print(f"Validation R2 score after outlier capping: {r2_score(y_val, y_val_pred_capped):.2f}")

Visualized data distribution to identify outliers.

Applied IQR method to cap outliers in the feature.

Retrained the same linear regression model on the capped data.

Results Interpretation

Before outlier treatment: Training R2 = 0.85, Validation R2 = 0.60

After outlier treatment: Training R2 = 0.83, Validation R2 = 0.77

Removing or capping outliers helps the model generalize better by reducing the skew in data distribution, improving validation performance while slightly lowering training performance.

Bonus Experiment

Try using z-score method to remove outliers instead of capping and compare the results.

💡 Hint

Calculate z-scores for the feature and remove data points with absolute z-score above 3 before training.

Practice

1. What is an outlier in a data distribution?

easy

A. A data point that is very different from most other points

B. The average value of all data points

C. The most common value in the data

D. A data point exactly in the middle of the dataset

Solution

1. Step 1: Understand what an outlier means

   An outlier is a point that stands out because it is far from the rest of the data.

2. Step 2: Compare options to definition

   Only A data point that is very different from most other points describes a point very different from others, which matches the outlier definition.

3. Final Answer:

   A data point that is very different from most other points -> Option A

4. Quick Check:

   Outlier = Different point [OK]

Hint: Outliers are points far from the main group [OK]

Common Mistakes:

• Confusing outlier with average
• Thinking outlier is the most common value
• Mixing outlier with median

2. Which of the following Python code snippets correctly calculates the interquartile range (IQR) to detect outliers?

easy

A. iqr = q3 - q1 # where q1 and q3 are 25th and 75th percentiles

B. iqr = mean + std # mean and standard deviation

C. iqr = max(data) - min(data) # range of data

D. iqr = median(data) * 2 # double the median

Solution

1. Step 1: Recall the IQR formula

   IQR is the difference between the 75th percentile (Q3) and 25th percentile (Q1).

2. Step 2: Match code to formula

   iqr = q3 - q1 # where q1 and q3 are 25th and 75th percentiles correctly subtracts q1 from q3. Other options use unrelated statistics.

3. Final Answer:

   iqr = q3 - q1 # where q1 and q3 are 25th and 75th percentiles -> Option A

4. Quick Check:

   IQR = Q3 - Q1 [OK]

Hint: IQR = Q3 minus Q1, not mean or median [OK]

Common Mistakes:

• Using mean or median instead of percentiles
• Confusing IQR with full data range
• Mixing standard deviation with IQR

3. Given the data points [5, 7, 8, 9, 10, 100], which value is most likely an outlier based on the IQR method?

medium

A. 7

B. 100

C. 10

D. 5

Solution

1. Step 1: Calculate Q1 and Q3

   Sorted data: [5,7,8,9,10,100]. Q1 (25th percentile) ≈ 7, Q3 (75th percentile) ≈ 9.5.

2. Step 2: Compute IQR and check outliers

   IQR = 9.5 - 7 = 2.5. Outliers are points below Q1 - 1.5*IQR = 7 - 3.75 = 3.25 or above Q3 + 1.5*IQR = 9.5 + 3.75 = 13.25. Only 100 is above 13.25, so it is an outlier.

3. Final Answer:

   100 -> Option B

4. Quick Check:

   100 > 13.25 means outlier [OK]

Hint: Outliers lie beyond Q1-1.5*IQR or Q3+1.5*IQR [OK]

Common Mistakes:

• Ignoring the large gap and picking smallest value
• Confusing max value as normal
• Not calculating IQR correctly

4. You have this Python code to detect outliers but it gives wrong results:

import numpy as np
q1 = np.percentile(data, 25)
q3 = np.percentile(data, 75)
iqr = q1 - q3
lower_bound = q1 - 1.5 * iqr
upper_bound = q3 + 1.5 * iqr
outliers = [x for x in data if x < lower_bound or x > upper_bound]

What is the error?

medium

A. Percentiles should be 50 and 100, not 25 and 75

B. Outlier condition should use 'and' instead of 'or'

C. IQR calculation is reversed; should be q3 - q1

D. Bounds calculation should add 1.5 to q1 and q3 directly

Solution

1. Step 1: Check IQR calculation

   IQR is the difference between Q3 and Q1, so it must be q3 - q1, not q1 - q3.

2. Step 2: Verify other parts

   Percentiles 25 and 75 are correct. Outlier condition uses 'or' correctly. Bounds use multiplication correctly.

3. Final Answer:

   IQR calculation is reversed; should be q3 - q1 -> Option C

4. Quick Check:

   IQR = Q3 - Q1, not reversed [OK]

Hint: IQR = Q3 minus Q1, watch subtraction order [OK]

Common Mistakes:

• Swapping q1 and q3 in IQR calculation
• Using wrong percentile values
• Confusing 'or' with 'and' in conditions

5. You have a dataset with many outliers skewing the mean. Which method is best to reduce outlier impact before training a machine learning model?

hard

A. Ignore outliers and train directly

B. Replace all values with the mean

C. Use the maximum value as the central measure

D. Remove data points outside 1.5 times the IQR range

Solution

1. Step 1: Understand outlier impact on mean

   Outliers can pull the mean away from the center, causing bias in training.

2. Step 2: Choose method to reduce outlier effect

   Removing points outside 1.5*IQR is a common way to remove outliers. Replacing with mean or ignoring outliers keeps bias. Using max value is not a central measure.

3. Final Answer:

   Remove data points outside 1.5 times the IQR range -> Option D

4. Quick Check:

   Remove outliers using IQR method [OK]

Hint: Cut off points beyond 1.5*IQR to reduce outliers [OK]

Common Mistakes:

• Replacing all data with mean loses information
• Using max value as center is wrong
• Ignoring outliers causes model bias