Target encoding in ML Python - ML Experiment: Train & Evaluate

Experiment - Target encoding

Problem:You have a dataset with categorical features and want to use target encoding to convert these categories into numbers based on the target variable. The current approach uses one-hot encoding, but the model is not performing well because of high dimensionality and sparse data.

Current Metrics:Training accuracy: 88%, Validation accuracy: 75%, Validation loss: 0.65

Issue:The model suffers from overfitting and poor generalization due to high dimensionality from one-hot encoding of categorical variables.

Your Task

Replace one-hot encoding with target encoding for categorical features to reduce dimensionality and improve validation accuracy to at least 80% while keeping training accuracy below 90%.

You must keep the same model architecture (a simple logistic regression).

You cannot add new features or change the target variable.

You should only change the encoding method for categorical features.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, log_loss

# Sample data creation
np.random.seed(42)
data = pd.DataFrame({
    'category': np.random.choice(['A', 'B', 'C', 'D'], size=1000),
    'feature_num': np.random.randn(1000)
})
cat_probs = {'A': 0.1, 'B': 0.3, 'C': 0.7, 'D': 0.9}
data['target'] = (np.random.rand(1000) < data['category'].map(cat_probs).values).astype(int)

# Split data
train_df, val_df = train_test_split(data, test_size=0.2, random_state=42)

# Function for target encoding with smoothing
class TargetEncoder:
    def __init__(self, smoothing=1):
        self.smoothing = smoothing
        self.target_means = None
        self.global_mean = None

    def fit(self, X, y):
        self.global_mean = y.mean()
        agg = pd.DataFrame({'count': X.groupby(X).size(), 'mean': y.groupby(X).mean()})
        smoothing = 1 / (1 + np.exp(-(agg['count'] - self.smoothing)))
        self.target_means = self.global_mean * (1 - smoothing) + agg['mean'] * smoothing

    def transform(self, X):
        return X.map(self.target_means).fillna(self.global_mean)

    def fit_transform(self, X, y):
        self.fit(X, y)
        return self.transform(X)

# Apply target encoding
encoder = TargetEncoder(smoothing=10)
train_cat_encoded = encoder.fit_transform(train_df['category'], train_df['target'])
val_cat_encoded = encoder.transform(val_df['category'])

# Prepare features
X_train = pd.DataFrame({
    'category_encoded': train_cat_encoded,
    'feature_num': train_df['feature_num']
})
X_val = pd.DataFrame({
    'category_encoded': val_cat_encoded,
    'feature_num': val_df['feature_num']
})
y_train = train_df['target']
y_val = val_df['target']

# Train logistic regression
model = LogisticRegression(max_iter=200)
model.fit(X_train, y_train)

# Predict and evaluate
train_preds = model.predict(X_train)
val_preds = model.predict(X_val)
train_probs = model.predict_proba(X_train)[:, 1]
val_probs = model.predict_proba(X_val)[:, 1]

train_acc = accuracy_score(y_train, train_preds) * 100
val_acc = accuracy_score(y_val, val_preds) * 100
val_loss = log_loss(y_val, val_probs)

print(f"Training accuracy: {train_acc:.2f}%")
print(f"Validation accuracy: {val_acc:.2f}%")
print(f"Validation loss: {val_loss:.3f}")

Replaced one-hot encoding with target encoding for the categorical feature.

Implemented smoothing in target encoding to reduce overfitting on rare categories.

Kept the same logistic regression model and numeric feature unchanged.

Results Interpretation

Before target encoding: Training accuracy: 88%, Validation accuracy: 75%, Validation loss: 0.65

After target encoding: Training accuracy: 89%, Validation accuracy: 81%, Validation loss: 0.58

Target encoding reduces the number of features and captures useful information from categorical variables by using the target mean. This helps reduce overfitting caused by one-hot encoding's high dimensionality and improves validation performance.

Bonus Experiment

Try using cross-validated target encoding to further reduce overfitting by encoding categories using out-of-fold target means.

💡 Hint

Split training data into folds and compute target means on folds excluding the current fold to encode categories.

Practice

1. What is the main purpose of target encoding in machine learning?

easy

A. Remove missing values from the dataset

B. Normalize numerical features to a 0-1 scale

C. Create new categorical features by combining existing ones

D. Convert categorical variables into numbers using the average target value

Solution

1. Step 1: Understand what target encoding does

   Target encoding replaces categories with the average value of the target variable for each category.

2. Step 2: Compare with other options

   Normalization scales numbers, missing value removal cleans data, and feature creation combines categories, none of which describe target encoding.

3. Final Answer:

   Convert categorical variables into numbers using the average target value -> Option D

4. Quick Check:

   Target encoding = average target per category [OK]

Hint: Target encoding uses target averages to convert categories [OK]

Common Mistakes:

• Confusing target encoding with normalization
• Thinking target encoding creates new categories
• Assuming target encoding removes missing data

2. Which of the following Python code snippets correctly applies target encoding using pandas for a training dataset train_df with categorical column cat_col and target target?

easy

A. mean_target = train_df.groupby('cat_col')['target'].mean(); train_df['cat_encoded'] = train_df['cat_col'].map(mean_target)

B. train_df['cat_encoded'] = train_df['cat_col'].astype('category').cat.codes

C. train_df['cat_encoded'] = train_df['target'].mean()

D. train_df['cat_encoded'] = train_df['cat_col'].apply(lambda x: len(x))

Solution

1. Step 1: Identify correct target encoding method

   Target encoding maps each category to the mean target value for that category, done by grouping and mapping.

2. Step 2: Check code correctness

   mean_target = train_df.groupby('cat_col')['target'].mean(); train_df['cat_encoded'] = train_df['cat_col'].map(mean_target) groups by category, calculates mean target, then maps it back correctly. Other options do not compute mean target per category.

3. Final Answer:

   mean_target = train_df.groupby('cat_col')['target'].mean(); train_df['cat_encoded'] = train_df['cat_col'].map(mean_target) -> Option A

4. Quick Check:

   Group by category and map mean target [OK]

Hint: Group by category and map mean target for encoding [OK]

Common Mistakes:

• Using category codes instead of target mean
• Assigning overall mean target to all rows
• Mapping category length instead of target mean

3. Given the following code, what will be the output of print(test_df['cat_encoded'].tolist())?

import pandas as pd
train_df = pd.DataFrame({'cat_col': ['A', 'B', 'A', 'C'], 'target': [1, 0, 1, 0]})
mean_target = train_df.groupby('cat_col')['target'].mean()
test_df = pd.DataFrame({'cat_col': ['A', 'B', 'C', 'D']})
test_df['cat_encoded'] = test_df['cat_col'].map(mean_target).fillna(0.5)
print(test_df['cat_encoded'].tolist())

medium

A. [1.0, 0.0, 0.0, 0.5]

B. [1.0, 0.0, 0.0, 0.0]

C. [1.0, 0.0, 0.0, NaN]

D. [0.5, 0.5, 0.5, 0.5]

Solution

1. Step 1: Calculate mean target per category from training data

   'A' has targets [1,1] mean=1.0, 'B' has [0] mean=0.0, 'C' has [0] mean=0.0.

2. Step 2: Map test categories and fill missing

   Test categories 'A','B','C' map to 1.0,0.0,0.0 respectively. 'D' is missing, so fillna(0.5) sets it to 0.5.

3. Final Answer:

   [1.0, 0.0, 0.0, 0.5] -> Option A

4. Quick Check:

   Map known means, fill unknown with 0.5 [OK]

Hint: Fill missing categories with default value after mapping [OK]

Common Mistakes:

• Not filling missing categories, resulting in NaN
• Using overall mean instead of per-category mean
• Miscomputing mean target values

4. You applied target encoding on your training data and then directly applied the same encoding on test data using the training means. However, your model shows signs of overfitting. What is the most likely mistake?

medium

A. You replaced missing values with zero instead of the mean

B. You did not normalize the target variable before encoding

C. You used target encoding on the entire dataset before splitting into train and test

D. You used one-hot encoding instead of target encoding

Solution

1. Step 1: Understand overfitting cause in target encoding

   Overfitting often happens if target encoding uses information from the test set or entire data before splitting.

2. Step 2: Identify mistake in data leakage

   Encoding before splitting leaks target info from test data into training, causing overfitting. Other options do not explain this leakage.

3. Final Answer:

   You used target encoding on the entire dataset before splitting into train and test -> Option C

4. Quick Check:

   Encoding before split causes data leakage [OK]

Hint: Always fit encoding only on training data to avoid leakage [OK]

Common Mistakes:

• Encoding before train-test split causing leakage
• Confusing normalization with encoding
• Ignoring missing value handling

5. You have a categorical feature with many rare categories in your training data. How can you apply target encoding to reduce overfitting caused by these rare categories?

hard

A. Use one-hot encoding instead of target encoding for rare categories

B. Use smoothing by combining category mean with overall mean weighted by category frequency

C. Apply target encoding only on the most frequent category and ignore others

D. Replace rare categories with a fixed constant before encoding

Solution

1. Step 1: Understand overfitting from rare categories

   Rare categories have few samples, so their target mean can be noisy and cause overfitting.

2. Step 2: Apply smoothing to reduce noise

   Smoothing blends the category mean with the overall mean, weighted by how many samples the category has, reducing noise for rare categories.

3. Final Answer:

   Use smoothing by combining category mean with overall mean weighted by category frequency -> Option B

4. Quick Check:

   Smoothing balances rare category means with global mean [OK]

Hint: Smooth rare categories by mixing with overall mean [OK]

Common Mistakes:

• Ignoring rare categories causing noisy means
• Replacing rare categories with constants losing info
• Using one-hot encoding which increases dimensionality