Handling categorical variables in ML Python - ML Experiment: Train & Evaluate

Experiment - Handling categorical variables

Problem:You want to build a model to predict if a customer will buy a product based on features including categorical variables like 'color' and 'city'. The current model uses raw categorical data without encoding.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%

Issue:The model is overfitting because it cannot properly understand categorical variables in their raw form.

Your Task

Reduce overfitting by properly encoding categorical variables so that validation accuracy improves to at least 85% while keeping training accuracy below 90%.

You must keep the same model architecture (a simple logistic regression).

You can only change how categorical variables are handled.

Do not change the dataset or add new features.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Sample data
data = pd.DataFrame({
    'color': ['red', 'blue', 'green', 'blue', 'red', 'green', 'red', 'blue'],
    'city': ['NY', 'LA', 'NY', 'LA', 'NY', 'LA', 'NY', 'LA'],
    'age': [25, 32, 47, 51, 23, 43, 36, 29],
    'bought': [1, 0, 1, 0, 1, 0, 1, 0]
})

# Split features and target
X = data[['color', 'city', 'age']]
y = data['bought']

# Split train and validation
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.25, random_state=42)

# One-hot encode categorical variables
encoder = OneHotEncoder(sparse=False, drop='first')
X_train_cat = encoder.fit_transform(X_train[['color', 'city']])
X_val_cat = encoder.transform(X_val[['color', 'city']])

# Combine encoded categorical with numerical
import numpy as np
X_train_final = np.hstack([X_train_cat, X_train[['age']].values])
X_val_final = np.hstack([X_val_cat, X_val[['age']].values])

# Train logistic regression
model = LogisticRegression(max_iter=1000)
model.fit(X_train_final, y_train)

# Predict and evaluate
train_preds = model.predict(X_train_final)
val_preds = model.predict(X_val_final)

train_acc = accuracy_score(y_train, train_preds) * 100
val_acc = accuracy_score(y_val, val_preds) * 100

print(f'Training accuracy: {train_acc:.2f}%')
print(f'Validation accuracy: {val_acc:.2f}%')

Applied one-hot encoding to categorical variables 'color' and 'city' to convert them into numeric format.

Kept the logistic regression model unchanged.

Combined encoded categorical features with numerical feature 'age' before training.

Results Interpretation

Before encoding: Training accuracy was 95%, validation accuracy was 70%, showing overfitting.

After encoding: Training accuracy dropped to 87.5%, validation accuracy improved to 87.5%, indicating better generalization.

Properly encoding categorical variables helps the model understand the data better, reducing overfitting and improving validation accuracy.

Bonus Experiment

Try using label encoding instead of one-hot encoding and compare the validation accuracy.

💡 Hint

Use sklearn's LabelEncoder on each categorical column separately and observe if the model performance changes.

Practice

1. Why do we need to convert categorical variables into numbers before using them in machine learning models?

easy

A. Because most machine learning algorithms only understand numbers, not words.

B. Because numbers take less memory than words.

C. Because categorical variables are always continuous data.

D. Because converting to numbers makes the data look nicer.

Solution

1. Step 1: Understand machine learning input requirements

   Most machine learning models work with numerical data and cannot process text or categories directly.

2. Step 2: Reason why conversion is necessary

   Converting categories to numbers allows models to perform calculations and learn patterns from the data.

3. Final Answer:

   Because most machine learning algorithms only understand numbers, not words. -> Option A

4. Quick Check:

   Machine learning needs numbers = D [OK]

Hint: Remember: ML models need numbers, not words [OK]

Common Mistakes:

• Thinking categorical variables are continuous data
• Believing conversion is just for memory saving
• Assuming conversion is for aesthetics

2. Which of the following is the correct way to apply one-hot encoding using pandas on a column named color?

easy

A. pd.encode_onehot(df.color)

B. df['color'].label_encode()

C. df.one_hot_encode('color')

D. pd.get_dummies(df['color'])

Solution

1. Step 1: Recall pandas one-hot encoding method

   Pandas provides get_dummies() to convert categorical columns into one-hot encoded columns.

2. Step 2: Match correct syntax

   The correct syntax is pd.get_dummies(df['color']). Other options are not valid pandas methods.

3. Final Answer:

   pd.get_dummies(df['color']) -> Option D

4. Quick Check:

   Use pd.get_dummies for one-hot encoding [OK]

Hint: Use pd.get_dummies for one-hot encoding in pandas [OK]

Common Mistakes:

• Using non-existent pandas methods
• Confusing label encoding with one-hot encoding
• Calling methods on dataframe incorrectly

3. Given this code snippet:

from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
df['size_encoded'] = le.fit_transform(df['size'])
print(df['size_encoded'].tolist())

If df['size'] contains ['small', 'medium', 'large', 'medium'], what will be the printed output?

medium

A. [0, 1, 2, 1]

B. [2, 1, 0, 1]

C. [1, 2, 0, 2]

D. [0, 2, 1, 2]

Solution

1. Step 1: Understand LabelEncoder mapping

   LabelEncoder assigns integers in alphabetical order: 'large'->0, 'medium'->1, 'small'->2.

2. Step 2: Map each value in the list

   Input list: ['small', 'medium', 'large', 'medium'] maps to [2, 1, 0, 1].

3. Final Answer:

   [2, 1, 0, 1] -> Option B

4. Quick Check:

   Alphabetical order encoding = A [OK]

Hint: LabelEncoder assigns numbers alphabetically [OK]

Common Mistakes:

• Assuming encoding follows input order
• Mixing up label encoding with one-hot encoding
• Forgetting zero-based indexing

4. You tried to apply one-hot encoding on a categorical column but got an error: ValueError: Input contains NaN, infinity or a value too large for dtype('float64'). What is the most likely cause and fix?

medium

A. The column has missing values; fill or drop them before encoding.

B. The column is numeric; convert it to string first.

C. The one-hot encoder only works on integers; convert categories to integers first.

D. The dataset is too large; reduce its size.

Solution

1. Step 1: Analyze the error message

   The error mentions NaN (missing) or infinite values, which cause encoding to fail.

2. Step 2: Identify fix for missing values

   Fill missing values with a placeholder or drop rows before encoding to avoid errors.

3. Final Answer:

   The column has missing values; fill or drop them before encoding. -> Option A

4. Quick Check:

   Handle missing data before encoding [OK]

Hint: Check for missing values before encoding [OK]

Common Mistakes:

• Ignoring missing values in categorical columns
• Trying to encode numeric columns as categories
• Assuming dataset size causes this error

5. You have a dataset with a categorical feature education_level having values: ['high school', 'bachelor', 'master', 'phd']. You want to use this feature in a regression model where order matters. Which encoding method should you choose and why?

hard

A. One-hot encoding, because it creates separate columns for each category.

B. Random encoding, to avoid bias in the model.

C. Label encoding, because it preserves the order of categories as numbers.

D. No encoding needed, use categories as strings directly.

Solution

1. Step 1: Understand the nature of the feature

   Education levels have a natural order: high school < bachelor < master < phd.

2. Step 2: Choose encoding that preserves order

   Label encoding assigns numbers reflecting order, suitable for regression models that consider magnitude.

3. Final Answer:

   Label encoding, because it preserves the order of categories as numbers. -> Option C

4. Quick Check:

   Ordered categories use label encoding [OK]

Hint: Use label encoding for ordered categories [OK]

Common Mistakes:

• Using one-hot encoding for ordered categories
• Ignoring order importance in encoding
• Trying to use strings directly in models