Why classification predicts categories in ML Python - Experiment to Prove It

Experiment - Why classification predicts categories

Problem:We want to teach a computer to recognize if an email is spam or not spam. The current model guesses correctly on training emails 95% of the time but only 70% on new emails it hasn't seen before.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%

Issue:The model is overfitting. It learns the training emails too well but does not generalize to new emails.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85%, while keeping training accuracy below 92%.

You can only change the model architecture and training settings.

Do not add more training data.

Keep the problem as a classification task predicting spam or not spam.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping
from sklearn.model_selection import train_test_split

# Simulated data: 1000 samples, 20 features
np.random.seed(42)
X = np.random.rand(1000, 20)
y = (np.sum(X, axis=1) > 10).astype(int)  # 0 = not spam, 1 = spam

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Build model with dropout to reduce overfitting
model = Sequential([
    Dense(32, activation='relu', input_shape=(20,)),
    Dropout(0.5),
    Dense(16, activation='relu'),
    Dropout(0.3),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Early stopping to prevent overfitting
early_stop = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

history = model.fit(X_train, y_train, epochs=50, batch_size=32, validation_data=(X_val, y_val), callbacks=[early_stop], verbose=0)

# Evaluate final model
train_loss, train_acc = model.evaluate(X_train, y_train, verbose=0)
val_loss, val_acc = model.evaluate(X_val, y_val, verbose=0)

print(f'Training accuracy: {train_acc*100:.2f}%')
print(f'Validation accuracy: {val_acc*100:.2f}%')

Added Dropout layers after Dense layers to reduce overfitting by randomly turning off neurons during training.

Implemented EarlyStopping callback to stop training when validation loss stops improving.

Reduced the number of neurons in Dense layers to lower model complexity.

Results Interpretation

Before: Training accuracy was 95%, validation accuracy was 70%. The model memorized training data but failed on new data.

After: Training accuracy dropped to 90%, validation accuracy improved to 86%. The model generalized better to new emails.

Classification models predict categories by learning patterns in data. Overfitting means the model learns too many details from training data and fails on new data. Techniques like dropout and early stopping help the model focus on important patterns, improving its ability to predict categories correctly on unseen data.

Bonus Experiment

Try using a simpler model like logistic regression and compare its validation accuracy to the neural network.

💡 Hint

Logistic regression is a basic classification model that often generalizes well on small datasets without overfitting.

Practice

1. Why does a classification model predict categories instead of numbers?

easy

A. Because it sorts data into distinct groups based on labels

B. Because it calculates the average value of data points

C. Because it predicts continuous values like temperature

D. Because it generates random outputs for data

Solution

1. Step 1: Understand classification purpose

   Classification models are designed to assign inputs to specific categories or groups.

2. Step 2: Differentiate from regression

   Regression models predict continuous numbers, while classification predicts labels or categories.

3. Final Answer:

   Because it sorts data into distinct groups based on labels -> Option A

4. Quick Check:

   Classification = category prediction [OK]

Hint: Classification = sorting into groups, not numbers [OK]

Common Mistakes:

• Confusing classification with regression
• Thinking classification predicts exact numbers
• Assuming classification outputs random values

2. Which of the following is the correct way to define a classification model in Python using scikit-learn?

easy

A. model = KNeighborsClassifier()

B. model = LinearRegression()

C. model = RandomForestRegressor()

D. model = PCA()

Solution

1. Step 1: Identify classification model

   KNeighborsClassifier is a classification model used to predict categories.

2. Step 2: Exclude non-classification models

   LinearRegression and RandomForestRegressor are for regression; PCA is for dimensionality reduction.

3. Final Answer:

   model = KNeighborsClassifier() -> Option A

4. Quick Check:

   KNeighborsClassifier is classification model [OK]

Hint: Look for 'Classifier' in model name for classification [OK]

Common Mistakes:

• Choosing regression models for classification tasks
• Confusing PCA with classification
• Using models without 'Classifier' suffix

3. Given this code snippet, what will be the predicted category for input X_new = [[5.1, 3.5]]?

from sklearn.neighbors import KNeighborsClassifier
X_train = [[5.0, 3.6], [6.7, 3.1], [5.9, 3.0]]
y_train = ['A', 'B', 'B']
model = KNeighborsClassifier(n_neighbors=1)
model.fit(X_train, y_train)
prediction = model.predict(X_new)
print(prediction)

medium

A. ['C']

B. ['B']

C. ['A']

D. Error: model not trained

Solution

1. Step 1: Understand nearest neighbor logic

   With n_neighbors=1, the model predicts the label of the closest training point to X_new.

2. Step 2: Find closest training point

   Distances: to [5.0,3.6] is small; label is 'A'. Others are farther with label 'B'.

3. Final Answer:

   ['A'] -> Option C

4. Quick Check:

   Nearest neighbor label = 'A' [OK]

Hint: Nearest neighbor label is prediction with k=1 [OK]

Common Mistakes:

• Ignoring nearest neighbor distance
• Assuming majority vote with k=1
• Confusing labels or input shapes

4. Identify the error in this classification code and fix it:

from sklearn.tree import DecisionTreeClassifier
X = [[1, 2], [3, 4]]
y = [0, 1]
model = DecisionTreeClassifier()
model.fit(X, y)
pred = model.predict([1, 2])
print(pred)

medium

A. No error, code runs fine

B. Change predict input to [[1, 2]]

C. Use DecisionTreeRegressor instead

D. Change y to [[0], [1]]

Solution

1. Step 1: Check input shape for predict

   Model expects 2D array for predict; [1, 2] is 1D, causing error.

2. Step 2: Fix input shape

   Wrap input in another list: [[1, 2]] to make it 2D.

3. Final Answer:

   Change predict input to [[1, 2]] -> Option B

4. Quick Check:

   Predict input must be 2D array [OK]

Hint: Predict input must be 2D array, even for one sample [OK]

Common Mistakes:

• Passing 1D array to predict
• Changing label shape unnecessarily
• Switching to regression model incorrectly

5. You have a dataset with three categories: 'cat', 'dog', and 'rabbit'. You want to build a classification model that predicts these categories from features. Which approach best explains why classification predicts categories here?

hard

A. The model calculates the average feature values and outputs a number

B. The model predicts continuous values that represent category probabilities

C. The model randomly assigns categories without learning

D. The model learns patterns from labeled examples to assign new inputs to one of the three groups

Solution

1. Step 1: Understand classification learning

   The model uses labeled data to learn how features relate to categories.

2. Step 2: Explain prediction process

   It predicts the category by matching new input features to learned patterns for each group.

3. Final Answer:

   The model learns patterns from labeled examples to assign new inputs to one of the three groups -> Option D

4. Quick Check:

   Classification = learning patterns to predict categories [OK]

Hint: Classification learns from labels to predict groups [OK]

Common Mistakes:

• Thinking classification outputs numbers directly
• Assuming random assignment without training
• Confusing classification with regression probabilities