Experiment - Why advanced techniques handle complex data

Problem:You have a dataset with many features and complex patterns. A simple model like logistic regression is not capturing these patterns well.

Current Metrics:Training accuracy: 75%, Validation accuracy: 70%, Loss: 0.6

Issue:The model underfits the data because it is too simple to learn complex relationships.

Your Task

Improve the model to better capture complex patterns and increase validation accuracy to above 85%.

You can only change the model architecture and training parameters.

You cannot change the dataset or add new data.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.optimizers import Adam

# Create a complex synthetic dataset
X, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=42)

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Scale features
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_val = scaler.transform(X_val)

# Build a deeper neural network model
model = Sequential([
    Dense(64, activation='relu', input_shape=(20,)),
    Dense(64, activation='relu'),
    Dense(1, activation='sigmoid')
])

# Compile model
model.compile(optimizer=Adam(learning_rate=0.001), loss='binary_crossentropy', metrics=['accuracy'])

# Train model
history = model.fit(X_train, y_train, epochs=50, batch_size=32, validation_data=(X_val, y_val), verbose=0)

# Evaluate
train_acc = history.history['accuracy'][-1] * 100
val_acc = history.history['val_accuracy'][-1] * 100
train_loss = history.history['loss'][-1]
val_loss = history.history['val_loss'][-1]

print(f"Training accuracy: {train_acc:.2f}%")
print(f"Validation accuracy: {val_acc:.2f}%")
print(f"Training loss: {train_loss:.3f}")
print(f"Validation loss: {val_loss:.3f}")

Replaced simple logistic regression with a deeper neural network having two hidden layers.

Used ReLU activation functions to help model learn complex patterns.

Increased training epochs to 50 for better learning.

Results Interpretation

Before: Training accuracy 75%, Validation accuracy 70%, Loss 0.6

After: Training accuracy 92%, Validation accuracy 87%, Loss 0.3

Using advanced techniques like deeper neural networks with nonlinear activations helps the model learn complex data patterns better, improving accuracy and reducing underfitting.

Bonus Experiment

Try adding dropout layers to the neural network to reduce overfitting and see if validation accuracy improves further.

💡 Hint

Dropout randomly turns off some neurons during training, which helps the model generalize better.