Bird
Raised Fist0

Feature Scaling - Normalization vs Standardization, When Each Works

THEORYbeginnerInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Feature Scaling - Normalization vs Standardization, When Each Works
Choosing the right feature scaling method for your data
theorybeginnerFreq: highData & Features
Prerequisites:Feature Engineering
💡
Why It Exists

Feature scaling was invented to address the problem of features having different units or scales, which can lead to biased model training. For example, in a dataset with height in centimeters and weight in kilograms, the model might give undue importance to weight due to its larger numerical values.

📖
What It Is

Feature scaling is a technique to standardize the range of independent variables or features of data. Imagine trying to compare the speed of a car in miles per hour with the speed of a snail in inches per minute; scaling puts them on a comparable scale.

∑
Math Intuition
Normalization rescales data to fit within a specific range, making it easier to compare different features. Standardization transforms data to have a mean of zero and a standard deviation of one, which helps in algorithms that assume normally distributed data.
⚙️
Core Mechanism

Normalization scales features to a range of [0, 1], while standardization scales them to have a mean of 0 and a standard deviation of 1.

📋
Step by Step
  1. Normalization: Subtract the minimum value of the feature and divide by the range (max-min).
  2. Standardization: Subtract the mean of the feature and divide by the standard deviation.
  3. Apply the scaling transformation to each feature independently.
  4. Ensure the same transformation is applied to both training and test data.
  5. Verify the scaled features have the desired properties (range or distribution).
🎯
Intuition
Think of normalization as squeezing all values into a small box (0 to 1), while standardization centers the data around zero and spreads it out evenly.
🐍
Python Example
Python
import numpy as np

data = np.array([[1, 2], [2, 3], [3, 4]])
# Normalization
normalized_data = (data - data.min(axis=0)) / (data.max(axis=0) - data.min(axis=0))
# Standardization
standardized_data = (data - data.mean(axis=0)) / data.std(axis=0)
print(normalized_data)  # output: [[0. 0.]
 [0.5 0.5]
 [1. 1.]]
print(standardized_data)  # output: [[-1.22474487 -1.22474487]
 [ 0.  0.]
 [ 1.22474487  1.22474487]]
✅
When It Works
  • Normalization is ideal when you know the data is bounded and you want to maintain the original distribution shape.
  • Standardization is preferred when the data follows a Gaussian distribution or when using algorithms that assume normally distributed data, like PCA.
  • Use normalization when features have different units and you want to bring them to a common scale.
  • Standardization is useful when the model is sensitive to the mean and variance of the data, such as in SVMs.
⚠️
When It Breaks
  • Normalization fails when outliers are present, as they can skew the min-max range significantly.
  • Standardization can be ineffective if the data is not approximately normally distributed, leading to misleading feature scaling.
  • Normalization is not suitable for algorithms that assume zero-centered data, like PCA.
  • Both methods can break if applied inconsistently between training and test datasets, leading to data leakage.
⇄
vs Robust Scaling
Use This When
Use normalization or standardization when the data is free of significant outliers and you want to maintain the original distribution or assume normality.
Use Alternative When
Use robust scaling when the data contains outliers, as it uses the median and interquartile range, making it less sensitive to extreme values.
🎯
Interview Angles
  • Interviewer asks: 'How would you preprocess features for a KNN model?' → testing understanding of distance-based algorithms and the need for feature scaling.
  • Interviewer asks: 'Why might normalization be inappropriate for data with outliers?' → assessing knowledge of how outliers affect scaling methods.
  • Interviewer asks: 'When would you choose standardization over normalization?' → evaluating understanding of data distribution assumptions and algorithm requirements.
  • Interviewer asks: 'How do you ensure consistent scaling between training and test data?' → checking awareness of data leakage and preprocessing consistency.
⚠️
Common Mistakes
Believing normalization and standardization are interchangeable, leading to incorrect assumptions about data distribution.
Assuming standardization is always better for all models, which can lead to poor performance in models sensitive to feature range.
Forgetting to apply the same scaling transformation to both training and test datasets, causing data leakage and inconsistent model performance.
Thinking normalization handles outliers well, which can result in skewed feature scaling and biased model training.
📌
Summary
  • Normalization scales features to a [0, 1] range, while standardization centers them around zero with unit variance.
  • Choose normalization for bounded data and standardization for normally distributed data or algorithms sensitive to mean and variance.
  • Be cautious of outliers and ensure consistent scaling across training and test datasets to avoid data leakage.

Practice

(1/5)
1. Given the following Python code snippet, what will be the output if the model's performance has decayed due to data drift?
import numpy as np
from sklearn.metrics import accuracy_score

y_true = np.array([1, 0, 1, 1, 0])
y_pred = np.array([0, 0, 1, 0, 0])

accuracy = accuracy_score(y_true, y_pred)
print(f'Accuracy: {accuracy:.2f}')
easy
A. Accuracy: 0.40
B. Accuracy: 0.80
C. Accuracy: 0.60
D. Accuracy: 0.20

Solution

  1. Step 1: Calculate accuracy

    Compare y_true and y_pred: only 2 out of 5 predictions are correct.

  2. Step 2: Compute accuracy

    Accuracy = 2/5 = 0.40.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Count correct predictions, divide by total [OK]
Hint: Count correct predictions, divide by total [OK]
Common Mistakes:
2. In the context of data collection for machine learning, which scenario best describes an offline data collection approach?
easy
A. Recording user feedback immediately after a transaction.
B. Collecting user interaction data in real-time as they browse a website.
C. Using sensors to monitor environmental conditions continuously.
D. Gathering historical sales data from a database for analysis.

Solution

  1. Step 1: Identify offline data collection characteristics

    Offline data collection involves gathering data that has already been generated and stored, not in real-time.

  2. Step 2: Eliminate real-time options

    Options A, C, and D involve real-time or near-real-time data collection, which is characteristic of online data collection.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Offline = historical, stored data. [OK]
Hint: Offline = historical, stored data. [OK]
Common Mistakes:
3. You conducted an A/B test using the interleaving method, but the results are inconsistent with previous tests. What is the most likely cause?
medium
A. Data leakage between test groups
B. Inadequate sample size
C. Temporal trends affecting results
D. Incorrect randomization of test subjects

Solution

  1. Step 1: Identify potential issues

    Interleaving requires proper randomization to ensure fair comparison.

  2. Step 2: Eliminate wrong options

    Data leakage and temporal trends are less likely in interleaving. Inadequate sample size affects power, not consistency.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Randomization is key in interleaving. [OK]
Hint: Randomization is key in interleaving. [OK]
Common Mistakes:
4. A model performs well on the validation set but poorly on the test set. What is the most likely cause?
medium
A. The test set is not representative of the validation set.
B. The validation set is too small.
C. The model is underfitting the data.
D. The model has too many parameters.

Solution

  1. Step 1: Identify the discrepancy

    Good validation performance but poor test performance suggests a mismatch between sets.

  2. Step 2: Eliminate less likely causes

    The model is underfitting the data. is incorrect as underfitting would affect both sets. The validation set is too small. is less likely unless the validation set is extremely small. The model has too many parameters. would likely affect both sets similarly.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Mismatch between validation and test sets. [OK]
Hint: Mismatch between validation and test sets. [OK]
Common Mistakes:
5. You need to train a model with a learning rate that decreases over time. Which approach would best suit this requirement?
hard
A. Use a constant learning rate with Adam
B. Switch to RMSProp with a fixed decay rate
C. Use AdaGrad, as it inherently decreases learning rate
D. Implement a learning rate scheduler with SGD

Solution

  1. Step 1: Identify the requirement.

    The requirement is for a learning rate that decreases over time.

  2. Step 2: Match the requirement to the solution.

    A learning rate scheduler with SGD allows for controlled decrease over time.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Scheduler with SGD for decreasing lr. [check]
Hint: Scheduler with SGD for decreasing lr. [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard