Bird
Raised Fist0

Feature Scaling - Normalization vs Standardization, When Each Works

THEORYbeginnerInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Normalization vs Standardization

When preparing features for a machine learning model, would you choose normalization or standardization? Justify your choice based on the data distribution and model type.

📊
Comparison
DimensionNormalizationStandardization
Handling OutliersPoorGood
Data Distribution AssumptionFlexibleNormal
Algorithm CompatibilityHigh for k-NN, NNHigh for SVM, LR
InterpretabilityIntuitiveLess Intuitive
Range ConsistencyFixedVariable
✅
Verdict
Default to normalization for algorithms sensitive to feature magnitude like neural networks, especially when data is not normally distributed. Switch to standardization when using linear models or when data is approximately normal.
💡
Key Insight
Choose normalization for scale-sensitive algorithms and standardization for normally distributed data.
⚠️
Interview Trap
❌ Wrong: Always use normalization because it scales data to [0, 1].
This ignores the data distribution and the specific algorithm requirements.
✅ Consider the data distribution and the algorithm's sensitivity to feature scale before choosing between normalization and standardization.
Normalization
"Scales features to a fixed range, typically [0, 1], which is useful for algorithms sensitive to feature magnitude."
Weaknesses
  • Sensitive to outliers, which can skew the scaling.
  • Assumes a bounded range, which may not be suitable for all data types.
  • Less effective if the data is normally distributed.
Ideal When
  • When using algorithms like k-NN or neural networks that are sensitive to feature magnitude.
  • When features have different units or scales.
  • When the data is not normally distributed.
Spotify uses normalization for audio features to ensure consistent input ranges for their recommendation algorithms.
Standardization
"Centers data around zero with a standard deviation of one, which is ideal for normally distributed data."
Weaknesses
  • Assumes data is normally distributed, which may not always be the case.
  • Does not bound the data to a specific range, which can be problematic for some algorithms.
  • Can be less intuitive to interpret compared to normalization.
Ideal When
  • When using algorithms like SVM or logistic regression that assume normally distributed data.
  • When the data is approximately normally distributed.
  • When dealing with features that have outliers.
Google uses standardization in their search algorithms to handle features with different distributions effectively.
🐍
Normalization - Example
▼ Show
Python
import numpy as np

def normalize(data):
    min_val = np.min(data, axis=0)
    max_val = np.max(data, axis=0)
    return (data - min_val) / (max_val - min_val)

# Example usage
features = np.array([[1, 2], [2, 3], [3, 4]])
normalized_features = normalize(features)
🐍
Standardization - Example
▼ Show
Python
import numpy as np

def standardize(data):
    mean = np.mean(data, axis=0)
    std_dev = np.std(data, axis=0)
    return (data - mean) / std_dev

# Example usage
features = np.array([[1, 2], [2, 3], [3, 4]])
standardized_features = standardize(features)
✅
Use Normalization When
  • •When using k-NN or neural networks.
  • •When feature scales vary significantly.
  • •When data is not normally distributed.
✅
Use Standardization When
  • •When using SVM or logistic regression.
  • •When data is approximately normally distributed.
  • •When dealing with outliers.
↳
Nuances
  • Normalization can fail if outliers are present, skewing the scale.
  • Standardization assumes normal distribution, which can mislead if the data is skewed.
  • Normalization is preferred for neural networks due to their sensitivity to input scale.
  • Standardization is often better for linear models that assume normally distributed inputs.

Practice

(1/5)
1. In the context of data labeling strategies, which method is best suited for generating large amounts of labeled data quickly but with potentially lower accuracy?
easy
A. Weak Supervision
B. Programmatic Labeling
C. Human Annotation
D. Active Learning

Solution

  1. Step 1: Understand the methods.

    Human annotation is accurate but slow. Weak supervision uses heuristics or external data sources to generate labels quickly but may be less accurate.

  2. Step 2: Eliminate wrong options.

    Programmatic labeling is similar to weak supervision but often requires more setup. Active learning involves human input and is not as fast.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Weak supervision = fast, less accurate [OK]
Hint: Weak supervision = fast, less accurate [OK]
Common Mistakes:
2. In a dataset with highly imbalanced classes, which sampling strategy is most appropriate to ensure each class is represented proportionally in the sample?
easy
A. Stratified Sampling
B. Weighted Sampling
C. Reservoir Sampling
D. Random Sampling

Solution

  1. Step 1: Identify the need for proportional representation.

    In imbalanced datasets, it's crucial to maintain class proportions to avoid bias.

  2. Step 2: Eliminate inappropriate strategies.

    Random Sampling (A) doesn't guarantee class proportions. Reservoir Sampling (C) is for streaming data, not class balance. Weighted Sampling (D) adjusts for importance, not proportions.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Stratified = Proportional representation [OK]
Hint: Stratified = Proportional representation [OK]
Common Mistakes:
3. A KNN model is performing poorly on a high-dimensional dataset. What is the most likely cause?
medium
A. The model has too few neighbors.
B. The distance metric is ineffective in high dimensions.
C. The dataset is too small.
D. The model is overfitting due to too many neighbors.

Solution

  1. Step 1: Consider high-dimensional effects

    In high dimensions, distance metrics often become less effective.

  2. Step 2: Eliminate wrong options

    The model has too few neighbors. is incorrect as increasing neighbors may not help. The dataset is too small. is wrong because size isn't the main issue. The model is overfitting due to too many neighbors. is incorrect as overfitting is less likely with many neighbors.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    High dimensions dilute distance metrics. [OK]
Hint: High dimensions dilute distance metrics. [OK]
Common Mistakes:
4. A model shows a high ROC-AUC but performs poorly on the minority class. What is the most likely cause?
medium
A. The ROC-AUC is misleading due to class imbalance.
B. The model is overfitting the training data.
C. The model has a high bias.
D. The model is underfitting the data.

Solution

  1. Step 1: Analyze the metrics

    ROC-AUC can be misleading in imbalanced datasets as it doesn't focus on the minority class.

  2. Step 2: Eliminate wrong options

    Overfitting (A) and underfitting (D) are not directly related to the discrepancy between ROC-AUC and minority class performance. High bias (C) would affect overall performance, not just minority class.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    ROC-AUC can mislead in imbalanced datasets. [OK]
Hint: ROC-AUC can mislead in imbalanced datasets. [OK]
Common Mistakes:
5. You need to track the lineage of a model from data ingestion to deployment using MLflow. Which approach best achieves this?
hard
A. Use MLflow Projects to encapsulate each stage of the pipeline.
B. Integrate MLflow with a data lineage tool to track transformations.
C. Utilize MLflow Registry to version control each data transformation.
D. Log all data transformations as metrics in MLflow Tracking.

Solution

  1. Step 1: Understand MLflow's capabilities

    MLflow alone does not track data lineage; integration is needed.

  2. Step 2: Evaluate options for lineage tracking

    Integrating with a dedicated lineage tool provides comprehensive tracking.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Lineage needs integration [OK]
Hint: Lineage needs integration [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard