Bird
Raised Fist0

Handling Missing Data - Imputation Strategies and When to Drop

THEORYintermediateInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Handling Missing Data - Imputation Strategies and When to Drop
Master the art of dealing with missing data in ML.
theoryintermediateFreq: highData & Features
Prerequisites:Feature Engineering
💡
Why It Exists

Missing data can lead to biased models and incorrect conclusions. For example, if a dataset has missing values in critical features, a model might learn patterns that don't generalize well to real-world data. Handling missing data ensures that models are trained on complete and representative datasets.

📖
What It Is

Handling missing data involves strategies to either fill in missing values or remove incomplete records. Think of it like filling in the blanks in a crossword puzzle: you can either guess the missing words based on context or skip the puzzle if too many blanks make it unsolvable.

∑
Math Intuition
Imputation strategies like mean or median imputation assume that the missing data is missing at random and that the mean or median of the observed data is a good estimate for the missing values. This approach can maintain the overall distribution of the data but may underestimate variability.
⚙️
Core Mechanism

The key idea is to either impute missing values using statistical methods or drop rows/columns with missing data.

📋
Step by Step
  1. Identify missing data in the dataset.
  2. Decide whether to impute or drop based on the extent and pattern of missingness.
  3. If imputing, choose a strategy: mean, median, mode, or model-based imputation.
  4. Apply the chosen strategy to fill in missing values.
  5. If dropping, remove rows or columns with missing data.
🎯
Intuition
Imagine a book with missing pages. You can either try to infer the missing content from the context of the story (imputation) or skip reading the book if too many pages are missing (dropping).
🐍
Python Example
Python
import numpy as np
from sklearn.impute import SimpleImputer

# Sample data with missing values
X = np.array([[1, 2, np.nan], [4, np.nan, 6], [7, 8, 9]])

# Impute missing values with the mean of each column
imputer = SimpleImputer(strategy='mean')
X_imputed = imputer.fit_transform(X)
print(X_imputed)  # output: [[1. 2. 7.5], [4. 5. 6.], [7. 8. 9.]]
✅
When It Works
  • When missing data is less than 5% of the dataset and is missing at random.
  • When the missingness pattern is not related to the target variable.
  • When the dataset is large enough that dropping rows with missing data won't significantly reduce its size.
  • When domain knowledge suggests a reliable imputation strategy.
⚠️
When It Breaks
  • When missing data is not random and is related to the target variable, leading to biased models. For example, if missing values in a medical dataset are more common in severe cases, imputing them might lead to underestimating the severity.
  • When more than 30% of the data is missing, making imputation unreliable. For instance, in a survey dataset where a large portion of responses are missing, any imputation might not reflect the true distribution.
  • When the dataset is small, and dropping rows with missing data leads to a significant loss of information. For example, in a small clinical trial dataset, dropping rows could lead to losing critical patient data.
  • When the imputation strategy introduces artificial patterns not present in the original data, such as using mean imputation in a dataset with a skewed distribution, which can distort the data's natural variability.
⇄
vs Dropping missing data
Use This When
Use imputation when the missing data is minimal and random, and you want to preserve the dataset size.
Use Alternative When
Use dropping when the missing data is extensive or when imputation would introduce bias.
🎯
Interview Angles
  • Interviewer asks: 'How would you handle missing data in a dataset with 10% missing values?' → testing your understanding of when to impute versus drop.
  • Interviewer presents a dataset with missing values and asks you to preprocess it → assessing your practical skills in data cleaning.
  • Interviewer asks about the impact of missing data on model performance → evaluating your awareness of bias and variance introduced by missing data.
  • Interviewer gives a scenario where imputation leads to overfitting → checking if you can identify and mitigate overfitting risks.
⚠️
Common Mistakes
Believing that mean imputation is always safe, which can lead to underestimated variance and biased models.
Assuming missing data is always random, which can result in incorrect imputation and biased predictions.
Dropping rows with missing data without considering the impact on dataset size and representativeness.
Using complex imputation methods without validating their assumptions, leading to overfitting.
📌
Summary
  • Handling missing data is crucial to prevent biased models and ensure generalization.
  • Choose imputation when missing data is minimal and random; drop when it's extensive or biased.
  • Understand the assumptions and limitations of each strategy to avoid introducing bias.

Practice

(1/5)
1. Given the following Python code snippet, what will be the output if the dataset contains data leakage?
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = LogisticRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(accuracy_score(y_test, y_pred))
easy
A. The accuracy will be lower than expected.
B. The code will throw an error due to data leakage.
C. The accuracy will be exactly as expected.
D. The accuracy will be higher than expected.

Solution

  1. Step 1: Understand the impact of data leakage

    Data leakage often results in overly optimistic performance metrics.

  2. Step 2: Analyze the code behavior

    The code will execute without errors, but if leakage exists, the model will perform better than it should.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Leakage inflates accuracy [OK]
Hint: Leakage inflates accuracy [OK]
Common Mistakes:
2. Given the following Python code using BPE tokenization, what is the output?
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.pre_tokenizers import Whitespace

# Initialize a BPE tokenizer
bpe_tokenizer = Tokenizer(BPE())
bpe_tokenizer.pre_tokenizer = Whitespace()

# Train the tokenizer
bpe_tokenizer.train(["hello world", "hello there"], vocab_size=10)

# Encode a sentence
output = bpe_tokenizer.encode("hello world")
print(output.tokens)
easy
A. ['h', 'e', 'l', 'l', 'o', 'world']
B. ['he', 'llo', 'world']
C. ['hello', 'wo', 'rld']
D. ['hello', 'world']

Solution

  1. Step 1: Understand BPE tokenization with whitespace pre-tokenization.

    BPE with whitespace pre-tokenization will initially split on spaces.

  2. Step 2: Analyze the output of the encode method.

    The sentence 'hello world' is tokenized as ['hello', 'world'] due to whitespace splitting.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    BPE with whitespace splits on spaces first. [OK]
Hint: BPE with whitespace splits on spaces first. [OK]
Common Mistakes:
3. A model trained on standardized data is underperforming. What could be a potential issue?
medium
A. The model is sensitive to the mean of the features.
B. The standard deviation of features was not considered.
C. The data was standardized but not normalized, causing issues.
D. The data was not normalized, affecting model performance.

Solution

  1. Step 1: Analyze the impact of standardization on model performance.

    Standardization centers data around zero, which can affect models sensitive to feature means.

  2. Step 2: Identify the potential issue.

    Some models may perform poorly if they rely on the original mean of the data.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Model sensitive to mean? Standardization affects it. [OK]
Hint: Model sensitive to mean? Standardization affects it. [OK]
Common Mistakes:
4. When evaluating a recommendation system, why might you choose NDCG over MAP?
medium
A. NDCG is easier to compute than MAP.
B. NDCG is more interpretable than MAP.
C. NDCG accounts for the position of relevant items, while MAP does not.
D. NDCG is less sensitive to the number of relevant items than MAP.

Solution

  1. Step 1: Understand metric differences

    NDCG considers the position of relevant items, giving higher weight to top-ranked items.

  2. Step 2: Eliminate wrong options

    MAP considers all relevant items equally, NDCG is not necessarily easier or more interpretable, and both metrics are sensitive to the number of relevant items.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    NDCG weights top positions more. [OK]
Hint: NDCG weights top positions more. [OK]
Common Mistakes:
5. A recurrent neural network is not learning effectively. The loss plateaus early during training. What is the most likely cause?
medium
A. The learning rate is too high.
B. The model is overfitting.
C. The vanishing gradient problem.
D. The dataset is too small.

Solution

  1. Step 1: Identify symptoms of vanishing gradient.

    Early plateauing of loss often indicates vanishing gradients.

  2. Step 2: Eliminate other causes.

    High learning rate (A) would cause erratic loss, overfitting (B) would show low training loss, and small dataset (D) would not cause early plateau.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Early plateau = vanishing gradient. [check]
Hint: Early plateau = vanishing gradient. [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard