Bird
Raised Fist0

Sampling Strategies - Random, Stratified, Reservoir, Weighted

THEORYintermediateInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Stratified Sampling Surprise
Started after the last data pipeline update two weeks ago.

A customer churn prediction model for a telecom company, processing 500k customer records monthly.

Symptom
Model accuracy is stable at 0.85, but recall has dropped to 0.0 for the minority class.
📡
Signals
metric
Recall for the minority class in the test set
Dropped from 0.65 to 0.0 after the last pipeline update
distribution
Class distribution in the training set
Minority class reduced from 20% to 5% after the update
feature_importance
Feature importance for 'customer_tenure'
Dropped from 0.12 to 0.01 post-update
log
Data pipeline logs showing sampling strategy
Switched from stratified to random sampling
metric
Overall model accuracy
Remained stable at 0.85
🔬
Hypotheses
The sampling strategy change caused class imbalance.
Supporting Signals
Class distribution in the training setData pipeline logs showing sampling strategy
Verify: Check the sampling strategy configuration in the data pipeline.
print(data_pipeline_config['sampling_strategy'])
Feature 'customer_tenure' is no longer predictive.
Supporting Signals
Feature importance for 'customer_tenure'
Verify: Analyze feature correlation with the target variable.
df[['customer_tenure', 'churn']].corr()
🎯
Root Cause
The data pipeline was incorrectly configured to use random sampling instead of stratified sampling.
Why It Happened

A recent update to the data pipeline inadvertently changed the sampling strategy configuration.

🔧
Fix Strategy
ImmediateRollback to the previous data pipeline configuration.
Short-termReconfigure the data pipeline to use stratified sampling.
Long-termImplement automated checks to ensure sampling strategy consistency.
🐍
Python Fix
Python
data_pipeline_config['sampling_strategy'] = 'stratified'
# Re-run the data pipeline
run_data_pipeline(data_pipeline_config)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting to the majority class.
Why candidates think this: The recall drop suggests the model isn't generalizing well.
Why wrong: The class distribution change indicates a data imbalance issue, not overfitting.
Feature 'customer_tenure' is no longer relevant.
Why candidates think this: The drop in feature importance suggests it's not predictive.
Why wrong: The feature importance drop is a consequence of the class imbalance, not the cause.
📊
Monitoring to Add
  • Monitor class distribution in training data after each pipeline run.
  • Set up alerts for drastic changes in feature importance.
  • Track sampling strategy changes in pipeline logs.
📋
Process Change

Introduce a review process for all data pipeline configuration changes.

🏗️
Architectural Change

Design the data pipeline to enforce stratified sampling by default for imbalanced datasets.

💡
Key Insight
Sampling strategy changes can silently introduce class imbalance, drastically affecting minority class performance.

Practice

(1/5)
1. In the context of image segmentation, which type of segmentation assigns a unique label to each pixel of an image based on the object class it belongs to, without distinguishing between different instances of the same class?
easy
A. Semantic segmentation
B. Instance segmentation
C. Panoptic segmentation
D. Object detection

Solution

  1. Step 1: Understand the segmentation types

    Semantic segmentation labels each pixel according to the object class, without distinguishing instances.

  2. Step 2: Eliminate wrong options

    Instance segmentation distinguishes between instances, panoptic combines both semantic and instance, and object detection is not segmentation.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Semantic = Class labels only [OK]
Hint: Semantic = Class labels only [OK]
Common Mistakes:
2. Given the following Python code snippet for a segmentation task, what will be the output shape if the input image shape is (256, 256, 3) and the model outputs a segmentation map with 10 classes?
import numpy as np
input_image = np.random.rand(256, 256, 3)
model_output = np.random.rand(256, 256, 10)
easy
A. (256, 256, 10)
B. (256, 256, 1)
C. (256, 256, 3)
D. (256, 256)

Solution

  1. Step 1: Understand the model output

    The model outputs a segmentation map with a shape corresponding to the input spatial dimensions and the number of classes.

  2. Step 2: Eliminate wrong options

    (256, 256, 3) is the input shape, (256, 256, 1) implies binary segmentation, and (256, 256) lacks class dimension.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Output shape = Input spatial + class channels [OK]
Hint: Output shape = Input spatial + class channels [OK]
Common Mistakes:
3. When should dropout not be used in a neural network model?
medium
A. When the model is overfitting on the training data.
B. When the model is underfitting and has high bias.
C. When the model is being used for time series forecasting.
D. When the model has a large number of parameters.

Solution

  1. Step 1: Identify when dropout is counterproductive

    Dropout is a regularization technique that helps prevent overfitting, not underfitting.

  2. Step 2: Eliminate wrong options

    When the model is overfitting on the training data. is incorrect because dropout is useful for overfitting. When the model is being used for time series forecasting. is incorrect because dropout can be used in time series if applied correctly. When the model has a large number of parameters. is incorrect because dropout is often used in models with many parameters to prevent overfitting.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Dropout not for underfitting. [OK]
Hint: Dropout not for underfitting. [OK]
Common Mistakes:
4. A deployed model is showing unexpected behavior with high variance in predictions. What is the most likely cause?
medium
A. Overfitting due to a complex model
B. Incorrect hyperparameter tuning
C. Underfitting due to a simple model
D. Data drift in the input features

Solution

  1. Step 1: Identify the symptom.

    High variance in predictions suggests the model is not generalizing well.

  2. Step 2: Determine the most likely cause.

    Data drift, where the input data distribution changes over time, is a common cause of unexpected behavior in production.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Data drift causes unexpected variance. [OK]
Hint: Data drift causes unexpected variance. [OK]
Common Mistakes:
5. You are designing a recommendation system for a video platform. Which metric would best capture user satisfaction if users tend to watch only the first few recommended videos?
hard
A. Precision at K
B. Mean Reciprocal Rank (MRR)
C. Normalized Discounted Cumulative Gain (NDCG)
D. Mean Average Precision (MAP)

Solution

  1. Step 1: Understand user behavior

    Users watching only the first few videos means early precision is crucial.

  2. Step 2: Choose the metric

    Precision at K directly measures the quality of the top K recommendations, aligning with user behavior.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Precision at K focuses on top results. [OK]
Hint: Precision at K focuses on top results. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard