Bird
Raised Fist0

Data Splits - Train, Validation, Test - Why Three Sets

THEORYbeginnerInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Train-Validation-Test Split vs Train-Test Split

When building a machine learning model, should you use a train-validation-test split or just a train-test split? Justify your choice.

📊
Comparison
DimensionTrain-Validation-Test SplitTrain-Test Split
Hyperparameter tuningHighLow
Overfitting riskLowHigh
Computational costHighLow
Implementation complexityHighLow
Data requirementHighLow
✅
Verdict
Default to a train-validation-test split for datasets larger than 10,000 samples and when hyperparameter tuning is crucial. Opt for a train-test split when data is limited and rapid iteration is needed.
💡
Key Insight
The choice between data splits hinges on balancing the need for robust model evaluation with resource constraints.
⚠️
Interview Trap
❌ Wrong: Always use a train-validation-test split for better performance.
Ignoring the computational and data overhead when resources are limited.
✅ Consider the dataset size and resource availability before deciding on the split strategy.
Train-Validation-Test Split
"Provides a separate dataset for hyperparameter tuning, reducing overfitting risk."
Weaknesses
  • Requires more data to maintain sufficient size for each split.
  • Increases computational cost due to additional validation phase.
  • Complexity in managing three datasets instead of two.
Ideal When
  • n > 10,000
  • Hyperparameter tuning is critical
  • Model complexity is high
  • Sufficient computational resources are available
Google uses train-validation-test splits for complex models like BERT to ensure robust hyperparameter tuning.
Train-Test Split
"Simplifies the process and reduces computational overhead."
Weaknesses
  • Higher risk of overfitting due to lack of validation set.
  • Limited ability to tune hyperparameters effectively.
  • Potential for biased performance estimates.
Ideal When
  • n < 10,000
  • Limited computational resources
  • Simple models with few hyperparameters
  • Quick prototyping is needed
Startups with limited data and resources often use train-test splits for rapid prototyping and iteration.
🐍
Train-Validation-Test Split - Example
▼ Show
Python
import numpy as np
from sklearn.model_selection import train_test_split

X, y = np.random.rand(10000, 10), np.random.randint(0, 2, 10000)
X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.3)
X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5)
🐍
Train-Test Split - Example
▼ Show
Python
import numpy as np
from sklearn.model_selection import train_test_split

X, y = np.random.rand(1000, 10), np.random.randint(0, 2, 1000)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
✅
Use Train-Validation-Test Split When
  • •n > 10,000
  • •Hyperparameter tuning is critical
  • •Model complexity is high
  • •Sufficient computational resources are available
✅
Use Train-Test Split When
  • •n < 10,000
  • •Limited computational resources
  • •Simple models with few hyperparameters
  • •Quick prototyping is needed
↳
Nuances
  • For very small datasets, a train-test split might be more practical despite the risk of overfitting.
  • In time-sensitive projects, the simplicity of a train-test split can outweigh the benefits of a validation set.
  • For models with few hyperparameters, the validation set may not provide significant benefits.

Practice

(1/5)
1. Which dimensionality reduction technique is most suitable for visualizing high-dimensional data while preserving local structures?
easy
A. t-SNE
B. PCA
C. UMAP
D. LDA

Solution

  1. Step 1: Understand the purpose of each technique

    PCA is used for linear dimensionality reduction, UMAP is similar to t-SNE but faster, and LDA is for supervised dimensionality reduction.

  2. Step 2: Identify the technique that preserves local structures

    t-SNE is specifically designed to preserve local structures in high-dimensional data.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    t-SNE for local structures [OK]
Hint: t-SNE for local structures [OK]
Common Mistakes:
2. A model registered in MLflow Registry is not showing the expected performance in production. What is the most likely cause?
medium
A. The model version was not updated in the registry.
B. The model was not packaged correctly in MLflow Models.
C. The model's hyperparameters were not logged correctly.
D. The model was trained with a different dataset than intended.

Solution

  1. Step 1: Analyze potential causes of performance issues

    Data discrepancies are a common cause of unexpected model performance.

  2. Step 2: Eliminate less likely causes

    Incorrect dataset usage is more likely than versioning or packaging issues.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Check dataset consistency [OK]
Hint: Check dataset consistency [OK]
Common Mistakes:
3. You have a model that is overfitting. Which approach is more suitable to address this issue?
medium
A. Add dropout layers to the model.
B. Reduce the size of the training dataset.
C. Use a smaller learning rate.
D. Increase the model complexity by adding more layers.

Solution

  1. Step 1: Identify methods to combat overfitting

    Dropout is a regularization technique that helps prevent overfitting by randomly dropping units during training.

  2. Step 2: Eliminate wrong options

    Increase the model complexity by adding more layers. increases overfitting risk, Reduce the size of the training dataset. reduces data, worsening generalization, and Use a smaller learning rate. affects convergence speed, not overfitting directly.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Dropout reduces overfitting [OK]
Hint: Dropout reduces overfitting [OK]
Common Mistakes:
4. A fine-tuned model is underperforming on a specific task despite using a well-known pretrained model. What is a likely cause?
medium
A. The pretrained model's architecture is incompatible with the task.
B. The learning rate during fine-tuning was too high, causing overfitting.
C. The pretrained model was not trained on a large enough dataset.
D. The task-specific dataset is too small, leading to poor fine-tuning.

Solution

  1. Step 1: Identify common issues in fine-tuning.

    Small task-specific datasets can lead to poor adaptation of pretrained models.

  2. Step 2: Eliminate wrong options.

    The pretrained model was not trained on a large enough dataset. is incorrect because the pretrained model's dataset size is not the issue. The learning rate during fine-tuning was too high, causing overfitting. is incorrect because a high learning rate typically causes instability, not overfitting. The pretrained model's architecture is incompatible with the task. is incorrect because architecture incompatibility would prevent fine-tuning altogether.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Small dataset? Poor fine-tuning. [check]
Hint: Small dataset? Poor fine-tuning. [check]
Common Mistakes:
5. You are tasked with building a multilingual language model. Which tokenization method would best handle multiple languages with different scripts and why?
hard
A. Byte Pair Encoding (BPE) because it efficiently compresses the vocabulary.
B. SentencePiece because it can handle multiple scripts without relying on whitespace.
C. WordPiece because it handles rare words across languages.
D. Character-level tokenization because it treats all scripts uniformly.

Solution

  1. Step 1: Consider the requirements for multilingual tokenization.

    SentencePiece is designed to handle multiple scripts and does not rely on whitespace, making it suitable for multilingual models.

  2. Step 2: Eliminate wrong options.

    BPE (A) and WordPiece (B) are less effective with diverse scripts; character-level (D) is too granular and inefficient.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    SentencePiece handles scripts without whitespace. [OK]
Hint: SentencePiece handles scripts without whitespace. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard