Bird
Raised Fist0

Transfer Learning and Fine-Tuning - Why Pretrain Then Finetune

THEORYintermediateInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Transfer Learning vs Training from Scratch

When building a model for a new image classification task with limited labeled data, would you use transfer learning or train from scratch? Justify your choice.

📊
Comparison
DimensionTransfer LearningTraining from Scratch
Data EfficiencyHighLow
Training SpeedFastSlow
Task SpecificityModerateHigh
Computational CostLowHigh
Risk of OverfittingLowHigh
✅
Verdict
Default to transfer learning when labeled data is limited and pre-trained models are available in a similar domain. Switch to training from scratch only when you have ample data and the task is significantly different from existing models.
💡
Key Insight
Transfer learning excels with limited data by leveraging pre-existing knowledge, but task similarity is crucial to avoid negative transfer.
⚠️
Interview Trap
❌ Wrong: Always use transfer learning because it's faster and requires less data.
Ignoring the importance of task similarity and the potential for negative transfer.
✅ Consider the similarity of the new task to the pre-trained model's domain and the availability of labeled data before deciding.
Transfer Learning
"Leverages pre-existing knowledge from large datasets to improve performance on new tasks with limited data."
Weaknesses
  • May not capture task-specific nuances if the pre-trained model's domain is too different.
  • Requires careful selection of pre-trained models to avoid negative transfer.
  • Fine-tuning can be computationally expensive if the model is large.
Ideal When
  • When labeled data < 10K
  • When pre-trained models are available in a similar domain
  • When computational resources are limited for training large models from scratch
Google uses transfer learning in BERT for NLP tasks, leveraging large corpora to improve performance on specific language tasks.
Training from Scratch
"Allows full control over model architecture and training process, potentially capturing task-specific features better."
Weaknesses
  • Requires large amounts of labeled data to achieve good performance.
  • Computationally expensive and time-consuming.
  • Higher risk of overfitting with limited data.
Ideal When
  • When labeled data > 100K
  • When the task is significantly different from existing pre-trained models
  • When there is ample computational power and time for training
OpenAI trained GPT models from scratch using vast amounts of text data to capture general language understanding.
🐍
Transfer Learning - Example
▼ Show
Python
import tensorflow as tf
base_model = tf.keras.applications.VGG16(weights='imagenet', include_top=False)
model = tf.keras.Sequential([
    base_model,
    tf.keras.layers.GlobalAveragePooling2D(),
    tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
🐍
Training from Scratch - Example
▼ Show
Python
import tensorflow as tf
model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(64, 64, 3)),
    tf.keras.layers.MaxPooling2D((2, 2)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
✅
Use Transfer Learning When
  • •When labeled data < 10K
  • •When pre-trained models are available in a similar domain
  • •When computational resources are limited
✅
Use Training from Scratch When
  • •When labeled data > 100K
  • •When the task is significantly different from existing pre-trained models
  • •When there is ample computational power and time for training
↳
Nuances
  • Transfer learning can fail if the pre-trained model's domain is too different, leading to negative transfer.
  • Training from scratch can be beneficial if the task requires capturing very specific features not present in pre-trained models.
  • Transfer learning is often more effective when computational resources are limited, as it reduces the need for extensive training.

Practice

(1/5)
1. Given the following Python code snippet, what is the output?
import numpy as np
features = np.array([1, 2, 3, 4])
normalized = (features - np.mean(features)) / np.std(features)
print(normalized)
easy
A. [-1.5, -0.5, 0.5, 1.5]
B. [0.0, 0.5, 1.0, 1.5]
C. [-1.3416, -0.4472, 0.4472, 1.3416]
D. [1, 2, 3, 4]

Solution

  1. Step 1: Understand the normalization process.

    Normalization involves subtracting the mean and dividing by the standard deviation.

  2. Step 2: Calculate the mean and standard deviation.

    Mean = 2.5, Standard deviation = 1.1180.

  3. Step 3: Apply normalization.

    Normalized values are [-1.5, -0.5, 0.5, 1.5].

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Normalize: subtract mean, divide by std. [OK]
Hint: Normalize: subtract mean, divide by std. [OK]
Common Mistakes:
2. In the context of Graph Neural Networks, which component is primarily responsible for aggregating information from neighboring nodes?
easy
A. Dropout layer
B. Activation function
C. Message passing mechanism
D. Batch normalization

Solution

  1. Step 1: Identify the role of each component

    The message passing mechanism is designed to aggregate information from neighboring nodes in a graph.

  2. Step 2: Eliminate wrong options

    Activation functions, dropout layers, and batch normalization do not specifically aggregate information from neighboring nodes.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Message passing aggregates neighbor info in GNNs. [check]
Hint: Message passing aggregates neighbor info in GNNs. [check]
Common Mistakes:
3. In the context of evaluating a recommendation system, which of the following best describes why offline evaluation metrics might disagree with online performance?
easy
A. Offline metrics are influenced by server latency.
B. Offline metrics are always more accurate than online metrics.
C. Online metrics are biased due to data leakage.
D. Offline metrics may not capture user interaction dynamics.

Solution

  1. Step 1: Understand the difference between offline and online evaluation.

    Offline evaluation uses historical data and predefined metrics, while online evaluation captures real-time user interactions.

  2. Step 2: Eliminate wrong options.

    Offline metrics are always more accurate than online metrics. is incorrect because offline metrics are not inherently more accurate. Online metrics are biased due to data leakage. is incorrect because online metrics are not biased due to data leakage; they reflect real-world interactions. Offline metrics are influenced by server latency. is incorrect because server latency affects online performance, not offline metrics.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Offline misses real user dynamics. [OK]
Hint: Offline misses real user dynamics. [OK]
Common Mistakes:
4. In a dataset with highly imbalanced classes, which sampling strategy is most appropriate to ensure each class is represented proportionally in the sample?
easy
A. Stratified Sampling
B. Weighted Sampling
C. Reservoir Sampling
D. Random Sampling

Solution

  1. Step 1: Identify the need for proportional representation.

    In imbalanced datasets, it's crucial to maintain class proportions to avoid bias.

  2. Step 2: Eliminate inappropriate strategies.

    Random Sampling (A) doesn't guarantee class proportions. Reservoir Sampling (C) is for streaming data, not class balance. Weighted Sampling (D) adjusts for importance, not proportions.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Stratified = Proportional representation [OK]
Hint: Stratified = Proportional representation [OK]
Common Mistakes:
5. When should you prefer using Gini Impurity over Entropy in decision trees?
medium
A. When computational efficiency is a priority.
B. When you need a more interpretable model.
C. When the dataset is highly imbalanced.
D. When you need to handle continuous features.

Solution

  1. Step 1: Compare computational efficiency.

    Gini Impurity is computationally less intensive than Entropy.

  2. Step 2: Eliminate wrong options.

    Interpretability is not affected by the choice, imbalance handling is not specific to Gini, and both can handle continuous features.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Gini is faster than Entropy. [OK]
Hint: Gini is faster than Entropy. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard