Feature extraction strategy in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Feature extraction strategy

Which metric matters for Feature Extraction Strategy and WHY

Feature extraction helps the model learn useful information from raw data. The key metrics to check are accuracy or loss on validation data. These show if the extracted features help the model make better predictions.

For classification tasks, accuracy, precision, and recall matter because they tell us how well the features separate classes.

For regression, mean squared error (MSE) or mean absolute error (MAE) show how well features predict continuous values.

In short, metrics that measure prediction quality after feature extraction are most important. They tell if the features are meaningful.

Confusion Matrix Example

Suppose we extract features from images to classify cats vs dogs. After training, the confusion matrix might look like this:

      | Predicted Cat | Predicted Dog |
      |--------------|---------------|
      | True Cat: 50 | False Dog: 10 |
      | False Cat: 5 | True Dog: 35  |
    

Here:

• TP (True Positive) = 50 (correct cat predictions)
• FP (False Positive) = 5 (dog predicted as cat)
• FN (False Negative) = 10 (cat predicted as dog)
• TN (True Negative) = 35 (correct dog predictions)

From this, precision = 50 / (50 + 5) = 0.91, recall = 50 / (50 + 10) = 0.83.

Precision vs Recall Tradeoff with Feature Extraction

Feature extraction affects precision and recall. For example:

• If features are too general, the model may predict many positives, increasing recall but lowering precision (more false alarms).
• If features are too strict, the model predicts fewer positives, increasing precision but lowering recall (misses some true cases).

Example: In medical image analysis, missing a disease (low recall) is worse than false alarms (low precision). So feature extraction should favor recall.

In spam detection, wrongly marking good emails as spam (low precision) is worse, so features should favor precision.

Good vs Bad Metric Values for Feature Extraction

Good feature extraction leads to:

• High accuracy (e.g., > 85% on validation)
• Balanced precision and recall (both > 0.8) for classification
• Low loss values (e.g., cross-entropy loss < 0.5)

Bad feature extraction shows:

• Low accuracy (close to random guessing, e.g., ~50% for binary)
• Very low precision or recall (below 0.5), meaning poor class separation
• High loss values or no improvement during training

Common Pitfalls in Feature Extraction Metrics

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced. Features may ignore minority classes.
• Data leakage: Features accidentally include future or test data info, inflating metrics falsely.
• Overfitting: Features too tuned to training data cause high training accuracy but poor validation results.
• Ignoring metric tradeoffs: Focusing only on accuracy without checking precision/recall can hide poor feature quality.

Self Check

Your model uses extracted features and shows 98% accuracy but only 12% recall on fraud detection. Is it good?

Answer: No. Despite high accuracy, the model misses most fraud cases (low recall). This is bad because catching fraud is critical. Feature extraction should improve recall to detect fraud better.

Key Result

Feature extraction quality is best judged by balanced precision and recall, not just accuracy.