Generative vs discriminative models in Prompt Engineering / GenAI - Metrics Comparison

Metrics & Evaluation - Generative vs discriminative models

Which metric matters for Generative vs Discriminative models and WHY

For discriminative models, metrics like accuracy, precision, recall, and F1-score matter most because these models focus on correctly classifying or predicting labels from input data.

For generative models, metrics that measure how well the model captures the data distribution are important. These include log-likelihood, perplexity, and Inception Score (for images). These metrics show how well the model can generate realistic new data.

In short, discriminative models are judged by how well they separate classes, while generative models are judged by how well they create or model data.

Confusion matrix example for a discriminative model

      Actual \ Predicted | Positive | Negative
      -------------------|----------|---------
      Positive           |    80    |   20
      Negative           |    10    |   90

      Total samples = 200
    

From this matrix:

• Precision = 80 / (80 + 10) = 0.89
• Recall = 80 / (80 + 20) = 0.80
• Accuracy = (80 + 90) / 200 = 0.85

Generative models do not use confusion matrices because they generate data rather than classify.

Precision vs Recall tradeoff with examples

Discriminative models: Imagine a spam email filter.

• High precision means most emails marked as spam really are spam (few good emails wrongly blocked).
• High recall means most spam emails are caught (few spam emails slip through).

Depending on what is worse (missing spam or blocking good emails), you choose to optimize precision or recall.

Generative models: The tradeoff is between quality and diversity of generated data.

• High quality means generated samples look very real.
• High diversity means generated samples cover many different types of data.

Improving one can reduce the other, so metrics like Inception Score balance this tradeoff.

What "good" vs "bad" metric values look like

Discriminative models:

• Good: Accuracy > 90%, Precision and Recall both > 85%
• Bad: Accuracy < 70%, Precision or Recall < 50%

Generative models:

• Good: High log-likelihood or low perplexity, Inception Score close to real data scores
• Bad: Low log-likelihood, high perplexity, generated data looks unrealistic or repetitive

Common pitfalls in metrics

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced (e.g., 95% accuracy by always predicting the majority class).
• Data leakage: When test data leaks into training, metrics look better but model fails in real use.
• Overfitting indicators: Very high training accuracy but low test accuracy means the model memorizes instead of learning.
• Generative model pitfalls: Metrics like Inception Score can be fooled by models that generate limited but high-quality samples, missing diversity.

Self-check question

Your discriminative model has 98% accuracy but only 12% recall on fraud cases. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of fraud cases (low recall), which is dangerous because many frauds go undetected. High accuracy is misleading here because fraud cases are rare. Improving recall is critical.

Key Result

Discriminative models focus on classification metrics like precision and recall, while generative models focus on data quality and diversity metrics like log-likelihood and Inception Score.