Choosing number of topics in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Choosing number of topics

Which metric matters for choosing number of topics and WHY

When picking how many topics to use in a topic model, we want metrics that tell us how clear and useful the topics are. Common metrics include:

• Coherence: Measures how related the top words in each topic are. Higher coherence means topics make more sense together.
• Perplexity: Measures how well the model predicts unseen data. Lower perplexity means better generalization.

Coherence is often preferred because it matches human understanding better. We want a number of topics that balances good coherence without making topics too broad or too narrow.

Confusion matrix or equivalent visualization

Topic modeling does not use a confusion matrix like classification. Instead, we look at metric trends across different topic counts. For example:

Number of Topics | Coherence Score
-----------------|----------------
       5         |      0.42
      10         |      0.51
      15         |      0.48
      20         |      0.44
    

This table shows coherence scores for different topic counts. The best coherence is at 10 topics here.

Precision vs Recall tradeoff (or equivalent)

Instead of precision and recall, topic modeling has a tradeoff between:

• Too few topics: Topics are broad and mix different ideas, making interpretation hard.
• Too many topics: Topics become too specific or noisy, splitting meaningful themes.

Choosing the right number balances clear, distinct topics without losing important themes.

What "good" vs "bad" metric values look like for this use case

Good: Coherence scores around 0.5 or higher usually mean topics are meaningful and interpretable. The number of topics chosen should show a peak or plateau in coherence.

Bad: Very low coherence (e.g., below 0.3) means topics are not related well. Also, if coherence keeps dropping as topics increase, the model may be overfitting or creating noisy topics.

Metrics pitfalls

• Relying only on perplexity: Lower perplexity does not always mean better topics for humans.
• Ignoring interpretability: Metrics can be high but topics may not make sense to people.
• Choosing too many topics: Leads to overfitting and fragmented topics.
• Data leakage: Using test data in training can give misleading metrics.

Self-check question

Your topic model has 25 topics with coherence 0.35 and perplexity improving as topics increase. Is this good?

Answer: No, because coherence is low, topics may not be meaningful. Even if perplexity improves, the topics might be too many and noisy. Consider fewer topics with higher coherence.

Key Result

Coherence score guides choosing number of topics by measuring topic interpretability; aim for highest coherence without too many topics.