Topic coherence evaluation in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What does topic coherence measure in topic modeling?

Topic coherence is a key metric in topic modeling. What does it measure?

A The number of topics generated by the model

B The speed at which the model converges during training

C The size of the vocabulary used in the model

D The semantic similarity between the top words in a topic

💡 Hint

Think about how well the words in a topic relate to each other in meaning.

Topic coherence measures how semantically related the top words in a topic are, indicating the quality of the topic.

❓ Predict Output

intermediate

Output of coherence score calculation code

What is the output of the following Python code that calculates topic coherence using Gensim?

NLP

from gensim.models import CoherenceModel
from gensim.corpora.dictionary import Dictionary
texts = [['apple', 'banana', 'fruit'], ['banana', 'orange', 'fruit'], ['apple', 'orange', 'fruit']]
dictionary = Dictionary(texts)
corpus = [dictionary.doc2bow(text) for text in texts]
topics = [['apple', 'banana', 'fruit'], ['banana', 'orange', 'fruit']]
coherence_model = CoherenceModel(topics=topics, texts=texts, dictionary=dictionary, coherence='c_v')
score = coherence_model.get_coherence()
print(round(score, 2))

A 0.91

B 0.45

C 1.00

D 0.00

💡 Hint

Coherence scores range between 0 and 1, higher means better semantic similarity.

The code calculates the c_v coherence score for the given topics and texts, resulting in approximately 0.91.

❓ Model Choice

advanced

Choosing a coherence measure for short texts

You want to evaluate topic coherence on very short texts like tweets. Which coherence measure is most suitable?

A u_mass coherence, which relies on document co-occurrence counts

B c_npmi coherence, which uses normalized pointwise mutual information

C c_v coherence, which uses sliding windows and a boolean sliding window approach

D c_uci coherence, which uses pointwise mutual information with a sliding window

💡 Hint

Consider which measure works best with sparse or short documents.

c_npmi coherence is better for short texts because it normalizes PMI and handles sparse data well.

❓ Hyperparameter

advanced

Effect of number of topics on coherence score

When increasing the number of topics in a topic model, what is the typical effect on the coherence score?

A Coherence score remains constant regardless of topic number

B Coherence score fluctuates randomly with no pattern

C Coherence score usually decreases because topics become less distinct and more noisy

D Coherence score usually increases because more topics capture more details

💡 Hint

Think about how splitting topics too much affects their quality.

Increasing topics too much often lowers coherence because topics overlap and contain less meaningful word groups.

🔧 Debug

expert

Why does this coherence calculation raise a ValueError?

Consider this code snippet that raises a ValueError when calculating coherence. What is the cause?

NLP

from gensim.models import CoherenceModel
texts = [['data', 'science'], ['machine', 'learning']]
topics = [['data', 'science'], ['machine', 'learning']]
coherence_model = CoherenceModel(topics=topics, texts=texts, coherence='c_v')
score = coherence_model.get_coherence()

A The dictionary parameter is missing, so the model cannot map words to ids

B The topics list is empty, causing no data to compute coherence

C The coherence type 'c_v' is not supported by Gensim

D The texts contain words not present in topics, causing mismatch

💡 Hint

Check if all required inputs for CoherenceModel are provided.

The CoherenceModel requires a dictionary to map words to ids for 'c_v' coherence. Missing it causes ValueError.