TF-IDF (TfidfVectorizer) in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of TF-IDF vectorization with simple corpus

What is the shape of the TF-IDF matrix produced by the following code?

NLP

from sklearn.feature_extraction.text import TfidfVectorizer
corpus = ['apple orange banana', 'banana fruit apple', 'fruit orange']
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus)
print(X.shape)

A (3, 4)

B (4, 3)

C (4, 4)

D (3, 3)

💡 Hint

Remember, rows correspond to documents and columns to unique words.

The corpus has 3 documents and 4 unique words: 'apple', 'orange', 'banana', 'fruit'. So the TF-IDF matrix shape is (3, 4).

🧠 Conceptual

intermediate

Understanding IDF effect on common words

Which statement best describes the effect of IDF (Inverse Document Frequency) in TF-IDF vectorization?

A IDF only considers the length of each document.

B IDF increases the weight of words that appear in many documents.

C IDF decreases the weight of words that appear in many documents.

D IDF assigns equal weight to all words regardless of frequency.

💡 Hint

Think about how common words like 'the' or 'and' should be treated.

IDF reduces the weight of words that appear in many documents to highlight more unique words.

❓ Hyperparameter

advanced

Effect of max_df parameter in TfidfVectorizer

What is the effect of setting max_df=0.5 in TfidfVectorizer?

A It normalizes the TF-IDF vectors to have max value 0.5.

B It ignores words that appear in less than 50% of the documents.

C It limits the maximum number of features to 50.

D It ignores words that appear in more than 50% of the documents.

💡 Hint

max_df controls filtering of very common words.

max_df=0.5 means words appearing in more than half the documents are ignored as too common.

❓ Metrics

advanced

Interpreting cosine similarity with TF-IDF vectors

Given two TF-IDF vectors A and B, which cosine similarity value indicates the most similar documents?

A 1.0

B 0.0

C -1.0

D 0.5

💡 Hint

Cosine similarity ranges from -1 to 1.

Cosine similarity of 1 means vectors point in the same direction, indicating maximum similarity.

🔧 Debug

expert

Identifying error in TF-IDF vectorization code

What error will the following code raise when executed? from sklearn.feature_extraction.text import TfidfVectorizer corpus = ['data science', 'machine learning'] vectorizer = TfidfVectorizer(stop_words='english') X = vectorizer.fit_transform(corpus) print(X.toarray()) print(vectorizer.get_feature_names_out()) print(vectorizer.stop_words_) print(vectorizer.vocabulary_['science'])

A AttributeError because 'stop_words_' attribute does not exist

B No error, code runs successfully

C TypeError because stop_words parameter expects a list, not a string

D KeyError because 'science' is removed as a stop word

💡 Hint

Check if 'science' is considered a stop word and if it exists in vocabulary.

No error occurs. 'science' is not an English stopword, so it remains in the vocabulary. The stop_words_ attribute exists after fitting, and stop_words='english' is valid.