Document-term matrix in NLP - ML Experiment: Train & Evaluate

Experiment - Document-term matrix

Problem:You want to convert a small set of text documents into a document-term matrix to prepare for text analysis.

Current Metrics:The current code creates a document-term matrix but includes all words, including very common words like 'the' and 'is', which may not be useful.

Issue:The matrix is too large and noisy because it includes stop words and very rare words, making it harder to analyze and slowing down further processing.

Your Task

Create a cleaner document-term matrix by removing common stop words and very rare words, reducing noise and matrix size.

Use Python and scikit-learn's CountVectorizer.

Keep the vocabulary size manageable (e.g., max 10 words).

Do not use any external datasets.

Hint 1

Hint 2

Hint 3

Solution

NLP

from sklearn.feature_extraction.text import CountVectorizer

# Sample documents
texts = [
    'The cat sat on the mat.',
    'Dogs and cats are great pets.',
    'I love my dog.',
    'Cats are playful and cute.',
    'The dog chased the cat.'
]

# Original vectorizer without stop words
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)
print(f'Original matrix shape: {X.shape}')
print(f'Original vocabulary: {vectorizer.get_feature_names_out()}')

# Vectorizer with stop words removal and min_df to remove rare words
clean_vectorizer = CountVectorizer(stop_words='english', min_df=2, max_features=10)
X_clean = clean_vectorizer.fit_transform(texts)
print(f'Cleaned matrix shape: {X_clean.shape}')
print(f'Cleaned vocabulary: {clean_vectorizer.get_feature_names_out()}')

Added stop_words='english' to remove common English words like 'the', 'and', 'is'.

Added min_df=2 to ignore words that appear in fewer than 2 documents.

Added max_features=10 to limit the vocabulary size to the 10 most frequent words.

Printed matrix shapes and vocabularies before and after cleaning to compare.

Results Interpretation

Before cleaning, the document-term matrix had 17 columns (words), including common stop words and rare words.

After cleaning, the matrix reduced to 3 columns, removing stop words and rare words, making it smaller and more focused.

Removing stop words and rare words helps create a cleaner, smaller document-term matrix that is easier to analyze and speeds up further text processing.

Bonus Experiment

Try creating a TF-IDF matrix instead of a simple count matrix to weigh words by importance.

💡 Hint

Use sklearn's TfidfVectorizer with similar parameters to see how word importance changes.

Practice

1. What does a document-term matrix represent in natural language processing?

easy

A. The length of each document

B. The order of words in a sentence

C. The meaning of each word

D. Counts of words in each document

Solution

1. Step 1: Understand the purpose of a document-term matrix

   A document-term matrix counts how many times each word appears in each document.

2. Step 2: Compare options with this definition

   Only Counts of words in each document correctly describes this counting process.

3. Final Answer:

   Counts of words in each document -> Option D

4. Quick Check:

   Document-term matrix = word counts [OK]

Hint: Remember: matrix counts words per document [OK]

Common Mistakes:

• Confusing word order with counts
• Thinking it shows word meanings
• Assuming it measures document length

2. Which Python library provides the CountVectorizer class to create a document-term matrix?

easy

A. numpy

B. pandas

C. scikit-learn

D. matplotlib

Solution

1. Step 1: Recall the library for text feature extraction

   CountVectorizer is part of scikit-learn, a popular machine learning library.

2. Step 2: Verify other options

   numpy is for arrays, pandas for data frames, matplotlib for plotting, so they don't provide CountVectorizer.

3. Final Answer:

   scikit-learn -> Option C

4. Quick Check:

   CountVectorizer = scikit-learn [OK]

Hint: CountVectorizer is from scikit-learn, not numpy [OK]

Common Mistakes:

• Choosing numpy because it handles arrays
• Confusing pandas with text vectorization
• Selecting matplotlib for visualization

3. What is the output of this Python code snippet?

from sklearn.feature_extraction.text import CountVectorizer
texts = ['cat dog', 'dog dog cat']
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)
print(X.toarray())

medium

A. [[1 1]
[1 2]]

B. [[1 1]
[2 1]]

C. [[2 1]
[1 2]]

D. [[1 2]
[1 1]]

Solution

1. Step 1: Identify the vocabulary and word counts

   The texts are 'cat dog' and 'dog dog cat'. Vocabulary sorted alphabetically is ['cat', 'dog']. First document has 1 'cat' and 1 'dog'. Second document has 1 'cat' and 2 'dog's.

2. Step 2: Form the document-term matrix

   Matrix rows correspond to documents, columns to words: [[1,1],[1,2]].

3. Final Answer:

   [[1 1] [1 2]] -> Option A

4. Quick Check:

   Word counts match matrix [OK]

Hint: Count words per document in alphabetical order [OK]

Common Mistakes:

• Mixing order of words in vocabulary
• Counting wrong number of word occurrences
• Confusing rows and columns

4. Identify the error in this code that tries to create a document-term matrix:

from sklearn.feature_extraction.text import CountVectorizer
texts = ['apple orange', 'orange apple apple']
vectorizer = CountVectorizer()
X = vectorizer.transform(texts)
print(X.toarray())

medium

A. toarray() is not a method of X

B. Missing fit() before transform()

C. texts should be a single string, not a list

D. CountVectorizer() should be CountVector()

Solution

1. Step 1: Understand CountVectorizer usage

   CountVectorizer requires calling fit() or fit_transform() before transform() to learn vocabulary.

2. Step 2: Check the code sequence

   The code calls transform() directly without fit(), causing an error.

3. Final Answer:

   Missing fit() before transform() -> Option B

4. Quick Check:

   fit() needed before transform() [OK]

Hint: Always fit before transform with CountVectorizer [OK]

Common Mistakes:

• Skipping fit() step
• Using wrong class name
• Passing wrong data type to vectorizer

5. You have three documents: ['sun moon', 'moon moon sun', 'star sun moon']. Using CountVectorizer, what is the shape of the document-term matrix and which word has the highest total count across all documents?

hard

A. Shape (3, 3), 'moon' has highest count

B. Shape (3, 4), 'sun' has highest count

C. Shape (3, 3), 'sun' has highest count

D. Shape (3, 4), 'moon' has highest count

Solution

1. Step 1: Identify unique words and matrix shape

   Unique words are 'sun', 'moon', 'star' -> 3 words. There are 3 documents, so shape is (3, 3).

2. Step 2: Count total occurrences of each word

   'sun': appears 1 + 1 + 1 = 3 times 'moon': appears 1 + 2 + 1 = 4 times 'star': appears 0 + 0 + 1 = 1 time Highest count is 'moon' with 4.

3. Final Answer:

   Shape (3, 3), 'moon' has highest count -> Option A

4. Quick Check:

   3 docs x 3 words, moon count highest [OK]

Hint: Count unique words for shape, sum counts for highest word [OK]

Common Mistakes:

• Counting duplicate words as unique
• Mixing up shape dimensions
• Incorrectly summing word counts