Python NLP ecosystem (NLTK, spaCy, Hugging Face) - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Tokenization Differences

Which statement best describes the difference between tokenization in NLTK and spaCy?

A spaCy tokenizes text into sentences only, while NLTK tokenizes into words only.

B NLTK uses rule-based tokenization while spaCy uses statistical models for tokenization.

C NLTK tokenization is slower because it uses deep learning models, spaCy uses simple regex.

D Both NLTK and spaCy use the exact same tokenization algorithms internally.

💡 Hint

Think about how each library approaches breaking text into pieces.

NLTK primarily uses rule-based tokenizers like word_tokenize which rely on regular expressions and rules. spaCy uses statistical models trained on large corpora to tokenize more accurately, handling edge cases better.

❓ Predict Output

intermediate

Output of spaCy Named Entity Recognition

What is the output of this code snippet?

NLP

import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp('Apple is looking at buying U.K. startup for $1 billion')
entities = [(ent.text, ent.label_) for ent in doc.ents]
print(entities)

A [('Apple', 'ORG'), ('U.K.', 'LOC'), ('$1 billion', 'MONEY')]

B [('Apple', 'PERSON'), ('U.K.', 'ORG'), ('$1 billion', 'QUANTITY')]

C [('Apple', 'ORG'), ('U.K.', 'GPE'), ('$1 billion', 'MONEY')]

D [('Apple', 'ORG'), ('U.K.', 'GPE'), ('$1 billion', 'QUANTITY')]

💡 Hint

Check spaCy's entity labels for organizations, geopolitical entities, and money.

spaCy labels 'Apple' as an organization (ORG), 'U.K.' as a geopolitical entity (GPE), and '$1 billion' as money (MONEY).

❓ Model Choice

advanced

Choosing a Hugging Face Model for Sentiment Analysis

You want to perform sentiment analysis on movie reviews using Hugging Face transformers. Which model is the best choice?

A bert-base-uncased

B gpt2

C roberta-base

D distilbert-base-uncased-finetuned-sst-2-english

💡 Hint

Look for a model fine-tuned specifically for sentiment tasks.

Option D is a DistilBERT model fine-tuned on the SST-2 dataset for sentiment analysis, making it the best choice. The others are general language models or not fine-tuned for sentiment.

❓ Hyperparameter

advanced

Effect of Changing Learning Rate in Fine-Tuning Transformers

During fine-tuning a Hugging Face transformer model, what is the most likely effect of setting the learning rate too high?

A The model training becomes unstable and loss may not decrease properly.

B The model converges faster and achieves higher accuracy.

C The model ignores the training data and uses pre-trained weights only.

D The model will underfit and have very low training loss.

💡 Hint

Think about what happens when updates are too large during training.

A too high learning rate causes large weight updates, making training unstable and preventing the loss from decreasing smoothly.

🔧 Debug

expert

Debugging Tokenization Output in NLTK

What error or unexpected output will this code produce?

NLP

from nltk.tokenize import word_tokenize
text = "Hello, world! Let's test tokenization."
tokens = word_tokenize(text)
print(tokens[10])

A IndexError: list index out of range

B 'tokenization'

C 'test'

D SyntaxError

💡 Hint

Count how many tokens are produced and check the index accessed.

The tokenized list has fewer than 11 tokens, so accessing index 10 causes an IndexError.

Practice

1. Which Python library is best known for providing pre-trained models for advanced NLP tasks?

easy

A. NLTK

B. Hugging Face

C. spaCy

D. Scikit-learn

Solution

1. Step 1: Understand the role of each library

   NLTK is mainly for learning and basic NLP tasks, spaCy is for fast real-world processing, and Hugging Face offers powerful pre-trained models.

2. Step 2: Identify the library specialized in pre-trained models

   Hugging Face is known for its large collection of pre-trained transformer models for advanced NLP.

3. Final Answer:

   Hugging Face -> Option B

4. Quick Check:

   Pre-trained models = Hugging Face [OK]

Hint: Remember: Hugging Face = pre-trained models [OK]

Common Mistakes:

• Confusing NLTK as the source of pre-trained models
• Thinking spaCy provides many pre-trained transformer models
• Choosing Scikit-learn which is not specialized for NLP

2. Which of the following is the correct way to import the English language model in spaCy?

easy

A. import spacy; nlp = spacy.load('en_core_web_sm')

B. import spacy; nlp = spacy.load('english')

C. from spacy import English; nlp = English()

D. import spacy; nlp = spacy.load('en')

Solution

1. Step 1: Recall spaCy's model loading syntax

   spaCy loads models using spacy.load() with the model name like 'en_core_web_sm'.

2. Step 2: Check each option's syntax

   import spacy; nlp = spacy.load('en_core_web_sm') uses the correct model name for the small English core model. 'en' loads a blank model without components, 'english' is not a valid model name, and from spacy import English; nlp = English() only initializes a basic tokenizer without trained pipelines.

3. Final Answer:

   import spacy; nlp = spacy.load('en_core_web_sm') -> Option A

4. Quick Check:

   spaCy model load = spacy.load('en_core_web_sm') [OK]

Hint: Use spacy.load('en_core_web_sm') to load English model [OK]

Common Mistakes:

• Using 'english' or 'en' instead of 'en_core_web_sm'
• Trying to import English class instead of loading model
• Forgetting to install the model before loading

3. What will be the output of this NLTK code snippet?

import nltk
from nltk.tokenize import word_tokenize
text = "Hello world!"
tokens = word_tokenize(text)
print(tokens)

medium

A. ['Hello world!']

B. ['Hello', 'world']

C. ['Hello', 'world!']

D. ['Hello', 'world', '!']

Solution

1. Step 1: Understand word_tokenize behavior

   NLTK's word_tokenize splits text into words and punctuation separately.

2. Step 2: Apply tokenization to 'Hello world!'

   The text splits into three tokens: 'Hello', 'world', and '!'.

3. Final Answer:

   ['Hello', 'world', '!'] -> Option D

4. Quick Check:

   word_tokenize splits punctuation separately [OK]

Hint: word_tokenize splits punctuation as separate tokens [OK]

Common Mistakes:

• Expecting punctuation to stay attached to words
• Confusing tokenization with simple split()
• Ignoring that '!' is a separate token

4. Identify the error in this Hugging Face transformers code snippet:

from transformers import pipeline
classifier = pipeline('sentiment-analysis')
result = classifier('I love NLP!')
print(result[0])

medium

A. Missing model download before pipeline creation

B. Incorrect pipeline task name

C. No error, code runs correctly

D. Result indexing should be result[1]

Solution

1. Step 1: Check pipeline usage

   The pipeline function with 'sentiment-analysis' is correct and downloads the default model automatically if needed.

2. Step 2: Verify result usage

   The classifier returns a list of dicts; accessing result[0] is correct to get the first prediction.

3. Final Answer:

   No error, code runs correctly -> Option C

4. Quick Check:

   Hugging Face pipeline auto-downloads models [OK]

Hint: Hugging Face pipelines auto-download models [OK]

Common Mistakes:

• Thinking model must be downloaded manually first
• Using wrong pipeline task name
• Accessing wrong index of result list

5. You want to extract named entities from a text quickly and accurately. Which combination of tools and steps is best?

hard

A. Use spaCy's pre-trained model with nlp = spacy.load('en_core_web_sm') and then nlp(text).ents

B. Use NLTK's word_tokenize and then manually match entity patterns

C. Use Hugging Face pipeline('ner') without loading any model

D. Use spaCy's tokenizer only and ignore entity recognition

Solution

1. Step 1: Identify fast and accurate named entity extraction

   spaCy provides pre-trained models that include named entity recognition (NER) ready to use.

2. Step 2: Evaluate options for NER

   Use spaCy's pre-trained model with nlp = spacy.load('en_core_web_sm') and then nlp(text).ents uses spaCy's model and extracts entities with nlp(text).ents, which is efficient and accurate. Use NLTK's word_tokenize and then manually match entity patterns requires manual pattern matching, which is slow and error-prone. Use Hugging Face pipeline('ner') without loading any model misses loading a model explicitly, which is needed. Use spaCy's tokenizer only and ignore entity recognition ignores entity recognition.

3. Final Answer:

   Use spaCy's pre-trained model with nlp = spacy.load('en_core_web_sm') and then nlp(text).ents -> Option A

4. Quick Check:

   spaCy pre-trained models = fast NER [OK]

Hint: spaCy pre-trained models provide fast named entity recognition [OK]

Common Mistakes:

• Trying to do NER manually with NLTK tokens
• Using pipeline('ner') without model loading
• Ignoring entity extraction step