Python NLP ecosystem (NLTK, spaCy, Hugging Face) - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Tokenization Differences

Which statement best describes the difference between tokenization in NLTK and spaCy?

A spaCy tokenizes text into sentences only, while NLTK tokenizes into words only.

B NLTK uses rule-based tokenization while spaCy uses statistical models for tokenization.

C NLTK tokenization is slower because it uses deep learning models, spaCy uses simple regex.

D Both NLTK and spaCy use the exact same tokenization algorithms internally.

💡 Hint

Think about how each library approaches breaking text into pieces.

NLTK primarily uses rule-based tokenizers like word_tokenize which rely on regular expressions and rules. spaCy uses statistical models trained on large corpora to tokenize more accurately, handling edge cases better.

❓ Predict Output

intermediate

Output of spaCy Named Entity Recognition

What is the output of this code snippet?

NLP

import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp('Apple is looking at buying U.K. startup for $1 billion')
entities = [(ent.text, ent.label_) for ent in doc.ents]
print(entities)

A [('Apple', 'ORG'), ('U.K.', 'LOC'), ('$1 billion', 'MONEY')]

B [('Apple', 'PERSON'), ('U.K.', 'ORG'), ('$1 billion', 'QUANTITY')]

C [('Apple', 'ORG'), ('U.K.', 'GPE'), ('$1 billion', 'MONEY')]

D [('Apple', 'ORG'), ('U.K.', 'GPE'), ('$1 billion', 'QUANTITY')]

💡 Hint

Check spaCy's entity labels for organizations, geopolitical entities, and money.

spaCy labels 'Apple' as an organization (ORG), 'U.K.' as a geopolitical entity (GPE), and '$1 billion' as money (MONEY).

❓ Model Choice

advanced

Choosing a Hugging Face Model for Sentiment Analysis

You want to perform sentiment analysis on movie reviews using Hugging Face transformers. Which model is the best choice?

A bert-base-uncased

B gpt2

C roberta-base

D distilbert-base-uncased-finetuned-sst-2-english

💡 Hint

Look for a model fine-tuned specifically for sentiment tasks.

Option D is a DistilBERT model fine-tuned on the SST-2 dataset for sentiment analysis, making it the best choice. The others are general language models or not fine-tuned for sentiment.

❓ Hyperparameter

advanced

Effect of Changing Learning Rate in Fine-Tuning Transformers

During fine-tuning a Hugging Face transformer model, what is the most likely effect of setting the learning rate too high?

A The model training becomes unstable and loss may not decrease properly.

B The model converges faster and achieves higher accuracy.

C The model ignores the training data and uses pre-trained weights only.

D The model will underfit and have very low training loss.

💡 Hint

Think about what happens when updates are too large during training.

A too high learning rate causes large weight updates, making training unstable and preventing the loss from decreasing smoothly.

🔧 Debug

expert

Debugging Tokenization Output in NLTK

What error or unexpected output will this code produce?

NLP

from nltk.tokenize import word_tokenize
text = "Hello, world! Let's test tokenization."
tokens = word_tokenize(text)
print(tokens[10])

A IndexError: list index out of range

B 'tokenization'

C 'test'

D SyntaxError

💡 Hint

Count how many tokens are produced and check the index accessed.

The tokenized list has fewer than 11 tokens, so accessing index 10 causes an IndexError.