Tokenization in spaCy in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of spaCy Tokenization Code

What is the output of the following code snippet that uses spaCy to tokenize a sentence?

NLP

import spacy
nlp = spacy.blank('en')
doc = nlp('Hello world! How are you?')
tokens = [token.text for token in doc]
print(tokens)

A ['Hello', 'world', '!', 'How', 'are', 'you', '?']

B ['Hello world!', 'How are you?']

C ['Hello', 'world!', 'How', 'are', 'you?']

D ['Hello', 'world', 'How', 'are', 'you']

💡 Hint

Think about how spaCy splits punctuation from words by default.

spaCy tokenizes text by separating punctuation from words, so 'world!' becomes 'world' and '!' as separate tokens.

🧠 Conceptual

intermediate

spaCy Tokenizer Behavior on Contractions

Which statement correctly describes how spaCy tokenizes contractions like "don't" by default?

A It removes the apostrophe and returns 'dont' as one token.

B It keeps "don't" as a single token.

C It splits "don't" into three tokens: 'do', 'n', and 't'.

D It splits "don't" into two tokens: 'do' and 'n't'.

💡 Hint

Think about how spaCy handles common English contractions.

spaCy splits contractions into base word and contraction part, so "don't" becomes 'do' and 'n't'.

❓ Hyperparameter

advanced

Changing spaCy Tokenizer Behavior

Which spaCy component or method would you customize to change how tokens are split, for example to keep 'New York' as one token?

A Modify the tokenizer exceptions or add special cases to the tokenizer.

B Change the pipeline's tagger component settings.

C Adjust the parser's dependency rules.

D Modify the lemmatizer's dictionary.

💡 Hint

Token splitting is controlled before tagging or parsing.

The tokenizer controls how text is split into tokens; adding special cases can keep multi-word expressions as one token.

❓ Metrics

advanced

Evaluating Tokenization Accuracy

You have a gold standard tokenization and a spaCy tokenizer output. Which metric best measures how well spaCy tokenized the text compared to the gold standard?

A Perplexity of the tokenizer output.

B Token-level F1 score comparing spaCy tokens to gold tokens.

C Sentence-level BLEU score.

D Accuracy of part-of-speech tags.

💡 Hint

Think about comparing sets of tokens for overlap.

Token-level F1 score measures precision and recall of tokens compared to gold standard, suitable for tokenization evaluation.

🔧 Debug

expert

Identifying Tokenization Bug in spaCy Customization

You added a special case to spaCy's tokenizer to keep 'San Francisco' as one token, but after running, it still splits into two tokens. What is the most likely cause?

NLP

import spacy
from spacy.symbols import ORTH

nlp = spacy.blank('en')
special_case = [{ORTH: 'San Francisco'}]
nlp.tokenizer.add_special_case('San Francisco', special_case)
doc = nlp('I visited San Francisco last year.')
tokens = [token.text for token in doc]
print(tokens)

A The ORTH symbol is incorrect; it should be LEMMA.

B The tokenizer needs to be rebuilt after adding special cases.

C The special case should be a list of dicts with separate tokens, not a single dict with the full phrase.

D The blank model 'en' does not support special cases.

💡 Hint

Check the format of the special case argument.

Special cases require a list of token dicts, each with ORTH for each token; a single dict with full phrase is invalid.