Tokenization in spaCy in NLP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is tokenization in spaCy?

Tokenization in spaCy is the process of breaking down text into smaller pieces called tokens, such as words, punctuation, or symbols, to help computers understand and analyze the text.

intermediate

How does spaCy handle tokenization differently from simple splitting by spaces?

spaCy uses rules and machine learning to split text into tokens, considering punctuation, contractions, and special cases, rather than just splitting by spaces, which helps keep meaningful parts together.

beginner

What Python code would you use to tokenize the sentence 'Hello, world!' using spaCy?

import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp('Hello, world!')
tokens = [token.text for token in doc]
print(tokens)  # Output: ['Hello', ',', 'world', '!']

beginner

Why is tokenization important before other NLP tasks?

Tokenization breaks text into manageable pieces, making it easier for models to analyze meaning, find patterns, and perform tasks like translation, sentiment analysis, or named entity recognition.

advanced

Can spaCy's tokenizer be customized? If yes, how?

Yes, spaCy's tokenizer can be customized by adding special cases, modifying rules, or changing how it splits tokens to better fit specific text types or languages.

What does spaCy use to split text into tokens?

A Only spaces

B Manual input

C Random splitting

D Rules and machine learning

spaCy uses a combination of rules and machine learning to accurately split text into tokens.

Which of these is NOT a token in the sentence 'Hello, world!' according to spaCy?

A Hello

B Hello world

C ,

D world

'Hello world' is two tokens, not one; spaCy splits them into separate tokens.

Why might you want to customize spaCy's tokenizer?

A To handle special text cases better

B To make it slower

C To remove punctuation always

D To ignore all spaces

Customization helps spaCy handle special cases or unique text formats more accurately.

What Python function is used to load a spaCy language model for tokenization?

A spacy.load()

B spacy.tokenize()

C spacy.split()

D spacy.model()

spacy.load() loads a language model needed for tokenization and other NLP tasks.

Tokenization helps in which of the following NLP tasks?

A Audio processing

B Image recognition

C Sentiment analysis

D Video editing

Tokenization prepares text for tasks like sentiment analysis by breaking it into meaningful parts.

Explain in your own words what tokenization in spaCy is and why it is useful.

Think about how breaking a sentence into words helps a computer understand it.

Describe how you would use spaCy to tokenize a sentence and get a list of tokens.

Remember the basic Python code to load a model and process text.

Practice

1. What does tokenization do in spaCy?

easy

A. It splits text into smaller pieces called tokens.

B. It trains a machine learning model.

C. It translates text into another language.

D. It visualizes text data.

Solution

1. Step 1: Understand tokenization concept

   Tokenization means breaking text into smaller parts called tokens, like words or punctuation.

2. Step 2: Relate to spaCy functionality

   spaCy uses tokenization to prepare text for analysis by splitting it into tokens.

3. Final Answer:

   It splits text into smaller pieces called tokens. -> Option A

4. Quick Check:

   Tokenization = splitting text [OK]

Hint: Tokenization means breaking text into tokens [OK]

Common Mistakes:

• Confusing tokenization with training models
• Thinking tokenization translates text
• Assuming tokenization visualizes data

2. Which of the following is the correct way to load the English model in spaCy for tokenization?

easy

A. import spacy; nlp = spacy.tokenize('en')

B. import spacy; nlp = spacy.load('en_core_web_sm')

C. import spacy; nlp = spacy.model('english')

D. import spacy; nlp = spacy.load_model('english')

Solution

1. Step 1: Recall spaCy model loading syntax

   spaCy loads models using spacy.load with the model name as a string.

2. Step 2: Identify correct model name and function

   The English small model is 'en_core_web_sm' and loaded by spacy.load('en_core_web_sm').

3. Final Answer:

   import spacy; nlp = spacy.load('en_core_web_sm') -> Option B

4. Quick Check:

   Use spacy.load('model_name') to load models [OK]

Hint: Use spacy.load('model_name') to load models [OK]

Common Mistakes:

• Using spacy.tokenize instead of spacy.load
• Wrong model names like 'english' instead of 'en_core_web_sm'
• Using non-existent functions like load_model

3. What will be the output tokens list from this code snippet?

import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp('Hello, world!')
tokens = [token.text for token in doc]
print(tokens)

medium

A. ['Hello', ',', 'world', '!']

B. ['Hello,', 'world!']

C. ['Hello world']

D. ['Hello', 'world!']

Solution

1. Step 1: Understand spaCy tokenization behavior

   spaCy splits punctuation from words, so commas and exclamation marks become separate tokens.

2. Step 2: Analyze the given text 'Hello, world!'

   Tokens will be 'Hello', ',', 'world', and '!' separately.

3. Final Answer:

   ['Hello', ',', 'world', '!'] -> Option A

4. Quick Check:

   spaCy separates punctuation as tokens [OK]

Hint: Remember spaCy splits punctuation into separate tokens [OK]

Common Mistakes:

• Keeping punctuation attached to words
• Combining words into one token
• Ignoring punctuation tokens

4. Identify the error in this spaCy tokenization code:

import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp('Test sentence.')
for token in doc:
print(token.text)

medium

A. The token.text attribute does not exist.

B. Wrong model name used in spacy.load.

C. Missing indentation for print inside the for loop.

D. The variable 'doc' is not defined.

Solution

1. Step 1: Check Python syntax for loops

   Python requires the code inside a for loop to be indented properly.

2. Step 2: Inspect the given code

   The print statement is not indented under the for loop, causing an IndentationError.

3. Final Answer:

   Missing indentation for print inside the for loop. -> Option C

4. Quick Check:

   Indent loop body code in Python [OK]

Hint: Indent code inside loops to avoid errors [OK]

Common Mistakes:

• Ignoring Python indentation rules
• Assuming model name is wrong
• Thinking token.text is invalid

5. You want to tokenize a sentence but keep contractions like "don't" as one token using spaCy. Which approach is best?

hard

A. Use the default spaCy tokenizer without changes.

B. Split contractions manually after tokenization.

C. Replace contractions with full words before tokenization.

D. Modify the tokenizer exceptions to keep contractions as single tokens.

Solution

1. Step 1: Understand spaCy's default tokenizer behavior

   By default, spaCy splits contractions like "don't" into two tokens: 'do' and "n't".

2. Step 2: Identify how to keep contractions as one token

   Modifying tokenizer exceptions allows spaCy to treat contractions as single tokens.

3. Final Answer:

   Modify the tokenizer exceptions to keep contractions as single tokens. -> Option D

4. Quick Check:

   Customize tokenizer exceptions to control token splits [OK]

Hint: Change tokenizer exceptions to keep contractions whole [OK]

Common Mistakes:

• Using default tokenizer expecting contractions as one token
• Splitting contractions manually after tokenization
• Replacing contractions before tokenization unnecessarily