Tokenization in spaCy in NLP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is tokenization in spaCy?

Tokenization in spaCy is the process of breaking down text into smaller pieces called tokens, such as words, punctuation, or symbols, to help computers understand and analyze the text.

intermediate

How does spaCy handle tokenization differently from simple splitting by spaces?

spaCy uses rules and machine learning to split text into tokens, considering punctuation, contractions, and special cases, rather than just splitting by spaces, which helps keep meaningful parts together.

beginner

What Python code would you use to tokenize the sentence 'Hello, world!' using spaCy?

import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp('Hello, world!')
tokens = [token.text for token in doc]
print(tokens)  # Output: ['Hello', ',', 'world', '!']

beginner

Why is tokenization important before other NLP tasks?

Tokenization breaks text into manageable pieces, making it easier for models to analyze meaning, find patterns, and perform tasks like translation, sentiment analysis, or named entity recognition.

advanced

Can spaCy's tokenizer be customized? If yes, how?

Yes, spaCy's tokenizer can be customized by adding special cases, modifying rules, or changing how it splits tokens to better fit specific text types or languages.

What does spaCy use to split text into tokens?

A Only spaces

B Manual input

C Random splitting

D Rules and machine learning

spaCy uses a combination of rules and machine learning to accurately split text into tokens.

Which of these is NOT a token in the sentence 'Hello, world!' according to spaCy?

A Hello

B Hello world

C ,

D world

'Hello world' is two tokens, not one; spaCy splits them into separate tokens.

Why might you want to customize spaCy's tokenizer?

A To handle special text cases better

B To make it slower

C To remove punctuation always

D To ignore all spaces

Customization helps spaCy handle special cases or unique text formats more accurately.

What Python function is used to load a spaCy language model for tokenization?

A spacy.load()

B spacy.tokenize()

C spacy.split()

D spacy.model()

spacy.load() loads a language model needed for tokenization and other NLP tasks.

Tokenization helps in which of the following NLP tasks?

A Audio processing

B Image recognition

C Sentiment analysis

D Video editing

Tokenization prepares text for tasks like sentiment analysis by breaking it into meaningful parts.

Explain in your own words what tokenization in spaCy is and why it is useful.

Think about how breaking a sentence into words helps a computer understand it.

Describe how you would use spaCy to tokenize a sentence and get a list of tokens.

Remember the basic Python code to load a model and process text.