Tokenization (word and sentence) in NLP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is tokenization in Natural Language Processing?

Tokenization is the process of breaking down text into smaller pieces called tokens, which can be words, sentences, or subwords. It helps computers understand and analyze text.

beginner

What is the difference between word tokenization and sentence tokenization?

Word tokenization splits text into individual words, while sentence tokenization splits text into sentences. Both help organize text for easier processing.

intermediate

Why is tokenization important before training an NLP model?

Tokenization converts raw text into manageable pieces so models can learn patterns. Without tokenization, models can't understand the structure of language.

beginner

Example: Tokenize the sentence 'Hello world! How are you?' into words.

The word tokens are: ['Hello', 'world', '!', 'How', 'are', 'you', '?']

intermediate

What challenges can arise during tokenization?

Challenges include handling punctuation, contractions (like "don't"), abbreviations, and languages without spaces. Good tokenization handles these well.

What does sentence tokenization do?

A Removes stopwords

B Splits text into words

C Splits text into sentences

D Converts text to lowercase

Sentence tokenization breaks text into sentences, helping analyze text at the sentence level.

Which of these is a word token from the sentence 'I can't go'?

A can't

B cant

C ca n't

D can

The word token 'can't' keeps the contraction intact in many tokenization methods.

Why do we tokenize text before feeding it to an NLP model?

A To convert text into smaller, understandable pieces

B To translate text into another language

C To remove all punctuation

D To increase text length

Tokenization breaks text into tokens so models can process and learn from the text.

Which punctuation is usually treated as a separate token in word tokenization?

A Space

B Comma

C Letter

D Number

Punctuation like commas, periods, and question marks are often separate tokens.

Which is NOT a common challenge in tokenization?

A Handling contractions

B Handling languages without spaces

C Handling abbreviations

D Handling spaces in English

English uses spaces to separate words, so handling spaces is straightforward compared to other challenges.

Explain what tokenization is and why it is important in NLP.

Think about how computers read text and why smaller pieces help.

Describe the difference between word tokenization and sentence tokenization with examples.

Consider how you would split 'Hello world! How are you?'