Tokenization (word and sentence) in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this word tokenization code?

Given the following Python code using NLTK for word tokenization, what is the output list?

NLP

from nltk.tokenize import word_tokenize
text = "Hello world! Let's test tokenization."
tokens = word_tokenize(text)
print(tokens)

A ['Hello', 'world', '!', 'Let's', 'test', 'tokenization', '.']

B ['Hello', 'world!', "Let's", 'test', 'tokenization.']

C ['Hello', 'world', '!', 'Let', "'s", 'test', 'tokenization', '.']

D ['Hello world!', "Let's test tokenization."]

💡 Hint

Remember that word_tokenize splits punctuation as separate tokens.

NLTK's word_tokenize splits words and punctuation separately. So "Let's" becomes ['Let', "'s"], and punctuation marks like '!' and '.' are separate tokens.

🧠 Conceptual

intermediate

Which option correctly describes sentence tokenization?

What does sentence tokenization do in Natural Language Processing?

A Converts words into numerical vectors for machine learning.

B Splits text into individual words, separating punctuation.

C Removes stopwords from the text.

D Splits text into sentences based on punctuation and capitalization.

💡 Hint

Think about how text is divided into meaningful chunks bigger than words.

Sentence tokenization breaks text into sentences, usually by detecting punctuation marks like periods and capitalization patterns.

❓ Metrics

advanced

How many tokens are produced by this sentence tokenizer?

Using NLTK's sent_tokenize on the text below, how many sentences are produced? "Dr. Smith loves AI. He works at OpenAI! Do you know him?"

NLP

from nltk.tokenize import sent_tokenize
text = "Dr. Smith loves AI. He works at OpenAI! Do you know him?"
sentences = sent_tokenize(text)
print(len(sentences))

A 3

B 1

C 4

D 2

💡 Hint

Consider how abbreviations like 'Dr.' affect sentence splitting.

NLTK's sent_tokenize correctly handles abbreviations and splits into 3 sentences: 'Dr. Smith loves AI.', 'He works at OpenAI!', and 'Do you know him?'.

🔧 Debug

advanced

What error does this tokenization code raise?

Consider this code snippet: from nltk.tokenize import word_tokenize text = None tokens = word_tokenize(text) print(tokens) What error will this code raise?

A TypeError: expected string or bytes-like object

B NameError: name 'word_tokenize' is not defined

C AttributeError: 'NoneType' object has no attribute 'split'

D ValueError: empty string passed to tokenizer

💡 Hint

Check what type word_tokenize expects as input.

word_tokenize expects a string input. Passing None causes a TypeError because it cannot process None as text.

❓ Model Choice

expert

Which tokenizer is best for splitting text into subword units for transformer models?

You want to prepare text input for a transformer-based language model that uses subword tokenization. Which tokenizer type should you choose?

A Character tokenizer that splits text into individual characters

B Byte-Pair Encoding (BPE) tokenizer that splits words into subword units

C Whitespace tokenizer that splits text only on spaces

D Sentence tokenizer that splits text into sentences

💡 Hint

Transformer models often use subword units to handle unknown words efficiently.

Byte-Pair Encoding (BPE) tokenizers split words into subword units, which helps transformer models handle rare or unknown words better than word or character tokenizers.