Tokenization basics in Data Analysis Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of simple whitespace tokenization

What is the output of this Python code that splits a sentence into tokens using whitespace?

Data Analysis Python

sentence = "Data science is fun and exciting"
tokens = sentence.split()
print(tokens)

A ['Data', 'science', 'is', 'fun', 'and', 'exciting']

B ['Data,', 'science,', 'is,', 'fun,', 'and,', 'exciting']

C ['Data science is fun and exciting']

D ['Data', 'science', 'is', 'fun', 'and']

💡 Hint

Remember that split() without arguments splits on spaces.

The split() method without arguments splits the string at each whitespace, producing a list of words.

❓ data_output

intermediate

Tokenizing with punctuation removal

Given the code below, what is the resulting list of tokens after removing punctuation and splitting?

Data Analysis Python

import string
text = "Hello, world! Let's learn tokenization."
tokens = text.translate(str.maketrans('', '', string.punctuation)).split()
print(tokens)

A ['Hello', 'world', 'Lets', 'learn', 'tokenization']

B ['Hello,', 'world!', "Let's", 'learn', 'tokenization.']

C ['Hello', 'world!', "Let's", 'learn', 'tokenization']

D ['Hello', 'world', "Let's", 'learn', 'tokenization']

💡 Hint

Removing punctuation means no commas or apostrophes remain.

The translate method removes all punctuation characters, so words like "Let's" become "Lets" before splitting.

❓ visualization

advanced

Visualizing token frequency distribution

Which option shows the correct bar chart of token frequencies from the given text?

Data Analysis Python

from collections import Counter
import matplotlib.pyplot as plt
text = "apple banana apple orange banana apple"
tokens = text.split()
counter = Counter(tokens)
plt.bar(counter.keys(), counter.values())
plt.show()

A Bar chart with all tokens equal height

B Bar chart with 'banana' highest, then 'apple', then 'orange'

C Bar chart with 'orange' highest, then 'banana', then 'apple'

D Bar chart with 'apple' highest, then 'banana', then 'orange'

💡 Hint

Count how many times each word appears in the text.

The word 'apple' appears 3 times, 'banana' 2 times, and 'orange' once, so the bar heights reflect this order.

🧠 Conceptual

advanced

Understanding tokenization challenges

Which option best describes a common challenge in tokenization for natural language processing?

A Removing all stopwords before tokenization

B Handling contractions and punctuation correctly to avoid losing meaning

C Converting all tokens to uppercase before analysis

D Sorting tokens alphabetically after splitting

💡 Hint

Think about how words like "don't" or "it's" are split.

Contractions and punctuation can change word meaning if not handled properly during tokenization.

🔧 Debug

expert

Identify the error in tokenization code

What error does this code produce when trying to tokenize text by splitting on spaces?

Data Analysis Python

text = None
tokens = text.split()
print(tokens)

A SyntaxError: invalid syntax

B TypeError: split() missing 1 required positional argument

C AttributeError: 'NoneType' object has no attribute 'split'

D ValueError: empty string cannot be split

💡 Hint

Check the type of the variable before calling split.

Since text is None, it has no split method, causing an AttributeError.