Tokenization basics in Data Analysis Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of simple whitespace tokenization

What is the output of this Python code that splits a sentence into tokens using whitespace?

Data Analysis Python

sentence = "Data science is fun and exciting"
tokens = sentence.split()
print(tokens)

A ['Data', 'science', 'is', 'fun', 'and', 'exciting']

B ['Data,', 'science,', 'is,', 'fun,', 'and,', 'exciting']

C ['Data science is fun and exciting']

D ['Data', 'science', 'is', 'fun', 'and']

💡 Hint

Remember that split() without arguments splits on spaces.

The split() method without arguments splits the string at each whitespace, producing a list of words.

❓ data_output

intermediate

Tokenizing with punctuation removal

Given the code below, what is the resulting list of tokens after removing punctuation and splitting?

Data Analysis Python

import string
text = "Hello, world! Let's learn tokenization."
tokens = text.translate(str.maketrans('', '', string.punctuation)).split()
print(tokens)

A ['Hello', 'world', 'Lets', 'learn', 'tokenization']

B ['Hello,', 'world!', "Let's", 'learn', 'tokenization.']

C ['Hello', 'world!', "Let's", 'learn', 'tokenization']

D ['Hello', 'world', "Let's", 'learn', 'tokenization']

💡 Hint

Removing punctuation means no commas or apostrophes remain.

The translate method removes all punctuation characters, so words like "Let's" become "Lets" before splitting.

❓ visualization

advanced

Visualizing token frequency distribution

Which option shows the correct bar chart of token frequencies from the given text?

Data Analysis Python

from collections import Counter
import matplotlib.pyplot as plt
text = "apple banana apple orange banana apple"
tokens = text.split()
counter = Counter(tokens)
plt.bar(counter.keys(), counter.values())
plt.show()

A Bar chart with all tokens equal height

B Bar chart with 'banana' highest, then 'apple', then 'orange'

C Bar chart with 'orange' highest, then 'banana', then 'apple'

D Bar chart with 'apple' highest, then 'banana', then 'orange'

💡 Hint

Count how many times each word appears in the text.

The word 'apple' appears 3 times, 'banana' 2 times, and 'orange' once, so the bar heights reflect this order.

🧠 Conceptual

advanced

Understanding tokenization challenges

Which option best describes a common challenge in tokenization for natural language processing?

A Removing all stopwords before tokenization

B Handling contractions and punctuation correctly to avoid losing meaning

C Converting all tokens to uppercase before analysis

D Sorting tokens alphabetically after splitting

💡 Hint

Think about how words like "don't" or "it's" are split.

Contractions and punctuation can change word meaning if not handled properly during tokenization.

🔧 Debug

expert

Identify the error in tokenization code

What error does this code produce when trying to tokenize text by splitting on spaces?

Data Analysis Python

text = None
tokens = text.split()
print(tokens)

A SyntaxError: invalid syntax

B TypeError: split() missing 1 required positional argument

C AttributeError: 'NoneType' object has no attribute 'split'

D ValueError: empty string cannot be split

💡 Hint

Check the type of the variable before calling split.

Since text is None, it has no split method, causing an AttributeError.

Practice

1. What is the main purpose of tokenization in text analysis?

easy

A. To split text into smaller parts called tokens

B. To translate text into another language

C. To remove all punctuation from text

D. To count the number of characters in text

Solution

1. Step 1: Understand tokenization concept

   Tokenization means breaking text into smaller pieces called tokens, like words or sentences.

2. Step 2: Compare options with definition

   Only To split text into smaller parts called tokens correctly describes tokenization as splitting text into tokens.

3. Final Answer:

   To split text into smaller parts called tokens -> Option A

4. Quick Check:

   Tokenization = splitting text [OK]

Hint: Tokenization means splitting text into pieces [OK]

Common Mistakes:

• Confusing tokenization with translation
• Thinking tokenization removes punctuation only
• Mixing tokenization with counting characters

2. Which of the following is the correct way to import the word_tokenize function from nltk in Python?

easy

A. from nltk import word_tokenize.tokenize

B. import nltk.word_tokenize

C. from nltk.tokenize import word_tokenize

D. import word_tokenize from nltk.tokenize

Solution

1. Step 1: Recall correct Python import syntax

   The correct syntax to import a function from a module is 'from module import function'.

2. Step 2: Match syntax with nltk tokenization import

   from nltk.tokenize import word_tokenize matches this syntax exactly for word_tokenize from nltk.tokenize.

3. Final Answer:

   from nltk.tokenize import word_tokenize -> Option C

4. Quick Check:

   Correct import syntax = from nltk.tokenize import word_tokenize [OK]

Hint: Use 'from module import function' to import specific functions [OK]

Common Mistakes:

• Using dot notation incorrectly in import
• Trying to import with 'import function from module'
• Confusing module and function names

3. What will be the output of the following code?

from nltk.tokenize import word_tokenize
text = "Hello, world! Let's tokenize this sentence."
tokens = word_tokenize(text)
print(tokens)

medium

A. ['Hello', ',', 'world', '!', 'Let', "'s", 'tokenize', 'this', 'sentence', '.']

B. ['Hello world Lets tokenize this sentence']

C. ['Hello', 'world', 'Lets', 'tokenize', 'this', 'sentence']

D. ['Hello,', 'world!', "Let's", 'tokenize', 'this', 'sentence.']

Solution

1. Step 1: Understand nltk.word_tokenize behavior

   word_tokenize splits text into words and punctuation as separate tokens, including contractions.

2. Step 2: Analyze the given text tokenization

   It splits "Hello," into 'Hello' and ',', "world!" into 'world' and '!', and "Let's" into 'Let' and "'s" separately.

3. Final Answer:

   ['Hello', ',', 'world', '!', 'Let', "'s", 'tokenize', 'this', 'sentence', '.'] -> Option A

4. Quick Check:

   word_tokenize splits punctuation and contractions separately [OK]

Hint: word_tokenize splits punctuation and contractions separately [OK]

Common Mistakes:

• Assuming punctuation stays attached to words
• Ignoring splitting of contractions
• Expecting a single string token

4. Identify the error in this tokenization code snippet:

import nltk
text = "Data Science is fun!"
tokens = nltk.word_tokenize(text)
print(tokens)

medium

A. word_tokenize is not a function in nltk

B. Missing import of word_tokenize from nltk.tokenize

C. text variable is not defined

D. print statement syntax error

Solution

1. Step 1: Check how word_tokenize is accessed

   word_tokenize is inside nltk.tokenize module, not directly under nltk.

2. Step 2: Identify import mistake

   Importing only nltk does not allow calling nltk.word_tokenize; must import from nltk.tokenize.

3. Final Answer:

   Missing import of word_tokenize from nltk.tokenize -> Option B

4. Quick Check:

   word_tokenize needs correct import [OK]

Hint: Import word_tokenize from nltk.tokenize, not just nltk [OK]

Common Mistakes:

• Calling nltk.word_tokenize without proper import
• Assuming all functions are directly under nltk
• Ignoring module structure

5. You have a list of sentences and want to tokenize each sentence into words, then count total tokens. Which code correctly does this using nltk's word_tokenize?

hard

A. from nltk.tokenize import word_tokenize
sentences = ["I love data.", "Tokenization is key."]
tokens = word_tokenize(sentences)
print(len(tokens))

B. import nltk
sentences = ["I love data.", "Tokenization is key."]
tokens = nltk.word_tokenize(sentences)
print(len(tokens))

C. from nltk.tokenize import word_tokenize
sentences = "I love data. Tokenization is key."
tokens = word_tokenize(sentences)
print(tokens)

D. from nltk.tokenize import word_tokenize
sentences = ["I love data.", "Tokenization is key."]
tokens = [word_tokenize(s) for s in sentences]
total = sum(len(t) for t in tokens)
print(total)

Solution

1. Step 1: Understand input and function usage

   word_tokenize expects a string, so to tokenize multiple sentences, tokenize each string separately.

2. Step 2: Analyze each option's approach

   from nltk.tokenize import word_tokenize sentences = ["I love data.", "Tokenization is key."] tokens = [word_tokenize(s) for s in sentences] total = sum(len(t) for t in tokens) print(total) tokenizes each sentence in a list comprehension, then sums lengths correctly. Some options wrongly pass list to word_tokenize. from nltk.tokenize import word_tokenize sentences = "I love data. Tokenization is key." tokens = word_tokenize(sentences) print(tokens) tokenizes a single string, not list.

3. Final Answer:

   The correct code tokenizes each sentence and counts total tokens -> Option D

4. Quick Check:

   Tokenize each sentence separately, then count tokens [OK]

Hint: Tokenize each sentence string separately, then count tokens [OK]

Common Mistakes:

• Passing list directly to word_tokenize
• Not summing token counts correctly
• Tokenizing entire text as one string when list given