Tokenization basics in Data Analysis Python - Step-by-Step Execution

Concept Flow - Tokenization basics

Start with text string

↓

Split text into tokens

↓

Clean tokens (optional)

↓

Output list of tokens

Tokenization breaks a text string into smaller pieces called tokens, usually words or symbols, to analyze text step-by-step.

Execution Sample

Data Analysis Python

text = "Hello world!"
tokens = text.split()
print(tokens)

This code splits a simple sentence into words (tokens) using space as separator.

Execution Table

Step	Action	Input	Output
1	Start with text string	"Hello world!"	"Hello world!"
2	Split text by spaces	"Hello world!"	["Hello", "world!"]
3	Print tokens	["Hello", "world!"]	["Hello", "world!"]
4	End	-	-

💡 All words separated by spaces become tokens; process ends after printing tokens.

Variable Tracker

Variable	Start	After split	Final
text	"Hello world!"	"Hello world!"	"Hello world!"
tokens	undefined	["Hello", "world!"]	["Hello", "world!"]

Key Moments - 2 Insights

Why does the token 'world!' include the exclamation mark?

Because split() only breaks text by spaces and does not remove punctuation, so 'world!' stays as one token as shown in execution_table step 2.

What if the text has multiple spaces between words?

split() treats any number of spaces as separators, so tokens will still be words without spaces, as split() removes extra spaces automatically.

Visual Quiz - 3 Questions

Look at the execution table at step 2, what is the output of splitting the text?

A ["Hello world!"]

B ["Hello", "world!"]

C ["Hello", "world"]

D ["Hello!", "world"]

💡 Hint

Check the Output column in execution_table row with Step 2.

According to variable_tracker, what is the value of 'tokens' before splitting?

A "Hello world!"

B []

C undefined

D ["Hello", "world!"]

💡 Hint

Look at the 'tokens' row under 'Start' column in variable_tracker.

If the text was "Hi there", how would split() treat the spaces?

A It would create tokens ["Hi", "there"]

B It would create tokens ["Hi", "", "there"]

C It would create one token ["Hi there"]

D It would remove all spaces and create ["Hithere"]

💡 Hint

Recall that split() removes extra spaces and splits on any whitespace.

Concept Snapshot

Tokenization basics:
- Tokenization splits text into tokens (words/symbols).
- Common method: split() breaks text by spaces.
- Tokens may include punctuation unless cleaned.
- Useful for text analysis and processing.

Full Transcript

Tokenization is the process of breaking a text string into smaller parts called tokens, usually words. We start with a text string, then split it by spaces to get tokens. For example, splitting "Hello world!" by spaces gives two tokens: "Hello" and "world!". Note that punctuation like the exclamation mark stays attached to the word because split() only separates by spaces. Variables like 'text' hold the original string, and 'tokens' hold the list of words after splitting. This process is important for analyzing text step-by-step in data science.