Tokenization basics in Data Analysis Python - Mini Project: Build & Apply

Tokenization basics

📖 Scenario: Imagine you work for a company that analyzes customer reviews. You want to break down each review into individual words to understand what customers are saying.

🎯 Goal: Build a simple program that splits a sentence into words using tokenization.

📋 What You'll Learn

Create a variable with a sentence string

Create a variable for the separator (space)

Use the split() method with the separator to tokenize the sentence

Print the list of tokens

💡 Why This Matters

🌍 Real World

Tokenization is the first step in analyzing text data like customer reviews, social media posts, or emails to understand what people are saying.

💼 Career

Data scientists and analysts use tokenization to prepare text data for tasks like sentiment analysis, topic modeling, and machine learning.

1

Create the sentence variable

Create a variable called sentence and set it to the string 'I love learning data science'.

Data Analysis Python

# Create the sentence variable
# Your code here

Hint

Use single or double quotes to create the string.

2

Create the separator variable

Create a variable called separator and set it to a single space string ' '.

Data Analysis Python

sentence = 'I love learning data science'
# Create the separator variable
# Your code here

Hint

The separator is a space character inside quotes.

3

Tokenize the sentence

Create a variable called tokens and set it to the result of calling sentence.split(separator).

Data Analysis Python

sentence = 'I love learning data science'
separator = ' '
# Tokenize the sentence using split
# Your code here

Hint

Use the split() method on the sentence with the separator.

4

Print the tokens

Write a print() statement to display the tokens list.

Data Analysis Python

sentence = 'I love learning data science'
separator = ' '
tokens = sentence.split(separator)
# Print the tokens list
# Your code here

Hint

Use print(tokens) to show the list of words.