str.replace() for substitution in Pandas

Introduction

We use str.replace() to change parts of text in data. It helps fix or update words or characters easily.

You want to fix typos in a list of names.

You need to change date formats in a text column.

You want to remove unwanted characters like extra spaces or symbols.

You want to replace abbreviations with full words in a dataset.

You want to clean data before analysis by standardizing text.

Syntax

Pandas

Series.str.replace(pat, repl, n=-1, case=None, regex=True)

pat is the pattern or text you want to replace.

repl is the new text you want to put instead.

Examples

Replace all 'John' with 'Jonathan' in the 'Name' column.

Pandas

df['Name'].str.replace('John', 'Jonathan')

Change all slashes '/' to dashes '-' in the 'Date' column.

Pandas

df['Date'].str.replace('/', '-')

Remove all spaces from the 'Code' column.

Pandas

df['Code'].str.replace(' ', '')

Replace only the first 'a' with 'A' in each string of the 'Text' column.

Pandas

df['Text'].str.replace('a', 'A', n=1)

Sample Program

This program creates a table with names, dates, and codes. It then fixes the names by changing 'John' to 'Jonathan', changes date slashes to dashes, and removes spaces from codes.

Pandas

import pandas as pd

# Create a sample DataFrame
sample_data = {'Name': ['John Doe', 'Jane Smith', 'Johnny Appleseed', 'Johnathan Davis'],
               'Date': ['2023/01/01', '2023/02/15', '2023/03/20', '2023/04/25'],
               'Code': ['A 123', 'B 456', 'C 789', 'D 012']}
df = pd.DataFrame(sample_data)

print('Original DataFrame:')
print(df)

# Replace 'John' with 'Jonathan' in 'Name'
df['Name'] = df['Name'].str.replace('John', 'Jonathan')

# Replace '/' with '-' in 'Date'
df['Date'] = df['Date'].str.replace('/', '-')

# Remove spaces in 'Code'
df['Code'] = df['Code'].str.replace(' ', '')

print('\nDataFrame after str.replace() operations:')
print(df)

Important Notes

Time complexity: Usually O(n) where n is the number of strings, because it checks each string once.

Space complexity: O(n) for the new strings created after replacement.

Common mistake: Forgetting that str.replace() returns a new Series and does not change the original unless assigned back.

Use str.replace() when you want to change text inside strings. For removing rows or columns, use other methods.

Summary

str.replace() helps change parts of text in pandas columns easily.

It works on each string in a column and returns a new column with changes.

Remember to assign the result back to keep the changes.

Practice

1. What does the pandas str.replace() method do when applied to a DataFrame column?

easy

A. It sorts the column values in ascending order.

B. It deletes the entire column from the DataFrame.

C. It converts all strings in the column to uppercase.

D. It replaces specified substrings in each string of the column with a new substring.

Solution

1. Step 1: Understand the purpose of str.replace()

   This method targets each string in a pandas column and looks for a substring to replace.

2. Step 2: Identify the effect on the column

   It creates a new column or series where the specified substring is replaced by the new substring in every string.

3. Final Answer:

   It replaces specified substrings in each string of the column with a new substring. -> Option D

4. Quick Check:

   str.replace() = substring substitution [OK]

Hint: Think: change parts of text in each cell [OK]

Common Mistakes:

• Confusing replace with deleting columns
• Thinking it changes case of strings
• Assuming it sorts data

2. Which of the following is the correct syntax to replace all occurrences of 'cat' with 'dog' in a pandas column named pets?

easy

A. df['pets'].str.replace('cat', 'dog')

B. df['pets'].replace('cat', 'dog')

C. df.str.replace('cat', 'dog')['pets']

D. df['pets'].replace_all('cat', 'dog')

Solution

1. Step 1: Identify the correct method for string replacement

   In pandas, str.replace() is used on a Series of strings to replace substrings.

2. Step 2: Check the syntax for applying str.replace() on a column

   The correct syntax is df['column'].str.replace(old, new). df['pets'].str.replace('cat', 'dog') matches this.

3. Final Answer:

   df['pets'].str.replace('cat', 'dog') -> Option A

4. Quick Check:

   Use .str.replace() on Series for string substitution [OK]

Hint: Use .str.replace() on the column, not .replace() alone [OK]

Common Mistakes:

• Using .replace() directly on the DataFrame column
• Trying to call .str.replace() on the whole DataFrame
• Using non-existent methods like replace_all

3. Given the DataFrame:

import pandas as pd
df = pd.DataFrame({'fruit': ['apple', 'banana', 'grape', 'pineapple']})
df['fruit_new'] = df['fruit'].str.replace('apple', 'orange')

What is the content of df['fruit_new']?

medium

A. ['orange', 'banana', 'grape', 'pineapple']

B. ['apple', 'banana', 'grape', 'pineapple']

C. ['orange', 'banana', 'grape', 'pineorange']

D. ['orange', 'orange', 'orange', 'orange']

Solution

1. Step 1: Understand what str.replace('apple', 'orange') does

   It replaces every occurrence of 'apple' in each string with 'orange'.

2. Step 2: Apply replacement to each value

   'apple' becomes 'orange', 'banana' unchanged, 'grape' unchanged, 'pineapple' contains 'apple' at the end, so it becomes 'pineorange'.

3. Final Answer:

   ['orange', 'banana', 'grape', 'pineorange'] -> Option C

4. Quick Check:

   Substring 'apple' replaced wherever found [OK]

Hint: Replace changes substrings inside words too [OK]

Common Mistakes:

• Assuming only exact matches replaced, not substrings
• Thinking 'pineapple' stays unchanged
• Confusing replacement with full string swap

4. What is wrong with this code snippet?

df['col'].str.replace('a', 'b')

It does not change df['col'] as expected.

medium

A. The result is not assigned back to df['col'] to save changes.

B. The method str.replace() does not exist in pandas.

C. The arguments 'a' and 'b' must be integers, not strings.

D. You must use df.replace() instead of str.replace().

Solution

1. Step 1: Understand that str.replace() returns a new Series

   The method does not change the original column in place; it returns a modified copy.

2. Step 2: Identify the missing assignment

   To keep changes, assign the result back: df['col'] = df['col'].str.replace('a', 'b').

3. Final Answer:

   The result is not assigned back to df['col'] to save changes. -> Option A

4. Quick Check:

   Assign result to keep changes [OK]

Hint: Always assign back to save string replacements [OK]

Common Mistakes:

• Expecting in-place change without assignment
• Confusing str.replace() with DataFrame.replace()
• Thinking str.replace() modifies original data silently

5. You have a DataFrame with a column emails containing email addresses. You want to anonymize the domain part by replacing everything after '@' with 'example.com'. Which code correctly does this?

hard

A. df['emails'].replace(r'@.*', '@example.com')

B. df['emails'].str.replace(r'@.*', '@example.com', regex=True)

C. df['emails'].str.replace('example.com', '@')

D. df['emails'].str.replace('@', '@example.com')

Solution

1. Step 1: Identify the pattern to replace

   The domain part starts with '@' and continues to the end, so the regex '@.*' matches it.

2. Step 2: Use str.replace() with regex=True to replace domain

   df['emails'].str.replace(r'@.*', '@example.com', regex=True) uses regex to replace everything after '@' with '@example.com'.

3. Step 3: Check other options

   A uses Series.replace() which defaults to regex=False, treating '@.*' literally. B replaces only '@' leaving the domain. C replaces in the wrong direction.

4. Final Answer:

   df['emails'].str.replace(r'@.*', '@example.com', regex=True) -> Option B

5. Quick Check:

   Use regex in str.replace() for domain substitution [OK]

Hint: Use regex '@.*' with regex=True to replace domain [OK]

Common Mistakes:

• Not using regex=True for pattern matching
• Replacing only '@' without domain
• Using DataFrame.replace instead of str.replace