Extracting with str.extract (regex) in Data Analysis Python

Introduction

We use str.extract to pull out specific parts of text from data using patterns. It helps us find and save useful pieces from messy text.

You want to get phone numbers from a list of customer messages.

You need to find dates hidden inside product reviews.

You want to separate area codes from full phone numbers in a contact list.

You need to extract email usernames from email addresses.

You want to pull out hashtags from social media posts.

Syntax

Data Analysis Python

Series.str.extract(pat, flags=0, expand=True)

pat is the pattern you want to find, written as a regular expression (regex).

If expand=True, the result is a DataFrame; if False, it returns a Series.

Examples

Extracts the first 3 digits (like area code) from phone numbers.

Data Analysis Python

df['phone'].str.extract(r'(\d{3})')

Extracts dates in YYYY-MM-DD format from text.

Data Analysis Python

df['text'].str.extract(r'(\d{4}-\d{2}-\d{2})')

Extracts the username part before '@' in email addresses.

Data Analysis Python

df['email'].str.extract(r'([^@]+)@')

Sample Program

This code creates a small table with messages. It then extracts area codes, dates, and email usernames using str.extract with regex patterns.

Data Analysis Python

import pandas as pd

data = {'messages': ['Call me at 415-555-1234', 'My birthday is 1990-05-21', 'Email: [e-mail]']}
df = pd.DataFrame(data)

# Extract area code (3 digits) from phone numbers
area_codes = df['messages'].str.extract(r'(\d{3})')

# Extract date in YYYY-MM-DD format
dates = df['messages'].str.extract(r'(\d{4}-\d{2}-\d{2})')

# Extract username from email
usernames = df['messages'].str.extract(r'([\w\.]+)@')

print('Area Codes:')
print(area_codes)
print('\nDates:')
print(dates)
print('\nUsernames:')
print(usernames)

Important Notes

Regex patterns use special symbols to match text. For example, \d means any digit.

If no match is found, str.extract returns NaN for that row.

Use parentheses () in regex to capture the part you want to extract.

Summary

str.extract helps pull out parts of text using patterns.

It returns a new table with the extracted pieces or NaN if nothing matches.

Useful for cleaning and organizing messy text data.

Practice

1. What does the str.extract method do in pandas when used with a regex pattern?

easy

A. It replaces matching text with a new string.

B. It deletes rows that don't match the pattern.

C. It pulls out matching parts of text into a new DataFrame.

D. It counts how many times the pattern appears.

Solution

1. Step 1: Understand str.extract purpose

   The method is designed to find parts of text that match a regex pattern and extract them.

2. Step 2: Check what it returns

   It returns a new DataFrame with the extracted groups or NaN if no match is found.

3. Final Answer:

   It pulls out matching parts of text into a new DataFrame. -> Option C

4. Quick Check:

   str.extract extracts text parts [OK]

Hint: Remember: extract pulls out text pieces using patterns [OK]

Common Mistakes:

• Thinking it deletes rows instead of extracting text
• Confusing extract with replace or count methods
• Expecting a single string instead of a DataFrame

2. Which of the following is the correct syntax to extract a 3-digit number from a pandas Series df['code'] using str.extract?

easy

A. df['code'].str.extract('\d{3}')

B. df['code'].str.extract(r'(\d{3})')

C. df['code'].str.extract(r'\d{3}')

D. df['code'].str.extract('(\d{3})')

Solution

1. Step 1: Identify correct regex string format

   Regex pattern must be a raw string (r'...') to handle backslashes properly.

2. Step 2: Check parentheses for capturing group

   Parentheses are needed to capture the group to extract.

3. Final Answer:

   df['code'].str.extract(r'(\d{3})') -> Option B

4. Quick Check:

   Use raw string with parentheses for capture [OK]

Hint: Use raw string and parentheses for capture groups [OK]

Common Mistakes:

• Missing parentheses around the pattern
• Not using raw string causing escape errors
• Using double backslashes incorrectly

3. Given the following code:

import pandas as pd
df = pd.DataFrame({'text': ['ID123', 'ID456', 'NoID', 'ID789']})
df['number'] = df['text'].str.extract(r'ID(\d{3})')

What will be the content of the new column number?

medium

A. [NaN, NaN, NaN, NaN]

B. ['ID123', 'ID456', 'NoID', 'ID789']

C. ['123', '456', 'NoID', '789']

D. ['123', '456', NaN, '789']

Solution

1. Step 1: Understand the regex pattern

   The pattern 'ID(\d{3})' looks for 'ID' followed by exactly 3 digits and extracts the digits.

2. Step 2: Apply extraction to each row

   'ID123' -> '123', 'ID456' -> '456', 'NoID' -> no match so NaN, 'ID789' -> '789'.

3. Final Answer:

   ['123', '456', NaN, '789'] -> Option D

4. Quick Check:

   Extract digits after 'ID' or NaN if no match [OK]

Hint: Extracted groups appear; no match gives NaN [OK]

Common Mistakes:

• Expecting full string instead of just digits
• Not realizing unmatched rows become NaN
• Confusing extracted text with original

4. Consider this code snippet:

import pandas as pd
df = pd.DataFrame({'info': ['abc123', 'def456', 'ghi789']})
df['extracted'] = df['info'].str.extract('\d{3}')

What is the issue with this code?

medium

A. The regex pattern lacks parentheses for capturing groups.

B. The regex pattern is invalid syntax.

C. The DataFrame column name is incorrect.

D. str.extract cannot extract digits.

Solution

1. Step 1: Check regex pattern for capture groups

   str.extract requires parentheses to capture groups; '\d{3}' has none.

2. Step 2: Understand effect of missing parentheses

   Without parentheses, str.extract returns NaN for all rows because no group is captured.

3. Final Answer:

   The regex pattern lacks parentheses for capturing groups. -> Option A

4. Quick Check:

   Parentheses needed to capture groups [OK]

Hint: Always use parentheses to capture groups in str.extract [OK]

Common Mistakes:

• Forgetting parentheses around regex group
• Assuming str.extract works without capture groups
• Confusing str.extract with str.contains

5. You have a DataFrame df with a column 'data' containing strings like 'Order: #12345, Date: 2023-05-10'. You want to extract the order number and date into separate columns using str.extract. Which regex pattern correctly extracts both parts?

hard

A. r'Order: #(\d+), Date: (\d{4}-\d{2}-\d{2})'

B. r'Order: #(\d+), Date: \d{4}-\d{2}-\d{2}'

C. r'Order: #\d+, Date: (\d{4}-\d{2}-\d{2})'

D. r'Order: #(\d+), Date: (\d{4}/\d{2}/\d{2})'

Solution

1. Step 1: Identify needed capture groups

   We need to capture order number digits and date in 'YYYY-MM-DD' format, so two groups are required.

2. Step 2: Check regex pattern correctness

   r'Order: #(\d+), Date: (\d{4}-\d{2}-\d{2})' has two groups: '(\d+)' for order number and '(\d{4}-\d{2}-\d{2})' for date, matching the format exactly.

3. Final Answer:

   r'Order: #(\d+), Date: (\d{4}-\d{2}-\d{2})' -> Option A

4. Quick Check:

   Two capture groups matching order and date format [OK]

Hint: Use parentheses for each part you want to extract [OK]

Common Mistakes:

• Missing parentheses around one or both groups
• Wrong date format with slashes instead of dashes
• Not capturing order number digits