Extracting with str.extract (regex) in Data Analysis Python - Step-by-Step Execution

Concept Flow - Extracting with str.extract (regex)

Start with a pandas Series

↓

Apply str.extract with regex

↓

Regex matches groups in each string

↓

Extract matched groups into new DataFrame columns

↓

Result: DataFrame with extracted parts

↓

End

We start with a pandas Series of strings, apply str.extract with a regex pattern that captures groups, and get a DataFrame with those extracted groups as columns.

Execution Sample

Data Analysis Python

import pandas as pd
s = pd.Series(['abc123', 'def456', 'ghi789'])
result = s.str.extract(r'([a-z]+)(\d+)')
print(result)

This code extracts letters and digits from each string in the Series into separate columns.

Execution Table

Step	Input String	Regex Pattern	Match Groups	Extracted Output
1	'abc123'	r'([a-z]+)(\d+)'	Group 1: 'abc', Group 2: '123'	['abc', '123']
2	'def456'	r'([a-z]+)(\d+)'	Group 1: 'def', Group 2: '456'	['def', '456']
3	'ghi789'	r'([a-z]+)(\d+)'	Group 1: 'ghi', Group 2: '789'	['ghi', '789']
4	End of Series	N/A	N/A	DataFrame with columns 0 and 1 containing extracted groups

💡 All strings processed; extraction complete with groups captured into DataFrame columns.

Variable Tracker

Variable	Start	After 1	After 2	After 3	Final
s	Series(['abc123', 'def456', 'ghi789'])	Same	Same	Same	Same
result	None	DataFrame with first row ['abc', '123']	DataFrame with first two rows ['abc', '123'], ['def', '456']	DataFrame with first three rows ['abc', '123'], ['def', '456'], ['ghi', '789']	DataFrame with all extracted groups

Key Moments - 2 Insights

Why does str.extract return a DataFrame instead of a Series?

Because the regex has multiple capturing groups, str.extract returns each group as a separate column in a DataFrame, as shown in execution_table rows 1-3.

What happens if a string does not match the regex pattern?

If no match is found, str.extract returns NaN for that row's columns. This is not shown here but would appear as missing values in the output DataFrame.

Visual Quiz - 3 Questions

Look at the execution_table at Step 2. What are the extracted groups for the input 'def456'?

A Group 1: 'def', Group 2: '456'

B Group 1: 'de', Group 2: 'f456'

C Group 1: 'def4', Group 2: '56'

D No match found

💡 Hint

Refer to execution_table row with Step 2 showing the matched groups.

At which step does the extraction process finish for all strings?

A Step 3

B Step 4

C Step 1

D Step 5

💡 Hint

Check the exit_note and execution_table last row describing completion.

If the regex pattern had only one group, what would be the type of the output?

A A DataFrame with one column

B A list of strings

C A Series with extracted strings

D An error

💡 Hint

Recall that str.extract returns a Series if there is only one capturing group.

Concept Snapshot

pandas.Series.str.extract(regex)
- Uses regex with capturing groups
- Extracts matched groups into DataFrame columns
- Returns DataFrame if multiple groups, Series if one group
- Missing matches become NaN
- Useful to split strings by patterns

Full Transcript

We start with a pandas Series of strings. We apply the str.extract method with a regex pattern that has capturing groups. For each string, the regex matches parts and extracts groups. These groups become columns in a new DataFrame. If a string does not match, the output is NaN for that row. The process repeats for all strings in the Series. The final result is a DataFrame showing extracted parts side by side.

Practice

1. What does the str.extract method do in pandas when used with a regex pattern?

easy

A. It replaces matching text with a new string.

B. It deletes rows that don't match the pattern.

C. It pulls out matching parts of text into a new DataFrame.

D. It counts how many times the pattern appears.

Solution

1. Step 1: Understand str.extract purpose

   The method is designed to find parts of text that match a regex pattern and extract them.

2. Step 2: Check what it returns

   It returns a new DataFrame with the extracted groups or NaN if no match is found.

3. Final Answer:

   It pulls out matching parts of text into a new DataFrame. -> Option C

4. Quick Check:

   str.extract extracts text parts [OK]

Hint: Remember: extract pulls out text pieces using patterns [OK]

Common Mistakes:

• Thinking it deletes rows instead of extracting text
• Confusing extract with replace or count methods
• Expecting a single string instead of a DataFrame

2. Which of the following is the correct syntax to extract a 3-digit number from a pandas Series df['code'] using str.extract?

easy

A. df['code'].str.extract('\d{3}')

B. df['code'].str.extract(r'(\d{3})')

C. df['code'].str.extract(r'\d{3}')

D. df['code'].str.extract('(\d{3})')

Solution

1. Step 1: Identify correct regex string format

   Regex pattern must be a raw string (r'...') to handle backslashes properly.

2. Step 2: Check parentheses for capturing group

   Parentheses are needed to capture the group to extract.

3. Final Answer:

   df['code'].str.extract(r'(\d{3})') -> Option B

4. Quick Check:

   Use raw string with parentheses for capture [OK]

Hint: Use raw string and parentheses for capture groups [OK]

Common Mistakes:

• Missing parentheses around the pattern
• Not using raw string causing escape errors
• Using double backslashes incorrectly

3. Given the following code:

import pandas as pd
df = pd.DataFrame({'text': ['ID123', 'ID456', 'NoID', 'ID789']})
df['number'] = df['text'].str.extract(r'ID(\d{3})')

What will be the content of the new column number?

medium

A. [NaN, NaN, NaN, NaN]

B. ['ID123', 'ID456', 'NoID', 'ID789']

C. ['123', '456', 'NoID', '789']

D. ['123', '456', NaN, '789']

Solution

1. Step 1: Understand the regex pattern

   The pattern 'ID(\d{3})' looks for 'ID' followed by exactly 3 digits and extracts the digits.

2. Step 2: Apply extraction to each row

   'ID123' -> '123', 'ID456' -> '456', 'NoID' -> no match so NaN, 'ID789' -> '789'.

3. Final Answer:

   ['123', '456', NaN, '789'] -> Option D

4. Quick Check:

   Extract digits after 'ID' or NaN if no match [OK]

Hint: Extracted groups appear; no match gives NaN [OK]

Common Mistakes:

• Expecting full string instead of just digits
• Not realizing unmatched rows become NaN
• Confusing extracted text with original

4. Consider this code snippet:

import pandas as pd
df = pd.DataFrame({'info': ['abc123', 'def456', 'ghi789']})
df['extracted'] = df['info'].str.extract('\d{3}')

What is the issue with this code?

medium

A. The regex pattern lacks parentheses for capturing groups.

B. The regex pattern is invalid syntax.

C. The DataFrame column name is incorrect.

D. str.extract cannot extract digits.

Solution

1. Step 1: Check regex pattern for capture groups

   str.extract requires parentheses to capture groups; '\d{3}' has none.

2. Step 2: Understand effect of missing parentheses

   Without parentheses, str.extract returns NaN for all rows because no group is captured.

3. Final Answer:

   The regex pattern lacks parentheses for capturing groups. -> Option A

4. Quick Check:

   Parentheses needed to capture groups [OK]

Hint: Always use parentheses to capture groups in str.extract [OK]

Common Mistakes:

• Forgetting parentheses around regex group
• Assuming str.extract works without capture groups
• Confusing str.extract with str.contains

5. You have a DataFrame df with a column 'data' containing strings like 'Order: #12345, Date: 2023-05-10'. You want to extract the order number and date into separate columns using str.extract. Which regex pattern correctly extracts both parts?

hard

A. r'Order: #(\d+), Date: (\d{4}-\d{2}-\d{2})'

B. r'Order: #(\d+), Date: \d{4}-\d{2}-\d{2}'

C. r'Order: #\d+, Date: (\d{4}-\d{2}-\d{2})'

D. r'Order: #(\d+), Date: (\d{4}/\d{2}/\d{2})'

Solution

1. Step 1: Identify needed capture groups

   We need to capture order number digits and date in 'YYYY-MM-DD' format, so two groups are required.

2. Step 2: Check regex pattern correctness

   r'Order: #(\d+), Date: (\d{4}-\d{2}-\d{2})' has two groups: '(\d+)' for order number and '(\d{4}-\d{2}-\d{2})' for date, matching the format exactly.

3. Final Answer:

   r'Order: #(\d+), Date: (\d{4}-\d{2}-\d{2})' -> Option A

4. Quick Check:

   Two capture groups matching order and date format [OK]

Hint: Use parentheses for each part you want to extract [OK]

Common Mistakes:

• Missing parentheses around one or both groups
• Wrong date format with slashes instead of dashes
• Not capturing order number digits