Why Pattern matching with str.contains in Data Analysis Python? - Purpose & Use Cases

The Big Idea

What if you could find any phrase in thousands of texts in just one line of code?

The Scenario

Imagine you have a long list of customer reviews and you want to find all reviews that mention the word "great" or "excellent". Doing this by reading each review one by one is like searching for a needle in a haystack.

The Problem

Manually scanning through hundreds or thousands of text entries is slow and tiring. You might miss some because of typos or different word forms. It's easy to make mistakes and hard to keep track of what you found.

The Solution

Using str.contains lets you quickly check if each text entry has the pattern you want. It works fast on big lists and can handle variations like uppercase or lowercase letters automatically.

Before vs After

✗ Before

matches = []
for review in reviews:
    if 'great' in review.lower() or 'excellent' in review.lower():
        matches.append(review)

✓ After

matches = reviews[reviews.str.contains('great|excellent', case=False, na=False)]

What It Enables

You can instantly filter and analyze large text data to find meaningful patterns without reading every word.

Real Life Example

A company can quickly find all customer feedback mentioning "slow service" or "friendly staff" to improve their business.

Key Takeaways

Manually searching text is slow and error-prone.

str.contains makes pattern matching fast and easy.

This helps analyze large text data efficiently.

Practice

1. What does the str.contains method do when used on a pandas Series?

easy

A. It replaces all occurrences of a pattern with another string.

B. It counts how many times a pattern appears in each string.

C. It splits each string into a list based on a pattern.

D. It checks if each string contains a specified pattern and returns true or false.

Solution

1. Step 1: Understand str.contains purpose

   The method checks each string in a Series for a pattern and returns true or false accordingly.

2. Step 2: Compare with other string methods

   Replacing, splitting, or counting are different methods like replace(), split(), or count(), not contains().

3. Final Answer:

   It checks if each string contains a specified pattern and returns true or false. -> Option D

4. Quick Check:

   str.contains returns boolean mask [OK]

Hint: Remember: contains returns true/false for pattern presence [OK]

Common Mistakes:

• Confusing contains with replace or split
• Thinking it returns counts instead of booleans
• Assuming it modifies the original data

2. Which of the following is the correct syntax to filter rows in a DataFrame df where the column 'name' contains the substring 'cat'?

easy

A. df[df['name'].str.contain('cat')]

B. df[df['name'].contains('cat')]

C. df[df['name'].str.contains('cat')]

D. df[df['name'].str.contains(cat)]

Solution

1. Step 1: Identify correct method usage

   The correct method is str.contains called on the Series with the pattern as a string argument.

2. Step 2: Check syntax details

   df[df['name'].str.contains('cat')] uses str.contains('cat') correctly with quotes around 'cat'. df[df['name'].contains('cat')] misses str, C has a typo, D misses quotes around 'cat'.

3. Final Answer:

   df[df['name'].str.contains('cat')] -> Option C

4. Quick Check:

   Use str.contains with string pattern in quotes [OK]

Hint: Use df['col'].str.contains('pattern') with quotes [OK]

Common Mistakes:

• Forgetting to use .str before contains
• Missing quotes around the pattern string
• Typo in method name like contain instead of contains

3. Given the DataFrame df below:

import pandas as pd
df = pd.DataFrame({'text': ['apple pie', 'banana split', 'cherry tart', 'apple turnover']})

What will df[df['text'].str.contains('apple')] return?

medium

A. Rows where 'text' contains 'apple': 'apple pie' and 'apple turnover'.

B. Rows where 'text' contains 'banana' only.

C. All rows because 'apple' is in every string.

D. An empty DataFrame because 'apple' is not found.

Solution

1. Step 1: Check which rows contain 'apple'

   Rows 0 ('apple pie') and 3 ('apple turnover') contain the substring 'apple'.

2. Step 2: Understand filtering result

   The filter returns only these rows, excluding others like 'banana split' and 'cherry tart'.

3. Final Answer:

   Rows where 'text' contains 'apple': 'apple pie' and 'apple turnover'. -> Option A

4. Quick Check:

   Filter keeps rows with 'apple' substring [OK]

Hint: Filter keeps only rows where pattern is found [OK]

Common Mistakes:

• Assuming all rows are returned
• Confusing 'apple' with other fruits
• Expecting a count instead of filtered rows

4. Consider this code snippet:

import pandas as pd
df = pd.DataFrame({'col': ['dog', 'cat', 'bird']})
filtered = df[df['col'].str.contains(cat)]
print(filtered)

What is the error and how to fix it?

medium

A. TypeError because df is not defined; import pandas first.

B. NameError because 'cat' is not quoted; fix by using 'cat' in quotes.

C. AttributeError because str.contains is misspelled; fix spelling.

D. No error; code runs correctly.

Solution

1. Step 1: Identify cause of error

   The pattern 'cat' is used without quotes, so Python treats it as a variable, causing NameError.

2. Step 2: Correct the syntax

   Put 'cat' in quotes: str.contains('cat') to pass a string pattern.

3. Final Answer:

   NameError because 'cat' is not quoted; fix by using 'cat' in quotes. -> Option B

4. Quick Check:

   Pattern must be a string in quotes [OK]

Hint: Always quote string patterns in str.contains [OK]

Common Mistakes:

• Forgetting quotes around pattern
• Assuming pattern can be variable without definition
• Ignoring error messages about undefined names

5. You have a DataFrame df with a column 'comments' containing text. You want to filter rows where 'comments' contain either 'error' or 'fail' (case insensitive). Which code correctly does this?

hard

A. df[df['comments'].str.contains('error|fail', case=false)]

B. df[df['comments'].str.contains('error' or 'fail', case=true)]

C. df[df['comments'].str.contains(['error', 'fail'], case=false)]

D. df[df['comments'].str.contains('error|fail', case=true)]

Solution

1. Step 1: Use regex pattern for multiple words

   To match 'error' or 'fail', use regex 'error|fail' inside str.contains.

2. Step 2: Set case insensitive matching

   Use case=false to ignore case differences.

3. Step 3: Check options

   df[df['comments'].str.contains('error|fail', case=false)] correctly uses regex and case=false. df[df['comments'].str.contains('error' or 'fail', case=true)] misuses 'or' inside string, C passes list instead of string, A uses case=true which is case sensitive.

4. Final Answer:

   df[df['comments'].str.contains('error|fail', case=false)] -> Option A

5. Quick Check:

   Use regex with | and case=false for case-insensitive OR [OK]

Hint: Use regex 'word1|word2' with case=false for OR matching [OK]

Common Mistakes:

• Using Python 'or' inside string instead of regex |
• Passing list instead of string pattern
• Forgetting case=false for case-insensitive search