Building cleaning pipelines with pipe() in Pandas - Mini Project: Build & Apply

Building cleaning pipelines with pipe()

📖 Scenario: You work as a data analyst for a small online store. You receive raw sales data that needs cleaning before analysis. The data has some missing values and inconsistent formatting.Using pandas, you will clean this data step-by-step by building a pipeline with the pipe() method. This method helps you apply multiple cleaning functions in a clear and organized way.

🎯 Goal: Create a pandas DataFrame with raw sales data, define cleaning functions, and use pipe() to apply these functions in a pipeline. Finally, display the cleaned DataFrame.

📋 What You'll Learn

Create a pandas DataFrame with given sales data

Define a function to fill missing values

Define a function to standardize product names

Use pipe() to apply cleaning functions in sequence

Print the cleaned DataFrame

💡 Why This Matters

🌍 Real World

Cleaning data is a common first step in data science projects. Using pipelines with <code>pipe()</code> helps keep your code organized and easy to read.

💼 Career

Data analysts and scientists often clean messy data before analysis. Knowing how to build pipelines with <code>pipe()</code> is a valuable skill for writing clean, maintainable code.

1

Create the raw sales DataFrame

Create a pandas DataFrame called sales with these exact columns and data:
Product: ['apple', 'Banana', None, 'orange', 'banana']
Quantity: [10, 5, 8, None, 7]
Price: [1.2, 0.5, 0.8, 1.0, None]

Pandas

# Create the DataFrame called sales with the exact data
# Your code here

Hint

Use pd.DataFrame with a dictionary of lists for columns.

2

Define cleaning functions

Define two functions:
1. fill_missing(df) that fills missing values in Quantity and Price columns with 0.
2. standardize_product(df) that converts all Product names to lowercase strings and fills missing Product values with 'unknown'.

Pandas

import pandas as pd

sales = pd.DataFrame({
    'Product': ['apple', 'Banana', None, 'orange', 'banana'],
    'Quantity': [10, 5, 8, None, 7],
    'Price': [1.2, 0.5, 0.8, 1.0, None]
})

# Define fill_missing(df) to fill missing Quantity and Price with 0
# Define standardize_product(df) to lowercase Product and fill missing with 'unknown'
# Your code here

Hint

Use fillna(0) for missing numbers and fillna('unknown') plus str.lower() for product names.

3

Build the cleaning pipeline with pipe()

Use the pipe() method on sales to apply fill_missing first, then standardize_product. Save the result in a new variable called cleaned_sales.

Pandas

import pandas as pd

def fill_missing(df):
    df['Quantity'] = df['Quantity'].fillna(0)
    df['Price'] = df['Price'].fillna(0)
    return df

def standardize_product(df):
    df['Product'] = df['Product'].fillna('unknown').str.lower()
    return df

sales = pd.DataFrame({
    'Product': ['apple', 'Banana', None, 'orange', 'banana'],
    'Quantity': [10, 5, 8, None, 7],
    'Price': [1.2, 0.5, 0.8, 1.0, None]
})

# Use pipe() on sales to apply fill_missing then standardize_product
# Save result in cleaned_sales
# Your code here

Hint

Chain pipe() calls: sales.pipe(fill_missing).pipe(standardize_product)

4

Display the cleaned DataFrame

Print the cleaned_sales DataFrame to see the cleaned data.

Pandas

import pandas as pd

def fill_missing(df):
    df['Quantity'] = df['Quantity'].fillna(0)
    df['Price'] = df['Price'].fillna(0)
    return df

def standardize_product(df):
    df['Product'] = df['Product'].fillna('unknown').str.lower()
    return df

sales = pd.DataFrame({
    'Product': ['apple', 'Banana', None, 'orange', 'banana'],
    'Quantity': [10, 5, 8, None, 7],
    'Price': [1.2, 0.5, 0.8, 1.0, None]
})

cleaned_sales = sales.pipe(fill_missing).pipe(standardize_product)

# Print cleaned_sales
# Your code here

Hint

Use print(cleaned_sales) to show the cleaned DataFrame.

Practice

1. What is the main purpose of using pipe() in pandas data cleaning pipelines?

easy

A. To create new DataFrame columns automatically

B. To speed up the execution of pandas code

C. To chain multiple data cleaning functions clearly and simply

D. To visualize data cleaning steps

Solution

1. Step 1: Understand the role of pipe()

   pipe() allows chaining functions that take and return DataFrames, making code easier to read and maintain.

2. Step 2: Compare with other options

   The other options describe unrelated features; pipe() does not speed execution, create columns automatically, or visualize steps.

3. Final Answer:

   To chain multiple data cleaning functions clearly and simply -> Option C

4. Quick Check:

   pipe() = chaining functions clearly [OK]

Hint: Remember: pipe() links functions for clear step-by-step cleaning [OK]

Common Mistakes:

• Thinking pipe() speeds up code execution
• Confusing pipe() with visualization tools
• Assuming pipe() creates new columns automatically

2. Which of the following is the correct syntax to use pipe() with a function named clean_data on a DataFrame df?

easy

A. df.pipe(clean_data())

B. df.pipe(clean_data)

C. pipe(df, clean_data)

D. df.clean_data.pipe()

Solution

1. Step 1: Recall pipe() syntax

   The pipe() method takes a function as an argument without calling it (no parentheses).

2. Step 2: Analyze options

   df.pipe(clean_data()) calls the function immediately, which is incorrect. pipe(df, clean_data) uses wrong syntax. df.clean_data.pipe() is invalid pandas syntax.

3. Final Answer:

   df.pipe(clean_data) -> Option B

4. Quick Check:

   pipe() needs function name without parentheses [OK]

Hint: Pass function name to pipe() without parentheses [OK]

Common Mistakes:

• Adding parentheses after function name in pipe()
• Using pipe() as a standalone function
• Trying to call pipe() on a function instead of DataFrame

3. Given the code below, what will be the output of result?

import pandas as pd
def drop_missing(df):
    return df.dropna()
def to_lowercase(df):
    df['Name'] = df['Name'].str.lower()
    return df

data = {'Name': ['Alice', None, 'Bob'], 'Age': [25, 30, None]}
df = pd.DataFrame(data)
result = df.pipe(drop_missing).pipe(to_lowercase)

medium

A. {'Name': ['alice'], 'Age': [25]}

B. DataFrame with all original rows but 'Name' lowercase

C. Error because to_lowercase does not return a DataFrame

D. DataFrame with rows where 'Name' is lowercase and no missing values

Solution

1. Step 1: Apply drop_missing function

   drop_missing removes rows with any missing values, leaving only rows with complete data.

2. Step 2: Apply to_lowercase function

   to_lowercase converts the 'Name' column to lowercase and returns the modified DataFrame.

3. Final Answer:

   DataFrame with rows where 'Name' is lowercase and no missing values -> Option D

4. Quick Check:

   pipe() chains dropna then lowercase correctly [OK]

Hint: pipe() applies functions stepwise, returning updated DataFrame each time [OK]

Common Mistakes:

• Assuming dropna removes only rows with missing 'Name'
• Thinking to_lowercase modifies DataFrame in place without return
• Confusing output with original DataFrame

4. Consider the following code snippet:

def fix_age(df):
    df['Age'] = df['Age'].fillna(0)

result = df.pipe(fix_age)

Why will this code cause an error when used with pipe()?

medium

A. fix_age does not return the DataFrame

B. pipe() cannot be used with functions that modify columns

C. fillna(0) is not a valid pandas method

D. df is not defined before pipe()

Solution

1. Step 1: Check function return value

   fix_age modifies the DataFrame but does not return it, so pipe() receives None and fails.

2. Step 2: Understand pipe() requirement

   Functions used with pipe() must accept and return a DataFrame to continue chaining.

3. Final Answer:

   fix_age does not return the DataFrame -> Option A

4. Quick Check:

   pipe() needs functions returning DataFrame [OK]

Hint: Always return the DataFrame in functions used with pipe() [OK]

Common Mistakes:

• Forgetting to return the DataFrame in cleaning functions
• Thinking pipe() forbids column modifications
• Assuming fillna(0) is invalid

5. You want to build a cleaning pipeline that removes rows with missing values, converts a 'City' column to uppercase, and adds a new column 'AgeGroup' based on 'Age'. Which of the following pipelines correctly uses pipe() to achieve this?

hard

A. df.pipe(lambda d: d.dropna()).pipe(lambda d: d.assign(City=d['City'].str.upper())).pipe(lambda d: d.assign(AgeGroup=d['Age'].apply(lambda x: 'Adult' if x>=18 else 'Child')))

B. df.pipe(dropna()).pipe(to_uppercase('City')).pipe(add_age_group())

C. df.pipe(dropna).pipe(to_uppercase).pipe(add_age_group)

D. df.pipe(dropna).pipe(to_uppercase('City')).pipe(add_age_group(df))

Solution

1. Step 1: Analyze df.pipe(lambda d: d.dropna()).pipe(lambda d: d.assign(City=d['City'].str.upper())).pipe(lambda d: d.assign(AgeGroup=d['Age'].apply(lambda x: 'Adult' if x>=18 else 'Child')))

   df.pipe(lambda d: d.dropna()).pipe(lambda d: d.assign(City=d['City'].str.upper())).pipe(lambda d: d.assign(AgeGroup=d['Age'].apply(lambda x: 'Adult' if x>=18 else 'Child'))) uses lambda functions with pipe(), each returning a DataFrame after applying the required transformation correctly.

2. Step 2: Analyze other options

   df.pipe(dropna()).pipe(to_uppercase('City')).pipe(add_age_group()) calls functions immediately (with parentheses), which is incorrect. df.pipe(dropna).pipe(to_uppercase).pipe(add_age_group) misses arguments for functions needing them. df.pipe(dropna).pipe(to_uppercase('City')).pipe(add_age_group(df)) passes df as argument incorrectly.

3. Final Answer:

   df.pipe(lambda d: d.dropna()).pipe(lambda d: d.assign(City=d['City'].str.upper())).pipe(lambda d: d.assign(AgeGroup=d['Age'].apply(lambda x: 'Adult' if x>=18 else 'Child'))) -> Option A

4. Quick Check:

   pipe() needs functions returning DataFrames, no immediate calls [OK]

Hint: Use lambdas or functions that accept and return DataFrames in pipe() [OK]

Common Mistakes:

• Calling functions inside pipe() instead of passing them
• Not returning DataFrame from functions
• Passing extra arguments incorrectly