Building cleaning pipelines with pipe() in Pandas - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of chained pipe() calls on a DataFrame

What is the output of the following code that uses pipe() to clean a DataFrame step-by-step?

Pandas

import pandas as pd

def drop_missing(df):
    return df.dropna()

def multiply_age(df):
    df['age'] = df['age'] * 2
    return df

def rename_columns(df):
    return df.rename(columns={'age': 'double_age'})

df = pd.DataFrame({
    'name': ['Alice', 'Bob', None, 'David'],
    'age': [25, None, 30, 40]
})

result = (df.pipe(drop_missing)
            .pipe(multiply_age)
            .pipe(rename_columns))

print(result)

A

   name  double_age
0  Alice          50
3  David          80

B {'name': ['Alice', 'David'], 'double_age': [50, 80]}

C

   name  age
0  Alice   50
3  David   80

D

   name  age
0  Alice   25
3  David   40

💡 Hint

Remember that pipe() passes the DataFrame to each function and returns the modified DataFrame.

The drop_missing function removes rows with any missing values, so rows with Bob and None name are dropped. Then multiply_age doubles the 'age' values. Finally, rename_columns renames 'age' to 'double_age'. The result shows only Alice and David with doubled ages.

❓ data_output

intermediate

Number of rows after cleaning pipeline

Given the following cleaning pipeline using pipe(), how many rows remain in the final DataFrame?

Pandas

import pandas as pd

def filter_adults(df):
    return df[df['age'] >= 18]

def drop_duplicates(df):
    return df.drop_duplicates(subset=['name'])

def add_status(df):
    df['status'] = 'adult'
    return df

df = pd.DataFrame({
    'name': ['Anna', 'Ben', 'Anna', 'Cara', 'Dan'],
    'age': [17, 22, 17, 19, 16]
})

result = (df.pipe(filter_adults)
            .pipe(drop_duplicates)
            .pipe(add_status))

print(len(result))

A 4

B 3

C 2

D 5

💡 Hint

Check which rows pass the age filter, then remove duplicates by name.

Rows with age >= 18 are Ben (22, index 1) and Cara (19, index 3). Both Anna rows (age 17) and Dan (16) are filtered out. The remaining rows have unique names, so drop_duplicates removes nothing. Final length: 2.

🔧 Debug

advanced

Identify the error in this pipe() cleaning function

What error will this code raise when run, and why?

Pandas

import pandas as pd

def add_column(df):
    df['new_col'] = df['age'] + 10
    # Missing return statement

df = pd.DataFrame({'age': [20, 30, 40]})

result = df.pipe(add_column)
print(result)

A

None
(empty output)

B AttributeError: 'NoneType' object has no attribute 'pipe'

C TypeError: unsupported operand type(s) for +: 'int' and 'str'

D KeyError: 'age'

💡 Hint

Check what the function returns when used with pipe().

The add_column function modifies the DataFrame in place but implicitly returns None (no return statement). Thus, df.pipe(add_column) returns None, and print(result) outputs 'None'. No error is raised.

🚀 Application

advanced

Using pipe() to apply multiple cleaning steps with parameters

You want to apply these cleaning steps to a DataFrame using pipe(): remove rows where 'score' is below a threshold, then add a new column 'passed' that is True if 'score' >= pass_mark. Which code correctly applies these steps with parameters using pipe()?

Pandas

import pandas as pd

def filter_score(df, threshold):
    return df[df['score'] >= threshold]

def add_passed_column(df, pass_mark):
    df['passed'] = df['score'] >= pass_mark
    return df

df = pd.DataFrame({'score': [55, 70, 40, 90]})

A result = df.pipe(filter_score, 50, pass_mark=60).pipe(add_passed_column)

B result = df.pipe(filter_score(threshold=50)).pipe(add_passed_column(pass_mark=60))

C result = df.pipe(filter_score, 50).pipe(add_passed_column, 60)

D result = df.pipe(filter_score, threshold=50).pipe(add_passed_column, pass_mark=60)

💡 Hint

Remember that extra arguments to pipe() after the first are passed as positional or keyword arguments to the function.

Option D correctly passes the threshold and pass_mark as keyword arguments to the respective functions in pipe(). Option D calls the functions immediately instead of passing them to pipe(). Option D passes positional arguments correctly but option D is more explicit and preferred. Option D passes extra arguments incorrectly.

🧠 Conceptual

expert

Why use pipe() in pandas cleaning pipelines?

Which of the following is the best explanation for why pipe() is useful in building data cleaning pipelines?

A <code>pipe()</code> automatically parallelizes DataFrame operations for faster execution.

B <code>pipe()</code> allows chaining functions that take and return DataFrames, improving readability and modularity.

C <code>pipe()</code> converts DataFrames into numpy arrays for faster numeric computation.

D <code>pipe()</code> replaces the need for writing custom functions by providing built-in cleaning methods.

💡 Hint

Think about how pipe() helps organize code.

pipe() is designed to pass a DataFrame through a sequence of functions that each take a DataFrame and return a DataFrame. This makes code easier to read and maintain by chaining steps clearly.

Practice

1. What is the main purpose of using pipe() in pandas data cleaning pipelines?

easy

A. To create new DataFrame columns automatically

B. To speed up the execution of pandas code

C. To chain multiple data cleaning functions clearly and simply

D. To visualize data cleaning steps

Solution

1. Step 1: Understand the role of pipe()

   pipe() allows chaining functions that take and return DataFrames, making code easier to read and maintain.

2. Step 2: Compare with other options

   The other options describe unrelated features; pipe() does not speed execution, create columns automatically, or visualize steps.

3. Final Answer:

   To chain multiple data cleaning functions clearly and simply -> Option C

4. Quick Check:

   pipe() = chaining functions clearly [OK]

Hint: Remember: pipe() links functions for clear step-by-step cleaning [OK]

Common Mistakes:

• Thinking pipe() speeds up code execution
• Confusing pipe() with visualization tools
• Assuming pipe() creates new columns automatically

2. Which of the following is the correct syntax to use pipe() with a function named clean_data on a DataFrame df?

easy

A. df.pipe(clean_data())

B. df.pipe(clean_data)

C. pipe(df, clean_data)

D. df.clean_data.pipe()

Solution

1. Step 1: Recall pipe() syntax

   The pipe() method takes a function as an argument without calling it (no parentheses).

2. Step 2: Analyze options

   df.pipe(clean_data()) calls the function immediately, which is incorrect. pipe(df, clean_data) uses wrong syntax. df.clean_data.pipe() is invalid pandas syntax.

3. Final Answer:

   df.pipe(clean_data) -> Option B

4. Quick Check:

   pipe() needs function name without parentheses [OK]

Hint: Pass function name to pipe() without parentheses [OK]

Common Mistakes:

• Adding parentheses after function name in pipe()
• Using pipe() as a standalone function
• Trying to call pipe() on a function instead of DataFrame

3. Given the code below, what will be the output of result?

import pandas as pd
def drop_missing(df):
    return df.dropna()
def to_lowercase(df):
    df['Name'] = df['Name'].str.lower()
    return df

data = {'Name': ['Alice', None, 'Bob'], 'Age': [25, 30, None]}
df = pd.DataFrame(data)
result = df.pipe(drop_missing).pipe(to_lowercase)

medium

A. {'Name': ['alice'], 'Age': [25]}

B. DataFrame with all original rows but 'Name' lowercase

C. Error because to_lowercase does not return a DataFrame

D. DataFrame with rows where 'Name' is lowercase and no missing values

Solution

1. Step 1: Apply drop_missing function

   drop_missing removes rows with any missing values, leaving only rows with complete data.

2. Step 2: Apply to_lowercase function

   to_lowercase converts the 'Name' column to lowercase and returns the modified DataFrame.

3. Final Answer:

   DataFrame with rows where 'Name' is lowercase and no missing values -> Option D

4. Quick Check:

   pipe() chains dropna then lowercase correctly [OK]

Hint: pipe() applies functions stepwise, returning updated DataFrame each time [OK]

Common Mistakes:

• Assuming dropna removes only rows with missing 'Name'
• Thinking to_lowercase modifies DataFrame in place without return
• Confusing output with original DataFrame

4. Consider the following code snippet:

def fix_age(df):
    df['Age'] = df['Age'].fillna(0)

result = df.pipe(fix_age)

Why will this code cause an error when used with pipe()?

medium

A. fix_age does not return the DataFrame

B. pipe() cannot be used with functions that modify columns

C. fillna(0) is not a valid pandas method

D. df is not defined before pipe()

Solution

1. Step 1: Check function return value

   fix_age modifies the DataFrame but does not return it, so pipe() receives None and fails.

2. Step 2: Understand pipe() requirement

   Functions used with pipe() must accept and return a DataFrame to continue chaining.

3. Final Answer:

   fix_age does not return the DataFrame -> Option A

4. Quick Check:

   pipe() needs functions returning DataFrame [OK]

Hint: Always return the DataFrame in functions used with pipe() [OK]

Common Mistakes:

• Forgetting to return the DataFrame in cleaning functions
• Thinking pipe() forbids column modifications
• Assuming fillna(0) is invalid

5. You want to build a cleaning pipeline that removes rows with missing values, converts a 'City' column to uppercase, and adds a new column 'AgeGroup' based on 'Age'. Which of the following pipelines correctly uses pipe() to achieve this?

hard

A. df.pipe(lambda d: d.dropna()).pipe(lambda d: d.assign(City=d['City'].str.upper())).pipe(lambda d: d.assign(AgeGroup=d['Age'].apply(lambda x: 'Adult' if x>=18 else 'Child')))

B. df.pipe(dropna()).pipe(to_uppercase('City')).pipe(add_age_group())

C. df.pipe(dropna).pipe(to_uppercase).pipe(add_age_group)

D. df.pipe(dropna).pipe(to_uppercase('City')).pipe(add_age_group(df))

Solution

1. Step 1: Analyze df.pipe(lambda d: d.dropna()).pipe(lambda d: d.assign(City=d['City'].str.upper())).pipe(lambda d: d.assign(AgeGroup=d['Age'].apply(lambda x: 'Adult' if x>=18 else 'Child')))

   df.pipe(lambda d: d.dropna()).pipe(lambda d: d.assign(City=d['City'].str.upper())).pipe(lambda d: d.assign(AgeGroup=d['Age'].apply(lambda x: 'Adult' if x>=18 else 'Child'))) uses lambda functions with pipe(), each returning a DataFrame after applying the required transformation correctly.

2. Step 2: Analyze other options

   df.pipe(dropna()).pipe(to_uppercase('City')).pipe(add_age_group()) calls functions immediately (with parentheses), which is incorrect. df.pipe(dropna).pipe(to_uppercase).pipe(add_age_group) misses arguments for functions needing them. df.pipe(dropna).pipe(to_uppercase('City')).pipe(add_age_group(df)) passes df as argument incorrectly.

3. Final Answer:

   df.pipe(lambda d: d.dropna()).pipe(lambda d: d.assign(City=d['City'].str.upper())).pipe(lambda d: d.assign(AgeGroup=d['Age'].apply(lambda x: 'Adult' if x>=18 else 'Child'))) -> Option A

4. Quick Check:

   pipe() needs functions returning DataFrames, no immediate calls [OK]

Hint: Use lambdas or functions that accept and return DataFrames in pipe() [OK]

Common Mistakes:

• Calling functions inside pipe() instead of passing them
• Not returning DataFrame from functions
• Passing extra arguments incorrectly