apply() function for custom logic in Data Analysis Python - Time & Space Complexity

Time Complexity: apply() function for custom logic

O(n)

Understanding Time Complexity

We want to understand how the time it takes to run the apply() function changes as the data grows.

Specifically, how does applying a custom function to each row or column affect the total work done?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

def custom_logic(row):
    return row['A'] * 2 + row['B']

# Create a DataFrame with columns A and B
# Apply custom_logic to each row
result = df.apply(custom_logic, axis=1)

This code applies a custom function to each row of a DataFrame to create a new series.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: The apply() function runs the custom function once for each row.
• How many times: It runs exactly as many times as there are rows in the DataFrame.

How Execution Grows With Input

As the number of rows grows, the total work grows in the same way.

Input Size (n)	Approx. Operations
10	10 function calls
100	100 function calls
1000	1000 function calls

Pattern observation: Doubling the rows doubles the number of function calls.

Final Time Complexity

Time Complexity: O(n)

This means the time grows linearly with the number of rows in the DataFrame.

Common Mistake

[X] Wrong: "The apply() function runs faster than looping manually because it's built-in."

[OK] Correct: apply() still calls the function once per row, so it does similar work as a manual loop. It's not magically faster in terms of time complexity.

Interview Connect

Understanding how apply() scales helps you explain your data processing choices clearly and shows you know what happens behind the scenes.

Self-Check

"What if we changed the custom function to use vectorized operations instead of apply()? How would the time complexity change?"

Practice

1. What does the apply() function do in a pandas DataFrame?

easy

A. It merges two DataFrames together.

B. It sorts the DataFrame by a column.

C. It filters rows based on a condition.

D. It applies a custom function to each row or column.

Solution

1. Step 1: Understand the purpose of apply()

   The apply() function is used to run a custom function on each row or column of a DataFrame.

2. Step 2: Compare with other DataFrame methods

   Sorting, filtering, and merging are different operations and not done by apply().

3. Final Answer:

   It applies a custom function to each row or column. -> Option D

4. Quick Check:

   apply() = custom function application [OK]

Hint: Remember: apply() runs your function on rows or columns [OK]

Common Mistakes:

• Confusing apply() with sort or filter methods
• Thinking apply() merges DataFrames
• Assuming apply() only works on columns

2. Which of the following is the correct syntax to apply a function f row-wise on a DataFrame df?

easy

A. df.apply(f, axis=0)

B. df.apply(f)

C. df.apply(f, axis=1)

D. df.apply(f, axis=2)

Solution

1. Step 1: Recall axis parameter meaning

   In pandas, axis=1 means apply function row-wise, while axis=0 means column-wise.

2. Step 2: Identify correct axis for row-wise

   Since the question asks for row-wise application, axis=1 is correct.

3. Final Answer:

   df.apply(f, axis=1) -> Option C

4. Quick Check:

   Row-wise apply = axis=1 [OK]

Hint: Use axis=1 for rows, axis=0 for columns [OK]

Common Mistakes:

• Using axis=0 for row-wise apply
• Omitting axis parameter and expecting row-wise
• Using invalid axis values like 2

3. Given the DataFrame:

import pandas as pd
df = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})

What is the output of:

df['Sum'] = df.apply(lambda row: row['A'] + row['B'], axis=1)
print(df)

medium

A. A B Sum
0 1 3 3
1 2 4 4

B. A B Sum
0 1 3 4
1 2 4 6

C. A B Sum
0 1 3 1
1 2 4 2

D. Error: axis parameter missing

Solution

1. Step 1: Understand the lambda function

   The lambda adds values from columns 'A' and 'B' for each row.

2. Step 2: Apply row-wise addition

   Row 0: 1 + 3 = 4; Row 1: 2 + 4 = 6.

3. Final Answer:

   A B Sum 0 1 3 4 1 2 4 6 -> Option B

4. Quick Check:

   Sum column = A + B per row [OK]

Hint: Sum columns row-wise with apply and axis=1 [OK]

Common Mistakes:

• Forgetting axis=1 causes wrong operation
• Adding columns incorrectly (e.g., column-wise)
• Expecting apply to work without lambda

4. What is wrong with this code snippet?

df = pd.DataFrame({'X': [10, 20], 'Y': [30, 40]})
df['Result'] = df.apply(lambda x: x['X'] * 2 if x['Y'] > 35 else x['X'], axis=0)

medium

A. Using axis=0 applies function column-wise, causing a KeyError.

B. Lambda function syntax is incorrect.

C. DataFrame is missing required columns.

D. No error; code runs correctly.

Solution

1. Step 1: Check axis parameter usage

   axis=0 applies function to each column (Series), not rows.

2. Step 2: Understand lambda function context

   Lambda expects row with keys 'X' and 'Y', but column Series only has index labels, causing KeyError.

3. Final Answer:

   Using axis=0 applies function column-wise, causing a KeyError. -> Option A

4. Quick Check:

   axis=0 = columns, lambda expects rows [OK]

Hint: Use axis=1 for row-wise logic to avoid key errors [OK]

Common Mistakes:

• Confusing axis=0 and axis=1
• Expecting row keys when applying column-wise
• Ignoring error messages about missing keys

5. You have a DataFrame df with columns "score" and "passed" (boolean). You want to create a new column "grade" where:
- If passed is True and score >= 90, grade is 'A'
- If passed is True and score >= 75 but < 90, grade is 'B'
- Otherwise, grade is 'F'
Which apply() function correctly implements this logic?

hard

A. df['grade'] = df.apply(lambda r: 'A' if r['passed'] and r['score'] >= 90 else ('B' if r['passed'] and r['score'] >= 75 else 'F'), axis=1)

B. df['grade'] = df.apply(lambda r: 'A' if r['score'] >= 90 else 'B' if r['score'] >= 75 else 'F', axis=0)

C. df['grade'] = df.apply(lambda r: 'A' if r['passed'] or r['score'] >= 90 else 'F', axis=1)

D. df['grade'] = df.apply(lambda r: 'A' if r['passed'] and r['score'] > 90 else 'B', axis=1)

Solution

1. Step 1: Check axis parameter for row-wise logic

   axis=1 is needed to apply function on each row.

2. Step 2: Verify conditional logic matches requirements

   df['grade'] = df.apply(lambda r: 'A' if r['passed'] and r['score'] >= 90 else ('B' if r['passed'] and r['score'] >= 75 else 'F'), axis=1) correctly checks if passed is True and score thresholds for 'A' and 'B', else 'F'.

3. Step 3: Identify errors in other options

   df['grade'] = df.apply(lambda r: 'A' if r['score'] >= 90 else 'B' if r['score'] >= 75 else 'F', axis=0) uses axis=0 (wrong axis). df['grade'] = df.apply(lambda r: 'A' if r['passed'] or r['score'] >= 90 else 'F', axis=1) uses 'or' instead of 'and'. df['grade'] = df.apply(lambda r: 'A' if r['passed'] and r['score'] > 90 else 'B', axis=1) misses 'F' grade and uses wrong comparison for 90.

4. Final Answer:

   df['grade'] = df.apply(lambda r: 'A' if r['passed'] and r['score'] >= 90 else ('B' if r['passed'] and r['score'] >= 75 else 'F'), axis=1) -> Option A

5. Quick Check:

   Correct logic and axis=1 for row-wise apply [OK]

Hint: Use axis=1 and nested if-else for multi-condition grading [OK]

Common Mistakes:

• Using axis=0 instead of axis=1
• Using 'or' instead of 'and' in conditions
• Missing else condition for failing grade