Why custom functions matter in Pandas - Performance Analysis

Time Complexity: Why custom functions matter

O(n)

Understanding Time Complexity

When using pandas, custom functions let us do special tasks on data. But how fast these functions run matters a lot.

We want to know how the time to run changes as our data grows bigger.

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

def custom_func(x):
    return x ** 2 + 1

df = pd.DataFrame({'A': range(1000)})
df['B'] = df['A'].apply(custom_func)

This code creates a DataFrame with 1000 numbers and applies a custom function to each number to make a new column.

Identify Repeating Operations

• Primary operation: Applying the custom function to each row value.
• How many times: Once for every row in the DataFrame (n times).

How Execution Grows With Input

Each new row means one more call to the custom function, so the total work grows steadily with data size.

Input Size (n)	Approx. Operations
10	10 function calls
100	100 function calls
1000	1000 function calls

Pattern observation: The time grows directly in line with the number of rows.

Final Time Complexity

Time Complexity: O(n)

This means the time to run grows in a straight line as the data gets bigger.

Common Mistake

[X] Wrong: "Using a custom function inside apply is always slow and complex like nested loops."

[OK] Correct: Actually, if the function runs once per row without extra loops inside, it grows simply with data size, not worse.

Interview Connect

Knowing how custom functions affect speed helps you write clear and efficient data code, a skill useful in many real projects.

Self-Check

"What if the custom function itself had a loop inside? How would the time complexity change?"

Practice

1. Why do we use custom functions when working with pandas DataFrames?

easy

A. To reuse code and make data changes clear and simple

B. To make the DataFrame larger automatically

C. To delete all data quickly

D. To change the pandas version

Solution

1. Step 1: Understand the purpose of custom functions

   Custom functions let us write a piece of code once and use it many times, which saves effort.

2. Step 2: Connect to pandas usage

   In pandas, custom functions help apply the same change or calculation to many rows or columns easily and clearly.

3. Final Answer:

   To reuse code and make data changes clear and simple -> Option A

4. Quick Check:

   Custom functions improve code reuse and clarity [OK]

Hint: Custom functions simplify repeated data changes [OK]

Common Mistakes:

• Thinking custom functions automatically change DataFrame size
• Believing custom functions delete data
• Confusing custom functions with pandas version control

2. Which of the following is the correct way to define a simple custom function in Python to double a number?

easy

A. function double(x) { return x * 2 }

B. def double x: return x * 2

C. def double(x): return x * 2

D. double = x => x * 2

Solution

1. Step 1: Identify Python function syntax

   Python functions start with 'def', followed by the function name and parentheses with parameters.

2. Step 2: Check each option

   def double(x): return x * 2 uses correct Python syntax. function double(x) { return x * 2 } is JavaScript syntax. def double x: return x * 2 misses parentheses. double = x => x * 2 is JavaScript arrow function syntax.

3. Final Answer:

   def double(x): return x * 2 -> Option C

4. Quick Check:

   Python function syntax = def name(params): [OK]

Hint: Python functions start with 'def' and use parentheses [OK]

Common Mistakes:

• Using JavaScript syntax in Python
• Omitting parentheses in function definition
• Confusing arrow functions with Python syntax

3. Given the code below, what will be the output of df['new']?

import pandas as pd
def add_five(x):
    return x + 5
df = pd.DataFrame({'num': [1, 2, 3]})
df['new'] = df['num'].apply(add_five)
print(df['new'])

medium

A. [1, 2, 3]

B. [6, 7, 8]

C. [5, 6, 7]

D. Error: add_five is not defined

Solution

1. Step 1: Understand the function add_five

   The function adds 5 to the input number.

2. Step 2: Apply the function to each element in 'num'

   Applying add_five to [1, 2, 3] results in [6, 7, 8].

3. Final Answer:

   [6, 7, 8] -> Option B

4. Quick Check:

   1+5=6, 2+5=7, 3+5=8 [OK]

Hint: Apply adds function result to each element [OK]

Common Mistakes:

• Thinking apply changes original column without assignment
• Confusing function name or missing definition
• Expecting original values without addition

4. What is wrong with this code snippet?

import pandas as pd
def multiply_by_two(x):
    x * 2
df = pd.DataFrame({'val': [4, 5, 6]})
df['result'] = df['val'].apply(multiply_by_two)
print(df['result'])

medium

A. The DataFrame is not imported correctly

B. The DataFrame is missing a column

C. apply() cannot be used with custom functions

D. The function multiply_by_two does not return a value

Solution

1. Step 1: Check the function multiply_by_two

   The function calculates x * 2 but does not return the result.

2. Step 2: Understand apply behavior

   apply expects the function to return a value to create the new column. Without return, it returns None for each row.

3. Final Answer:

   The function multiply_by_two does not return a value -> Option D

4. Quick Check:

   Functions must return values for apply to work [OK]

Hint: Always return a value in your custom function [OK]

Common Mistakes:

• Forgetting the return statement in functions
• Assuming apply works without returned values
• Thinking DataFrame import affects apply

5. You have a DataFrame with a column 'score' containing numbers. You want to create a new column 'grade' that assigns 'Pass' if the score is 50 or more, and 'Fail' otherwise. Which custom function and pandas code correctly achieve this?

hard

A. def grade_func(x): return 'Pass' if x >= 50 else 'Fail'
df['grade'] = df['score'].apply(grade_func)

B. def grade_func(x): if x >= 50: 'Pass' else: 'Fail'
df['grade'] = df['score'].apply(grade_func)

C. def grade_func(x): return x >= 50 or 'Pass' and 'Fail'
df['grade'] = df['score'].apply(grade_func)

D. def grade_func(x): return 'Pass' if x > 50 else 'Fail'
df['grade'] = df['score'].apply(grade_func)

Solution

1. Step 1: Check the function logic for grading

   def grade_func(x): return 'Pass' if x >= 50 else 'Fail' df['grade'] = df['score'].apply(grade_func) correctly returns 'Pass' if x is 50 or more, else 'Fail'. def grade_func(x): if x >= 50: 'Pass' else: 'Fail' df['grade'] = df['score'].apply(grade_func) misses return statement. def grade_func(x): return x >= 50 or 'Pass' and 'Fail' df['grade'] = df['score'].apply(grade_func) uses incorrect logical operators, resulting in True or 'Fail'. def grade_func(x): return 'Pass' if x > 50 else 'Fail' df['grade'] = df['score'].apply(grade_func) uses > 50, which excludes 50.

2. Step 2: Confirm pandas apply usage

   All options use apply correctly, but only def grade_func(x): return 'Pass' if x >= 50 else 'Fail' df['grade'] = df['score'].apply(grade_func) has correct function logic and syntax.

3. Final Answer:

   def grade_func(x): return 'Pass' if x >= 50 else 'Fail' df['grade'] = df['score'].apply(grade_func) -> Option A

4. Quick Check:

   Use return with correct condition for grading [OK]

Hint: Use return and correct condition in custom function [OK]

Common Mistakes:

• Forgetting return in function
• Using wrong comparison operator
• Confusing logical expressions without return