apply() on columns in Pandas - Time & Space Complexity

Time Complexity: apply() on columns

O(n x m)

Understanding Time Complexity

We want to understand how the time needed to run apply() on DataFrame columns changes as the data grows.

Specifically, how does the work increase when we apply a function to each column?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.rand(1000, 10))
result = df.apply(lambda col: col.sum(), axis=0)

This code creates a DataFrame with 1000 rows and 10 columns, then sums each column using apply().

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: The function runs once per column, processing all rows in that column.
• How many times: It repeats for each of the columns (here, 10 times).

How Execution Grows With Input

As the number of rows grows, the work to sum each column grows linearly. More rows mean more numbers to add per column.

Input Size (n rows)	Approx. Operations
10	10 columns x 10 rows = 100 operations
100	10 columns x 100 rows = 1,000 operations
1000	10 columns x 1000 rows = 10,000 operations

Pattern observation: Operations grow directly with the number of rows. Doubling rows doubles work.

Final Time Complexity

Time Complexity: O(n x m)

This means the time grows proportionally with the number of rows (n) times the number of columns (m).

Common Mistake

[X] Wrong: "Applying a function on columns only depends on the number of columns, so it's O(m)."

[OK] Correct: Each column's function processes all rows, so the total work depends on both rows and columns, not just columns.

Interview Connect

Understanding how data size affects function application helps you write efficient code and explain your choices clearly in real projects.

Self-Check

What if we changed axis=0 to axis=1 to apply the function on rows instead of columns? How would the time complexity change?

Practice

1. What does the apply() function do when used on a pandas DataFrame column?

easy

A. It applies a function to each value in the column.

B. It deletes the column from the DataFrame.

C. It sorts the column values in ascending order.

D. It merges the column with another DataFrame.

Solution

1. Step 1: Understand the purpose of apply() on columns

   The apply() method runs a function on each element of the column, transforming or analyzing the data.

2. Step 2: Compare with other options

   Deleting, sorting, or merging are not actions performed by apply() on a column.

3. Final Answer:

   It applies a function to each value in the column. -> Option A

4. Quick Check:

   apply() runs function on each column value [OK]

Hint: Remember: apply() runs a function on each column value [OK]

Common Mistakes:

• Thinking apply() deletes or sorts data
• Confusing apply() with merge or drop functions
• Assuming apply() works on entire DataFrame without axis

2. Which of the following is the correct syntax to apply a lambda function that doubles each value in the 'age' column of a DataFrame df?

easy

A. df['age'].apply = lambda x: x * 2

B. df.apply['age'](lambda x: x * 2)

C. df['age'].apply(lambda x: x * 2)

D. apply(df['age'], lambda x: x * 2)

Solution

1. Step 1: Identify correct method call syntax

   The correct way to use apply on a column is df['column'].apply(function). df['age'].apply(lambda x: x * 2) matches this.

2. Step 2: Check lambda function usage

   df['age'].apply(lambda x: x * 2) correctly passes a lambda function that doubles the value. Other options misuse syntax or assignment.

3. Final Answer:

   df['age'].apply(lambda x: x * 2) -> Option C

4. Quick Check:

   Use df['col'].apply(lambda x: ...) [OK]

Hint: Use df['col'].apply(lambda x: ...) for column functions [OK]

Common Mistakes:

• Using square brackets with apply like a list
• Assigning apply instead of calling it
• Calling apply as a standalone function

3. Given the DataFrame df below:

   score
0     10
1     20
2     30
3     40

What is the output of df['score'].apply(lambda x: x // 15)?

medium

A. 0 10
1 20
2 30
3 40
Name: score, dtype: int64

B. 0 0
1 1
2 2
3 3
Name: score, dtype: int64

C. 0 0
1 0
2 1
3 2
Name: score, dtype: int64

D. 0 0
1 1
2 2
3 2
Name: score, dtype: int64

Solution

1. Step 1: Apply the lambda function to each value

   Divide each score by 15 using floor division:
   10 // 15 = 0
   20 // 15 = 1
   30 // 15 = 2
   40 // 15 = 2

2. Step 2: Write the resulting Series

   The output Series is:
   0 0
   1 1
   2 2
   3 2

3. Final Answer:

   0 0 1 1 2 2 3 2 Name: score, dtype: int64 -> Option D

4. Quick Check:

   Floor division on each value = output [OK]

Hint: Apply lambda to each value and calculate carefully [OK]

Common Mistakes:

• Using normal division instead of floor division
• Misreading floor division results
• Confusing output format with DataFrame instead of Series

4. What is wrong with this code snippet?

df['new_col'] = df['value'].apply(x: x + 5)

How can it be fixed?

medium

A. Missing lambda keyword; fix: df['value'].apply(lambda x: x + 5)

B. apply() cannot add numbers; use map() instead

C. Should use square brackets around x; fix: apply([x]: x + 5)

D. Function must be defined outside apply; fix: def f(x): return x+5

Solution

1. Step 1: Identify syntax error in lambda function

   The code uses x: x + 5 without the lambda keyword, which is required to define an anonymous function.

2. Step 2: Correct the syntax

   Add lambda before the function: df['value'].apply(lambda x: x + 5).

3. Final Answer:

   Missing lambda keyword; fix: df['value'].apply(lambda x: x + 5) -> Option A

4. Quick Check:

   Lambda keyword needed for inline functions [OK]

Hint: Always use 'lambda' before inline functions in apply() [OK]

Common Mistakes:

• Omitting the lambda keyword
• Using incorrect brackets or syntax for lambda
• Thinking apply() can't use inline functions

5. You have a DataFrame df with a column 'temperature' in Celsius. You want to create a new column 'temp_f' with Fahrenheit values using apply(). Which code correctly does this?

hard

A. df['temp_f'] = df.apply(lambda c: (c * 9/5) + 32)

B. df['temp_f'] = df['temperature'].apply(lambda c: (c * 9/5) + 32)

C. df['temp_f'] = df['temperature'].apply(lambda c: c + 32 * 9/5)

D. df['temp_f'] = df['temperature'].apply(lambda c: c * (9/5 + 32))

Solution

1. Step 1: Recall the formula for Celsius to Fahrenheit

   The correct formula is Fahrenheit = (Celsius * 9/5) + 32.

2. Step 2: Check each option's lambda function

   df['temp_f'] = df['temperature'].apply(lambda c: (c * 9/5) + 32) correctly applies the formula to each value in 'temperature'. df['temp_f'] = df.apply(lambda c: (c * 9/5) + 32) applies to whole DataFrame, C and D have incorrect formulas.

3. Final Answer:

   df['temp_f'] = df['temperature'].apply(lambda c: (c * 9/5) + 32) -> Option B

4. Quick Check:

   Use correct formula inside apply() on the column [OK]

Hint: Use correct formula inside lambda for each column value [OK]

Common Mistakes:

• Applying function to whole DataFrame instead of column
• Mixing order of operations in formula
• Forgetting parentheses in formula