When to use apply vs vectorized operations in Pandas - When to Use Which

The Big Idea

Discover how to speed up your data work and avoid slow, error-prone loops!

The Scenario

Imagine you have a huge spreadsheet with thousands of rows of sales data. You want to calculate a new column based on some complex rule for each row. Doing this by hand or with simple loops feels like a never-ending chore.

The Problem

Manually looping through each row or using slow functions can take forever and often leads to mistakes. It's easy to lose track, make errors, or wait minutes for your computer to finish.

The Solution

Using vectorized operations lets you perform calculations on entire columns at once, making it super fast and less error-prone. When you need custom logic that vectorized methods can't handle, apply lets you run your own function on each row or column efficiently.

Before vs After

✗ Before

for i in range(len(df)):
    df.loc[i, 'new'] = df.loc[i, 'a'] + df.loc[i, 'b']

✓ After

df['new'] = df['a'] + df['b']

What It Enables

You can quickly and correctly transform large datasets, choosing the fastest method for your task and avoiding slow, error-prone loops.

Real Life Example

A data analyst calculates total sales by adding columns with vectorized operations, but uses apply to categorize customers based on complex rules that don't fit simple math.

Key Takeaways

Vectorized operations are fast and work on whole columns at once.

apply is useful for custom row- or column-wise logic.

Choosing the right method saves time and reduces errors.

Practice

1. Which of the following best describes when to use apply in pandas?

easy

A. When you want to merge two DataFrames.

B. When you want to perform simple arithmetic on entire columns quickly.

C. When you want to sort a DataFrame by a column.

D. When you need to run a custom function on each row or column.

Solution

1. Step 1: Understand the purpose of apply

   apply is used to apply custom or complex functions to rows or columns in a DataFrame.

2. Step 2: Compare with vectorized operations

   Vectorized operations are for simple, fast math on entire columns, not custom row-wise functions.

3. Final Answer:

   When you need to run a custom function on each row or column. -> Option D

4. Quick Check:

   Custom function use = apply = A [OK]

Hint: Use apply for custom row/column functions, vectorized for simple math [OK]

Common Mistakes:

• Confusing apply with vectorized operations for simple math
• Using apply for sorting or merging
• Thinking apply is always faster

2. Which of these is the correct syntax to use apply on a DataFrame column df['col'] with a function func?

easy

A. apply(df['col'], func)

B. df['col'].apply(func)

C. df.apply['col'](func)

D. df['col'].apply = func

Solution

1. Step 1: Recall the method call syntax

   In pandas, apply is a method called on a Series or DataFrame with the function as argument.

2. Step 2: Check each option

   df['col'].apply(func) uses correct syntax: df['col'].apply(func). Others have syntax errors or wrong usage.

3. Final Answer:

   df['col'].apply(func) -> Option B

4. Quick Check:

   Method call with function argument = C [OK]

Hint: Use dot apply with parentheses and function inside [OK]

Common Mistakes:

• Using square brackets instead of parentheses for apply
• Trying to assign function to apply attribute
• Calling apply as a standalone function

3. Given the DataFrame:

import pandas as pd
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

What is the output of:

df['C'] = df['A'] + df['B']
df['D'] = df['A'].apply(lambda x: x * 2)
df[['C', 'D']]

medium

A. C D
0 5 2
1 7 4
2 9 6

B. C D
0 5 1
1 7 2
2 9 3

C. C D
0 4 2
1 5 4
2 6 6

D. C D
0 5 4
1 7 6
2 9 8

Solution

1. Step 1: Calculate column C using vectorized addition

   Column C = A + B = [1+4, 2+5, 3+6] = [5, 7, 9]

2. Step 2: Calculate column D using apply with lambda

   Column D = A * 2 = [1*2, 2*2, 3*2] = [2, 4, 6]

3. Final Answer:

   C D 0 5 2 1 7 4 2 9 6 -> Option A

4. Quick Check:

   Vectorized + apply results = B [OK]

Hint: Add columns directly, use apply for custom row-wise ops [OK]

Common Mistakes:

• Mixing up which columns are added
• Forgetting to multiply by 2 in apply
• Confusing apply output with original values

4. You want to double the values in column 'A' of DataFrame df. Which code has an error?

medium

A. df['A'] = df['A'] * 2

B. df['A'] = df['A'].apply(lambda x: x * 2)

C. df['A'] = df.apply(lambda x: x['A'] * 2)

D. df['A'] = df['A'].map(lambda x: x * 2)

Solution

1. Step 1: Analyze df['A'] = df.apply(lambda x: x['A'] * 2)'s use of apply

   Using df.apply without specifying axis defaults to axis=0 (columns), so x['A'] fails because x is a column name string, not a row.

2. Step 2: Compare with other options

   Options B, C, and D correctly double values using apply on Series, vectorized, or map on Series.

3. Final Answer:

   df['A'] = df.apply(lambda x: x['A'] * 2) -> Option C

4. Quick Check:

   apply on DataFrame needs axis=1 for row-wise, else error = A [OK]

Hint: Use axis=1 with apply on DataFrame rows to avoid errors [OK]

Common Mistakes:

• Forgetting axis=1 in DataFrame apply
• Confusing apply on DataFrame vs Series
• Using map on DataFrame instead of Series

5. You have a DataFrame df with columns 'X' and 'Y'. You want to create a new column 'Z' that is 'X' squared if 'Y' is even, else 'X' cubed. Which is the best approach?

hard

A. Use vectorized operations with np.where and powers.

B. Use apply with a custom function checking Y and computing Z.

C. Use a for loop to iterate rows and assign Z values.

D. Use map on column 'X' only.

Solution

1. Step 1: Understand the condition and operations

   We want Z = X^2 if Y is even, else Z = X^3.

2. Step 2: Choose efficient method

   Using np.where with vectorized power operations is fast and clear. apply works but is slower. For loops are slow and verbose. map on X alone can't check Y.

3. Final Answer:

   Use vectorized operations with np.where and powers. -> Option A

4. Quick Check:

   Conditional vectorized ops > apply or loops = D [OK]

Hint: Use np.where for conditional vectorized operations [OK]

Common Mistakes:

• Using apply when vectorized is faster
• Using loops instead of vectorized code
• Trying to map without condition on another column