When to use apply vs vectorized operations in Pandas - Visual Side-by-Side Comparison

Concept Flow - When to use apply vs vectorized operations

Start with DataFrame

↓

Need to transform data?

Yes↓

Is there a vectorized function?

No→Use apply (row/column wise)

Yes↓

Use vectorized operation (fast)

↓

Get transformed DataFrame

↓

End

Decide if a fast built-in vectorized function fits your task; if not, use apply for custom row/column operations.

Execution Sample

Pandas

import pandas as pd

df = pd.DataFrame({'A': [1, 2, 3]})

# Vectorized
squared = df['A'] ** 2

# Apply
plus_one = df['A'].apply(lambda x: x + 1)

Shows squaring a column with vectorized operation and adding one using apply.

Execution Table

Step	Operation	Input	Output	Notes
1	Vectorized square	[1, 2, 3]	[1, 4, 9]	Fast, uses built-in operator
2	Apply plus one	1	2	Apply calls lambda on each element
3	Apply plus one	2	3	Apply calls lambda on each element
4	Apply plus one	3	4	Apply calls lambda on each element
5	End	-	-	All elements processed

💡 All elements processed, vectorized and apply results ready

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
df['A']	[1, 2, 3]	[1, 2, 3]	[1, 2, 3]	[1, 2, 3]	[1, 2, 3]	[1, 2, 3]
squared	N/A	[1, 4, 9]	[1, 4, 9]	[1, 4, 9]	[1, 4, 9]	[1, 4, 9]
plus_one	N/A	N/A	2	3	4	[2, 3, 4]

Key Moments - 2 Insights

Why is vectorized operation faster than apply?

Vectorized operations use optimized C code internally and work on whole arrays at once, while apply calls a Python function for each element, which is slower (see execution_table steps 1 vs 2-4).

When should I use apply instead of vectorized operations?

Use apply when no built-in vectorized function exists for your custom logic, as shown in the example where adding one uses apply with a lambda (execution_table steps 2-4).

Visual Quiz - 3 Questions

Look at the execution table, what is the output of the vectorized operation at step 1?

A [2, 3, 4]

B [1, 4, 9]

C [1, 2, 3]

D [0, 1, 2]

💡 Hint

Check the 'Output' column at step 1 in the execution_table.

At which step does apply process the element 3?

A Step 2

B Step 3

C Step 4

D Step 1

💡 Hint

Look at the 'Input' column for apply steps in execution_table rows 2-4.

If you replaced apply with a vectorized operation for adding one, what would change in the execution_table?

A Fewer steps, faster execution

B More steps, slower execution

C No change in steps

D Output would be incorrect

💡 Hint

Vectorized operations handle all elements at once, reducing per-element calls (see key_moments about speed).

Concept Snapshot

Use vectorized operations for speed and simplicity when possible.
Use apply for custom row/column functions without vectorized equivalents.
Vectorized ops work on whole arrays at once.
Apply calls a function on each element, slower but flexible.
Check if pandas has a built-in vectorized method before apply.

Full Transcript

This lesson shows when to use pandas vectorized operations versus apply. Vectorized operations are fast because they work on whole columns or arrays at once using optimized code. Apply is slower because it runs a Python function on each element individually. Use vectorized operations when pandas has a built-in method for your task, like squaring a column. Use apply when you need custom logic not available as vectorized functions, like adding one with a lambda. The execution table traces squaring with vectorized power operator and adding one with apply step by step. Variable tracker shows how values change after each step. Key moments explain why vectorized is faster and when apply is needed. The quiz checks understanding of outputs and steps. Remember: prefer vectorized for speed, apply for flexibility.

Practice

1. Which of the following best describes when to use apply in pandas?

easy

A. When you want to merge two DataFrames.

B. When you want to perform simple arithmetic on entire columns quickly.

C. When you want to sort a DataFrame by a column.

D. When you need to run a custom function on each row or column.

Solution

1. Step 1: Understand the purpose of apply

   apply is used to apply custom or complex functions to rows or columns in a DataFrame.

2. Step 2: Compare with vectorized operations

   Vectorized operations are for simple, fast math on entire columns, not custom row-wise functions.

3. Final Answer:

   When you need to run a custom function on each row or column. -> Option D

4. Quick Check:

   Custom function use = apply = A [OK]

Hint: Use apply for custom row/column functions, vectorized for simple math [OK]

Common Mistakes:

• Confusing apply with vectorized operations for simple math
• Using apply for sorting or merging
• Thinking apply is always faster

2. Which of these is the correct syntax to use apply on a DataFrame column df['col'] with a function func?

easy

A. apply(df['col'], func)

B. df['col'].apply(func)

C. df.apply['col'](func)

D. df['col'].apply = func

Solution

1. Step 1: Recall the method call syntax

   In pandas, apply is a method called on a Series or DataFrame with the function as argument.

2. Step 2: Check each option

   df['col'].apply(func) uses correct syntax: df['col'].apply(func). Others have syntax errors or wrong usage.

3. Final Answer:

   df['col'].apply(func) -> Option B

4. Quick Check:

   Method call with function argument = C [OK]

Hint: Use dot apply with parentheses and function inside [OK]

Common Mistakes:

• Using square brackets instead of parentheses for apply
• Trying to assign function to apply attribute
• Calling apply as a standalone function

3. Given the DataFrame:

import pandas as pd
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

What is the output of:

df['C'] = df['A'] + df['B']
df['D'] = df['A'].apply(lambda x: x * 2)
df[['C', 'D']]

medium

A. C D
0 5 2
1 7 4
2 9 6

B. C D
0 5 1
1 7 2
2 9 3

C. C D
0 4 2
1 5 4
2 6 6

D. C D
0 5 4
1 7 6
2 9 8

Solution

1. Step 1: Calculate column C using vectorized addition

   Column C = A + B = [1+4, 2+5, 3+6] = [5, 7, 9]

2. Step 2: Calculate column D using apply with lambda

   Column D = A * 2 = [1*2, 2*2, 3*2] = [2, 4, 6]

3. Final Answer:

   C D 0 5 2 1 7 4 2 9 6 -> Option A

4. Quick Check:

   Vectorized + apply results = B [OK]

Hint: Add columns directly, use apply for custom row-wise ops [OK]

Common Mistakes:

• Mixing up which columns are added
• Forgetting to multiply by 2 in apply
• Confusing apply output with original values

4. You want to double the values in column 'A' of DataFrame df. Which code has an error?

medium

A. df['A'] = df['A'] * 2

B. df['A'] = df['A'].apply(lambda x: x * 2)

C. df['A'] = df.apply(lambda x: x['A'] * 2)

D. df['A'] = df['A'].map(lambda x: x * 2)

Solution

1. Step 1: Analyze df['A'] = df.apply(lambda x: x['A'] * 2)'s use of apply

   Using df.apply without specifying axis defaults to axis=0 (columns), so x['A'] fails because x is a column name string, not a row.

2. Step 2: Compare with other options

   Options B, C, and D correctly double values using apply on Series, vectorized, or map on Series.

3. Final Answer:

   df['A'] = df.apply(lambda x: x['A'] * 2) -> Option C

4. Quick Check:

   apply on DataFrame needs axis=1 for row-wise, else error = A [OK]

Hint: Use axis=1 with apply on DataFrame rows to avoid errors [OK]

Common Mistakes:

• Forgetting axis=1 in DataFrame apply
• Confusing apply on DataFrame vs Series
• Using map on DataFrame instead of Series

5. You have a DataFrame df with columns 'X' and 'Y'. You want to create a new column 'Z' that is 'X' squared if 'Y' is even, else 'X' cubed. Which is the best approach?

hard

A. Use vectorized operations with np.where and powers.

B. Use apply with a custom function checking Y and computing Z.

C. Use a for loop to iterate rows and assign Z values.

D. Use map on column 'X' only.

Solution

1. Step 1: Understand the condition and operations

   We want Z = X^2 if Y is even, else Z = X^3.

2. Step 2: Choose efficient method

   Using np.where with vectorized power operations is fast and clear. apply works but is slower. For loops are slow and verbose. map on X alone can't check Y.

3. Final Answer:

   Use vectorized operations with np.where and powers. -> Option A

4. Quick Check:

   Conditional vectorized ops > apply or loops = D [OK]

Hint: Use np.where for conditional vectorized operations [OK]

Common Mistakes:

• Using apply when vectorized is faster
• Using loops instead of vectorized code
• Trying to map without condition on another column