When to use apply vs vectorized operations in Pandas

Introduction

We use apply and vectorized operations to change or analyze data in tables. Vectorized operations are faster and simpler, but apply is useful when we need custom or complex changes.

You want to quickly add or multiply columns in a table.

You need to apply a simple math operation to every value in a column.

You want to run a custom function that is not built-in on each row or column.

You have complex logic that depends on multiple columns and cannot be done with simple math.

You want to transform data row by row or column by column with a specific rule.

Syntax

Pandas

df['new_column'] = df['existing_column'].apply(function)
df['new_column'] = df['existing_column'] + 10  # vectorized example

apply runs a function on each element or row/column.

Vectorized operations use built-in fast math on whole columns at once.

Examples

This is a vectorized operation that doubles each value fast.

Pandas

df['double'] = df['value'] * 2

This does the same using apply and a custom function.

Pandas

df['double'] = df['value'].apply(lambda x: x * 2)

Use apply with axis=1 to check multiple columns in each row.

Pandas

df['category'] = df.apply(lambda row: 'high' if row['value'] > 10 else 'low', axis=1)

Sample Program

This code shows adding 10 to each number using vectorized operation, which is fast and simple.

Then it uses apply with a function to label values as 'high' or 'low'.

Pandas

import pandas as pd

data = {'value': [5, 15, 8, 20]}
df = pd.DataFrame(data)

# Vectorized operation: add 10 to each value
added = df['value'] + 10
print('Vectorized result:')
print(added)

# Apply with a custom function: label high or low
labels = df['value'].apply(lambda x: 'high' if x > 10 else 'low')
print('\nApply result:')
print(labels)

Important Notes

Vectorized operations are usually faster and preferred for simple math or built-in functions.

Use apply when you need custom logic that vectorized operations cannot do.

apply with axis=1 works row-wise, axis=0 works column-wise.

Summary

Use vectorized operations for simple, fast math on columns.

Use apply for custom or complex functions on rows or columns.

apply is slower but more flexible than vectorized operations.

Practice

1. Which of the following best describes when to use apply in pandas?

easy

A. When you want to merge two DataFrames.

B. When you want to perform simple arithmetic on entire columns quickly.

C. When you want to sort a DataFrame by a column.

D. When you need to run a custom function on each row or column.

Solution

1. Step 1: Understand the purpose of apply

   apply is used to apply custom or complex functions to rows or columns in a DataFrame.

2. Step 2: Compare with vectorized operations

   Vectorized operations are for simple, fast math on entire columns, not custom row-wise functions.

3. Final Answer:

   When you need to run a custom function on each row or column. -> Option D

4. Quick Check:

   Custom function use = apply = A [OK]

Hint: Use apply for custom row/column functions, vectorized for simple math [OK]

Common Mistakes:

• Confusing apply with vectorized operations for simple math
• Using apply for sorting or merging
• Thinking apply is always faster

2. Which of these is the correct syntax to use apply on a DataFrame column df['col'] with a function func?

easy

A. apply(df['col'], func)

B. df['col'].apply(func)

C. df.apply['col'](func)

D. df['col'].apply = func

Solution

1. Step 1: Recall the method call syntax

   In pandas, apply is a method called on a Series or DataFrame with the function as argument.

2. Step 2: Check each option

   df['col'].apply(func) uses correct syntax: df['col'].apply(func). Others have syntax errors or wrong usage.

3. Final Answer:

   df['col'].apply(func) -> Option B

4. Quick Check:

   Method call with function argument = C [OK]

Hint: Use dot apply with parentheses and function inside [OK]

Common Mistakes:

• Using square brackets instead of parentheses for apply
• Trying to assign function to apply attribute
• Calling apply as a standalone function

3. Given the DataFrame:

import pandas as pd
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

What is the output of:

df['C'] = df['A'] + df['B']
df['D'] = df['A'].apply(lambda x: x * 2)
df[['C', 'D']]

medium

A. C D
0 5 2
1 7 4
2 9 6

B. C D
0 5 1
1 7 2
2 9 3

C. C D
0 4 2
1 5 4
2 6 6

D. C D
0 5 4
1 7 6
2 9 8

Solution

1. Step 1: Calculate column C using vectorized addition

   Column C = A + B = [1+4, 2+5, 3+6] = [5, 7, 9]

2. Step 2: Calculate column D using apply with lambda

   Column D = A * 2 = [1*2, 2*2, 3*2] = [2, 4, 6]

3. Final Answer:

   C D 0 5 2 1 7 4 2 9 6 -> Option A

4. Quick Check:

   Vectorized + apply results = B [OK]

Hint: Add columns directly, use apply for custom row-wise ops [OK]

Common Mistakes:

• Mixing up which columns are added
• Forgetting to multiply by 2 in apply
• Confusing apply output with original values

4. You want to double the values in column 'A' of DataFrame df. Which code has an error?

medium

A. df['A'] = df['A'] * 2

B. df['A'] = df['A'].apply(lambda x: x * 2)

C. df['A'] = df.apply(lambda x: x['A'] * 2)

D. df['A'] = df['A'].map(lambda x: x * 2)

Solution

1. Step 1: Analyze df['A'] = df.apply(lambda x: x['A'] * 2)'s use of apply

   Using df.apply without specifying axis defaults to axis=0 (columns), so x['A'] fails because x is a column name string, not a row.

2. Step 2: Compare with other options

   Options B, C, and D correctly double values using apply on Series, vectorized, or map on Series.

3. Final Answer:

   df['A'] = df.apply(lambda x: x['A'] * 2) -> Option C

4. Quick Check:

   apply on DataFrame needs axis=1 for row-wise, else error = A [OK]

Hint: Use axis=1 with apply on DataFrame rows to avoid errors [OK]

Common Mistakes:

• Forgetting axis=1 in DataFrame apply
• Confusing apply on DataFrame vs Series
• Using map on DataFrame instead of Series

5. You have a DataFrame df with columns 'X' and 'Y'. You want to create a new column 'Z' that is 'X' squared if 'Y' is even, else 'X' cubed. Which is the best approach?

hard

A. Use vectorized operations with np.where and powers.

B. Use apply with a custom function checking Y and computing Z.

C. Use a for loop to iterate rows and assign Z values.

D. Use map on column 'X' only.

Solution

1. Step 1: Understand the condition and operations

   We want Z = X^2 if Y is even, else Z = X^3.

2. Step 2: Choose efficient method

   Using np.where with vectorized power operations is fast and clear. apply works but is slower. For loops are slow and verbose. map on X alone can't check Y.

3. Final Answer:

   Use vectorized operations with np.where and powers. -> Option A

4. Quick Check:

   Conditional vectorized ops > apply or loops = D [OK]

Hint: Use np.where for conditional vectorized operations [OK]

Common Mistakes:

• Using apply when vectorized is faster
• Using loops instead of vectorized code
• Trying to map without condition on another column