diff() for differences in Pandas - Step-by-Step Execution

Concept Flow - diff() for differences

Start with DataFrame or Series

↓

Call diff() method

↓

Calculate difference between current and previous row

↓

Return new DataFrame/Series with differences

↓

Use or display result

The diff() method calculates the difference between each element and the previous one in a DataFrame or Series, returning a new object showing these differences.

Execution Sample

Pandas

import pandas as pd
s = pd.Series([10, 15, 20, 18])
diff_s = s.diff()
print(diff_s)

This code creates a Series and uses diff() to find the difference between each value and the one before it.

Execution Table

Step	Index	Current Value	Previous Value	Difference (Current - Previous)
1	0	10	None	NaN (no previous value)
2	1	15	10	5
3	2	20	15	5
4	3	18	20	-2
5	End	No more rows		Stop calculation

💡 Reached end of Series, no more previous values to subtract.

Variable Tracker

Variable	Start	After 1	After 2	After 3	Final
s	[10, 15, 20, 18]	[10, 15, 20, 18]	[10, 15, 20, 18]	[10, 15, 20, 18]	[10, 15, 20, 18]
diff_s	[NaN, NaN, NaN, NaN]	[NaN, 5, NaN, NaN]	[NaN, 5, 5, NaN]	[NaN, 5, 5, -2]	[NaN, 5, 5, -2]

Key Moments - 3 Insights

Why is the first difference value NaN?

The first value has no previous value to subtract from, so diff() returns NaN as shown in execution_table step 1.

What happens if the Series has missing values?

diff() subtracts the immediately previous value by position; if the previous value is NaN, the difference will be NaN.

Can diff() be used on DataFrames?

Yes, diff() works on DataFrames by default calculating difference row-wise for each column.

Visual Quiz - 3 Questions

Look at the execution_table at step 3, what is the difference calculated?

A 5

B 10

C -5

D NaN

💡 Hint

Check the 'Difference' column at step 3 in the execution_table.

At which step does the difference become negative?

A Step 2

B Step 4

C Step 3

D Never

💡 Hint

Look at the 'Difference' values in the execution_table rows.

If the first value was 0 instead of 10, what would be the difference at step 2?

A 15

B 10

C 5

D NaN

💡 Hint

Difference is current minus previous; changing first value affects subsequent differences.

Concept Snapshot

diff() method in pandas:
- Calculates difference between current and previous row values
- Returns NaN for first element (no previous)
- Works on Series and DataFrames
- Default axis=0 (row-wise differences)
- Useful for time series and sequential data analysis

Full Transcript

The diff() method in pandas calculates the difference between each element and the one before it in a Series or DataFrame. The first element has no previous value, so its difference is NaN. This method helps find changes or trends in data over time or sequence. For example, given a Series of numbers, diff() subtracts each number from the one before it and returns a new Series with these differences. This is useful in many data science tasks like analyzing stock price changes or sensor readings.

Practice

1. What does the diff() function in pandas do when applied to a DataFrame column?

easy

A. Calculates the difference between each value and the previous value in the column

B. Calculates the sum of all values in the column

C. Sorts the column values in ascending order

D. Replaces missing values with zeros

Solution

1. Step 1: Understand the purpose of diff()

   The diff() function calculates the difference between each element and the one before it in the same column.

2. Step 2: Compare options with the function's behavior

   Only Calculates the difference between each value and the previous value in the column correctly describes this behavior. Other options describe different functions.

3. Final Answer:

   Calculates the difference between each value and the previous value in the column -> Option A

4. Quick Check:

   diff() = difference between consecutive values [OK]

Hint: Remember, diff() shows step-by-step changes in data [OK]

Common Mistakes:

• Thinking diff() sums values
• Confusing diff() with sorting
• Assuming diff() fills missing data

2. Which of the following is the correct syntax to calculate the difference between rows in a pandas DataFrame column named sales?

easy

A. df.diff('sales')

B. df['sales'].diff()

C. df.diff[sales]()

D. df['sales'].difference()

Solution

1. Step 1: Identify correct method call on a DataFrame column

   To calculate differences on a column, call diff() on the Series: df['sales'].diff().

2. Step 2: Check syntax of each option

   df['sales'].diff() uses correct syntax. df.diff('sales') wrongly passes column name to diff. df.diff[sales]() uses wrong brackets. df['sales'].difference() uses a non-existent method.

3. Final Answer:

   df['sales'].diff() -> Option B

4. Quick Check:

   Correct syntax = df['column'].diff() [OK]

Hint: Use diff() directly on the column Series like df['col'].diff() [OK]

Common Mistakes:

• Passing column name as argument to diff()
• Using square brackets instead of parentheses
• Calling non-existent methods like difference()

3. Given the DataFrame:

import pandas as pd
df = pd.DataFrame({'A': [10, 15, 20, 18]})
df['diff'] = df['A'].diff()

What is the content of the diff column?

medium

A. [NaN, 10, 15, 20]

B. [10, 15, 20, 18]

C. [NaN, 5.0, 5.0, -2.0]

D. [0, 5, 5, -2]

Solution

1. Step 1: Calculate differences between consecutive values in column 'A'

   Values: 10, 15, 20, 18. Differences: NaN (first row), 15-10=5, 20-15=5, 18-20=-2.

2. Step 2: Match calculated differences with options

   [NaN, 5.0, 5.0, -2.0] matches the list [NaN, 5.0, 5.0, -2.0]. Others do not.

3. Final Answer:

   [NaN, 5.0, 5.0, -2.0] -> Option C

4. Quick Check:

   diff() = [NaN, 5, 5, -2] [OK]

Hint: First diff is always NaN; subtract previous value from current [OK]

Common Mistakes:

• Assuming first diff is zero instead of NaN
• Confusing original values with differences
• Ignoring negative differences

4. You run this code:

import pandas as pd
df = pd.DataFrame({'X': [3, 7, 2]})
df['change'] = df.diff(axis=1)

But you get an error or unexpected output. What is the problem?

medium

A. diff() cannot be used with axis=1 on a single-column DataFrame

B. You must specify the period argument in diff()

C. The DataFrame must have numeric column names for diff()

D. diff() requires the DataFrame to have at least 5 rows

Solution

1. Step 1: Understand axis parameter in diff()

   axis=1 means calculate difference across columns (left to right). With only one column, no previous column exists to subtract.

2. Step 2: Identify why error or unexpected output occurs

   Since only one column exists, diff(axis=1) returns NaN or error because no previous column to compare.

3. Final Answer:

   diff() cannot be used with axis=1 on a single-column DataFrame -> Option A

4. Quick Check:

   diff(axis=1) needs multiple columns [OK]

Hint: diff(axis=1) needs 2+ columns; single column causes NaN/error [OK]

Common Mistakes:

• Thinking period argument is mandatory
• Assuming column names must be numeric
• Believing diff() needs many rows

5. You have a DataFrame showing daily temperatures:

import pandas as pd
data = {'Day': ['Mon', 'Tue', 'Wed', 'Thu'], 'Temp': [20, 22, 21, 25]}
df = pd.DataFrame(data)

You want to add a column Temp_change showing how much temperature changed from the previous day. Which code correctly does this?

hard

A. df['Temp_change'] = df['Temp'].cumsum()

B. df['Temp_change'] = df['Temp'].diff(periods=2)

C. df['Temp_change'] = df['Temp'].shift()

D. df['Temp_change'] = df['Temp'].diff()

Solution

1. Step 1: Identify how to calculate daily temperature change

   Temperature change from previous day is difference between current and previous value, which diff() calculates by default with period=1.

2. Step 2: Evaluate each option

   df['Temp_change'] = df['Temp'].diff() uses diff() correctly. df['Temp_change'] = df['Temp'].cumsum() sums temperatures, not differences. df['Temp_change'] = df['Temp'].shift() shifts values but does not calculate difference. df['Temp_change'] = df['Temp'].diff(periods=2) calculates difference over 2 days, not 1.

3. Final Answer:

   df['Temp_change'] = df['Temp'].diff() -> Option D

4. Quick Check:

   Use diff() for stepwise changes [OK]

Hint: Use diff() without arguments for previous row difference [OK]

Common Mistakes:

• Using cumsum() instead of diff()
• Using shift() which only moves data
• Using diff(periods=2) for daily change