Forward fill and backward fill in Pandas - Step-by-Step Execution

Concept Flow - Forward fill and backward fill

Start with DataFrame with missing values

↓

Choose fill method: forward fill or backward fill

↓

Forward fill

↓

Replace NaN with

↓

previous valid

↓

Return filled DataFrame

The process starts with a DataFrame containing missing values. You choose to fill missing values either by carrying forward the last known value (forward fill) or by using the next known value (backward fill).

Execution Sample

Pandas

import pandas as pd

df = pd.DataFrame({'A': [1, None, None, 4, None, 6]})
ffill = df.fillna(method='ffill')
bfill = df.fillna(method='bfill')

This code creates a DataFrame with missing values and fills them using forward fill and backward fill methods.

Execution Table

Step	DataFrame 'A' values	Action	Resulting 'A' values
1	[1, NaN, NaN, 4, NaN, 6]	Start with original data	[1, NaN, NaN, 4, NaN, 6]
2	[1, NaN, NaN, 4, NaN, 6]	Apply forward fill	[1, 1, 1, 4, 4, 6]
3	[1, NaN, NaN, 4, NaN, 6]	Apply backward fill	[1, 4, 4, 4, 6, 6]
4	[1, NaN, NaN, 4, NaN, 6]	End of filling operations	N/A

💡 All missing values replaced by forward or backward fill; no NaNs remain after fill.

Variable Tracker

Variable	Start	After forward fill	After backward fill
df['A']	[1, NaN, NaN, 4, NaN, 6]	[1, 1, 1, 4, 4, 6]	[1, 4, 4, 4, 6, 6]

Key Moments - 3 Insights

Why does forward fill replace NaN at index 1 with 1 instead of 4?

Forward fill replaces NaN with the last valid value seen before it. At index 1, the last valid value is 1 at index 0, so it fills with 1 (see execution_table row 2).

Why does backward fill replace NaN at index 1 with 4 instead of 1?

Backward fill replaces NaN with the next valid value after it. At index 1, the next valid value is 4 at index 3, so it fills with 4 (see execution_table row 3).

What happens if the first value is NaN and we use forward fill?

Forward fill cannot fill NaN at the start because there is no previous valid value. The NaN remains unchanged unless a different method is used.

Visual Quiz - 3 Questions

Look at the execution table at step 2. What is the value at index 4 after forward fill?

A 4

B NaN

C 6

D 1

💡 Hint

Check the 'Resulting A values' column at step 2 in the execution_table.

At which step does backward fill replace the NaN at index 2?

A Step 2

B Step 3

C Step 1

D Step 4

💡 Hint

Look at the 'Action' column and the changes in 'A' values in execution_table rows.

If the first value in column 'A' was NaN, what would forward fill do to it?

A Replace it with the next valid value

B Replace it with zero

C Leave it as NaN

D Replace it with the last valid value

💡 Hint

Refer to key_moments about forward fill behavior at the start of data.

Concept Snapshot

Forward fill (ffill) replaces missing values with the last known value above.
Backward fill (bfill) replaces missing values with the next known value below.
Use df.fillna(method='ffill') or df.fillna(method='bfill').
Missing values at start (ffill) or end (bfill) may remain if no valid value exists.
Useful for filling gaps in time series or ordered data.

Full Transcript

Forward fill and backward fill are methods to fill missing data in pandas DataFrames. Forward fill replaces missing values with the last valid value seen before them, moving down the column. Backward fill replaces missing values with the next valid value seen after them, moving up the column. In the example, a DataFrame with missing values is filled using both methods. Forward fill fills NaNs by carrying the previous value forward, while backward fill fills NaNs by carrying the next value backward. If missing values appear at the start (for forward fill) or at the end (for backward fill) with no valid values to fill from, those NaNs remain. These methods help clean data for analysis, especially when data is ordered like time series.

Practice

1. What does the forward fill method do in pandas when handling missing data?

easy

A. It copies the next known value backward to fill missing spots.

B. It copies the last known value forward to fill missing spots.

C. It replaces missing values with zero.

D. It removes rows with missing values.

Solution

1. Step 1: Understand forward fill concept

   Forward fill means filling missing data by copying the last valid value forward.

2. Step 2: Compare options with definition

   It copies the last known value forward to fill missing spots. matches this exactly, while others describe different methods.

3. Final Answer:

   It copies the last known value forward to fill missing spots. -> Option B

4. Quick Check:

   Forward fill = copy last known value forward [OK]

Hint: Forward fill copies previous value downwards [OK]

Common Mistakes:

• Confusing forward fill with backward fill
• Thinking forward fill removes missing data
• Assuming forward fill replaces with zero

2. Which of the following is the correct pandas syntax to perform a backward fill on a DataFrame df?

easy

A. df.fillna(method='backfill')

B. df.fillna(method='ffill')

C. df.fillna(backward=True)

D. df.fillna(direction='backward')

Solution

1. Step 1: Recall pandas fillna syntax

   Backward fill uses method='backfill' or method='bfill' in fillna.

2. Step 2: Check options for correct syntax

   df.fillna(method='backfill') uses method='backfill', which is correct. Others are invalid or incorrect syntax.

3. Final Answer:

   df.fillna(method='backfill') -> Option A

4. Quick Check:

   Backward fill syntax = method='backfill' [OK]

Hint: Use method='backfill' for backward fill in fillna [OK]

Common Mistakes:

• Using method='ffill' for backward fill
• Passing invalid parameters like backward=True
• Confusing method names in fillna

3. Given the DataFrame:

import pandas as pd
df = pd.DataFrame({'A': [1, None, None, 4, None]})
df_filled = df.fillna(method='ffill')

What is the content of df_filled?

medium

A. [1, None, None, 4, 1]

B. [1, 1, 1, 4, None]

C. [1, None, None, 4, None]

D. [1, 1, 1, 4, 4]

Solution

1. Step 1: Understand forward fill on the DataFrame

   Forward fill copies last valid value downwards to fill missing values.

2. Step 2: Apply forward fill step-by-step

   Index 0: 1 (valid)
   Index 1: filled with 1
   Index 2: filled with 1
   Index 3: 4 (valid)
   Index 4: filled with 4

3. Final Answer:

   [1, 1, 1, 4, 4] -> Option D

4. Quick Check:

   Forward fill propagates last valid value forward [OK]

Hint: Forward fill fills missing with last valid above [OK]

Common Mistakes:

• Thinking forward fill leaves trailing NAs unfilled
• Failing to propagate through consecutive NAs
• Confusing forward fill with backward fill

4. Consider this code snippet:

import pandas as pd
df = pd.DataFrame({'B': [None, 2, None, 4]})
df.fillna(method='bfill', inplace=True)

After running this, what is the content of column 'B' in df?

medium

A. [2, 2, 4, 4]

B. [None, 2, None, 4]

C. [2, 2, 2, 4]

D. [None, 2, 4, 4]

Solution

1. Step 1: Understand backward fill with inplace=True

   Backward fill copies next valid value upwards, modifying df directly.

2. Step 2: Apply backward fill to column 'B'

   Index 0: None filled with next valid 2
   Index 1: 2 (valid)
   Index 2: None filled with next valid 4
   Index 3: 4 (valid)

3. Final Answer:

   [2, 2, 4, 4] -> Option A

4. Quick Check:

   Backward fill copies next valid value up [OK]

Hint: Backward fill copies next valid value upward [OK]

Common Mistakes:

• Not using inplace=True and expecting df to change
• Confusing backward fill with forward fill
• Leaving None values unchanged

5. You have a DataFrame df with missing values scattered in multiple columns. You want to fill missing values by first forward filling, then backward filling any remaining missing values. Which code snippet correctly achieves this?

hard

A. df.fillna(method='ffill', method='bfill')

B. df.fillna(method='bfill').fillna(method='ffill')

C. df.fillna(method='ffill').fillna(method='bfill')

D. df.fillna(method=['ffill', 'bfill'])

Solution

1. Step 1: Understand chaining fillna methods

   First forward fill fills missing values from previous valid data, then backward fill fills remaining missing values from next valid data.

2. Step 2: Analyze options for correct chaining

   df.fillna(method='ffill').fillna(method='bfill') chains fillna with method='ffill' then method='bfill', which is correct.
   df.fillna(method='bfill').fillna(method='ffill') reverses order, which may fill differently.
   Options C and D use invalid syntax.

3. Final Answer:

   df.fillna(method='ffill').fillna(method='bfill') -> Option C

4. Quick Check:

   Chain forward then backward fill to cover all missing [OK]

Hint: Chain fillna with ffill then bfill to fill all gaps [OK]

Common Mistakes:

• Using invalid syntax with multiple methods in one fillna
• Reversing order and missing some fills
• Expecting one fillna call to do both fills