Forward fill and backward fill in Pandas

Introduction

Sometimes data has missing values. Forward fill and backward fill help fill these gaps using nearby known values.

You have a time series with missing measurements and want to fill gaps with the last known value.

You want to fill missing data in a table by copying the next available value backward.

You need to prepare data for analysis or visualization without losing rows due to missing values.

You want to keep data continuity by filling missing entries with nearby known values.

You want a simple way to handle missing data before applying calculations.

Syntax

Pandas

DataFrame.fillna(method='ffill')  # forward fill
DataFrame.fillna(method='bfill')  # backward fill

ffill means forward fill: fill missing values with the last known value above.

bfill means backward fill: fill missing values with the next known value below.

Examples

Fill missing values by copying the last known value forward.

Pandas

df.fillna(method='ffill')

Fill missing values by copying the next known value backward.

Pandas

df.fillna(method='bfill')

Forward fill but only fill up to 1 consecutive missing value.

Pandas

df.fillna(method='ffill', limit=1)

Backward fill along columns instead of rows.

Pandas

df.fillna(method='bfill', axis=1)

Sample Program

This code creates a table with some missing numbers. It shows the original table, then fills missing spots by copying the last known value down (forward fill), and then by copying the next known value up (backward fill).

Pandas

import pandas as pd
import numpy as np

# Create a DataFrame with missing values
data = {'A': [1, np.nan, np.nan, 4, 5],
         'B': [np.nan, 2, np.nan, 4, np.nan],
         'C': [1, 2, 3, np.nan, 5]}

df = pd.DataFrame(data)

print('Original DataFrame:')
print(df)

# Forward fill missing values
ffill_df = df.fillna(method='ffill')
print('\nDataFrame after forward fill:')
print(ffill_df)

# Backward fill missing values
bfill_df = df.fillna(method='bfill')
print('\nDataFrame after backward fill:')
print(bfill_df)

Important Notes

If missing values are at the start (for forward fill) or end (for backward fill), they may remain missing because there is no value to copy.

You can use the limit parameter to control how many consecutive missing values get filled.

Forward and backward fill work well for ordered data like time series.

Summary

Forward fill copies the last known value down to fill missing spots.

Backward fill copies the next known value up to fill missing spots.

These methods help keep data complete for analysis without guessing new values.

Practice

1. What does the forward fill method do in pandas when handling missing data?

easy

A. It copies the next known value backward to fill missing spots.

B. It copies the last known value forward to fill missing spots.

C. It replaces missing values with zero.

D. It removes rows with missing values.

Solution

1. Step 1: Understand forward fill concept

   Forward fill means filling missing data by copying the last valid value forward.

2. Step 2: Compare options with definition

   It copies the last known value forward to fill missing spots. matches this exactly, while others describe different methods.

3. Final Answer:

   It copies the last known value forward to fill missing spots. -> Option B

4. Quick Check:

   Forward fill = copy last known value forward [OK]

Hint: Forward fill copies previous value downwards [OK]

Common Mistakes:

• Confusing forward fill with backward fill
• Thinking forward fill removes missing data
• Assuming forward fill replaces with zero

2. Which of the following is the correct pandas syntax to perform a backward fill on a DataFrame df?

easy

A. df.fillna(method='backfill')

B. df.fillna(method='ffill')

C. df.fillna(backward=True)

D. df.fillna(direction='backward')

Solution

1. Step 1: Recall pandas fillna syntax

   Backward fill uses method='backfill' or method='bfill' in fillna.

2. Step 2: Check options for correct syntax

   df.fillna(method='backfill') uses method='backfill', which is correct. Others are invalid or incorrect syntax.

3. Final Answer:

   df.fillna(method='backfill') -> Option A

4. Quick Check:

   Backward fill syntax = method='backfill' [OK]

Hint: Use method='backfill' for backward fill in fillna [OK]

Common Mistakes:

• Using method='ffill' for backward fill
• Passing invalid parameters like backward=True
• Confusing method names in fillna

3. Given the DataFrame:

import pandas as pd
df = pd.DataFrame({'A': [1, None, None, 4, None]})
df_filled = df.fillna(method='ffill')

What is the content of df_filled?

medium

A. [1, None, None, 4, 1]

B. [1, 1, 1, 4, None]

C. [1, None, None, 4, None]

D. [1, 1, 1, 4, 4]

Solution

1. Step 1: Understand forward fill on the DataFrame

   Forward fill copies last valid value downwards to fill missing values.

2. Step 2: Apply forward fill step-by-step

   Index 0: 1 (valid)
   Index 1: filled with 1
   Index 2: filled with 1
   Index 3: 4 (valid)
   Index 4: filled with 4

3. Final Answer:

   [1, 1, 1, 4, 4] -> Option D

4. Quick Check:

   Forward fill propagates last valid value forward [OK]

Hint: Forward fill fills missing with last valid above [OK]

Common Mistakes:

• Thinking forward fill leaves trailing NAs unfilled
• Failing to propagate through consecutive NAs
• Confusing forward fill with backward fill

4. Consider this code snippet:

import pandas as pd
df = pd.DataFrame({'B': [None, 2, None, 4]})
df.fillna(method='bfill', inplace=True)

After running this, what is the content of column 'B' in df?

medium

A. [2, 2, 4, 4]

B. [None, 2, None, 4]

C. [2, 2, 2, 4]

D. [None, 2, 4, 4]

Solution

1. Step 1: Understand backward fill with inplace=True

   Backward fill copies next valid value upwards, modifying df directly.

2. Step 2: Apply backward fill to column 'B'

   Index 0: None filled with next valid 2
   Index 1: 2 (valid)
   Index 2: None filled with next valid 4
   Index 3: 4 (valid)

3. Final Answer:

   [2, 2, 4, 4] -> Option A

4. Quick Check:

   Backward fill copies next valid value up [OK]

Hint: Backward fill copies next valid value upward [OK]

Common Mistakes:

• Not using inplace=True and expecting df to change
• Confusing backward fill with forward fill
• Leaving None values unchanged

5. You have a DataFrame df with missing values scattered in multiple columns. You want to fill missing values by first forward filling, then backward filling any remaining missing values. Which code snippet correctly achieves this?

hard

A. df.fillna(method='ffill', method='bfill')

B. df.fillna(method='bfill').fillna(method='ffill')

C. df.fillna(method='ffill').fillna(method='bfill')

D. df.fillna(method=['ffill', 'bfill'])

Solution

1. Step 1: Understand chaining fillna methods

   First forward fill fills missing values from previous valid data, then backward fill fills remaining missing values from next valid data.

2. Step 2: Analyze options for correct chaining

   df.fillna(method='ffill').fillna(method='bfill') chains fillna with method='ffill' then method='bfill', which is correct.
   df.fillna(method='bfill').fillna(method='ffill') reverses order, which may fill differently.
   Options C and D use invalid syntax.

3. Final Answer:

   df.fillna(method='ffill').fillna(method='bfill') -> Option C

4. Quick Check:

   Chain forward then backward fill to cover all missing [OK]

Hint: Chain fillna with ffill then bfill to fill all gaps [OK]

Common Mistakes:

• Using invalid syntax with multiple methods in one fillna
• Reversing order and missing some fills
• Expecting one fillna call to do both fills