Interpolation for missing values in Pandas - Mini Project: Build & Apply

Interpolation for missing values

📖 Scenario: You have collected daily temperature data for a week, but some days are missing values. You want to fill in these missing temperatures by estimating them based on the available data.

🎯 Goal: Build a small program that creates a pandas DataFrame with temperature data including missing values, sets up a method to fill those missing values using interpolation, applies the interpolation, and then shows the completed data.

📋 What You'll Learn

Create a pandas DataFrame named temps with a Day column and a Temperature column containing some missing values (NaN).

Create a variable called method and set it to the string 'linear' to specify the interpolation method.

Use the interpolate() method on the Temperature column of temps with the method variable to fill missing values.

Print the resulting DataFrame after interpolation.

💡 Why This Matters

🌍 Real World

Data collected from sensors or surveys often have missing values. Interpolation helps estimate these missing points to keep data analysis accurate.

💼 Career

Data scientists and analysts use interpolation to clean and prepare data before building models or reports.

1

Create the temperature data with missing values

Create a pandas DataFrame called temps with two columns: Day containing the days from 'Mon' to 'Sun', and Temperature containing these values exactly: 22.0, NaN, 24.5, NaN, 26.0, 27.5, NaN. Use import pandas as pd at the start.

Pandas

import pandas as pd
# Create the DataFrame temps with Day and Temperature columns
# Your code here

Hint

Use pd.DataFrame with a dictionary for columns. Use float('nan') for missing values.

2

Set the interpolation method

Create a variable called method and set it to the string 'linear' to specify the interpolation method.

Pandas

import pandas as pd

temps = pd.DataFrame({
    'Day': ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun'],
    'Temperature': [22.0, float('nan'), 24.5, float('nan'), 26.0, 27.5, float('nan')]
})

# Set the interpolation method to 'linear'
# Your code here

Hint

Just assign the string 'linear' to the variable method.

3

Apply interpolation to fill missing values

Use the interpolate() method on the Temperature column of temps with the method variable to fill missing values. Assign the result back to the Temperature column of temps.

Pandas

import pandas as pd

temps = pd.DataFrame({
    'Day': ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun'],
    'Temperature': [22.0, float('nan'), 24.5, float('nan'), 26.0, 27.5, float('nan')]
})

method = 'linear'

# Fill missing Temperature values using interpolate with method
# Your code here

Hint

Use temps['Temperature'].interpolate(method=method) and assign it back to temps['Temperature'].

4

Print the DataFrame after interpolation

Write a print statement to display the temps DataFrame after interpolation.

Pandas

import pandas as pd

temps = pd.DataFrame({
    'Day': ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun'],
    'Temperature': [22.0, float('nan'), 24.5, float('nan'), 26.0, 27.5, float('nan')]
})

method = 'linear'

temps['Temperature'] = temps['Temperature'].interpolate(method=method)

# Print the temps DataFrame
# Your code here

Hint

Use print(temps) to show the DataFrame.

Practice

1. What does the interpolate() method in pandas do when applied to a DataFrame with missing values?

easy

A. It fills missing values by estimating them based on nearby known values.

B. It deletes all rows containing missing values.

C. It replaces missing values with zeros.

D. It converts missing values to strings.

Solution

1. Step 1: Understand the purpose of interpolation

   Interpolation estimates missing data points by using values around them to create a smooth transition.

2. Step 2: Recognize what pandas interpolate() does

   The method fills missing values by calculating estimates based on existing nearby values, not by deleting or replacing with fixed values.

3. Final Answer:

   It fills missing values by estimating them based on nearby known values. -> Option A

4. Quick Check:

   Interpolation = estimate missing values [OK]

Hint: Interpolation estimates missing data using nearby values [OK]

Common Mistakes:

• Thinking it deletes rows with missing data
• Assuming it fills with zeros automatically
• Confusing interpolation with data type conversion

2. Which of the following is the correct syntax to interpolate missing values in a pandas DataFrame named df using the default method?

easy

A. df.interpolate(method='fill')

B. df.interpolate()

C. df.fillna(method='interpolate')

D. df.fillna(interpolate=True)

Solution

1. Step 1: Recall pandas interpolation syntax

   The method to fill missing values by interpolation is called directly as interpolate() on the DataFrame.

2. Step 2: Check each option for correctness

   df.interpolate() uses df.interpolate() which is correct. Options A, B, and D misuse fillna() or wrong parameters.

3. Final Answer:

   df.interpolate() -> Option B

4. Quick Check:

   Correct method call = df.interpolate() [OK]

Hint: Use df.interpolate() to fill missing values by default [OK]

Common Mistakes:

• Using fillna with wrong parameters
• Confusing interpolate with fillna method
• Adding invalid arguments to interpolate

3. Given the DataFrame df below, what will be the output of df.interpolate()?

import pandas as pd
import numpy as np
df = pd.DataFrame({'A': [1, np.nan, np.nan, 4, 5]})
print(df.interpolate())

medium

A. A
0 1.0
1 NaN
2 NaN
3 4.0
4 5.0

B. A
0 1.0
1 1.0
2 1.0
3 4.0
4 5.0

C. A
0 1.0
1 2.0
2 3.0
3 4.0
4 5.0

D. A
0 1.0
1 2.5
2 3.5
3 4.0
4 5.0

Solution

1. Step 1: Understand linear interpolation on missing values

   Linear interpolation fills missing values by evenly spacing values between known points. Here, missing values between 1 and 4 are filled as 2.0 and 3.0.

2. Step 2: Apply interpolation to the DataFrame

   Positions 1 and 2 are NaN. Interpolated values are 2.0 and 3.0 respectively, resulting in the sequence 1.0, 2.0, 3.0, 4.0, 5.0.

3. Final Answer:

   A 0 1.0 1 2.0 2 3.0 3 4.0 4 5.0 -> Option C

4. Quick Check:

   Linear interpolation fills gaps evenly [OK]

Hint: Linear interpolate fills gaps evenly between known values [OK]

Common Mistakes:

• Assuming missing values remain NaN
• Filling missing with previous value only
• Using wrong interpolation method

4. You run the code below but get an error. What is the most likely cause?

import pandas as pd
df = pd.DataFrame({'date': ['2023-01-01', '2023-01-02', '2023-01-03'], 'value': [1, None, 3]})
df['date'] = pd.to_datetime(df['date'])
df.interpolate(method='time')

medium

A. The DataFrame index is not set to the datetime column.

B. The 'value' column contains None instead of np.nan.

C. The 'date' column is not converted to datetime.

D. The interpolate method 'time' is not supported.

Solution

1. Step 1: Understand 'time' interpolation requirement

   The 'time' method requires the DataFrame index to be a datetime type to interpolate based on time differences.

2. Step 2: Check DataFrame index and columns

   Here, 'date' is a column but not set as index. Without datetime index, 'time' interpolation raises an error.

3. Final Answer:

   The DataFrame index is not set to the datetime column. -> Option A

4. Quick Check:

   Time interpolation needs datetime index [OK]

Hint: Set datetime column as index before time interpolation [OK]

Common Mistakes:

• Not setting datetime column as index
• Confusing None with np.nan (both work here)
• Assuming 'time' method unsupported

5. You have a DataFrame df with missing values in multiple columns and a datetime index. You want to fill missing values using time-based interpolation only on column 'temperature' without affecting others. Which code snippet correctly does this?

hard

A. df['temperature'].fillna(method='time')

B. df.interpolate(method='time', columns=['temperature'])

C. df.interpolate(method='time').loc[:, 'temperature']

D. df['temperature'] = df['temperature'].interpolate(method='time')

Solution

1. Step 1: Understand column-specific interpolation

   To interpolate only one column, call interpolate() on that column's Series.

2. Step 2: Check each option's correctness

   df['temperature'] = df['temperature'].interpolate(method='time') applies interpolation with method='time' directly on 'temperature' column and assigns back, which is correct. df.interpolate(method='time', columns=['temperature']) uses invalid 'columns' parameter. df.interpolate(method='time').loc[:, 'temperature'] interpolates whole DataFrame then selects column but does not assign back. df['temperature'].fillna(method='time') uses invalid fillna method.

3. Final Answer:

   df['temperature'] = df['temperature'].interpolate(method='time') -> Option D

4. Quick Check:

   Interpolate single column by calling method on that column [OK]

Hint: Interpolate single column by calling interpolate() on it [OK]

Common Mistakes:

• Using invalid 'columns' parameter in interpolate
• Not assigning interpolated result back to column
• Using fillna with method='time' which is invalid