shift() for lagging data in Pandas - Mini Project: Build & Apply

Using shift() for Lagging Data in pandas

📖 Scenario: Imagine you are analyzing daily sales data for a small store. You want to compare each day's sales with the previous day's sales to see how sales change day by day.

🎯 Goal: You will create a pandas DataFrame with sales data, then use the shift() function to add a new column showing the previous day's sales. Finally, you will print the updated DataFrame.

📋 What You'll Learn

Create a pandas DataFrame with exact sales data for 5 days

Create a variable to hold the number of days to lag

Use the shift() function with the lag variable to create a new column

Print the final DataFrame showing original and lagged sales

💡 Why This Matters

🌍 Real World

Lagging data is useful in time series analysis to compare current values with past values, such as sales, stock prices, or weather data.

💼 Career

Data analysts and data scientists often use lagging data to find trends, calculate changes, and build predictive models.

1

Create the sales DataFrame

Create a pandas DataFrame called sales_data with two columns: 'Day' and 'Sales'. Use these exact values: Days are ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday'] and Sales are [200, 220, 210, 230, 240].

Pandas

# Create the sales_data DataFrame with Day and Sales columns
# Your code here

Hint

Use pd.DataFrame with a dictionary containing the two lists for 'Day' and 'Sales'.

2

Set the lag variable

Create a variable called lag_days and set it to 1. This will represent how many days to look back for lagging.

Pandas

import pandas as pd

sales_data = pd.DataFrame({
    'Day': ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday'],
    'Sales': [200, 220, 210, 230, 240]
})

# Create lag_days variable and set it to 1
# Your code here

Hint

Just create a variable named lag_days and assign it the value 1.

3

Create lagged sales column using shift()

Use the shift() function on the 'Sales' column of sales_data with the lag value lag_days. Store the result in a new column called 'Previous_Sales' in the sales_data DataFrame.

Pandas

import pandas as pd

sales_data = pd.DataFrame({
    'Day': ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday'],
    'Sales': [200, 220, 210, 230, 240]
})

lag_days = 1

# Create 'Previous_Sales' column using shift() with lag_days
# Your code here

Hint

Use sales_data['Sales'].shift(lag_days) to create the lagged sales and assign it to sales_data['Previous_Sales'].

4

Print the final DataFrame

Print the sales_data DataFrame to show the original sales and the previous day's sales side by side.

Pandas

import pandas as pd

sales_data = pd.DataFrame({
    'Day': ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday'],
    'Sales': [200, 220, 210, 230, 240]
})

lag_days = 1

sales_data['Previous_Sales'] = sales_data['Sales'].shift(lag_days)

# Print the sales_data DataFrame
# Your code here

Hint

Use print(sales_data) to display the DataFrame.

Practice

1. What does the shift() function do in pandas?

easy

A. It changes the data type of a column.

B. It moves data up or down to compare current and past values.

C. It removes missing values from the dataset.

D. It sorts the data in ascending order.

Solution

1. Step 1: Understand the purpose of shift()

   The shift() function moves data up or down in a column, creating a lag or lead effect.

2. Step 2: Compare with other options

   Sorting, removing missing values, or changing data types are different pandas functions, not shift().

3. Final Answer:

   It moves data up or down to compare current and past values. -> Option B

4. Quick Check:

   shift() moves data for lagging [OK]

Hint: Remember: shift() moves data to compare past values [OK]

Common Mistakes:

• Confusing shift() with sort()
• Thinking shift() removes missing data
• Assuming shift() changes data types

2. Which of the following is the correct syntax to shift a pandas Series named data by 2 rows downward?

easy

A. data.shift(2)

B. data.shift(-2)

C. data.shift(0)

D. data.shift()

Solution

1. Step 1: Understand shift() parameter

   The argument inside shift() is the number of periods to move. Positive values shift downward (lag), negative shift upward (lead).

2. Step 2: Apply to the question

   To shift downward by 2 rows, use data.shift(2). Negative or zero values do not match the requirement.

3. Final Answer:

   data.shift(2) -> Option A

4. Quick Check:

   Positive number shifts down = data.shift(2) [OK]

Hint: Positive number in shift() moves data down [OK]

Common Mistakes:

• Using negative number to shift downward
• Omitting the argument defaults to shift by 1
• Using zero which means no shift

3. Given the pandas Series:

import pandas as pd
data = pd.Series([10, 20, 30, 40, 50])
shifted = data.shift(1)

What is the value of shifted at index 2?

medium

A. NaN

B. 30.0

C. 20.0

D. 10.0

Solution

1. Step 1: Understand shift(1) effect on Series

   Shifting by 1 moves all values down by one index. The value at index 2 in shifted is the original value at index 1.

2. Step 2: Find original value at index 1

   Original data at index 1 is 20, so shifted at index 2 is 20.0 (float due to NaN presence).

3. Final Answer:

   20.0 -> Option C

4. Quick Check:

   shift(1) moves index 1 value to index 2 [OK]

Hint: Value at index i after shift(1) is original at i-1 [OK]

Common Mistakes:

• Confusing index positions after shift
• Expecting integer type instead of float
• Thinking shifted index 2 is original index 2

4. What is wrong with this code snippet?

import pandas as pd
data = pd.Series([5, 10, 15])
shifted = data.shift('1')

medium

A. shift() cannot be used on Series objects.

B. shift() requires a second argument for axis.

C. The data list must be a DataFrame, not a Series.

D. The argument to shift() should be an integer, not a string.

Solution

1. Step 1: Check shift() argument type

   The shift() function expects an integer or a timedelta, not a string.

2. Step 2: Identify error cause

   Passing a string like '1' causes a TypeError because pandas cannot interpret it as a number of periods.

3. Final Answer:

   The argument to shift() should be an integer, not a string. -> Option D

4. Quick Check:

   shift() needs integer periods, not string [OK]

Hint: Use integer, not string, inside shift() [OK]

Common Mistakes:

• Passing string instead of integer to shift()
• Thinking shift() only works on DataFrames
• Forgetting shift() default axis is 0

5. You have a DataFrame df with a column 'sales'. You want to create a new column 'sales_change' that shows the difference between current sales and sales one day before. Which code correctly achieves this?

hard

A. df['sales_change'] = df['sales'] - df['sales'].shift(1)

B. df['sales_change'] = df['sales'].shift(1) - df['sales']

C. df['sales_change'] = df['sales'].diff(-1)

D. df['sales_change'] = df['sales'].shift(-1) - df['sales']

Solution

1. Step 1: Understand the goal

   We want the change from previous day to current day, so current minus previous.

2. Step 2: Use shift(1) to get previous day sales

   df['sales'].shift(1) gives previous day's sales aligned with current row.

3. Step 3: Calculate difference

   Subtract previous sales from current sales: df['sales'] - df['sales'].shift(1).

4. Final Answer:

   df['sales_change'] = df['sales'] - df['sales'].shift(1) -> Option A

5. Quick Check:

   Current minus previous = sales_change [OK]

Hint: Subtract shifted column from current for change [OK]

Common Mistakes:

• Reversing subtraction order
• Using negative shift which looks ahead
• Using diff(-1) which is future difference