Shift and lag operations in Data Analysis Python - Mini Project: Build & Apply

Shift and lag operations

📖 Scenario: You work in a sales department. You have daily sales data for a small store. You want to compare each day's sales with the previous day's sales to see if sales went up or down.

🎯 Goal: Build a small program that uses shift and lag operations on sales data to compare each day's sales with the previous day.

📋 What You'll Learn

Create a pandas DataFrame with daily sales data.

Add a new column that shows the previous day's sales using shift.

Calculate the difference between current day sales and previous day sales.

Print the final DataFrame showing the comparison.

💡 Why This Matters

🌍 Real World

Shift and lag operations are used in time series analysis to compare current values with past values, such as daily sales, stock prices, or sensor readings.

💼 Career

Data analysts and data scientists use shift and lag operations to detect trends, calculate changes, and prepare data for forecasting models.

1

Create the sales data DataFrame

Create a pandas DataFrame called sales_data with two columns: 'date' and 'sales'. Use these exact values for 'date': '2024-04-01', '2024-04-02', '2024-04-03', '2024-04-04', '2024-04-05'. Use these exact values for 'sales': 100, 150, 130, 160, 170. Import pandas as pd first.

Data Analysis Python

import pandas as pd
# Create the sales_data DataFrame with the exact dates and sales values
# Your code here

Hint

Use pd.DataFrame with a dictionary containing the 'date' and 'sales' lists.

2

Add a column for previous day's sales

Create a new column in sales_data called 'prev_day_sales' that contains the previous day's sales using the shift(1) method on the 'sales' column.

Data Analysis Python

import pandas as pd
sales_data = pd.DataFrame({
    'date': ['2024-04-01', '2024-04-02', '2024-04-03', '2024-04-04', '2024-04-05'],
    'sales': [100, 150, 130, 160, 170]
})
# Add the 'prev_day_sales' column using shift(1) on 'sales'
# Your code here

Hint

Use sales_data['prev_day_sales'] = sales_data['sales'].shift(1) to get the previous day's sales.

3

Calculate sales difference from previous day

Create a new column in sales_data called 'sales_diff' that calculates the difference between 'sales' and 'prev_day_sales'.

Data Analysis Python

import pandas as pd
sales_data = pd.DataFrame({
    'date': ['2024-04-01', '2024-04-02', '2024-04-03', '2024-04-04', '2024-04-05'],
    'sales': [100, 150, 130, 160, 170]
})
sales_data['prev_day_sales'] = sales_data['sales'].shift(1)
# Calculate the difference between 'sales' and 'prev_day_sales' in 'sales_diff'
# Your code here

Hint

Subtract sales_data['prev_day_sales'] from sales_data['sales'] and assign to sales_diff.

4

Print the final DataFrame

Print the sales_data DataFrame to show the 'date', 'sales', 'prev_day_sales', and 'sales_diff' columns.

Data Analysis Python

import pandas as pd
sales_data = pd.DataFrame({
    'date': ['2024-04-01', '2024-04-02', '2024-04-03', '2024-04-04', '2024-04-05'],
    'sales': [100, 150, 130, 160, 170]
})
sales_data['prev_day_sales'] = sales_data['sales'].shift(1)
sales_data['sales_diff'] = sales_data['sales'] - sales_data['prev_day_sales']
# Print the sales_data DataFrame
# Your code here

Hint

Use print(sales_data) to display the DataFrame.

Practice

1. What does the shift(1) function do to a pandas DataFrame column?

easy

A. Moves the data down by one row, creating a lag of previous values

B. Moves the data up by one row, creating a lead of next values

C. Sorts the column in ascending order

D. Removes the first row of the column

Solution

1. Step 1: Understand the shift function

   The shift(1) moves data down by one row, so each value is replaced by the previous row's value.

2. Step 2: Identify lag vs lead

   Positive shift values create lag (previous rows), so shift(1) creates lag by one row.

3. Final Answer:

   Moves the data down by one row, creating a lag of previous values -> Option A

4. Quick Check:

   shift(1) = lag by 1 row [OK]

Hint: Positive shift means lag (previous rows), negative means lead [OK]

Common Mistakes:

• Confusing shift(1) with shift(-1)
• Thinking shift sorts data
• Assuming shift removes rows

2. Which of the following is the correct syntax to create a new column 'prev_value' that contains the previous row's value of column 'value' in pandas?

easy

A. df['prev_value'] = df['value'].shift(-1)

B. df['prev_value'] = shift(df['value'], 1)

C. df['prev_value'] = df.shift('value', 1)

D. df['prev_value'] = df['value'].shift(1)

Solution

1. Step 1: Recall pandas shift syntax

   The correct syntax is to call shift() on the Series, like df['value'].shift(1).

2. Step 2: Check each option

   df['prev_value'] = df['value'].shift(1) uses correct syntax. df['prev_value'] = df['value'].shift(-1) shifts in the wrong direction. df['prev_value'] = df.shift('value', 1) uses wrong method call. df['prev_value'] = shift(df['value'], 1) uses undefined function.

3. Final Answer:

   df['prev_value'] = df['value'].shift(1) -> Option D

4. Quick Check:

   Correct pandas shift syntax = df['prev_value'] = df['value'].shift(1) [OK]

Hint: Use Series.shift(n) to lag by n rows [OK]

Common Mistakes:

• Using negative shift for previous row
• Calling shift on DataFrame with wrong arguments
• Using undefined shift function

3. Given the DataFrame:

import pandas as pd
df = pd.DataFrame({'value': [10, 20, 30, 40]})
df['lag1'] = df['value'].shift(1)
df['lead1'] = df['value'].shift(-1)
print(df)

What is the output?

medium

A. value lag1 lead1
0 10 10.0 20.0
1 20 20.0 30.0
2 30 30.0 40.0
3 40 40.0 NaN

B. value lag1 lead1
0 10 20.0 NaN
1 20 30.0 10.0
2 30 40.0 20.0
3 40 NaN 30.0

C. value lag1 lead1
0 10 NaN 20.0
1 20 10.0 30.0
2 30 20.0 40.0
3 40 30.0 NaN

D. value lag1 lead1
0 10 NaN NaN
1 20 10.0 20.0
2 30 20.0 30.0
3 40 30.0 40.0

Solution

1. Step 1: Calculate lag1 column

   shift(1) moves values down by one, so first row is NaN, second row gets 10, third 20, fourth 30.

2. Step 2: Calculate lead1 column

   shift(-1) moves values up by one, so last row is NaN, third row gets 40, second 30, first 20.

3. Final Answer:

   value lag1 lead1 0 10 NaN 20.0 1 20 10.0 30.0 2 30 20.0 40.0 3 40 30.0 NaN -> Option C

4. Quick Check:

   Positive shift = lag, negative shift = lead [OK]

Hint: Positive shift adds NaN on top, negative shift adds NaN at bottom [OK]

Common Mistakes:

• Mixing lag and lead directions
• Forgetting NaN appears at shifted edges
• Confusing shift with sorting

4. What is wrong with this code snippet?

df['prev'] = df.shift(1)['value']

Assuming df is a pandas DataFrame with a column 'value'.

medium

A. It works correctly and creates a lagged column

B. It shifts the entire DataFrame, not just the 'value' column, which is inefficient

C. It causes a syntax error due to wrong indexing

D. It will cause a KeyError because shift returns a Series, not DataFrame

Solution

1. Step 1: Understand what df.shift(1) does

   df.shift(1) shifts the entire DataFrame down by one row, including all columns.

2. Step 2: Check indexing and efficiency

   Accessing ['value'] after shift works, but shifting entire DataFrame is inefficient if only one column is needed.

3. Final Answer:

   It shifts the entire DataFrame, not just the 'value' column, which is inefficient -> Option B

4. Quick Check:

   Shift on DataFrame shifts all columns [OK]

Hint: Shift only the needed column, not whole DataFrame [OK]

Common Mistakes:

• Assuming shift returns Series when called on DataFrame
• Expecting syntax error from indexing
• Ignoring performance impact

5. You have a time series DataFrame df with a column 'sales'. You want to create a new column 'pct_change' that shows the percentage change from the previous day. Which code correctly uses shift to calculate this?

hard

A. df['pct_change'] = (df['sales'] - df['sales'].shift(1)) / df['sales'].shift(1) * 100

B. df['pct_change'] = (df['sales'].shift(1) - df['sales']) / df['sales'] * 100

C. df['pct_change'] = df['sales'].pct_change(1)

D. df['pct_change'] = df['sales'].shift(-1) / df['sales'] * 100

Solution

1. Step 1: Understand percentage change formula

   Percentage change = (current - previous) / previous * 100.

2. Step 2: Apply shift to get previous value

   df['sales'].shift(1) gives previous day's sales, so formula in df['pct_change'] = (df['sales'] - df['sales'].shift(1)) / df['sales'].shift(1) * 100 matches the formula.

3. Step 3: Check other options

   df['pct_change'] = (df['sales'].shift(1) - df['sales']) / df['sales'] * 100 reverses numerator and denominator incorrectly. df['pct_change'] = df['sales'].pct_change(1) uses built-in method but question asks for shift usage. df['pct_change'] = df['sales'].shift(-1) / df['sales'] * 100 uses lead instead of lag.

4. Final Answer:

   df['pct_change'] = (df['sales'] - df['sales'].shift(1)) / df['sales'].shift(1) * 100 -> Option A

5. Quick Check:

   Percentage change uses lag with shift(1) [OK]

Hint: Use (current - shift(1)) / shift(1) for percent change [OK]

Common Mistakes:

• Using shift(-1) instead of shift(1)
• Reversing numerator and denominator
• Confusing built-in pct_change with manual shift