shift() for lagging data in Pandas - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does the shift() function do in pandas?

The shift() function moves the data in a column up or down by a specified number of rows, creating lagged or lead data. It is often used to compare current values with past or future values.

beginner

How do you create a new column with the previous day's value using shift()?

Use df['previous_day'] = df['value'].shift(1). This moves the 'value' column down by one row, so each row shows the value from the previous day.

beginner

What happens to the first row when you use shift(1) on a column?

The first row becomes NaN because there is no previous row to fill it with. This is normal when creating lagged data.

intermediate

Can shift() be used to lead data instead of lag data? How?

Yes. Use a negative number like shift(-1) to move data up, creating lead data (future values).

beginner

Why is lagging data useful in data science?

Lagging data helps compare current values with past values. It is useful for time series analysis, trend detection, and building features for models that depend on past information.

What does df['value'].shift(2) do?

A Moves the 'value' column down by 2 rows, creating lagged data.

B Moves the 'value' column up by 2 rows, creating lead data.

C Deletes 2 rows from the 'value' column.

D Duplicates the 'value' column twice.

shift(2) moves data down by 2 rows, so each row shows the value from 2 rows above.

What value appears in the first row after applying shift(1)?

A NaN (missing value)

B The original first row value

C Zero

D The last row value

The first row has no previous row to fill from, so it becomes NaN.

How do you create a column with next day's data using shift()?

A <code>shift(1)</code>

B <code>shift()</code> without arguments

C <code>shift(-1)</code>

D <code>shift(0)</code>

Using shift(-1) moves data up by one row, showing next day's data.

Which of these is a common use of lagged data?

A Changing data types

B Comparing current sales with past sales

C Sorting data alphabetically

D Removing duplicates

Lagged data helps compare current values with past values, useful in time series.

If you want to compare today's temperature with the temperature 3 days ago, which code is correct?

A df['temp_3days_ago'] = df['temperature']

B df['temp_3days_ago'] = df['temperature'].shift(-3)

C df['temp_3days_ago'] = df['temperature'].shift(1)

D df['temp_3days_ago'] = df['temperature'].shift(3)

Shift by 3 moves data down 3 rows, showing temperature from 3 days ago.

Explain how the shift() function helps create lagged data in pandas and why this is useful.

Think about how you compare today's data with yesterday's.

Describe what happens to the first few rows of a DataFrame column when you apply shift(1) and how you might handle those values.

Consider what data is missing after shifting.

Practice

1. What does the shift() function do in pandas?

easy

A. It changes the data type of a column.

B. It moves data up or down to compare current and past values.

C. It removes missing values from the dataset.

D. It sorts the data in ascending order.

Solution

1. Step 1: Understand the purpose of shift()

   The shift() function moves data up or down in a column, creating a lag or lead effect.

2. Step 2: Compare with other options

   Sorting, removing missing values, or changing data types are different pandas functions, not shift().

3. Final Answer:

   It moves data up or down to compare current and past values. -> Option B

4. Quick Check:

   shift() moves data for lagging [OK]

Hint: Remember: shift() moves data to compare past values [OK]

Common Mistakes:

• Confusing shift() with sort()
• Thinking shift() removes missing data
• Assuming shift() changes data types

2. Which of the following is the correct syntax to shift a pandas Series named data by 2 rows downward?

easy

A. data.shift(2)

B. data.shift(-2)

C. data.shift(0)

D. data.shift()

Solution

1. Step 1: Understand shift() parameter

   The argument inside shift() is the number of periods to move. Positive values shift downward (lag), negative shift upward (lead).

2. Step 2: Apply to the question

   To shift downward by 2 rows, use data.shift(2). Negative or zero values do not match the requirement.

3. Final Answer:

   data.shift(2) -> Option A

4. Quick Check:

   Positive number shifts down = data.shift(2) [OK]

Hint: Positive number in shift() moves data down [OK]

Common Mistakes:

• Using negative number to shift downward
• Omitting the argument defaults to shift by 1
• Using zero which means no shift

3. Given the pandas Series:

import pandas as pd
data = pd.Series([10, 20, 30, 40, 50])
shifted = data.shift(1)

What is the value of shifted at index 2?

medium

A. NaN

B. 30.0

C. 20.0

D. 10.0

Solution

1. Step 1: Understand shift(1) effect on Series

   Shifting by 1 moves all values down by one index. The value at index 2 in shifted is the original value at index 1.

2. Step 2: Find original value at index 1

   Original data at index 1 is 20, so shifted at index 2 is 20.0 (float due to NaN presence).

3. Final Answer:

   20.0 -> Option C

4. Quick Check:

   shift(1) moves index 1 value to index 2 [OK]

Hint: Value at index i after shift(1) is original at i-1 [OK]

Common Mistakes:

• Confusing index positions after shift
• Expecting integer type instead of float
• Thinking shifted index 2 is original index 2

4. What is wrong with this code snippet?

import pandas as pd
data = pd.Series([5, 10, 15])
shifted = data.shift('1')

medium

A. shift() cannot be used on Series objects.

B. shift() requires a second argument for axis.

C. The data list must be a DataFrame, not a Series.

D. The argument to shift() should be an integer, not a string.

Solution

1. Step 1: Check shift() argument type

   The shift() function expects an integer or a timedelta, not a string.

2. Step 2: Identify error cause

   Passing a string like '1' causes a TypeError because pandas cannot interpret it as a number of periods.

3. Final Answer:

   The argument to shift() should be an integer, not a string. -> Option D

4. Quick Check:

   shift() needs integer periods, not string [OK]

Hint: Use integer, not string, inside shift() [OK]

Common Mistakes:

• Passing string instead of integer to shift()
• Thinking shift() only works on DataFrames
• Forgetting shift() default axis is 0

5. You have a DataFrame df with a column 'sales'. You want to create a new column 'sales_change' that shows the difference between current sales and sales one day before. Which code correctly achieves this?

hard

A. df['sales_change'] = df['sales'] - df['sales'].shift(1)

B. df['sales_change'] = df['sales'].shift(1) - df['sales']

C. df['sales_change'] = df['sales'].diff(-1)

D. df['sales_change'] = df['sales'].shift(-1) - df['sales']

Solution

1. Step 1: Understand the goal

   We want the change from previous day to current day, so current minus previous.

2. Step 2: Use shift(1) to get previous day sales

   df['sales'].shift(1) gives previous day's sales aligned with current row.

3. Step 3: Calculate difference

   Subtract previous sales from current sales: df['sales'] - df['sales'].shift(1).

4. Final Answer:

   df['sales_change'] = df['sales'] - df['sales'].shift(1) -> Option A

5. Quick Check:

   Current minus previous = sales_change [OK]

Hint: Subtract shifted column from current for change [OK]

Common Mistakes:

• Reversing subtraction order
• Using negative shift which looks ahead
• Using diff(-1) which is future difference