shift() for lagging data in Pandas - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of shift() with default parameters

What is the output of the following code snippet using pandas shift() with default lag?

Pandas

import pandas as pd

df = pd.DataFrame({'value': [10, 20, 30, 40]})
df['lagged'] = df['value'].shift()
print(df)

A

   value  lagged
0     10     NaN
1     20    10.0
2     30    20.0
3     40    30.0

B

   value  lagged
0     10    10.0
1     20    20.0
2     30    30.0
3     40    40.0

C

   value  lagged
0     10    20.0
1     20    30.0
2     30    40.0
3     40     NaN

D

   value  lagged
0     10     NaN
1     20     NaN
2     30     NaN
3     40     NaN

💡 Hint

Remember that shift() moves data down by 1 row by default, introducing NaN at the top.

The shift() function moves the data down by one row, so the first row in the new column is NaN, and each subsequent row contains the previous row's value.

❓ data_output

intermediate

Result of shift() with negative periods

What is the resulting DataFrame after applying shift() with periods=-1 on the 'value' column?

Pandas

import pandas as pd

df = pd.DataFrame({'value': [5, 15, 25, 35]})
df['lead'] = df['value'].shift(periods=-1)
print(df)

A

   value  lead
0      5  35.0
1     15  25.0
2     25  15.0
3     35   NaN

B

   value  lead
0      5   NaN
1     15   5.0
2     25  15.0
3     35  25.0

C

   value  lead
0      5  15.0
1     15  25.0
2     25  35.0
3     35   NaN

D

   value  lead
0      5   NaN
1     15   NaN
2     25   NaN
3     35   NaN

💡 Hint

Negative periods shift data up, so the last row becomes NaN.

Using periods=-1 shifts the data up by one row, so the last row in the new column is NaN, and each row contains the next row's value.

🔧 Debug

advanced

Identify the error in shift() usage

What error will this code raise and why? import pandas as pd df = pd.DataFrame({'score': [100, 200, 300]}) df['shifted'] = df['score'].shift(periods='2')

Pandas

import pandas as pd

df = pd.DataFrame({'score': [100, 200, 300]})
df['shifted'] = df['score'].shift(periods='2')

A TypeError: unsupported operand type(s) for +: 'int' and 'str'

B SyntaxError: invalid syntax

C No error, code runs successfully

D ValueError: periods must be an integer

💡 Hint

Check the type of the 'periods' argument in shift().

The periods argument must be an integer. Passing a string like '2' causes a ValueError.

🚀 Application

advanced

Using shift() to calculate daily returns

Given a DataFrame with daily stock prices, which code correctly calculates the daily return as percentage change using shift()?

Pandas

import pandas as pd

df = pd.DataFrame({'price': [100, 105, 110, 120]})

A df['return'] = df['price'] - df['price'].shift(1) * 100

B df['return'] = (df['price'] - df['price'].shift(1)) / df['price'].shift(1) * 100

C df['return'] = (df['price'] / df['price'].shift(1)) * 100

D df['return'] = (df['price'].shift(1) - df['price']) / df['price'] * 100

💡 Hint

Daily return is (today's price - yesterday's price) divided by yesterday's price.

Option B correctly computes the percentage change from the previous day using shift(1).

🧠 Conceptual

expert

Effect of shift() on time series with missing dates

If a time series DataFrame has missing dates and you apply shift() on the value column, what is the effect on the lagged data?

A The lagged data aligns with the previous row regardless of missing dates, so gaps remain unfilled.

B shift() automatically fills missing dates with NaN before shifting, so lagged data aligns with actual previous dates.

C shift() fills missing dates with the last available value before shifting, so lagged data has no gaps.

D shift() removes rows with missing dates before shifting, so lagged data only contains continuous dates.

💡 Hint

shift() works row-wise and does not modify the index or fill missing dates.

shift() moves data down by rows without considering the index values, so missing dates cause gaps in lagged data.

Practice

1. What does the shift() function do in pandas?

easy

A. It changes the data type of a column.

B. It moves data up or down to compare current and past values.

C. It removes missing values from the dataset.

D. It sorts the data in ascending order.

Solution

1. Step 1: Understand the purpose of shift()

   The shift() function moves data up or down in a column, creating a lag or lead effect.

2. Step 2: Compare with other options

   Sorting, removing missing values, or changing data types are different pandas functions, not shift().

3. Final Answer:

   It moves data up or down to compare current and past values. -> Option B

4. Quick Check:

   shift() moves data for lagging [OK]

Hint: Remember: shift() moves data to compare past values [OK]

Common Mistakes:

• Confusing shift() with sort()
• Thinking shift() removes missing data
• Assuming shift() changes data types

2. Which of the following is the correct syntax to shift a pandas Series named data by 2 rows downward?

easy

A. data.shift(2)

B. data.shift(-2)

C. data.shift(0)

D. data.shift()

Solution

1. Step 1: Understand shift() parameter

   The argument inside shift() is the number of periods to move. Positive values shift downward (lag), negative shift upward (lead).

2. Step 2: Apply to the question

   To shift downward by 2 rows, use data.shift(2). Negative or zero values do not match the requirement.

3. Final Answer:

   data.shift(2) -> Option A

4. Quick Check:

   Positive number shifts down = data.shift(2) [OK]

Hint: Positive number in shift() moves data down [OK]

Common Mistakes:

• Using negative number to shift downward
• Omitting the argument defaults to shift by 1
• Using zero which means no shift

3. Given the pandas Series:

import pandas as pd
data = pd.Series([10, 20, 30, 40, 50])
shifted = data.shift(1)

What is the value of shifted at index 2?

medium

A. NaN

B. 30.0

C. 20.0

D. 10.0

Solution

1. Step 1: Understand shift(1) effect on Series

   Shifting by 1 moves all values down by one index. The value at index 2 in shifted is the original value at index 1.

2. Step 2: Find original value at index 1

   Original data at index 1 is 20, so shifted at index 2 is 20.0 (float due to NaN presence).

3. Final Answer:

   20.0 -> Option C

4. Quick Check:

   shift(1) moves index 1 value to index 2 [OK]

Hint: Value at index i after shift(1) is original at i-1 [OK]

Common Mistakes:

• Confusing index positions after shift
• Expecting integer type instead of float
• Thinking shifted index 2 is original index 2

4. What is wrong with this code snippet?

import pandas as pd
data = pd.Series([5, 10, 15])
shifted = data.shift('1')

medium

A. shift() cannot be used on Series objects.

B. shift() requires a second argument for axis.

C. The data list must be a DataFrame, not a Series.

D. The argument to shift() should be an integer, not a string.

Solution

1. Step 1: Check shift() argument type

   The shift() function expects an integer or a timedelta, not a string.

2. Step 2: Identify error cause

   Passing a string like '1' causes a TypeError because pandas cannot interpret it as a number of periods.

3. Final Answer:

   The argument to shift() should be an integer, not a string. -> Option D

4. Quick Check:

   shift() needs integer periods, not string [OK]

Hint: Use integer, not string, inside shift() [OK]

Common Mistakes:

• Passing string instead of integer to shift()
• Thinking shift() only works on DataFrames
• Forgetting shift() default axis is 0

5. You have a DataFrame df with a column 'sales'. You want to create a new column 'sales_change' that shows the difference between current sales and sales one day before. Which code correctly achieves this?

hard

A. df['sales_change'] = df['sales'] - df['sales'].shift(1)

B. df['sales_change'] = df['sales'].shift(1) - df['sales']

C. df['sales_change'] = df['sales'].diff(-1)

D. df['sales_change'] = df['sales'].shift(-1) - df['sales']

Solution

1. Step 1: Understand the goal

   We want the change from previous day to current day, so current minus previous.

2. Step 2: Use shift(1) to get previous day sales

   df['sales'].shift(1) gives previous day's sales aligned with current row.

3. Step 3: Calculate difference

   Subtract previous sales from current sales: df['sales'] - df['sales'].shift(1).

4. Final Answer:

   df['sales_change'] = df['sales'] - df['sales'].shift(1) -> Option A

5. Quick Check:

   Current minus previous = sales_change [OK]

Hint: Subtract shifted column from current for change [OK]

Common Mistakes:

• Reversing subtraction order
• Using negative shift which looks ahead
• Using diff(-1) which is future difference