shift() for lagging data in Pandas

Introduction

We use shift() to move data up or down in a table. This helps us compare current values with past values easily.

When you want to compare today's sales with yesterday's sales.

When you need to calculate the change in stock prices from one day to the next.

When analyzing time series data to see trends over time.

When creating features for machine learning that depend on past values.

When you want to find the difference between current and previous measurements.

Syntax

Pandas

DataFrame.shift(periods=1, freq=None, axis=0, fill_value=None)

periods tells how many steps to move the data. Positive moves data down (lag), negative moves up (lead).

axis=0 means shift rows, axis=1 means shift columns.

Examples

This creates a new column with values shifted down by 1 row (previous row's value).

Pandas

df['lagged'] = df['value'].shift(1)

This shifts values up by 1 row, showing next row's value.

Pandas

df['lead'] = df['value'].shift(-1)

This shifts all rows down by 2 steps, adding NaN at the top.

Pandas

df.shift(2)

Sample Program

This code creates a table of sales by day. It adds a column with the previous day's sales using shift(1). Then it calculates the change in sales from the previous day.

Pandas

import pandas as pd

data = {'day': [1, 2, 3, 4, 5], 'sales': [100, 150, 200, 130, 170]}
df = pd.DataFrame(data)

df['previous_day_sales'] = df['sales'].shift(1)
df['sales_change'] = df['sales'] - df['previous_day_sales']

print(df)

Important Notes

Missing values appear where there is no previous data after shifting (usually at the start).

You can use fill_value to replace missing values after shifting.

Summary

shift() moves data up or down to compare current and past values.

It is useful for time-based comparisons and feature creation.

Remember that shifting creates missing values where data is moved out of range.