Slide a column up or down, and every row gets a neighbor to compare itself to.
How do I compare this row to the one before it? Pandas has no built-in notion of "previous row" in an expression — but if you move the whole column down by one position, the previous row's value lands next to the current one, and the comparison becomes ordinary arithmetic. That is all .shift() does, and it is the engine underneath diff and pct_change: s.diff() is precisely s - s.shift(). Reach for shift directly when you want the neighbor itself rather than the subtraction — a previous timestamp, a previous label, a previous category.
Official documentation: DataFrame.shift and Series.shift.
The arguments that earn their keep
s.shift(periods=1, # how far, and negative to look forward
freq=None, # move the index labels instead of the values
fill_value=…) # what to put in the vacated rows
periods=1 moves values down, so each row sees the one above it. Negative values move them up, which is how you get the next row: periods=-1 for the following observation, periods=-12 for next year on monthly data.
freq is the odd one out: without it shift moves the data and leaves the index alone, and with it shift moves the index and leaves the data alone. fill_value matters more than it looks, because shifting an integer column introduces NaN and promotes the whole thing to float64; fill_value=0 keeps it int64.
A worked example, on real data
The USGS earthquake catalog is a CSV endpoint with no key required. Here is every magnitude-5-and-up event in the first half of 2026, with the country pulled off the end of the place string:
import pandas as pd
url = ('https://earthquake.usgs.gov/fdsnws/event/1/query?format=csv'
'&starttime=2026-01-01&endtime=2026-07-01&minmagnitude=5')
quakes = (
pd.read_csv(url, parse_dates=['time'], usecols=['time', 'place', 'mag'])
.assign(region=lambda df_: df_['place'].str.split(', ').str[-1])
.drop(columns='place')
)
918 events, give or take the revisions USGS makes to older entries. How long does a region typically wait between quakes? That is a question about neighbors, so it is a question for shift — and because the neighbor has to be in the same region, the shift belongs inside a groupby:
(
quakes
.sort_values('time')
.assign(gap=lambda d: d['time'] - d.groupby('region')['time'].shift())
.loc[lambda d: d['region'] == 'Japan']
.head(5)
)
time mag region gap
892 2026-01-06 01:18:48.586000+00:00 5.7 Japan NaT
891 2026-01-06 01:37:48.746000+00:00 5.0 Japan 0 days 00:19:00.160000
851 2026-01-14 22:13:16.026000+00:00 5.5 Japan 8 days 20:35:27.280000
848 2026-01-15 05:48:04.538000+00:00 5.1 Japan 0 days 07:34:48.512000
841 2026-01-16 19:34:14.532000+00:00 5.3 Japan 1 days 13:46:09.994000
Nineteen minutes, then nearly nine days: aftershocks and quiet weeks. The NaT sits at the top of each group, because the first event has nothing before it. Aggregate that column and you get a real result — a median of four hours between magnitude-5 events in the Philippines, and four and a half days in Alaska.
Three mistakes people make
Shifting a frame that is not sorted. The USGS feed arrives newest-first, so running the same calculation on it as downloaded gives this:
quakes.assign(gap=lambda d: d['time'] - d['time'].shift()).head(3)
time mag region gap
0 2026-06-30 23:44:54.898000+00:00 5.3 China NaT
1 2026-06-30 21:01:34.117000+00:00 5.3 Mexico -1 days +21:16:39.219000
2 2026-06-30 19:45:39.422000+00:00 6.0 Mexico -1 days +22:44:05.305000
Negative gaps, because shift moves rows, not time. It never consults the index, and it never warns. Here the sign gives the mistake away; on a column of prices or counts it would not, and you would publish the wrong number. sort_index or sort_values before every shift — pct_change and diff carry the identical trap for the identical reason.
Shifting across group boundaries. Drop the groupby from the query above, keep everything else, and the same Japanese rows come back looking calmer and completely wrong:
time mag region gap
892 2026-01-06 01:18:48.586000+00:00 5.7 Japan 0 days 02:46:26.342000
891 2026-01-06 01:37:48.746000+00:00 5.0 Japan 0 days 00:19:00.160000
851 2026-01-14 22:13:16.026000+00:00 5.5 Japan 0 days 01:36:55.610000
The 8-day gap became 1 hour 37 minutes, because the previous row is now whatever happened next anywhere on Earth. Every value is plausible; none answers the question asked. Whenever a frame stacks several series on top of each other, the shift goes inside the groupby.
Reading periods as an amount of time. It is a number of rows, and the two only agree when the rows are evenly spaced. Japan's 59 gaps above run from 12 seconds to 27 days, all of them shift(1). On monthly data with one month missing, shift(12) reaches thirteen months back and nothing says so; if you need a genuine calendar offset, resample onto a complete calendar first. And since the values move while the labels stay put, name the new column for what it holds — prev_time, mag_lag1 — rather than reusing the original name.
Looking the other way is the same argument with the sign flipped: d.groupby('region')['mag'].shift(-1) puts the next quake's magnitude on each row, which is how you ask whether a big one tends to be followed by a bigger one.
Where it shows up in Bamboo Weekly
Bamboo Weekly #115: Sahm rule is shift doing the job nothing else can. The Sahm recession rule needs the mean of the three months before each month, and s.rolling(window=3).mean() labels that mean with the last month inside its own window. The fix is s.rolling(window=3).mean().shift(1), and the 12-month floor beside it is s.rolling(window=12).min().shift(1). One row of misalignment would move every recession call by a month.
Bamboo Weekly #84: Central banks uses shift to test a lag hypothesis: after resampling monthly and taking pct_change, it builds lagged_fed with df_['US Federal Reserve System'].shift(lag_factor), drops the unlagged column, and runs corr() to see which other central banks follow the Fed, and after how many months.
Practice it
Work through a shift exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/shift/
Go deeper
diff is shift plus subtraction and pct_change is shift plus division. rolling and expanding are the other two order-dependent windows, and shift is regularly what lines their output up correctly. sort_index comes first in all of them; groupby keeps each comparison inside its own series.
More Pandas videos on Python and Pandas with Reuven Lerner.
Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.
Related methods
.diff()— which is shift and subtract in one call, and usually what you meant.sort_index()— which has to happen first, or you are comparing arbitrary neighbours
See it on real data
Below are the 2 Bamboo Weekly exercises that use shift on real-world data — try each one, then study the worked solution.
Part of the Pandas Methods Index. See also practice by skill.