Skip to content
Bamboo Weekly

pandas diff

Subtract each value from the one before it, and turn a column of levels into a column of changes.

Have you ever loaded a time series, looked at a column of prices or populations or debt balances, and realized that the number you actually care about is not in the data at all? You want the change — how much did it move since last month? That number is one method call away, and the method is diff.

diff subtracts each value from the value that came before it and hands back an object of the same shape. Row 5 becomes row 5 minus row 4, and so on down the frame. There is nothing before the first row, so the first row is NaN. That is the entire idea, and everything else here is a variation on it.

Official documentation: DataFrame.diff and Series.diff

The arguments that earn their keep

There are only two, and you can learn both in a minute:

s.diff()                     # each value minus the previous one
s.diff(periods=12)           # each value minus the one 12 rows earlier
s.diff(periods=-1)           # each value minus the NEXT one
df.diff(axis='columns')      # each column minus the column to its left

periods is how far back to look. The default of 1 means "the previous row," but monthly data with periods=12 gives you a year-over-year change in a single call, and weekly data with periods=52 does the same. A negative periods reverses the direction: periods=-1 subtracts the next value rather than the previous one, and it is then the last rows that come back NaN, not the first.

axis decides whether "previous" means up or sideways. The default, axis='rows', walks down the frame. Pass axis='columns' and Pandas subtracts each column from the one to its left instead — which is exactly what you need after reshaping years or categories into columns.

One relationship worth carrying in your head: s.diff(n) is precisely s - s.shift(n). If you already understand shifting, you already understand diff.

A worked example, on real data

Ember publishes monthly wholesale electricity prices for European countries, going back to 2015. It is a good diff dataset because it is long — one row per country per month, the shape most real time series arrive in.

import pandas as pd

url = ('https://files.ember-energy.org/public-downloads/price/outputs/'
       'european_wholesale_electricity_price_data_monthly.csv')

df = pd.read_csv(url, parse_dates=['Date'])

df.head()

   Country ISO3 Code       Date  Price (EUR/MWhe)
0  Austria       AUT 2015-01-01             29.94
1  Belgium       BEL 2015-01-01             42.33
2  Czechia       CZE 2015-01-01             29.47
3  Denmark       DNK 2015-01-01             27.12
4  Estonia       EST 2015-01-01             33.84

That is 3,978 rows covering 32 countries. Start with one country, so the shape stays simple:

(
    df
    .loc[pd.col('Country') == 'Germany']
    .set_index('Date')
    ['Price (EUR/MWhe)']
    .loc['2021-09':'2022-03']
)

Date
2021-09-01    128.98
2021-10-01    142.22
2021-11-01    176.28
2021-12-01    225.08
2022-01-01    167.81
2022-02-01    129.86
2022-03-01    252.07
Name: Price (EUR/MWhe), dtype: float64

Those are levels. Add .diff() before the slice and they become movements:

(
    df
    .loc[pd.col('Country') == 'Germany']
    .set_index('Date')
    ['Price (EUR/MWhe)']
    .diff()
    .loc['2021-09':'2022-03']
)

Date
2021-09-01     46.80
2021-10-01     13.24
2021-11-01     34.06
2021-12-01     48.80
2022-01-01    -57.27
2022-02-01    -37.95
2022-03-01    122.21
Name: Price (EUR/MWhe), dtype: float64

Now the story is visible: a steady climb through autumn 2021, a two-month retreat, and then a jump of 122 euros in the month Russia invaded Ukraine.

Monthly prices are seasonal, though, and comparing March with February partly measures the weather. Comparing March with the previous March does not, and that is what periods=12 is for:

(
    df
    .loc[pd.col('Country') == 'Germany']
    .set_index('Date')
    ['Price (EUR/MWhe)']
    .diff(periods=12)
    .nlargest(5)
)

Date
2022-08-01    381.41
2022-07-01    235.53
2022-09-01    223.83
2022-03-01    205.39
2021-12-01    181.33
Name: Price (EUR/MWhe), dtype: float64

German power in August 2022 cost 381 euros per megawatt-hour more than it had a year earlier. Same method, same column, different periods.

Make periods negative and the arithmetic runs the other way — each value minus the one below it, which is how you label a row with the change that is coming rather than the change that has happened:

(
    df
    .loc[pd.col('Country') == 'Germany']
    .set_index('Date')
    ['Price (EUR/MWhe)']
    .diff(periods=-1)
    .loc['2021-09':'2021-12']
)

Date
2021-09-01   -13.24
2021-10-01   -34.06
2021-11-01   -48.80
2021-12-01    57.27
Name: Price (EUR/MWhe), dtype: float64

These are the same numbers as before with the signs flipped and the labels moved up a row.

Diffing within groups

All 32 countries live in one frame. Call .diff() on it and Pandas will happily subtract Austria's last month from Belgium's first, because it has no idea that the country column means anything. Say so with groupby:

(
    df
    .assign(change=lambda df_: df_.groupby('Country')['Price (EUR/MWhe)'].diff())
    .nlargest(5, 'change')
)

        Country ISO3 Code       Date  Price (EUR/MWhe)  change
3909   Slovenia       SVN 2026-06-01            324.79  223.18
2505    Denmark       DNK 2022-08-01            454.43  194.89
2515  Lithuania       LTU 2022-08-01            480.50  175.12
2484      Italy       ITA 2022-07-01            441.77  170.10
2514     Latvia       LVA 2022-08-01            467.09  162.11

groupby(...).diff() restarts the subtraction at the top of every group and returns a series aligned to the original index, so it drops straight into assign. Each country gets its own leading NaN, which is the correct answer.

Diffing across columns

Reshape the same data so that years become columns, and the interesting comparison runs sideways instead of downward:

(
    df
    .assign(year=pd.col('Date').dt.year)
    .loc[pd.col('year').isin([2019, 2022, 2026])]
    .loc[pd.col('Country').isin(['France', 'Germany', 'Norway', 'Poland', 'Spain'])]
    .pivot_table(index='Country', columns='year', values='Price (EUR/MWhe)')
    .round(2)
    .diff(axis='columns')
)

year     2019    2022    2026
Country
France    NaN  235.58 -200.80
Germany   NaN  197.72 -131.93
Norway    NaN   99.78  -47.36
Poland    NaN  112.84  -51.89
Spain     NaN  119.92 -101.84

The 2019 column is NaN because it is the baseline; 2022 shows the crisis, and 2026 shows how much of it has unwound. Reshaping with pivot_table or unstack and then diffing across columns is a Bamboo Weekly staple; two of the four issues cited below use it.

diff or pct_change?

diff answers "by how much?" and pct_change answers "by what proportion?" They are two views of the same movement, and they are literally related: s.pct_change() equals s.diff() / s.shift(). Put the three side by side on Germany's winter of 2021–2022:

prices = (
    df
    .loc[pd.col('Country') == 'Germany']
    .set_index('Date')
    ['Price (EUR/MWhe)']
)

(
    pd.DataFrame({'price': prices,
                  'diff': prices.diff(),
                  'pct_change': (prices.pct_change() * 100).round(1)})
    .loc['2021-12':'2022-03']
)

             price    diff  pct_change
Date
2021-12-01  225.08   48.80        27.7
2022-01-01  167.81  -57.27       -25.4
2022-02-01  129.86  -37.95       -22.6
2022-03-01  252.07  122.21        94.1

Use diff when the units matter — euros, seats in Congress, degrees. Use pct_change when you are comparing series of wildly different sizes, where a 5-euro move means one thing in Norway and another in Italy. Its own page covers its arguments and traps.

Three mistakes people make

The first row is always NaN, and it quietly poisons what comes next. Germany has 140 monthly prices, so .diff() produces 139 real numbers and one NaN. Ask how often the price rose and it is tempting to write (prices.diff() > 0).mean(), but comparing NaN with zero gives False, so the missing month is silently counted as a fall. That yields 0.5571 where the honest answer, 78 rises out of 139 changes, is 0.5612. Small here, embarrassing on a short series. Reach for .count() rather than len(), or drop the row you know is empty.

Diffing an unsorted frame gives nonsense, without an error. diff compares each row with the row physically above it, not with the row that comes earlier in time. Sort by price rather than by date and you still get numbers:

(
    df
    .loc[pd.col('Country') == 'Germany']
    .sort_values('Price (EUR/MWhe)')
    .set_index('Date')
    ['Price (EUR/MWhe)']
    .diff()
    .head()
)

Date
2020-04-01     NaN
2020-05-01    0.24
2020-02-01    3.94
2016-02-01    0.31
2020-03-01    0.47
Name: Price (EUR/MWhe), dtype: float64

Those are the gaps between adjacent prices in a sorted list, which is a real quantity and almost certainly not the one you wanted. Call sort_values or sort_index on the date first, every time.

Diffing across groups bleeds one group's last value into the next group's first. This is the failure that costs people the most time, because the number looks plausible. Sort the electricity data by country and date, call a plain .diff(), and look at the seam:

(
    df
    .sort_values(['Country', 'Date'])
    .assign(change=lambda df_: df_['Price (EUR/MWhe)'].diff())
    .iloc[140:144]
)

      Country ISO3 Code       Date  Price (EUR/MWhe)  change
3915  Austria       AUT 2026-07-01            117.30    9.20
3947  Austria       AUT 2026-08-01            146.49   29.19
1     Belgium       BEL 2015-01-01             42.33 -104.16
25    Belgium       BEL 2015-02-01             50.54    8.21

Belgium's first month appears to have fallen 104 euros. It did not: that is Belgium's January 2015 price minus Austria's August 2026 price, two countries and eleven years apart. With groupby('Country') in front of the diff, that cell is NaN instead — and NaN is a much better lie detector than a plausible negative number.

Where it shows up in Bamboo Weekly

Thirty-three Bamboo Weekly exercises use diff on real data. Four worth studying, all free to read:

Bamboo Weekly #47: Minimum wage builds a frame with one row per year and one column per state, then calls a plain .diff() to find which states raised their minimum wage on January 1st — and by how much.

Bamboo Weekly #28: Pret a Manger is the clearest illustration of periods I have. The data is weekly, so comparing with a year ago means .diff(periods=52), which is a great deal nicer than subtracting iloc[-53] from iloc[-1] by hand.

Bamboo Weekly #75: Refugees uses both arguments at once. The UNHCR figures are cumulative, and the years are columns, so the total for 2000 through 2023 is .diff(periods=23, axis='columns') — the last column minus the first.

Bamboo Weekly #34: House of Representatives groups, unstacks years into columns, and then diffs across them to show which states gained and lost congressional seats between 2020 and 2022.

Practice it

Work through a .diff() exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/diff/

Go deeper

diff belongs to a family of methods that compare a row with its neighbors: shift moves values without subtracting, pct_change divides instead of subtracting, and cumsum runs the whole thing backwards, turning changes into levels. The Pandas time series user guide covers the resampling and shifting that usually happen on either side of a diff.

More Pandas videos on Python and Pandas with Reuven Lerner.

Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.

See it on real data

Below are the 33 Bamboo Weekly exercises that use diff on real-world data — try each one, then study the worked solution.

Part of the Pandas Methods Index. See also practice by skill.