Skip to content
Bamboo Weekly

pandas pct_change

Turn a column of levels into a column of growth rates — and know when a percentage is the honest way to say it.

How much did it change? If you have been reading Bamboo Weekly for a while, you know this is the question I ask more often than any other. Prices, populations, emissions, rents: the raw numbers are rarely the story, and the movement between them almost always is. When I want that movement as a proportion rather than a quantity, I reach for pct_change.

pct_change divides each value by the value that came before it and subtracts one. Row 5 becomes row 5 divided by row 4, minus 1. There is nothing before the first row, so the first row is NaN. The result comes back as a fraction — a rise from 100 to 105 gives you 0.05, not 5 — and remembering that is most of the battle. It also helps to know that s.pct_change(n) is exactly s / s.shift(n) - 1. Not approximately: I checked, and the two produce bit-identical results.

Official documentation: DataFrame.pct_change and Series.pct_change

The arguments that earn their keep

s.pct_change()                 # each value against the previous one
s.pct_change(periods=12)       # each value against the one 12 rows earlier
s.pct_change(periods=-1)       # each value against the NEXT one
df.pct_change(axis='columns')  # each column against the column to its left

periods is how far back to look. Its default of 1 means "the previous row," but monthly data with periods=12 gives you a year-over-year change in a single call, and annual data with periods=10 gives you a decade.

A negative periods runs the comparison the other way, against the row below rather than the row above. That sounds like a curiosity until you meet a data set whose most recent year is at the top — which is how a surprising number of agencies publish — and then it is the argument that saves you a sort.

axis decides whether "previous" means up or sideways. The default walks down the frame; axis='columns' compares each column with the one to its left, which is what you want after reshaping years or categories into columns. That turns out to be the most common form in Bamboo Weekly, because so many of these exercises begin with a pivot.

There is a fourth argument, fill_method, which used to matter a great deal and now exists only to reject you. It has earned a section of its own further down.

A worked example, on real data

Our World in Data publishes a CO2 file with one row per country per year, going back to 1750. It suits this method because the question people actually ask of it — is this country's output rising or falling, and how fast? — is a percentage question.

import pandas as pd

url = 'https://nyc3.digitaloceanspaces.com/owid-public/data/co2/owid-co2-data.csv'

df = pd.read_csv(url, usecols=['country', 'iso_code', 'year', 'co2'])

df.tail()

        country  year iso_code     co2
50186  Zimbabwe  2019      ZWE  10.263
50187  Zimbabwe  2020      ZWE   8.495
50188  Zimbabwe  2021      ZWE  10.204
50189  Zimbabwe  2022      ZWE  10.425
50190  Zimbabwe  2023      ZWE  11.164

That is 50,191 rows. The country column also holds continents and income groups, so iso_code is there to tell the 218 real countries from the aggregates. Start with one:

(
    df
    .loc[pd.col('country') == 'United Kingdom']
    .set_index('year')
    ['co2']
    .loc[2015:2023]
)

year
2015    422.461
2016    399.430
2017    387.367
2018    379.730
2019    364.753
2020    326.263
2021    344.510
2022    313.835
2023    305.146
Name: co2, dtype: float64

Those are millions of tonnes. Add .pct_change() before the slice and the levels become rates:

(
    df
    .loc[pd.col('country') == 'United Kingdom']
    .set_index('year')
    ['co2']
    .pct_change()
    .loc[2015:2023]
)

year
2015   -0.037251
2016   -0.054516
2017   -0.030201
2018   -0.019715
2019   -0.039441
2020   -0.105523
2021    0.055927
2022   -0.089040
2023   -0.027687
Name: co2, dtype: float64

Notice that these are fractions. Nobody says "British emissions fell by 0.037 in 2015," so multiply by 100 and round before showing anyone. I will keep that series around as uk:

uk = (
    df
    .loc[pd.col('country') == 'United Kingdom']
    .set_index('year')
    ['co2']
)

(uk.pct_change() * 100).round(1).loc[2015:2023]

year
2015    -3.7
2016    -5.5
2017    -3.0
2018    -2.0
2019    -3.9
2020   -10.6
2021     5.6
2022    -8.9
2023    -2.8
Name: co2, dtype: float64

Now the shape is visible: a steady grind downward, a pandemic-sized drop of 10.6% in 2020, a rebound in 2021, and back to falling. Year-on-year figures are noisy, though. Ask for the change against a decade earlier and the trend stops arguing with itself:

(uk.pct_change(periods=10) * 100).round(1).loc[2015:2023]

year
2015   -25.9
2016   -29.7
2017   -30.8
2018   -30.3
2019   -26.2
2020   -36.3
2021   -26.7
2022   -35.6
2023   -36.1
Name: co2, dtype: float64

British emissions in 2023 were 36% below their 2013 level. Same column, same method, one argument. A negative periods runs the same arithmetic backwards; it earns its own demonstration in the mistakes below, where it is the fix for a real problem.

Percent change within groups

All 218 countries live in one frame, and since it is sorted alphabetically, a plain .pct_change() on it would cheerfully divide Algeria's first year by Albania's last. Say what you mean with groupby:

g7 = ['Canada', 'France', 'Germany', 'Italy', 'Japan',
      'United Kingdom', 'United States']

(
    df
    .assign(pct=lambda df_: df_.groupby('country')['co2'].pct_change() * 100)
    .loc[pd.col('country').isin(g7) & pd.col('year').between(2020, 2023)]
    .pivot(index='year', columns='country', values='pct')
    .round(1)
)

country  Canada  France  Germany  Italy  Japan  United Kingdom  United States
year
2020       -9.5   -11.0     -8.7  -10.9   -5.9           -10.6          -10.4
2021        2.6     9.0      4.7   11.0    2.1             5.6            6.7
2022        1.9    -4.0     -1.1    1.4   -2.8            -8.9            0.9
2023       -0.2    -7.2    -11.2   -8.0   -4.3            -2.8           -3.3

groupby(...).pct_change() restarts the division at the top of each group and hands back a series aligned to the original index, so it drops straight into assign. Watch the order of operations: I compute the percentages on the whole frame and filter afterwards. Filter first and the 2020 row would be NaN, because 2020 would then be each group's first row.

Percent change across columns

Reshape so that years become columns, and the comparison runs sideways:

(
    df
    .loc[pd.col('country').isin(g7) & pd.col('year').isin([2003, 2013, 2023])]
    .pivot(index='country', columns='year', values='co2')
    .pct_change(axis='columns')
    .mul(100)
    .round(1)
)

year            2003  2013  2023
country
Canada           NaN  -2.1  -3.4
France           NaN -12.9 -24.0
Germany          NaN  -7.0 -28.3
Italy            NaN -25.4 -15.2
Japan            NaN   2.2 -24.8
United Kingdom   NaN -16.4 -36.1
United States    NaN  -8.8 -10.4

The 2003 column is NaN because it is the baseline; every other cell is one decade's change for one country. Reshaping with pivot_table or unstack and then running pct_change across the columns is the most common shape this method takes in Bamboo Weekly.

What happened to fill_method

If you learned this method a few years ago, you learned it with a fill_method argument that defaulted to forward-filling. Pandas 2.1 deprecated that default; in Pandas 3 the argument survives only so that it can turn you down:

uk.pct_change(fill_method='ffill')

ValueError: fill_method must be None; got fill_method='ffill'.

This is a good change, and this data set shows why. Greek emissions in the 1870s were recorded only every fourth year or so:

gr = (
    df
    .loc[pd.col('country') == 'Greece']
    .set_index('year')
    ['co2']
)

gr.loc[1875:1883]

year
1875    0.165
1876      NaN
1877      NaN
1878      NaN
1879    0.004
1880      NaN
1881      NaN
1882      NaN
1883    0.004
Name: co2, dtype: float64

Today a missing value produces a missing answer, which is the honest one:

(gr.pct_change() * 100).round(1).loc[1875:1883]

year
1875    10.0
1876     NaN
1877     NaN
1878     NaN
1879     NaN
1880     NaN
1881     NaN
1882     NaN
1883     NaN
Name: co2, dtype: float64

Fill first — which is what the old default did without telling you — and Pandas reports four years in which Greek emissions supposedly held perfectly steady, then dumps the entire 1875-to-1879 collapse onto one of them:

(gr.ffill().pct_change() * 100).round(1).loc[1875:1883]

year
1875    10.0
1876     0.0
1877     0.0
1878     0.0
1879   -97.6
1880     0.0
1881     0.0
1882     0.0
1883     0.0
Name: co2, dtype: float64

Those zeros are inventions. Filling gaps is sometimes the right call, but it is a judgment about your data, not a formatting detail — so make it yourself, in the open, where a reader of your code can argue with you. That is the entire point of the deprecation.

pct_change or diff?

diff answers "by how much?" and pct_change answers "by what proportion?" You can have both at once — s.agg(['diff', 'pct_change']) returns them as two columns — but which one is honest depends on your data, and the failure runs in both directions.

A percentage of a small base is close to meaningless. Ask which countries increased their emissions most in 2023, ranked by percentage:

changes = (
    df
    .assign(pct=lambda df_: df_.groupby('country')['co2'].pct_change() * 100,
            change=lambda df_: df_.groupby('country')['co2'].diff())
    .loc[pd.col('iso_code').notna() & (pd.col('year') == 2023)]
)

changes.nlargest(6, 'pct')[['country', 'co2', 'change', 'pct']].round(2)

          country     co2  change    pct
36437      Panama   14.02    2.66  23.44
48947   Venezuela   99.57   12.05  13.77
49121     Vietnam  334.73   37.09  12.46
46675      Tuvalu    0.01    0.00   9.09
37805       Qatar  115.71    9.56   9.01
4045   Azerbaijan   43.94    3.55   8.78

Tuvalu is the fourth-fastest-growing emitter on Earth, on an increase of roughly one thousandth of a million tonnes — a true number and a worthless one.

Rank by the absolute change instead and a different world appears:

changes.nlargest(6, 'change')[['country', 'co2', 'change', 'pct']].round(2)

       country       co2  change    pct
9885     China  11902.50  551.97   4.86
21658    India   3062.32  231.16   8.16
49121  Vietnam    334.73   37.09  12.46
29571   Mexico    482.62   17.29   3.71
22588     Iran    817.88   17.13   2.14
38157   Russia   1815.92   13.73   0.76

China's 4.86% is the smallest percentage in this table and by far the largest number of tonnes. So use diff when the units are comparable and the quantity is what matters — tonnes, euros, seats in Congress. Use pct_change when the series are of wildly different sizes, or the units do not add up, and a proportion is the only fair basis. When your data spans orders of magnitude, as this does, show both.

Four mistakes people make

Running it on a frame that is not in time order. This is the big one, and it is silent. pct_change compares each row with the row physically above it, with no idea whether your index means anything. Sorting newest-first for a readable display is a reasonable thing to do, and it destroys the result:

(
    df
    .loc[pd.col('country') == 'United Kingdom']
    .sort_values('year', ascending=False)
    .set_index('year')
    ['co2']
    .pct_change()
    .mul(100)
    .round(1)
    .head(6)
)

year
2023     NaN
2022     2.8
2021     9.8
2020    -5.3
2019    11.8
2018     4.1
Name: co2, dtype: float64

Every sign is flipped, every label is off by a year, and the magnitudes are wrong too, because each division now uses the wrong denominator. The UK's real 2.8% fall in 2023 has become a 2.8% rise recorded against 2022. Nothing there looks wrong, which is why people ship it.

Call sort_values or sort_index on the date before you compute. Or keep the newest-first order and pass periods=-1, which walks the frame in the direction it is actually sorted:

(
    df
    .loc[pd.col('country') == 'United Kingdom']
    .sort_values('year', ascending=False)
    .set_index('year')
    ['co2']
    .pct_change(periods=-1)
    .mul(100)
    .round(1)
    .head(6)
)

year
2023    -2.8
2022    -8.9
2021     5.6
2020   -10.6
2019    -3.9
2018    -2.0
Name: co2, dtype: float64

The right numbers on the right years — and note that they are not the broken output with its signs flipped. The broken 2018 reads 4.1% where the true 2019 reads -3.9%, though both describe the same 15 million tonnes. Dividing by the smaller number gives the bigger percentage.

Forgetting groupby when the frame holds more than one series. Stack two countries, compute a plain pct_change, and the seam between them produces a plausible, completely fictional number:

(
    df
    .loc[pd.col('country').isin(['France', 'Germany']) & pd.col('year').between(2021, 2023)]
    .assign(pct=lambda df_: df_['co2'].pct_change() * 100)
    .round(2)
)

       country  year iso_code     co2     pct
17297   France  2021      FRA  305.61     NaN
17298   France  2022      FRA  293.50   -3.96
17299   France  2023      FRA  272.48   -7.16
18228  Germany  2021      DEU  678.78  149.11
18229  Germany  2022      DEU  671.47   -1.08
18230  Germany  2023      DEU  596.15  -11.22

France's 2021 is NaN because it is the first row, which is fine. Germany's 2021 is the problem: emissions there did not rise 149%. That cell is nothing but Germany's 2021 divided by France's 2023, and it will sit in your data unchallenged for as long as you let it. Put groupby('country') in front and it comes back NaN instead — a far better lie detector than a plausible number.

Treating the result as a percentage when it is a fraction. The classic version is multiplying twice. Python's percent format specifier scales by 100 itself, so applying it to something you already scaled gives you this:

uk.pct_change().mul(100).map('{:.1%}'.format).loc[2020:2023]

year
2020    -1055.2%
2021      559.3%
2022     -890.4%
2023     -276.9%
Name: co2, dtype: str

Drop the mul(100) and let the formatter do its job:

uk.pct_change().map('{:.1%}'.format).loc[2020:2023]

year
2020    -10.6%
2021      5.6%
2022     -8.9%
2023     -2.8%
Name: co2, dtype: str

Multiply by 100 or format as a percent. Never both.

Letting a zero denominator quietly become inf. Dividing by zero does not raise here. It returns infinity, and infinity contaminates everything downstream. Compute annual changes for every country since 1990 and ask for the average:

pct = (
    df
    .assign(pct=lambda df_: df_.groupby('country')['co2'].pct_change() * 100)
    .loc[pd.col('iso_code').notna() & pd.col('year').between(1990, 2023)]
)

pct['pct'].mean()

inf

One row out of 7,412 did that:

import numpy as np

pct.loc[np.isinf(pct['pct'])][['country', 'year', 'co2', 'pct']]

          country  year    co2  pct
13297  East Timor  1998  0.061  inf

East Timor recorded exactly zero emissions in 1997 and 0.061 in 1998, and dividing by that zero was enough to make the average of seven thousand honest numbers infinite. Pandas does not warn you, and .max() will not help — it returns inf as well.

An infinite mean at least announces itself. The dangerous case is an inf and a -inf in one column, which cancel and hand you a number that looks perfectly reasonable. Clear them out before aggregating — pct['pct'].replace([np.inf, -np.inf], np.nan).mean() gives 3.54 — or use .median(), which shrugs at infinities and returns 1.8.

Where it shows up in Bamboo Weekly

Fifty-eight Bamboo Weekly exercises use pct_change on real data. Four worth studying, all free to read:

Bamboo Weekly #9: US house prices is the grouped version at its clearest. The index has one level for location and one for date, so the answer is groupby(level='location').pct_change(periods=11) — a year of monthly data, computed separately for each region.

Bamboo Weekly #78: Stock markets puts the sorting trap front and center. The scraped prices arrive newest-first, and rather than resorting them I used pct_change(periods=-1). It is my best example of a negative periods being the natural choice rather than a trick.

Bamboo Weekly #25: Entrepreneurship pivots countries against years and compares sideways with pct_change(axis='columns'), first year over year and then with periods=10 for the decade.

Bamboo Weekly #56: Rent increases transposes national rent data so months run across the columns, then plots the month-to-month change. It also passes fill_method=None explicitly, which was how you asked for today's default back when the default was something else.

Practice it

Work through a pct_change exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/pct-change/

Go deeper

pct_change sits in a small family of methods that compare a row with its neighbors: shift moves values without doing arithmetic, diff subtracts instead of dividing, and cumprod runs the whole thing backwards, compounding growth rates back into levels. The Pandas time series user guide covers the resampling and shifting that usually happen on either side of a percentage change.

More Pandas videos on Python and Pandas with Reuven Lerner.

Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.

See it on real data

Below are the 56 Bamboo Weekly exercises that use pct_change on real-world data — try each one, then study the worked solution.

Part of the Pandas Methods Index. See also practice by skill.