expanding in pandas — practice examples (Bamboo Weekly)

A window with a fixed start and a moving end.

How unusual is this year, given everything that came before it? That question needs a window that grows: at row 100 you want all 100 rows, not the last seven. .expanding() gives you exactly that, and then hands you the full aggregation vocabulary to run over it.

Like rolling, it computes nothing on its own. .expanding() returns an Expanding object, and the aggregation you chain onto it does the work: .mean() for a running average, .max() for a record-so-far, .std() for a running spread.

Official documentation: DataFrame.expanding and Series.expanding.

The arguments that earn their keep

There is essentially one:

s.expanding(min_periods=1)   # rows required before an answer appears

The default of 1 is the important difference from rolling, where min_periods defaults to the window size and costs you a block of NaN at the top. An expanding window answers from row 0 onward. Raise min_periods when the early answers would be silly — a running standard deviation over three observations, say.

How it differs from cumsum, and from rolling

.expanding().sum() and .cumsum() compute the same running total, and .expanding().max() is exactly .cummax(). So why does expanding exist?

Because Pandas ships only four cum methods — cumsum, cumprod, cummax, cummin. There is no cummean, no cumstd, no cumquantile. .expanding() opens all of them, plus median, var, rank, nunique, corr, apply and agg. If you want a running average, this is the method.

They also disagree about missing values. cumsum writes NaN into the row where the gap sits; expanding().sum() skips it and keeps counting:

t = pd.Series([1.0, float('nan'), 3.0, 4.0])

pd.DataFrame({'cumsum': t.cumsum(), 'expanding_sum': t.expanding().sum()})

   cumsum  expanding_sum
0     1.0            1.0
1     NaN            1.0
2     4.0            4.0
3     8.0            8.0

Against rolling the difference is memory. A rolling window forgets; an expanding one never does. That makes rolling the right tool for "what is happening now" and expanding the right one for "what is the record" or "how does this compare to everything so far."

A worked example, on real data

NASA's GISTEMP series gives one global temperature anomaly per year since 1880, in degrees Celsius against the 1951–1980 baseline. GISS rebuilds the file monthly, so your numbers may run slightly past mine.

import pandas as pd

url = 'https://data.giss.nasa.gov/gistemp/tabledata_v4/GLB.Ts+dSST.csv'

temps = (
    pd.read_csv(url, skiprows=1, index_col='Year', na_values='***',
                usecols=['Year', 'J-D'])
    .dropna()
    .rename(columns={'J-D': 'anomaly'})
    ['anomaly']
)

A record year is one that equals the running maximum, which is a single comparison:

temps.loc[temps == temps.expanding().max()].tail(8)

Year
2005    0.68
2010    0.73
2014    0.75
2015    0.90
2016    1.01
2020    1.01
2023    1.17
2024    1.28
Name: anomaly, dtype: float64

Twenty-five record years in 146. Now put the two windows next to each other:

pd.DataFrame({'anomaly': temps,
              'so_far': temps.expanding().mean().round(3),
              'last_30': temps.rolling(30).mean().round(3)}).tail(5)

      anomaly  so_far  last_30
Year
2021     0.85   0.053    0.627
2022     0.89   0.059    0.649
2023     1.17   0.066    0.680
2024     1.28   0.075    0.713
2025     1.19   0.082    0.738

Two honest answers to two different questions. The expanding mean is 0.082, because it is still carrying 1880; the 30-year mean is 0.738, because it has forgotten everything before 1996. Neither is the "real" average, and quoting one while meaning the other is how a chart misleads.

Two mistakes people make

Running it on a frame that is not in the order you think. This is the whole risk of the method. expanding walks rows in the order they sit in the frame, not in index order, and it will not complain. Reverse this series and ask the same question:

rev = temps.sort_index(ascending=False)

(rev == rev.expanding().max()).sum()

Twenty-five record years became two, and every row after 2024 reports a record-so-far of 1.28. It is a perfectly well-formed column of numbers about a history running backwards. Call sort_index first.

The flip side is that a deliberate sort makes expanding answer a different question entirely. Sort descending and .expanding().sum() becomes "the total of the top n," which is how you find out how many countries it takes to match some threshold.

Reading ties as records. 2020 shows up in the list above at 1.01, and so does 2016. Equality against the running maximum counts a tie as a new record. For strictly-hotter-than-ever, compare against the previous maximum with temps > temps.expanding().max().shift(1), which finds 20 years rather than 25.

Where it shows up in Bamboo Weekly

Bamboo Weekly #103: CDC data pivots weekly deaths into one column per pathogen and then calls .expanding().sum() on the whole frame, turning weekly counts into cumulative ones — every column at once, no loop. The post uses the phrase "window function" deliberately, and the running total goes straight into a line plot.

Bamboo Weekly #124: NATO spending is the sorted case. It drops the United States, sorts the remaining members by 2024 spending descending, and runs .expanding().sum() to ask how many allies it takes to out-spend Washington. The answer comes back as an empty series — none of them do — which is only meaningful because the sort was chosen on purpose.

Practice it

Work through an expanding exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/expanding/

Go deeper

rolling is this method with a window that forgets, and shift is how you line a running figure up against the row it should be compared to. sort_index is the prerequisite for all three. For fixed calendar buckets instead of a growing window, see resample; for the aggregations themselves, mean and max.

More Pandas videos on Python and Pandas with Reuven Lerner.

Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.

.rolling() — when the window should be a fixed width rather than growing
.sum() — for a plain total, or cumsum for a running one without the window machinery

See it on real data

Below are the 2 Bamboo Weekly exercises that use expanding on real-world data — try each one, then study the worked solution.

Part of the Pandas Methods Index. See also practice by skill.

pandas expanding