Fill the holes in a series along a straight line — or, with ffill, along a staircase.
What should go in the hole? interpolate says the gap lies on a line between the values on either side. ffill says nothing changed until you were told otherwise. Both leave a column with no NaN in it, both look equally tidy, and only one is right for any given column.
That is the lesson of this page: filling a gap is not a cleaning step on the way to the analysis. It is a claim about what happened while you were not looking, and it belongs in the analysis, where someone can argue with it.
Official documentation: DataFrame.interpolate, Series.interpolate, DataFrame.ffill and DataFrame.bfill
The arguments that earn their keep
s.interpolate() # straight line, counting rows
s.interpolate(method='time') # straight line, counting days
s.interpolate(method='nearest') # copy the closer neighbor
s.interpolate(limit=2) # fill at most 2 in a row
s.interpolate(limit_area='inside') # only between real values
s.interpolate(limit_direction='both') # also fill before the first value
s.ffill() # carry the last value forward
s.bfill() # carry the next value backward
s.ffill(limit=3) # ...but not forever
method is the one that changes your answers. The default, 'linear', draws its line across row positions and ignores the index entirely — the middle row of a three-row hole sits halfway along, whether those rows are three seconds or three years apart. 'time' draws the same line across the actual timestamps, and needs a DatetimeIndex. 'nearest' gives up on lines and copies whichever real value is closer, and needs SciPy.
The other three decide where the filling stops: limit caps how many consecutive gaps get filled, limit_direction says which way it travels, and limit_area='inside' fills only between two real observations.
A worked example, on real data
The St. Louis Fed publishes FRED as plain CSV, no key required. Three daily series make a good frame, because each is missing values for a different reason.
import pandas as pd
url = ('https://fred.stlouisfed.org/graph/fredgraph.csv'
'?id={}&cosd=2001-01-01&coed=2025-06-30')
rates = (
pd.read_csv(url.format('DGS10'), parse_dates=['observation_date'],
index_col='observation_date')
.join([pd.read_csv(url.format('DFEDTARU'), parse_dates=['observation_date'],
index_col='observation_date'),
pd.read_csv(url.format('DTWEXM'), parse_dates=['observation_date'],
index_col='observation_date')])
.rename(columns={'DGS10': 'yield_10y', 'DFEDTARU': 'fed_target',
'DTWEXM': 'dollar'})
.rename_axis('date')
)
rates.isna().sum()
yield_10y 265
fed_target 2075
dollar 1577
dtype: int64
Three columns, 6,390 rows, three different stories. The 10-year Treasury yield is missing on market holidays, scattered through the middle. The Fed's target range is missing at the start: it only began announcing a range in December 2008. The dollar index is missing at the end, discontinued on the last day of 2019.
linear counts rows; time counts days
Veterans Day 2024 fell on a Monday, and the bond market was shut — a hole with Friday on one side and Tuesday on the other:
yields = rates['yield_10y']
(
pd.DataFrame({'yield_10y': yields,
'linear': yields.interpolate(),
'time': yields.interpolate(method='time'),
'nearest': yields.interpolate(method='nearest')})
.loc['2024-11-07':'2024-11-13']
)
yield_10y linear time nearest
date
2024-11-07 4.31 4.310 4.3100 4.31
2024-11-08 4.30 4.300 4.3000 4.30
2024-11-11 NaN 4.365 4.3975 4.43
2024-11-12 4.43 4.430 4.4300 4.43
2024-11-13 4.44 4.440 4.4400 4.44
Three answers for one Monday. 'linear' sees one row before and one after, so it splits the difference. 'time' sees a Monday three days after Friday and one day before Tuesday, and puts it three-quarters of the way along. 'nearest' shrugs and copies Tuesday.
That is the trap, and a quiet one: an index made of dates is not an index made of evenly spaced dates, and plain 'linear' cannot tell the difference. Business days, event logs, survey waves — the rows are uneven and the default pretends otherwise. Across these 24 years the two methods disagree on 198 of the 265 missing days, by a median of one basis point and a maximum of six. Small here; not small in a series that jumps, and never a number you measured.
Where the filling stops
Markets were closed for two days after September 11, 2001. limit decides how far into a hole like that you will go:
window = yields.loc['2001-09-07':'2001-09-17']
pd.DataFrame({'yield_10y': window,
'time': window.interpolate(method='time'),
'limit_1': window.interpolate(method='time', limit=1)})
yield_10y time limit_1
date
2001-09-07 4.80 4.800000 4.800000
2001-09-10 4.84 4.840000 4.840000
2001-09-11 NaN 4.773333 4.773333
2001-09-12 NaN 4.706667 NaN
2001-09-13 4.64 4.640000 4.640000
2001-09-14 4.57 4.570000 4.570000
2001-09-17 4.63 4.630000 4.630000
The fill runs forward, so with limit=1 September 11 is filled and September 12 stays empty — a good house rule, since a one-day hole is a holiday and a ten-day hole is a story you have not investigated yet.
The dollar index is the argument I would make mandatory if I could. It ends on December 31, 2019, and then nothing:
rates['dollar'].interpolate().loc['2025-06-24':'2025-06-30']
date
2025-06-24 90.8221
2025-06-25 90.8221
2025-06-26 90.8221
2025-06-27 90.8221
2025-06-30 90.8221
Name: dollar, dtype: float64
A plain interpolate took a series that stopped existing five and a half years ago and carried its last value to the end of the frame — 1,577 missing values in, zero out. Only 143 were interior gaps; the other 1,434 are the flat line of a dead index, and nothing in the output says so.
rates['dollar'].interpolate(limit_area='inside').isna().sum() # 1434
limit_area='inside' fills the 143 holes between two genuine observations and refuses the rest. The leading edge is safer by default: fed_target has 2,075 missing rows before December 2008, and a plain interpolate leaves every one alone. Pass limit_direction='both' and Pandas backfills 0.25 into January 2001, when the target was 6.5.
When the world is a staircase
The Fed's target rate does not drift between meetings. It sits still, then jumps. Pull out the 29 rows where it moved and put them back on the daily grid, which is how announcement data usually arrives:
decisions = rates['fed_target'].dropna().loc[lambda s_: s_.diff() != 0]
announced = decisions.reindex(rates.index)
(
pd.DataFrame({'announced': announced,
'ffill': announced.ffill(),
'time': announced.interpolate(method='time')})
.loc['2024-09-16':'2024-09-25']
)
announced ffill time
date
2024-09-16 NaN 5.5 5.003571
2024-09-17 NaN 5.5 5.002381
2024-09-18 NaN 5.5 5.001190
2024-09-19 5.0 5.0 5.000000
2024-09-20 NaN 5.0 4.995000
2024-09-23 NaN 5.0 4.980000
2024-09-24 NaN 5.0 4.975000
2024-09-25 NaN 5.0 4.970000
The ffill column is the truth: 5.5 right up to September 19, when the cut took effect, and 5.0 from then on. The time column is fiction — a gentle slide through 5.0035 and 4.995, rates the Fed has never set and never will, because it moves in quarter-point steps. Interpolation assumes the quantity was doing something in between. For a policy rate, a price list or a status flag, it was not.
ffill also handles what interpolate will not touch: in Pandas 3 a text column raises TypeError: Cannot interpolate with str dtype, while ffill carries the label down without complaint. Which is why the most common ffill in Bamboo Weekly has nothing to do with time — read a spreadsheet with merged cells, and the group label appears once followed by blanks. Putting it back on every row is not a modeling decision, it is repairing a file format, and it is the one gap-fill you can do without arguing with anybody.
interpolate, ffill, or a constant?
Three questions, in order. Does the quantity vary continuously between observations, so a value in between means something? Then interpolate, with method='time' if the index is dates. Does it hold its last value until something changes it? Then ffill. Is the absence itself the measurement — no sales that month, no earthquakes above 5.5 in Japan in February? Then neither: the honest fill is a constant, and fillna is the page for that.
Four mistakes people make
Using the default on a datetime index with uneven spacing. The one above. If your rows are dates and are not evenly spaced, method='time' is what you meant, and it fails loudly rather than quietly when it cannot help: ValueError: time-weighted interpolation only works on Series or DataFrames with a DatetimeIndex.
Interpolating past the end of your real data. A plain interpolate fills trailing gaps by repeating the last observation, so a series that stopped in 2019 becomes a flat line running to today — a chart of a stable market rather than a dead one. Make limit_area='inside' a habit rather than a rescue.
Forward-filling a price, then computing a change from it. ffill puts Friday's yield on Monday's holiday, so the change on that Monday is exactly zero and Tuesday absorbs the full two-day move:
z = pd.DataFrame({'yield_10y': yields, 'ffilled': yields.ffill()})
z.assign(change=z['ffilled'].diff()).loc['2024-11-07':'2024-11-13']
yield_10y ffilled change
date
2024-11-07 4.31 4.31 -0.11
2024-11-08 4.30 4.30 -0.01
2024-11-11 NaN 4.30 0.00
2024-11-12 4.43 4.43 0.13
2024-11-13 4.44 4.44 0.01
In 2024 that turns 16 genuinely flat days into 27 and drops the standard deviation of the daily change from 0.0575 to 0.0563. Fill for a chart if you like; drop the missing rows before you diff or pct_change.
Interpolating something you can count. Take the USGS grid of monthly magnitude-5.5 earthquake counts from the fillna page, and run the months through interpolate:
url = ('https://earthquake.usgs.gov/fdsnws/event/1/query.csv'
'?starttime=2024-01-01&endtime=2024-12-31&minmagnitude=5.5')
quakes = (
pd.read_csv(url, usecols=['time', 'place', 'mag'], parse_dates=['time'])
.assign(country=pd.col('place').str.split(', ').str[-1],
month=pd.col('time').dt.month)
)
top = list(quakes['country'].value_counts().head(5).index)
(
quakes
.loc[pd.col('country').isin(top)]
.pivot_table(index='country', columns='month', values='mag', aggfunc='count')
.interpolate(axis='columns')
)
month 1 2 3 4 5 6 7 8 9 10 11 12
country
Alaska 1.0 1.0 1.0 1.5 2.0 1.5 1.0 1.0 1.0 1.0 1.0 10.0
Indonesia 1.0 1.0 4.0 4.0 3.0 2.0 1.5 1.0 2.0 3.0 2.5 2.0
Japan 6.0 4.0 2.0 4.0 1.0 1.0 1.0 2.0 3.0 3.0 3.0 3.0
Papua New Guinea NaN 1.0 3.0 1.0 5.0 1.0 1.0 2.5 4.0 1.0 2.0 2.0
Tonga 5.0 2.0 1.0 1.0 2.0 3.0 2.0 3.0 1.0 2.0 2.0 2.0
One and a half earthquakes in Alaska in April, three in Japan in December. Neither happened: those months had no quake above 5.5, which is exactly what the NaN was telling us, and the honest fill is zero. Anything countable, and anything categorical, wants fillna or ffill — never a line drawn between two integers.
Where it shows up in Bamboo Weekly
Thirteen Bamboo Weekly exercises fill gaps this way. Four worth reading:
Bamboo Weekly #60: Iceland is this whole page in one sentence. Iceland's tourism figures are missing for 2018, and the solution says so out loud before calling interpolate: "Assuming — and this isn't always a good assumption — that the missing data would make sense as the mean of the values on either side."
Bamboo Weekly #54: Household debt interpolates with axis='columns', because the quarters run sideways, then computes pct_change on the result — the third mistake above, worth studying deliberately.
Bamboo Weekly #25: Entrepreneurship interpolates to make a line chart readable, the use I am most comfortable with: the filled points are presentation, not measurement.
Bamboo Weekly #84: Central banks is the staircase on real data: announcements arrive on the days banks make them, and ffill turns them into a rate for every day. Subscribers only, as is Bamboo Weekly #122: Economic growth, which uses ffill for the other reason — repairing merged cells in a World Bank spreadsheet.
Practice it
Work through an interpolate exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/interpolate/
Go deeper
interpolate sits in a small family: fillna for constants, ffill and bfill for carrying values, and resample for changing the grid the gaps sit on in the first place. The Pandas missing data user guide has the full list of method values, including the SciPy-backed 'polynomial' and 'spline'.
More Pandas videos on Python and Pandas with Reuven Lerner.
Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.
Related methods
.isna()— when you want to see where the gaps are before filling them.dropna()— when removing the rows is more honest than inventing values for them.pct_change()— which is where invented values do the most damage.fillna()— when a constant is more honest than a line drawn through the gap
See it on real data
Below are the 8 Bamboo Weekly exercises that use interpolate on real-world data — try each one, then study the worked solution.
- Bamboo Weekly #145: Economic indicators
- Bamboo Weekly #125: Shrinking dollars
- Bamboo Weekly #97: Drones
- Bamboo Weekly #95: Tariffs
- Bamboo Weekly #93: Anti-politics
- Bamboo Weekly #60: Iceland
- Bamboo Weekly #54: Household debt
- Bamboo Weekly #25: Entrepreneurship
Part of the Pandas Methods Index. See also practice by skill.