Skip to content
Bamboo Weekly

pandas apply

Run your own function across rows, columns, or groups — and know when not to.

Have you ever written apply, watched it work, and then wondered why your notebook suddenly felt sluggish? That is this method in one sentence. apply is the escape hatch: you hand Pandas a Python function, and Pandas calls it for you — once per element, once per column, once per row, or once per group, depending on what you called it on. It will do anything, which is why it deserves a page that spends as much space on when to avoid it as on how to use it.

The rule I teach in every class: if a vectorized operation can express what you want, use the vectorized operation. apply is for when nothing else fits, and that is rarer than most people think.

Official documentation: Series.apply, DataFrame.apply, and GroupBy.apply

The arguments that earn their keep

There are really two methods here wearing one name, and the difference is what your function receives:

s.apply(f)                        # a series: f gets one value at a time
df.apply(f)                       # a data frame: f gets one whole column
df.apply(f, axis='columns')       # a data frame: f gets one whole row
df.groupby('x').apply(f)          # f gets one sub-frame per group

Series.apply is element-wise. DataFrame.apply is never element-wise; it hands your function an entire series and lets you look across it. That is the single most useful thing to know about this method, and it is why map, not apply, is the right element-wise tool on a data frame.

Beyond axis, four arguments matter:

s.apply(f, args=(70,))            # extra positional arguments
s.apply(f, threshold=70)          # extra keyword arguments, passed straight through
df.apply(f, axis='columns', result_type='expand')   # a list per row becomes columns
df.apply(f, axis='columns', raw=True)               # f gets a NumPy array, not a series

A worked example, on real data

Bamboo Weekly #36 worked with the Nobel Prize API, which is still the best small data set I know for practicing apply, because the interesting parts arrive as nested Python objects that no vectorized operation can touch:

import pandas as pd

url = 'https://api.nobelprize.org/2.1/laureates?limit=100000'

df = pd.json_normalize(
    pd.read_json(url, typ='series',
                 storage_options={'User-Agent': 'Mozilla/5.0'})
    ['laureates'])

That is 1,018 laureates and 110 columns. The API refuses Pandas' default user-agent, hence the storage_options — the same trick Wikipedia needs.

Each laureate's nobelPrizes column holds a list of dictionaries. Counting them is a job for Series.apply, because there is no vectorized way to call len on a Python list:

df['nobelPrizes'].apply(len).value_counts()

nobelPrizes
1    1011
2       6
3       1
Name: count, dtype: int64

The one three-time winner is not a person at all: it is the International Committee of the Red Cross. Now to flatten those dictionaries into columns. When your function returns a series, apply stacks the results into a data frame — the trick I called "pretty cool" back in #36, and I stand by that:

def prize_fields(prize):
    return pd.Series({'year': int(prize['awardYear']),
                      'category': prize['category']['en'],
                      'portion': prize['portion']})

df['nobelPrizes'].explode().apply(prize_fields).head()

   year           category portion
0  2001  Economic Sciences     1/3
1  1975            Physics     1/3
2  2004          Chemistry     1/3
3  1982          Chemistry       1
4  2021         Literature       1

Joined back onto the names and birth years, that gives one row per prize:

prizes = (
    df
    .assign(name=pd.col('knownName.en'),
            born=lambda df_: pd.to_numeric(df_['birth.year'], errors='coerce'))
    .explode('nobelPrizes')
    .pipe(lambda df_: df_[['name', 'born', 'gender']]
                      .join(df_['nobelPrizes'].apply(prize_fields)))
    .assign(age=pd.col('year') - pd.col('born'))
    .reset_index(drop=True)
)

1,044 rows. Note that the age calculation is not apply — it is plain subtraction inside assign, and the next section shows what that choice is worth.

axis='rows': one whole column at a time

The default axis hands your function each column. This is where apply shines, because a function called six times costs nothing. first_valid_index is a series method with no data-frame equivalent, so apply is the only way to run it across a table — exactly the move in #47:

(
    prizes
    .assign(decade=pd.col('year') // 10 * 10)
    .pivot_table(index='decade', columns='category', values='name',
                 aggfunc='count')
    .apply(pd.Series.first_valid_index)
)

category
Chemistry                 1900
Economic Sciences         1960
Literature                1900
Peace                     1900
Physics                   1900
Physiology or Medicine    1900
dtype: int64

Five prizes date from 1901; the economics prize was added almost seventy years later. Passing pd.Series.first_valid_index — an unbound method — instead of a lambda is worth copying.

axis='columns': one whole row at a time

Same method, opposite axis, and a completely different performance profile:

(
    prizes
    .set_index('name')
    .apply(lambda row: row['year'] - row['born'], axis='columns')
    .nlargest(4)
)

name
John B. Goodenough    97.0
Arthur Ashkin         96.0
John J. Hopfield      91.0
Klaus Hasselmann      90.0
dtype: float64

Goodenough was 97. The answer is right, and the code is a mistake. Here is the same subtraction four ways, timed on these 1,044 rows:

apply(axis='columns')             2.428 ms
apply(axis='columns', raw=True)   0.789 ms
assign + pd.col                   0.085 ms
df['year'] - df['born']           0.023 ms

The row-wise apply is 106 times slower than the subtraction, and 29 times slower than the same subtraction written with pd.col inside assign. On a thousand rows nobody notices. In #128, on a climate data set of hundreds of millions of temperature readings, I measured the identical comparison at 2.38 seconds versus two minutes.

Column-wise apply, by contrast, cost 0.177 ms on the same frame — cheaper than the built-in aggregation I compared it against, because the function ran three times rather than 1,044. The cost of apply is not apply. It is how many times Python has to call your function.

args, keywords, and result_type

Anything after the function goes to the function:

def in_words(age, unit='years old', prefix=''):
    return f'{prefix}{age:.0f} {unit}'

youngest = prizes.set_index('name')['age'].nsmallest(4)

youngest.apply(in_words, args=('at the ceremony',))   # positional
youngest.apply(in_words, prefix='aged ')              # keyword

The second call gives:

name
Malala Yousafzai    aged 17 years old
Lawrence Bragg      aged 25 years old
Nadia Murad         aged 25 years old
Carl D. Anderson    aged 31 years old
Name: age, dtype: str

result_type matters only when axis='columns' and your function returns something list-like. A returned series already becomes columns, named after its index. A returned list does not, until you ask:

def share_list(row):
    numerator, _, denominator = row['portion'].partition('/')
    return [int(numerator) / int(denominator or 1), denominator == '']

prizes.head(4).apply(share_list, axis='columns', result_type='expand')

          0      1
0  0.333333  False
1  0.333333  False
2  0.333333  False
3  1.000000   True

Without it you get a single column of Python lists. Return a series instead and you get named columns for free, which is why I almost never type result_type.

groupby(...).apply()

The group form hands your function one sub-frame per group. It is the right tool when the answer for a group needs more than one column, or needs whole rows:

(
    prizes
    .dropna(subset='age')
    .groupby('category')
    .apply(lambda sub: sub.nsmallest(1, 'age')[['name', 'year', 'age']].squeeze())
)

                                        name  year   age
category
Chemistry                    Frédéric Joliot  1935  35.0
Economic Sciences               Esther Duflo  2019  47.0
Literature                   Rudyard Kipling  1907  42.0
Peace                       Malala Yousafzai  2014  17.0
Physics                       Lawrence Bragg  1915  25.0
Physiology or Medicine  Frederick G. Banting  1923  32.0

Malala Yousafzai at 17, and no chemist under 35. If all you need is one column summarized, agg does it faster and reads better; see groupby.

Five mistakes people make

Reaching for apply when an operator would do. The numbers above are the argument. Arithmetic, comparisons, and the str and dt accessors all run in compiled code across a whole column at once. Before writing apply, ask whether assign with pd.col says the same thing; in Pandas 3 it usually does, and pd.col has quietly removed most of the remaining excuses. If you genuinely need a row-wise function, raw=True hands it a NumPy array instead of a series and buys back a factor of three.

Passing axis to a series. Series.apply has no axis parameter, so Pandas forwards it to your function, and you get one of the most confusing error messages in the library:

s.apply(lambda x: x, axis=1)
# TypeError: <lambda>() got an unexpected keyword argument 'axis'

Expecting the grouping column inside groupby.apply. This changed, and most advice online is out of date. In Pandas 3 the grouping columns are excluded from the sub-frame your function receives, so sub['category'] raises KeyError: 'category'. The include_groups=True escape hatch that Pandas 2.2 offered is gone: it now raises ValueError: include_groups=True is no longer allowed. The group key is not lost, though — it is on sub.name.

While correcting old advice: groupby(...).apply() used to call your function twice on the first group to pick a fast code path, which made functions with side effects behave bizarrely. Verified on Pandas 3.0.5, that is over — six groups, six calls, in order. The phantom call that remains is on an empty frame, where df.head(0).apply(f, axis='columns') still runs your function once, on a row of NaN.

Returning different shapes from different rows. This one fails silently, which makes it the worst of the five. Here one branch returns a scalar and the other returns a series:

def share(row):
    numerator, _, denominator = row['portion'].partition('/')
    if not denominator:            # a sole winner: portion is just '1'
        return 1.0
    return pd.Series({'share': int(numerator) / int(denominator),
                      'sole_winner': False})

prizes.head(4).set_index('name').apply(share, axis='columns')

                      share sole_winner
name
A. Michael Spence  0.333333       False
Aage N. Bohr       0.333333       False
Aaron Ciechanover  0.333333       False
Aaron Klug         1.000000         1.0

The scalar was broadcast across both columns, so sole_winner reads 1.0 instead of True. Worse, run the same function on a slice where everyone was a sole winner and you get a plain series rather than a data frame. The shape of your result now depends on which rows happen to be present. Make every branch return the same thing.

Using apply where map is clearer. On a series the two are close cousins, and for formatting they time identically. Use map for a dictionary or a lookup series, and apply for a function that takes arguments.

Where it shows up in Bamboo Weekly

Thirty-eight solutions use apply, one of the most-used methods on the site — but the pattern is telling. In 21 of them every single call is Series.apply with a format string, only three ever pass axis='columns', and not one uses groupby(...).apply(). The escape hatch stays mostly shut.

#36: Nobel Prize is the source of the data above, and of the .apply(Series) trick for turning a column of dictionaries into a table.

#47: Minimum wage finds when each US state first reported a minimum wage with df.apply(pd.Series.first_valid_index) — column-wise apply with an unbound method, and no vectorized alternative. #68: Dangerously hot weather asks the same of weather hazards, written as a lambda.

#45: Netflix runs a third-party language detector over show titles with df['Title'].apply(guess_language.guess_language). When the work is an outside Python function, apply is the answer, not a compromise.

#128: Extreme heat is the timing post: a Kelvin-to-Celsius conversion done four ways, with apply finishing dead last by roughly fiftyfold. My verdict there was that the result is "clear evidence, if you ever needed it, that you should avoid apply as much as possible," and nothing since has changed my mind.

Practice it

Work through an .apply() exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/apply/

Go deeper

Most of the time the better answer is another method. map translates values through a dictionary or a function. assign adds computed columns without leaving the chain, and with pd.col the expression stays vectorized. pipe hands a whole data frame to a function of yours exactly once, which is what people usually want when they reach for a row-wise apply. And groupby with agg covers nearly everything groupby(...).apply() gets used for.

More Pandas videos on Python and Pandas with Reuven Lerner.

Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.

See it on real data

Below are the 38 Bamboo Weekly exercises that use apply on real-world data — try each one, then study the worked solution.

Part of the Pandas Methods Index. See also practice by skill.