isin in pandas — practice examples (Bamboo Weekly)

Ask a whole column "are you one of these?" and get a boolean mask back.

How do you filter for several values at once? One value is easy — df['LocationAbbr'] == 'CA' — and two is bearable with an |. By the fifth you have typed the column name five times, inside a line of parentheses and ampersands nobody wants to read a month later.

isin fixes that. Hand it a collection of values, and it asks every element of the column "are you in here?", answering True or False for each. What comes back is a boolean series the same length as the column, so it drops straight into .loc as a row selector.

One distinction first, because readers confuse these two constantly. isin tests membership: the whole value has to equal one of the values you supplied. str.contains tests for a substring, anywhere inside the value. df['type'].isin(['Rule']) matches only the rows whose type is exactly Rule; df['type'].str.contains('Rule') also drags in Proposed Rule. Categories want isin. Sentences want str.contains.

Official documentation: Series.isin, DataFrame.isin, Index.isin

The one argument

s.isin(['CA', 'NY', 'TX'])       # a list, the usual case
s.isin({'CA', 'NY'})             # a set, tuple or range works too
s.isin(other['state'])           # a series: its values, not its index
s.isin(other.index)              # an Index
~s.isin(['CA'])                  # ~ negates: everything else
df.index.isin(['CA', 'NY'])      # ask the index instead of a column
df.isin({'state': ['CA'],        # per-column membership, on a data frame
         'year': [2022]})

values is the whole signature. Passing a series is worth remembering: isin reads its values and ignores its index, which is what you want when the allowed values came out of an earlier result.

The dict form is the one most readers have never met. On a data frame, isin takes a mapping of column name to the values allowed in that column, and returns a boolean data frame of the same shape. Columns you leave out come back all False. Pair it with .all(axis='columns') and you have an AND across several columns in one call.

A worked example, on real data

The CDC publishes results from every fertility clinic in the United States — the same data set behind Bamboo Weekly #55:

import pandas as pd

url = ('https://data.cdc.gov/api/views/9tjt-seye/'
       'rows.csv?accessType=DOWNLOAD')

df = pd.read_csv(url)
df.shape

(61260, 28)

One row per clinic, question and patient group, for 457 clinics in 51 locations. Three states at once:

df['LocationAbbr'].isin(['CA', 'NY', 'TX']).value_counts()

LocationAbbr
False    38191
True     23069
Name: count, dtype: int64

That mask is the whole point. Chain it, and ask something real — how the success rate for intended egg retrievals falls with the patient's age:

live_births = 'Percentage of intended retrievals resulting in live-birth deliveries'

(
    df
    .loc[lambda df_: df_['Question'] == live_births]
    .loc[lambda df_: df_['Breakout'].isin(['<35', '35-37', '38-40', '>40'])]
    .groupby('Breakout')['data_value_num']
    .median()
    .reindex(['<35', '35-37', '38-40', '>40'])
)

Breakout
<35      46.40
35-37    31.95
38-40    19.50
>40       5.30
Name: data_value_num, dtype: float64

The median clinic turns 46% of retrievals into a live birth for patients under 35, and 5% for patients over 40.

Two columns at once is where the dict form earns its place:

(
    df[['LocationAbbr', 'Breakout']]
    .isin({'LocationAbbr': ['CA', 'NY', 'TX'], 'Breakout': ['<35', '>40']})
    .all(axis='columns')
    .sum()
)

And after a groupby, the labels you care about have moved into the index, so ask the index:

(
    df
    .loc[lambda df_: df_['Question'] == live_births]
    .loc[lambda df_: df_['Breakout'] == '<35']
    .groupby('LocationAbbr')['data_value_num']
    .median()
    .loc[lambda s_: s_.index.isin(['CA', 'MA', 'NY', 'TX'])]
)

LocationAbbr
CA    39.60
MA    52.45
NY    44.55
TX    46.60
Name: data_value_num, dtype: float64

Five mistakes people make

Passing a string instead of a list. This is the classic — a string is iterable, so you expect isin to test the characters. In Pandas 3 it refuses, which is a mercy:

df['LocationAbbr'].isin('CA')
# TypeError: only list-like objects are allowed to be passed to isin(),
#            you passed a `str`

The fix is two brackets. What Pandas cannot catch is the same mistake in disguise: any other iterable of characters passes silently. df['LocationAbbr'].isin(set('CA')) asks whether each state code is 'C' or 'A', returns 0, and says nothing. Zero rows with no error is the signature of this whole family of bugs.

A dtype mismatch that matches nothing and complains about nothing. The Data_Value column holds its numbers as text:

df['Data_Value'].isin([87, 92]).sum()      # 0
df['Data_Value'].isin(['87', '92']).sum()  # 69

isin compares by equality, and '87' == 87 is False in Python. Nothing matched, nothing was raised. Check the dtype before you believe a zero, and fix the column with astype rather than rewriting your list to match the mess.

Case and whitespace. The same failure, one layer up:

df['Clinic Status'].isin(['open']).sum()                          # 0
df['Clinic Status'].str.strip().str.lower().isin(['open']).sum()  # 56570

Normalize the column, then compare against normalized values. Some mismatches survive normalizing: this file's TopicId column holds TOP06 60,803 times and TOP6 457 times, which no amount of strip and lower will reconcile.

Asking isin about missing values. You have probably read that NaN never matches. In Pandas 3 it depends on the dtype, which is worse:

import numpy as np

df['Breakout'].isin([np.nan]).sum()   # 6855 -- the same as .isna()

On the new str dtype, np.nan, None and pd.NA all match. On a float64 column, np.nan matches and None does not. On a nullable Int64 column it is the other way round. Do not memorize that table — use .isna(), which means the same thing everywhere.

Using not instead of ~.

df.loc[not df['Clinic Status'].isin(['Closed'])]
# ValueError: The truth value of a Series is ambiguous.

not wants one yes-or-no answer, and a mask is 61,260 of them. ~ negates element by element: (~df['Clinic Status'].isin(['Closed'])).sum() gives 60,590. It is the same rule as & and | inside .loc — Python's words are for scalars, the operators are for series.

Where it shows up in Bamboo Weekly

51 Bamboo Weekly solutions call isin, which makes it one of the most-used methods in the archive. Four worth reading, all free:

Bamboo Weekly #55: IVF is this page's data set, one edition earlier. It pulls two of the 48 question types out at once with .isin(['Number of retrievals', 'Number of transfers']), then pivots by state.

Bamboo Weekly #75: Refugees is the .index.isin() example. The World Bank's file mixes real countries with aggregates like "Euro area" and "Sub-Saharan Africa," so I build a list of genuine countries once, then reuse .loc[lambda df_: df_.index.isin(countries)] in answer after answer.

Bamboo Weekly #41: Wine production passes an index rather than a list: df['Region/Country'].isin(greatest_producers.index) keeps only the countries that survived an earlier calculation.

Bamboo Weekly #29: Auto accidents uses df['TIME'].isin(range(2017, 2023)) — any iterable will do, and a range of years reads better than typing six of them.

Practice it

Work through an isin exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/isin/

Go deeper

isin produces a mask, so its partner is always .loc, which turns the mask into rows. filter is the similar-sounding method that does something else entirely — it selects by label, not by value. When your column holds sentences rather than categories, str.contains is what you wanted. And value_counts is the fastest way to see what a column really holds, which is how most isin mysteries end.

More Pandas videos on Python and Pandas with Reuven Lerner.

Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.

.str.contains() — when you want a substring match rather than membership in a set
.filter() — when you want to select columns by their names rather than rows by their values
.query() — when the condition is easier to read written out as a string
.loc() — when the mask should also choose which columns come back

See it on real data

Below are the 51 Bamboo Weekly exercises that use isin on real-world data — try each one, then study the worked solution.

Part of the Pandas Methods Index. See also practice by skill.

pandas isin