Skip to content
Bamboo Weekly

pandas notna

The inverse mask of isna, and a short page on purpose.

Why would you want a method that does exactly the opposite of one you already have? Because ~df['x'].isna() is a double negative, and half the time you are not asking where the holes are. You are asking for the rows that have something in them.

.notna() returns a boolean object the same shape as whatever you called it on, True wherever a value is present. It takes no arguments, and .notnull() is an alias that does exactly the same thing. Everything about measuring missingness — the audit, the sentinel values that .isna() cannot see, NaN versus None versus pd.NA — lives on isna, and this page will not repeat it.

Official documentation: DataFrame.notna and Series.notna.

Filtering is the reason it exists

dropna already removes rows with missing values, so it is fair to ask what notna adds. The answer is composition. dropna(subset=['x']) is a whole method call that can only say one thing; .notna() is a term you can drop into a boolean expression alongside every other condition you have.

Our World in Data's CO2 file, filtered three ways at once — real countries only, 2023 only, and only where the reading exists:

import pandas as pd

url = 'https://nyc3.digitaloceanspaces.com/owid-public/data/co2/owid-co2-data.csv'

co2 = pd.read_csv(url,
                  usecols=['country', 'iso_code', 'year',
                           'co2_per_capita', 'energy_per_capita'],
                  storage_options={'User-Agent': 'Mozilla/5.0'})

(co2
 .loc[pd.col('iso_code').notna()
      & pd.col('energy_per_capita').notna()
      & (pd.col('year') == 2023)]
 .nlargest(5, 'energy_per_capita'))

                    country  year iso_code  co2_per_capita  energy_per_capita
37805                 Qatar  2023      QAT          38.841         226847.656
21484               Iceland  2023      ISL           9.708         167421.547
41190             Singapore  2023      SGP           8.508         160276.984
47301  United Arab Emirates  2023      ARE          21.554         149830.328
45601   Trinidad and Tobago  2023      TTO          22.834         106746.781

A dropna would need its own call in the middle of the chain, with a subset= listing the same column again — and a bare .dropna() would also throw away every row missing co2_per_capita, which nobody asked for. The ~ ... .isna() version does work, and ~ binds tighter than & so it needs no extra parentheses, but a stray tilde in a long & chain is genuinely hard to see. Python's own not is not an option at all: not df['x'].isna() raises ValueError: The truth value of a Series is ambiguous, because not wants one answer and you handed it 50,191.

Counting what is there

Summing a boolean gives you the number of True values, so .notna().sum() is a count of the values that exist:

co2.notna().sum()

country              50191
year                 50191
iso_code             42262
co2_per_capita       26182
energy_per_capita    10109
dtype: int64

Two things worth noticing. .count() gives the identical numbers — that is all .count() has ever been. And a count of what is present is often the finding itself. Ask how many countries reported energy use in each of the last few years:

(co2
 .loc[pd.col('iso_code').notna()]
 .assign(reported=pd.col('energy_per_capita').notna())
 .groupby('year')['reported'].sum()
 .tail(6))

year
2018    204
2019    204
2020    204
2021    204
2022     79
2023     79
Name: reported, dtype: int64

Coverage falls off a cliff in 2022. Any chart of "the world in 2023" built on this column is a chart of 79 countries, and nothing in the numbers themselves says so.

Where it shows up in Bamboo Weekly

Bamboo Weekly #166: Income tax is the composition argument in the wild: & df['OBS_VALUE'].notna() sits as the last term of a five-part OECD filter, and later gets bound to a variable and reused across several questions. No dropna call can be stored and passed around like that.

Bamboo Weekly #139: Chinese exports calls it somewhere people forget it works — the index. After a set_index, a few labels are NaN, and .loc[lambda df_: df_.index.notna()] removes those rows.

Bamboo Weekly #132: JetBrains survey uses presence as the answer, for subscribers: an unticked box in the survey is blank, so .filter(regex='exploration.tools.pandas').notna().value_counts(normalize=True) is the share of respondents who use Pandas. Nothing in that chain looks at a value; it only asks whether one is there.

Practice it

Work through a notna exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/notna/

Go deeper

isna is the substantial half of this pair and where the audit material lives. Once you know what is missing, dropna removes it, fillna substitutes a constant, ffill carries the last value forward, and interpolate estimates what lay in between. A notna mask turns into rows through loc, and into a number through count.

More Pandas videos on Python and Pandas with Reuven Lerner.

Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.

See it on real data

Below are the 5 Bamboo Weekly exercises that use notna on real-world data — try each one, then study the worked solution.

Part of the Pandas Methods Index. See also practice by skill.