Find the missing values, count them, and look at where they cluster.
How complete is the file you just loaded? Nobody knows that from the first head(), and guessing is how a chart ends up describing a quarter of the data it claims to describe. .isna() is how you stop guessing. It returns a boolean object the same shape as whatever you called it on, True wherever a value is missing. .notna() is its exact opposite. Neither one changes your data or removes anything. They only report.
Three lines will cover most of what you ever want from them:
df.isna() # a True/False frame, same shape as df
df.isna().sum() # how many are missing, per column
df.isna().sum().sort_values(ascending=False) # ...worst column first
The second line works because True is 1 and False is 0, so summing a boolean column counts its True values. That one expression is the first thing I run on any file I have not seen before, and it decides what I am willing to ask of the data.
Official documentation: DataFrame.isna, DataFrame.notna, Series.isna and Series.notna.
No arguments, and two spare names
.isna() and .notna() take no arguments at all. There is nothing to configure and nothing to get wrong; the whole question is what you do with the boolean they hand back.
isnull and notnull are not similar methods. They are the same methods under a second name. Ask Pandas 3.0 for the source and the body is one line:
import inspect
import pandas as pd
print(inspect.getsource(pd.DataFrame.isnull))
def isnull(self) -> DataFrame:
"""
DataFrame.isnull is an alias for DataFrame.isna.
...
"""
return self.isna()
The null spelling is inherited vocabulary, from the SQL and R world a lot of Pandas's early users came from. It has never been deprecated, so there is no correctness argument for either name — only a consistency one, which is the argument that matters. Code that says isnull in one chain and isna in the next makes a reader stop and wonder whether the difference means something.
Pick one and hold to it. I would pick isna, because it matches dropna, fillna, NaN and pd.NA — every other missing-data name in the library uses "na" — and because null in Python already means None, which is only one of the several things Pandas counts as missing. Across the 455 Bamboo Weekly posts, .isna() and .notna() appear in 18 solutions and .isnull() and .notnull() appear in none.
There is also a top-level pd.isna(), which takes a single value where the method cannot: pd.isna(np.nan), pd.isna(None), pd.isna(pd.NaT) and pd.isna(pd.NA) are all True, and pd.isna('') is False. That last one comes back later.
The audit, on real data
Our World in Data publishes a single CSV with every country's carbon emissions back to 1750, no key and no signup:
import pandas as pd
url = ('https://nyc3.digitaloceanspaces.com/owid-public/data/co2/'
'owid-co2-data.csv')
df = pd.read_csv(url)
df.shape
(50191, 79)
Seventy-nine columns is more than anyone reads by hand, and the file goes back 274 years, so most of it cannot possibly be filled in. Start with the mask itself, on five rows of Vietnam, to see the shape of what comes back:
(
df
.loc[(pd.col('country') == 'Vietnam') & pd.col('year').between(1988, 1992),
['year', 'iso_code', 'co2', 'consumption_co2', 'methane']]
.set_index('year')
.isna()
)
iso_code co2 consumption_co2 methane
year
1988 False False True False
1989 False False True False
1990 False False False False
1991 False False False False
1992 False False False False
Same rows, same columns, True and False instead of values. Notice that consumption_co2 flips in 1990 and stays flipped — that is not damage, that is a data source that starts in 1990, and we will come back to it.
Nobody eyeballs 50,191 rows, though. Sum the mask and sort it:
df.isna().sum().sort_values(ascending=False).head(6)
share_global_cumulative_other_co2 48083
share_global_other_co2 48083
other_co2_per_capita 47717
other_industry_co2 46989
cumulative_other_co2 46989
consumption_co2_per_gdp 45747
dtype: int64
Six nearly empty columns, found in one line. Sorting descending is what makes this an audit rather than a wall of numbers: the columns that will ruin your analysis come first, and you can stop reading once the numbers get small.
Counts are hard to judge without the denominator, though, so I usually want the percentage instead. .mean() on a boolean gives you exactly that, because the mean of a column of ones and zeros is the fraction of ones:
(df.isna().mean() * 100).round(1).sort_values(ascending=False).head(6)
share_global_cumulative_other_co2 95.8
share_global_other_co2 95.8
other_co2_per_capita 95.1
other_industry_co2 93.6
cumulative_other_co2 93.6
consumption_co2_per_gdp 91.1
dtype: float64
Same ranking, and now I know what 48,083 meant: 95.8 percent. Take the mean twice and the whole file collapses to one number — df.isna().mean().mean() is 0.531, so this data set is 53 percent holes. Across 274 years and 79 indicators that is not a scandal, but it is a shape worth knowing before you average anything.
Where the gaps are is often the finding
The audit tells you how much is missing. The more interesting question is which rows, and the answer is frequently not "at random."
The iso_code column in this file is missing 7,929 times. That is 15.8 percent, which is small enough that most people scroll past it on the way to the real problems. Scrolling past it is the mistake. Ask which countries those rows belong to:
df['iso_code'].isna().sum() # 7929
df.loc[pd.col('iso_code').isna(), 'country'].nunique() # 37
Thirty-seven, out of 255 names in that column. And here they are:
df.loc[pd.col('iso_code').isna(), 'country'].drop_duplicates().head(12).to_list()
['Africa', 'Africa (GCP)', 'Asia', 'Asia (GCP)',
'Asia (excl. China and India)', 'Central America (GCP)', 'Europe',
'Europe (GCP)', 'Europe (excl. EU-27)', 'Europe (excl. EU-28)',
'European Union (27)', 'European Union (28)']
None of those is a country. The rows missing an ISO code are the continents, the income bands, the trading blocs, "International aviation," "World" — every aggregate row, mixed into the same column as the real countries, because that is how OWID ships the file. There is no is_aggregate flag. The missing ISO code is the flag.
Which means the mask is not a nuisance here, it is the tool. Ask for the biggest emitters of 2023 without it:
df.loc[pd.col('year') == 2023].set_index('country')['co2'].nlargest(5)
country
World 37791.570
Non-OECD (GCP) 25442.199
Asia 22600.221
Asia (GCP) 19747.367
Upper-middle-income countries 17581.070
Name: co2, dtype: float64
Five answers, no countries, and every one of them technically correct. Now add .notna():
(
df
.loc[pd.col('iso_code').notna() & (pd.col('year') == 2023)]
.set_index('country')
['co2']
.nlargest(5)
)
country
China 11902.503
United States 4911.391
India 3062.324
Russia 1815.925
Japan 988.785
Name: co2, dtype: float64
That is the question I actually asked. One boolean, and the aggregates step out of the ranking.
The general form of this move is to assign the mask as a column and group by something. Here is consumption_co2, the column that flipped in 1990 up above, across real countries only:
(
df
.loc[pd.col('iso_code').notna()]
.assign(decade=pd.col('year') // 10 * 10,
missing=pd.col('consumption_co2').isna())
.groupby('decade')
['missing'].mean()
.loc[1960:]
.round(2)
)
decade
1960 1.00
1970 1.00
1980 1.00
1990 0.45
2000 0.45
2010 0.45
2020 0.59
Name: missing, dtype: float64
Nothing before 1990, then a flat 45 percent. Consumption-based emissions require trade data that simply was not compiled earlier, and the 45 percent is the set of countries that still do not report it. So a column that is 90 percent empty across the whole file is better than half populated for the years it actually covers, which is a completely different fact — and .isna() grouped by decade is what tells you which of the two you are looking at.
Four mistakes people make
Comparing to np.nan or None instead of asking .isna(). This one is silent, which is why it is first:
import numpy as np
(df['iso_code'] == np.nan).sum() # 0
(df['iso_code'] == None).sum() # 0
df['iso_code'].isna().sum() # 7929
np.nan == np.nan is False. That is IEEE 754 behaving as designed: a missing value is not equal to anything, including itself, because "unknown" and "unknown" are not evidence of a match. So the comparison is False on every row, the filter selects nothing, and no error is raised. There is no way to write this test with ==; .isna() is the only way to ask.
Trusting .isna() about strings. Empty and whitespace-only strings are perfectly good strings, and Pandas counts them as present. NOAA publishes a daily weather summary for every station in the world; here is Chicago O'Hare for 2024:
gsod = ('https://www.ncei.noaa.gov/data/global-summary-of-the-day/'
'access/2024/72530094846.csv')
weather = pd.read_csv(gsod)
weather.shape # (366, 28)
weather.isna().sum().sum() # 0
Not one missing value in 10,248 cells. A perfect file — except that its MAX_ATTRIBUTES column holds exactly two distinct values, and one of them is a space:
weather['MAX_ATTRIBUTES'].unique()
<ArrowStringArray>
[' ', '*']
Length: 2, dtype: str
weather['MAX_ATTRIBUTES'].isna().sum() # 0
weather['MAX_ATTRIBUTES'].str.strip().eq('').sum() # 331
Three hundred and thirty-one days out of 366 carry a blank there, and .isna() reports zero. When a text column looks suspiciously complete, run .str.strip().eq('') on it before you believe the audit.
Assuming a sentinel is a NaN. The same weather file has a worse version of the problem, in its numbers. Watch what describe says about the wind:
weather[['TEMP', 'GUST', 'SNDP']].describe().round(1)
TEMP GUST SNDP
count 366.0 366.0 366.0
mean 55.0 280.8 923.6
std 18.5 429.4 265.5
min -6.9 14.0 1.2
25% 40.2 21.0 999.9
50% 58.4 27.5 999.9
75% 70.9 999.9 999.9
max 87.7 999.9 999.9
The mean wind gust at O'Hare in 2024 was 280 miles per hour. Of course it was not: 999.9 is NOAA's code for "not recorded," on 96 days for GUST and 338 for SNDP, and .isna() cannot see any of it, because those cells contain a number. Every agency invents its own marker — 999.9, -1, 9999, Unspecified, N/A — and you catch them in the min and max rows of describe for numbers, and in value_counts for text. Then declare them at load time with na_values= in read_csv, so the rest of the chain can trust .isna().
Putting a whole frame in an if. .isna() on a data frame returns a data frame, and a data frame has no single truth value:
if weather.isna():
print('there are gaps')
ValueError: The truth value of a DataFrame is ambiguous.
Use a.empty, a.bool(), a.item(), a.any() or a.all().
What you meant was .any(), twice — once to collapse each column to a single boolean, and once to collapse those:
weather.isna().any().any() # False
On a series one .any() is enough. And if you want the count rather than the answer, df.isna().sum().sum() totals the whole frame the same way.
Where it shows up in Bamboo Weekly
Bamboo Weekly #13: Python developers is the ranking audit turned into an actual answer. The JetBrains survey stores each respondent's languages across a family of primary_proglang columns, where an unused language is simply blank — so (~df[proglang_cols].isna()).sum().sort_values(ascending=False).head(10) counts the answers that were given, and ranks the languages. The surprise in that issue is that Python is not first.
Bamboo Weekly #159: State of the Union takes it further and measures nothing but the missingness. In a table of every State of the Union address by president and year, a gap means no address that year, so df.notna().sum(axis='columns') is a count of speeches. The absence is the data, and the whole chart is built out of .notna().
Bamboo Weekly #133: Wind power is the ISO-code trick above, in the wild. Ember's global electricity file mixes countries with regions and income groups exactly as OWID does, and every query in that solution opens with .loc[lambda df_: df_['ISO 3 code'].notna()] before it does anything else — seventeen times in one issue.
Bamboo Weekly #12: Tourism applies .isna() somewhere people forget it works: the index. After set_index('country') a few rows have NaN for a label, and df.loc[~df.index.isna()] removes them. An index is close enough to a series that most of these methods are available on it.
Practice it
Work through an .isna() exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/isna/
Go deeper
Measuring is the easy half. Once you know what is missing you have three choices, and this page deliberately does not make them for you: dropna removes the rows or columns and works out what that costs, fillna substitutes a constant or a per-column mapping, and interpolate estimates values that lie between their neighbors. Keeping them is a fourth choice people forget: value_counts(dropna=False) and groupby(dropna=False) both give the missing group a line of its own.
Nearby: loc is what turns an .isna() mask into rows, describe is where the sentinel values hide, and the Pandas missing data user guide explains pd.NA and the nullable dtypes, which are how you keep an integer column integer when it has gaps in it.
More Pandas videos on Python and Pandas with Reuven Lerner.
Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.
Related methods
.dropna()— when you have found the missing values and want to remove the rows.interpolate()— when you would rather fill the gaps than remove or count them.describe()— when the missing values are hiding as a sentinel like 999 rather than as NaN.count()— when you want to count what is there rather than what is absent.fillna()— when you have decided to fill the gaps rather than count or drop them
See it on real data
Below are the 13 Bamboo Weekly exercises that use isna on real-world data — try each one, then study the worked solution.
- Bamboo Weekly #163: Daylight saving time
- Bamboo Weekly #156: Winter Olympics
- Bamboo Weekly #138: Federal workers
- Bamboo Weekly #122: Economic growth
- Bamboo Weekly #121: Research funding
- Bamboo Weekly #97: Drones
- Bamboo Weekly #92: Climate disaster costs
- Bamboo Weekly #90: Voter participation
- Bamboo Weekly #61: Solar eclipse
- Bamboo Weekly #45: Netflix
- Bamboo Weekly #33: Fracking
- Bamboo Weekly #13: Python developers
- Bamboo Weekly #12: Tourism
Part of the Pandas Methods Index. See also practice by skill.