nunique in pandas — practice examples (Bamboo Weekly)

How many distinct values a column holds.

How many different things are in this column? The answer decides what you can do next. Two distinct values and the column is a flag. Twenty and it is a category worth grouping by. Seventy thousand distinct values in seventy thousand rows and it is an identifier, useful as a key and meaningless as a grouping. .nunique() is the one line that tells you which of those you are holding, and it is the cheapest thing you can run before committing to an analysis.

Official documentation: Series.nunique and DataFrame.nunique

The arguments that earn their keep

s.nunique()                    # distinct values, missing ones ignored
s.nunique(dropna=False)        # NaN counts as one more distinct value
df.nunique()                   # a series: one count per column
df.nunique(axis='columns')     # one count per row, which you rarely want

dropna is the whole story. The default is True, matching value_counts and disagreeing with unique, whose array does include NaN. So len(s.unique()) equals s.nunique(dropna=False) and not the plain call — a difference of exactly one, on every column that has a gap in it.

The data frame form is the one I reach for most, because it profiles a whole file in a single line.

A worked example, on real data

FEMA publishes every United States disaster declaration since 1953 in one CSV. It grows constantly, so your numbers will run a little above mine:

import pandas as pd

url = 'https://www.fema.gov/api/open/v2/DisasterDeclarationsSummaries.csv'

df = pd.read_csv(url)

df[['disasterNumber', 'state', 'incidentType', 'designatedArea',
    'declarationTitle', 'hash']].nunique()

disasterNumber       5248
state                  59
incidentType           27
designatedArea       2962
declarationTitle     2485
hash                70248
dtype: int64

The file has 70,248 rows, and that column of numbers is a map of it. hash has one distinct value per row, so it is an identifier — and it also explains why df.drop_duplicates() removes nothing from this file, the trap that drop_duplicates is built around. state at 59 and incidentType at 27 are the columns to group by. disasterNumber at 5,248 says the 70,248 rows describe 5,248 disasters, because FEMA writes one row per county per declaration.

That last number is the one that changes an answer. Compare counting rows with counting distinct disasters:

df.groupby('state')['disasterNumber'].size().nlargest(5)

state
TX    5424
KY    3376
MO    2874
FL    2794
GA    2768
Name: disasterNumber, dtype: int64

df.groupby('state')['disasterNumber'].nunique().nlargest(5)

state
CA    397
TX    389
OK    259
WA    223
FL    188
Name: disasterNumber, dtype: int64

By rows, Texas leads and Kentucky is second. By disasters, California leads and Kentucky is nowhere near the top — Kentucky has 120 counties, and a statewide flood declaration produces 120 rows. Inside a groupby, .nunique() is how you ask "how many different ones," while .size() and .count() answer "how many rows."

Is it a category?

The other use is deciding whether a text column should become a Categorical. The rule of thumb is the ratio of distinct values to rows, and .nunique() gives you the numerator:

for col in ['incidentType', 'designatedArea', 'hash']:
    plain = df[col].memory_usage(deep=True)
    cat = df[col].astype('category').memory_usage(deep=True)
    print(f'{col:16s} {df[col].nunique():6d} {plain:>10,} {cat:>10,}')

incidentType         27  1,221,738     70,896
designatedArea     2962  1,728,973    219,732
hash              70248  3,372,036  3,661,809

Twenty-seven distinct values in 70,248 rows: a category costs 6 percent of the strings. Nearly three thousand distinct values still pays, at 13 percent. But hash, with one distinct value per row, costs 9 percent more as a category than it did as text, because you are now storing both the strings and an integer code pointing at each one. That is the crossover, and memory_usage walks through the rest of it.

Three mistakes people make

Forgetting that NaN is dropped. The count is of real values only, and when you are deciding whether to group by a column the blanks are part of what you are deciding about. On FEMA's incidentEndDate, .nunique() gives 3,383 and .nunique(dropna=False) gives 3,384. That difference of exactly one is the whole signal: the file has gaps there, and it has none in designatedArea, where the two calls agree.

Confusing it with .count(). .count() is how many non-missing values there are; .nunique() is how many different ones. On designatedArea those are 70,248 and 2,962. Both are useful and they answer entirely different questions.

Reaching for len(df[col].unique()). It works, it costs an extra allocation, and it quietly gives an answer one larger whenever the column has a missing value. .nunique() says what you mean.

Where it shows up in Bamboo Weekly

Bamboo Weekly #51: Academy Awards is the best example of the grouped form. Films are identified by a FilmId, but several films share a title, so .groupby('Film')['FilmId'].nunique() followed by a filter for counts above 1 finds them: four different movies called A Star Is Born, four called Little Women. The same solution uses .groupby('Year')['Category'].nunique() to plot how the number of Oscar categories grew over a century.

Bamboo Weekly #45: Netflix guesses the language of every title, then asks .groupby(df['Release Date'].dt.year)['language'].nunique() — how many distinct languages Netflix released in, per year. Counting rows there would have measured catalog size instead of diversity.

Bamboo Weekly #55: IVF uses the plain series form as an opening move on an unfamiliar file: how many clinics, how many questions asked of each. That is the audit this page is about, done before anything else.

Bamboo Weekly #180: Movies counts distinct titles per distributor per year with .groupby(['year', 'distributor'])['title'].nunique(), then unstacks it into a line chart.

Practice it

Work through a nunique exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/nunique/

Go deeper

unique gives you the values rather than the count, value_counts gives you both at once, and drop_duplicates is what you want when the thing you are counting is rows. The grouped form belongs to groupby, which will take 'nunique' by name inside .agg(). Once the audit says a column is low-cardinality, astype makes it a category and memory_usage shows the bill.

Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.

.unique() — when you want the values themselves rather than a count of them
.value_counts() — when you want each value with its frequency rather than a single count

See it on real data

Below are the 5 Bamboo Weekly exercises that use nunique on real-world data — try each one, then study the worked solution.

Part of the Pandas Methods Index. See also practice by skill.

pandas nunique