Skip to content

pandas unique

The distinct values in a column, in the order they first appear.

What is actually in this column? That is the first question I ask of a data set I have not seen before, and .unique() is the shortest way to ask it. It hands back every distinct value in a series, once each, in the order Pandas met them going down the rows. Not sorted, not by frequency — file order.

Two things about the result catch people out, and both matter before you chain anything onto it. The order is first-appearance order, which is rarely the order you would have chosen. And what comes back is an array, not a series, so most of the Pandas vocabulary you were about to reach for is not there.

Official documentation: Series.unique and pandas.unique

The arguments that earn their keep

There are none. The signature is Series.unique(self), and the parentheses are the whole method:

df['column'].unique()      # the distinct values, as an array
pd.unique(df['column'])    # the same thing, as a function
df.unique()                # AttributeError -- there is no data frame version

That last line is worth knowing. .unique() lives on a series and on an index, never on a data frame. For distinct combinations across several columns you want drop_duplicates, which takes a subset.

A worked example, on real data

The CDC publishes a yearly summary of every fertility clinic in the United States, one row per clinic per statistic. It is the file behind Bamboo Weekly #55:

import pandas as pd

url = 'https://data.cdc.gov/api/views/9tjt-seye/rows.csv?accessType=DOWNLOAD'

df = pd.read_csv(url, usecols=['LocationAbbr', 'FacilityName',
                               'Clinic Status', 'Breakout_Category'])

df['Clinic Status'].unique()
<ArrowStringArray>
['Reorganized', 'Open', 'Closed']
Length: 3, dtype: str

Three statuses, and look at the order: Reorganized, Open, Closed. Alphabetically that would be Closed, Open, Reorganized, and by frequency it would be Open first by a mile — 56,570 rows against 670. What you actually get is the order in which those words appear in the file, which is an accident of how the CDC sorted its export.

Now the type. That is an ArrowStringArray, because string columns are backed by PyArrow when it is installed; ask for a numeric column and you get a plain NumPy ndarray instead. Either way it is an array. It has no index, no .head(), no .str accessor and no .sort_values().

.drop_duplicates() answers the same question and gives you a series:

df['Clinic Status'].drop_duplicates()
0        Reorganized
268             Open
15812         Closed
Name: Clinic Status, dtype: str

Same three values, same order, but now with the row label where each first appeared, and with every series method still available. The CDC re-orders its export between downloads, so your row numbers will differ from mine — which is a decent reminder of how little first-appearance order guarantees. When the distinct values are the end of the road, .unique() is fine. When they are the middle of a chain, use .drop_duplicates() and keep the series.

Three mistakes people make

Calling .value_counts() on the result. On a string column this is the cruel one, because it runs:

df['Clinic Status'].unique().value_counts()
Reorganized    1
Open           1
Closed         1
Name: count, dtype: int64

Every count is 1, which is arithmetically perfect and completely useless — you counted a list in which each value appears exactly once. On a numeric column the same line raises AttributeError: 'numpy.ndarray' object has no attribute 'value_counts', which is the kinder outcome. Drop the .unique(): value_counts already gives you the distinct values and their frequencies together.

Assuming the values come back sorted. They do not, ever. If you need order, say so: sorted(s.unique()) gives a plain Python list, and s.drop_duplicates().sort_values() keeps you in Pandas.

Forgetting that missing values count as a value. value_counts() drops NaN by default; .unique() includes it. The CDC's Breakout_Category column shows the gap:

df['Breakout_Category'].unique()
<ArrowStringArray>
[nan, 'Age of Patient', 'Egg/embryo type', 'Yes/No']
Length: 4, dtype: str

Four entries, one of them nan. So len(df['Breakout_Category'].unique()) is 4 while .nunique() is 3. The rule to remember is that len(s.unique()) equals s.nunique(dropna=False), never the plain .nunique().

Where it shows up in Bamboo Weekly

Bamboo Weekly #55: IVF is the one solution where I use it in Pandas, and it teaches the whole family in three consecutive lines: df['FacilityName'].nunique() for the count, df['FacilityName'].unique() for the values, then df['FacilityName'].drop_duplicates() for the same values as a series. The file lists 457 distinct clinics today.

Bamboo Weekly #156: Winter Olympics also calls .unique(), twice — but on a Polars frame, in an issue that solves everything both ways. The Pandas half of that solution uses .nunique().

Two posts is a thin showing for a method this well known, and the reason is worth saying out loud: when I want the distinct values I usually want their counts too, so value_counts wins, and when I want distinct rows rather than distinct values, drop_duplicates wins. .unique() is for the moment you are reading a column for the first time and just want to see what is in there.

Practice it

Work through a unique exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/unique/

Go deeper

The neighbors are nunique for how many rather than which, value_counts for the values with their frequencies, and drop_duplicates for the series form and for whole-row deduplication. Once you know how few distinct values a column holds, astype can turn it into a category and memory_usage will show you what that saved.

Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.

See it on real data

Below are the 2 Bamboo Weekly exercises that use unique on real-world data — try each one, then study the worked solution.

Part of the Pandas Methods Index. See also practice by skill.