Skip to content
Bamboo Weekly

pandas describe

The two survey methods — describe for what the values are, info for what the columns are.

Is this data what I think it is? That is the question every analysis rests on, and scrolling through rows is a terrible way to answer it. describe answers it in one line: for every numeric column, it returns eight numbers describing the whole column, not the top of it. info answers the companion question — how many rows, which dtypes, how many non-missing values per column, and how much memory the whole thing occupies.

I treat them as a pair, which is why they share a page here rather than getting one each. head shows you five rows that somebody else chose; describe and info show you the shape of all of them. If you only ever run head after loading a file, you are looking at a sample of size five and calling it a survey.

Official documentation: DataFrame.describe, Series.describe and DataFrame.info

The arguments that earn their keep

df.describe()                       # numeric columns only
df.describe(include='all')          # every column, numeric and not
df.describe(include='str')          # only the string columns
df.describe(percentiles=[.1, .9])   # your choice of percentiles

df.info()                           # dtypes, non-null counts, memory
df.info(memory_usage='deep')        # memory you can actually believe

Three things about that list.

The default is numeric-only, and it is silent about what it left out. A frame of twenty columns where four are numbers gives you a four-column summary and no mention of the other sixteen.

include='str' is the Pandas 3 spelling. include=object still works and still selects string columns, but it now raises a Pandas4Warning telling you to be explicit, so write 'str' in new code.

percentiles= replaces the default 25/50/75 rather than adding to it. If you pass percentiles=[.9] you get a summary with no median in it at all — ask for 0.5 when you want one.

The eight rows

For a numeric column: count is the number of non-missing values, mean and std are the average and the sample standard deviation, min and max are the extremes, and 25%, 50%, 75% are the quartiles, with 50% being the median. That is it — eight rows, and the two that catch the most bugs are count and min/max.

For a string column you get four different rows: count, unique, top (the most common value) and freq (how often the top value appears).

A worked example, on real data

Here is a year of daily weather from Boston's Logan airport, straight from NOAA:

import pandas as pd

url = ('https://www.ncei.noaa.gov/data/global-summary-of-the-day/'
       'access/2024/72509014739.csv')

cols = ['DATE', 'NAME', 'TEMP', 'VISIB', 'WDSP',
        'GUST', 'MAX', 'MIN', 'PRCP', 'SNDP']

df = pd.read_csv(url, usecols=cols)

df.describe().round(1)

        TEMP  VISIB   WDSP   GUST    MAX    MIN   PRCP   SNDP
count  366.0  366.0  366.0  366.0  366.0  366.0  366.0  366.0
mean    53.5    9.3    9.2  361.2   63.5   45.3    0.1  999.9
std     16.0    1.5    3.2  463.5   17.7   15.2    0.3    0.0
min     15.8    1.3    2.7   15.0   26.1   10.0    0.0  999.9
25%     40.2    9.4    7.0   22.9   48.9   34.0    0.0  999.9
50%     52.7   10.0    8.7   28.9   63.5   45.0    0.0  999.9
75%     67.1   10.0   10.8  999.9   78.1   59.0    0.1  999.9
max     84.0   10.0   25.9  999.9   98.1   73.9    2.6  999.9

TEMP, MAX, MIN and WDSP look like Boston. Two columns do not.

The average wind gust at Logan in 2024 was 361 mph. That is not a windy year; that is a broken column. Read down the GUST column and you can see the machinery: the 75th percentile and the maximum are both exactly 999.9, which means at least a quarter of the days carry that value, and it is not a measurement. It is NOAA's code for "no gust recorded." Ask .mean() on its own and you get 361.2 with no hint that anything is wrong. describe shows you the same 361.2 sitting next to a max of 999.9, and the shape of the problem is immediately obvious.

SNDP, snow depth, is worse. Its min, its max and all three quartiles are 999.9, and its standard deviation is 0. Every single value in that column is the sentinel. The column contains no data whatsoever, and nothing about df.head() would ever have told you.

NOAA documents its sentinels, so the fix belongs at load time:

df = pd.read_csv(url, usecols=cols,
                 na_values={'GUST': '999.9', 'SNDP': '999.9',
                            'VISIB': '999.9', 'PRCP': '99.99'})

df.describe().round(1)

        TEMP  VISIB   WDSP   GUST    MAX    MIN   PRCP  SNDP
count  366.0  366.0  366.0  240.0  366.0  366.0  366.0   0.0
mean    53.5    9.3    9.2   25.8   63.5   45.3    0.1   NaN
std     16.0    1.5    3.2    6.6   17.7   15.2    0.3   NaN
min     15.8    1.3    2.7   15.0   26.1   10.0    0.0   NaN
25%     40.2    9.4    7.0   21.0   48.9   34.0    0.0   NaN
50%     52.7   10.0    8.7   25.1   63.5   45.0    0.0   NaN
75%     67.1   10.0   10.8   28.9   78.1   59.0    0.1   NaN
max     84.0   10.0   25.9   51.1   98.1   73.9    2.6   NaN

The average gust is 25.8 mph, the strongest of the year was 51.1, and — look at the top row — count is now 240 for GUST, 366 for everything else, and 0 for SNDP. Three different counts in one frame. That row is the honest answer to "how much data do I actually have," and the difference between 366 and 240 is exactly the 126 days with no gust reading.

Now the columns describe refused to mention:

df.describe(include='str')

              DATE                                       NAME
count          366                                        366
unique         366                                          1
top     2024-01-01  BOSTON LOGAN INTERNATIONAL AIRPORT, MA US
freq             1                                        366

Two useful facts in four rows. DATE has 366 unique values across 366 rows, so there are no duplicate days and none are missing — 2024 was a leap year, and all of it is here. NAME has exactly one unique value, repeated 366 times, which means it is a constant carrying no information at all. Constant columns are common in files assembled per station or per region, and they are worth spotting before you group by one.

Then info, which answers the questions describe does not:

df.info(memory_usage='deep')

<class 'pandas.DataFrame'>
RangeIndex: 366 entries, 0 to 365
Data columns (total 10 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   DATE    366 non-null    str
 1   NAME    366 non-null    str
 2   TEMP    366 non-null    float64
 3   VISIB   366 non-null    float64
 4   WDSP    366 non-null    float64
 5   GUST    240 non-null    float64
 6   MAX     366 non-null    float64
 7   MIN     366 non-null    float64
 8   PRCP    366 non-null    float64
 9   SNDP    0 non-null      float64
dtypes: float64(8), str(2)
memory usage: 76.3 KB

The same 366 / 240 / 0 story, plus the dtypes. DATE came in as text rather than a datetime, which is worth knowing before you try to sort by it.

I always pass memory_usage='deep', and here is why. Run the same call without it and the last line reads:

memory usage: 28.7 KB

Same frame, and the shallow number is 37 percent of the real one. When a string column is stored as pointers to Python string objects, the default calculation measures the pointers and ignores the strings. NAME reports 2,928 bytes shallow and 32,940 deep — off by a factor of eleven, because the shallow count never looks at the 41-character station name sitting behind each pointer. Pandas 2 at least flagged this by printing the total as 28.7+ KB, with a plus sign meaning "and then some." Pandas 3 prints the undercount with no marker at all.

One honest caveat: if PyArrow is installed, str columns are backed by Arrow rather than by Python objects, Pandas knows the true buffer size without walking it, and both calls report the same 47.0 KB. Passing deep costs you nothing in that case, which is the best argument for always passing it — it is either free or it is the only correct answer, and you should not have to remember which.

Four mistakes people make

Reading count as the number of rows. It is the number of non-missing values, which is why GUST shows 240 above while its neighbors show 366. Two columns of the same frame will happily report different counts, and that difference is information, not a rendering glitch. For the row count use len(df) or df.shape[0], and when the gaps matter, dropna is the next stop.

Trusting a mean without looking at min and max. A sentinel like 999.9 or -999 drags the average without ever announcing itself. df['GUST'].mean() returns 361.2 and looks like a number; the same column in describe shows 361.2 next to a maximum of 999.9 and the fraud is visible instantly. This is the single best reason to reach for describe rather than for the aggregations one at a time.

Not noticing which columns were skipped. describe quietly drops everything non-numeric, so the survey you just ran may have covered four of your twenty columns. If the answer feels thin, it is: use include='all' to see the string and numeric columns side by side, or include='str' for just the ones the default hid.

Running info without memory_usage='deep'. The default undercounts string-heavy frames badly — 28.7 KB against 76.3 KB on a small file like this one, and far worse on a big one. Type the deep every time; see also memory_usage for the per-column breakdown. One related surprise: on frames wider than 100 columns, info collapses to a one-line summary and shows you no per-column detail at all. Pass verbose=True, show_counts=True to get it back.

Where it shows up in Bamboo Weekly

Bamboo Weekly #3: Earthquakes is describe at its simplest and most effective. Filtering the USGS feed down to Turkey on February 6, 2023 and calling .describe() on the magnitude column reports nine earthquakes that day, a median of 6.0 and a maximum of 7.8 — the whole story of the event in eight numbers. Free to read.

Bamboo Weekly #61: Solar eclipse runs describe on a timedelta column, the length of totality in each US county. It works exactly as it does on numbers, printing durations rather than floats: a mean of 3 minutes 5 seconds, a minimum of 2 seconds and a maximum just under 4 minutes. Also free to read.

Bamboo Weekly #150: Kalshi uses describe the way this page argues for. Summarizing the bid-ask spread on a prediction market gives a median of 100 — meaning sellers wanted 100 and buyers offered nothing — which is obvious nonsense and sends the analysis back to filter for markets with real volume. The same post runs df.info(memory_usage='deep') on a 3.3 GB frame and cuts it by 75 percent with astype('category').

Bamboo Weekly #149: Flu season is the deep argument in full, on CDC data: 1.7 MB reported by the default, 3.1 MB by the honest count, with the Pandas 2 plus sign explained along the way.

Practice it

Work through a describe exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/describe/

Go deeper

head is the other half of this argument — the first look that should never be the only look. isna counts the gaps that count implies and dropna removes them, value_counts does for a single categorical column what describe(include='str') sketches for all of them, and astype is how you act on the dtypes info just showed you. If a column turned out to be text when it should be numbers, that argument belongs in read_csv, not in a cleanup step afterward.

More Pandas videos on Python and Pandas with Reuven Lerner.

Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.

See it on real data

Below are the 11 Bamboo Weekly exercises that use describe on real-world data — try each one, then study the worked solution.

Part of the Pandas Methods Index. See also practice by skill.