Skip to content
Bamboo Weekly

pandas memory_usage

Find out what each column of your data frame actually costs — and then make it cost less.

How big is your data frame? Not how many rows, but how many bytes. Most people never ask until Pandas runs out of memory, at which point the answer arrives as a MemoryError and there is nothing to do but restart the kernel.

memory_usage is how you ask early. It returns the number of bytes used by each column, as a Series, and that Series is the beginning of every memory conversation worth having: which column is the expensive one, what happened when I converted it, did the change I just made help or hurt. I measure before, I make the change, and I measure again. Almost everything I know about keeping data frames small came from doing that a few hundred times.

Official documentation: DataFrame.memory_usage

The arguments that earn their keep

There are two, and one of them behaves differently in Pandas 3 than in every tutorial written before it.

index defaults to True, which adds an Index entry at the top of the returned Series. Pass index=False and you get only the columns.

deep defaults to False. Historically this was the argument you always passed, because Pandas stored strings as pointers to Python objects and the shallow measurement counted only the pointers. That is still true when strings are stored the old way — and it is no longer true when they are not. Which one you have depends on whether PyArrow is installed, so before you trust any number, ask any string column which kind it is:

pd.Series(['a']).dtype

<StringDtype(na_value=nan)>

pd.Series(['a']).dtype.storage

'pyarrow'

Without PyArrow, the same two lines give <StringDtype(storage='python', na_value=nan)> and 'python'. Both spellings print as str in df.dtypes, which is why the repr and the .storage attribute are worth knowing. PyArrow is an optional dependency of Pandas 3, not a required one, so both worlds are out there — and the difference between them is enormous. Everything below is measured on Pandas 3.0.5 with PyArrow 25.0.1, and then measured again with PyArrow uninstalled.

A worked example, on real data

The National Transportation Safety Board publishes every aviation investigation it has conducted, and casting the widest possible net on its aviation investigation search produces a CSV file of 176,884 rows and 38 columns. This is the data set from Bamboo Weekly #104, and it is a good specimen because almost every column is text that repeats.

import pandas as pd

df = pd.read_csv('ntsb.csv', low_memory=False)

df.memory_usage()

Index            132
NtsbNo       3184110
EventType    1967694
Mkey         1415072
EventDate    4974383
City         2952840
State        2786359
Country      3688427
dtype: int64

The result is a Series indexed by column name, which means every Series method is available to you. That is most of the value. Sort it, and you know immediately where your memory went:

df.memory_usage().sort_values(ascending=False).head()

ProbableCause          11560715
EventDate               4974383
OriginalPublishDate     4569723
Country                 3688427
ReportType              3641316
dtype: int64

One free-text column, ProbableCause, is 11.5 MB of the total by itself. Sum the Series for the whole-frame figure:

df.memory_usage().sum()

93109809

Ninety-three megabytes, for a 66 MB CSV file.

The deep=True surprise

Now the argument every older tutorial insists on:

df.memory_usage(deep=True).sum()

93109809

Identical. Not close — identical, to the byte, on all 38 columns. PyArrow stores strings in contiguous Arrow buffers rather than as a scattered heap of Python str objects, so there is nothing hiding off to the side for deep to go and find. The shallow number was already the whole truth.

That is worth stating plainly, because it reverses standard advice: with PyArrow installed, deep=True on a string column is a no-op. The same holds for info — df.info() and df.info(memory_usage='deep') print the same figure.

Uninstall PyArrow and run exactly the same code, and the old world comes right back:

df['State'].dtype.storage

'python'

df.memory_usage().head(8)

Index            132
NtsbNo       1415072
EventType    1415072
Mkey         1415072
EventDate    1415072
City         1415072
State        1415072
Country      1415072
dtype: int64

Every column is 1,415,072 bytes, which is 176,884 rows times eight bytes. That is the giveaway: Pandas is reporting the size of the pointer array and nothing else. The real figure is five and a half times larger:

df.memory_usage(deep=True).sum()

281199474

Fifty-one megabytes claimed, 281 MB actually used. So the rule is not "always pass deep" and it is not "never bother". It is: check .storage once, and if it says 'python', deep=True is mandatory.

Where deep=True still earns its keep

Even with PyArrow, some columns genuinely hold Python objects, and those are exactly the columns deep was written for. Splitting a text column into lists is a common way to create one:

words = df['ProbableCause'].str.split()

words.dtype

object

words.memory_usage(index=False)             # 1415072
words.memory_usage(deep=True, index=False)  # 21978696

A factor of fifteen. Whenever dtypes says object rather than str, the shallow number is meaningless — and object is what you get from lists, dicts, tuples, Decimal, and anything else Pandas has no native storage for.

index=

The index is a column too, and on a default RangeIndex it is free:

df.memory_usage(deep=True).head(3)

Index            132
NtsbNo       3184110
EventType    1967694
dtype: int64

A hundred and thirty-two bytes, because a RangeIndex stores a start, a stop and a step rather than 176,884 numbers. Make a real column into the index and those bytes move rather than disappear:

df.set_index('NtsbNo').memory_usage(deep=True).head(3)

Index        3184110
EventType    1967694
Mkey         1415072
dtype: int64

The 3.2 MB that NtsbNo cost as a column is now what Index costs. Pass index=False and it vanishes from the report, which is fine as long as you do it on both sides of a before-and-after comparison.

The three levers

Measure, change, measure again. Our starting figure is 93,109,809 bytes.

Categories, for text that repeats

A category stores each distinct string once and keeps a small integer per row. Twelve of these columns have fewer than a hundred distinct values:

low_card = df.select_dtypes('str').nunique().loc[lambda s_: s_ < 100].index

list(low_card)

['EventType', 'State', 'ReportType', 'HighestInjuryLevel', 'AirCraftCategory',
 'AmateurBuilt', 'NumberOfEngines', 'Scheduled', 'FAR', 'AirCraftDamage',
 'WeatherCondition', 'ReportStatus']

AirCraftDamage alone, with 30 distinct values across 176,884 rows, goes from 3,253,969 bytes to 177,575 — a factor of eighteen, for a column that behaves the same afterwards. The mechanics of the conversion, including what a category will not let you do later, are on the astype page, which measures the same effect on a smaller column: 6,709 bytes down to 1,053.

Smaller integers, for numbers that are small

Pandas reads whole numbers as int64 because it cannot know your range. Check the range and you often find eight bytes doing one byte's work:

df[['FatalInjuryCount', 'SeriousInjuryCount', 'MinorInjuryCount']].max()

FatalInjuryCount      574
SeriousInjuryCount    111
MinorInjuryCount      380
dtype: int64

Nothing above 574, so int16 — which reaches 32,767 — has room to spare.

Both levers at once, in a single astype:

counts = ['FatalInjuryCount', 'SeriousInjuryCount', 'MinorInjuryCount']

smaller = df.astype({c: 'category' for c in low_card} |
                    {c: 'int16' for c in counts})

smaller.memory_usage(deep=True).sum()

64274774

From 93,109,809 to 64,274,774 — 69 percent of the original, in one line, with no loss of functionality. The categories account for 25.7 MB of that and the integers for 3.2 MB.

usecols=, so the memory is never allocated

The first two levers shrink a frame you have already built, which means you paid for the full version at least once. If you know which columns you want, say so at read time and Pandas never allocates the rest:

sub = pd.read_csv('ntsb.csv', low_memory=False,
                  usecols=['EventDate', 'State',
                           'AirCraftDamage', 'FatalInjuryCount'],
                  dtype={'State': 'category',
                         'AirCraftDamage': 'category',
                         'FatalInjuryCount': 'int16'},
                  parse_dates=['EventDate'])

sub.memory_usage(deep=True)

Index                   132
EventDate           1415072
State                179972
FatalInjuryCount     353768
AirCraftDamage       178647
dtype: int64

sub.memory_usage(deep=True).sum()

2127591

Two megabytes instead of ninety-three. dtype= and parse_dates= in read_csv apply the other two levers during the read as well, so the large version never exists at all. This is the lever to reach for first when a file is close to the size of your RAM, because it is the only one that lowers your peak.

Five mistakes people make

Trusting the shallow default on a string column. If .storage says 'python', the default under-reports the NTSB frame by 230 MB — 51 MB claimed against 281 MB used. The tell is that every column reports the same number, and that number divided by the row count is exactly 8. I have written this warning into Bamboo Weekly solutions more times than any other, because it caught me first.

Turning a high-cardinality column into a category. A category is only a saving when values repeat. When they do not, you pay for the dictionary of distinct values and the integer codes, and get nothing back. Here is the crossover, on four real columns of this frame:

cols = ['AirCraftDamage', 'Operator', 'N', 'NtsbNo']

pd.DataFrame({
    'distinct': df[cols].nunique(),
    'as str': df[cols].memory_usage(deep=True, index=False),
    'as category': df[cols].astype('category').memory_usage(deep=True, index=False)})

                distinct   as str  as category
AirCraftDamage        30  3253969       177575
Operator           40485  2347207      1787854
N                 144854  2492258      2752857
NtsbNo            176868  3184110      3913467

Thirty distinct values save 95 percent. Forty thousand save 24 percent. And NtsbNo, which is nearly unique — 176,868 distinct values in 176,884 rows — gets 23 percent bigger. Loop astype('category') over every text column without looking, as it is tempting to do, and you will make some of them worse. Check nunique first.

Adding up frames that share their data. Copy-on-Write means a copy is mostly a promise, not an allocation, and memory_usage reports what a frame's columns would cost rather than what was newly allocated:

copies = [df.copy() for _ in range(9)]

That is ten data frames counting the original, each reporting 93.1 MB, for 931 MB of claimed memory. Watching the process from outside, it grew by about 90 MB — one frame's worth, not ten, because the string buffers are shared and only the fixed-width columns were actually duplicated. Measure the frames you care about one at a time, and never total memory_usage across a frame and a copy or slice of it.

Choosing an integer width from today's maximum. SeriousInjuryCount tops out at 111, which fits in an int8 with sixteen to spare. It works today, and it will keep working right up until one accident injures 130 people:

pd.Series([111, 130]).astype('int8')

0    111
1   -126
dtype: int8

No error, no warning, just a negative number of injuries in your data. The astype page shows the same trap one size up, where 40,000 becomes -25,536 under int16. Leave headroom, or use pd.to_numeric(downcast=...) and re-check it whenever the data is refreshed.

Comparing a figure that includes the index against one that does not. After set_index('NtsbNo'), 3.2 MB of this frame lives in the index. Measure the before with the default and the after with index=False and you will report a saving that is really just a column you stopped counting. Pick one convention and stay with it. The good news, if you learned otherwise from an older tutorial, is that an index in Pandas 3 keeps whatever dtype you gave it — a float32 column set as the index stays four bytes per row rather than being widened back to eight — so the index is no longer the place where memory optimizations go to die.

Where it shows up in Bamboo Weekly

Twelve Bamboo Weekly solutions call memory_usage directly, and six more reach it through info. The memory optimization tag collects them.

#64: Coal power is the one to read first, because the whole question is this page. It works through the suspicious all-columns-equal output, explains why deep=True is needed, converts every non-numeric column to a category, and lands on 4,301,719 bytes down to 416,504 — a saving of 3.8 MB on 13,906 rows. It also asks the question this page's second mistake answers: are there columns you could turn into categories but should not?

#43: Financial protection is the same lesson at a scale that hurts. A CFPB complaints file loads in 19.6 seconds and reports 6.7 GB with deep=True, of which a single free-text narrative column is 1.8 GB. Declaring categories and dates in the read_csv call brings it to 2.1 GB — a 70 percent cut, applied at read time rather than afterwards.

#51: Academy Awards measures the storage backend itself: the same six columns cost 1,120,354 bytes with PyArrow and 3,735,417 with NumPy. That three-to-one gap is the change described at the top of this page, arriving early.

#104: Aviation accidents is the data set used above. I measured 249 MB before optimizing and 87 MB after — 35 percent of the original — using categories plus uint16 on the injury-count columns. Those figures come from the object-dtype era, which is why they differ from the 93 MB and 64 MB measured here on the same file. A memory number is only true for the Pandas that produced it, which is the best argument I know for measuring rather than remembering.

Practice it

Work through a memory_usage() exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/memory-usage/

Go deeper

astype is the method that makes the changes you measure here, and it covers what a category costs you in flexibility as well as what it saves you in bytes. describe covers info, which is the whole-frame survey — dtypes, non-null counts and one memory total in a single glance — where memory_usage is the per-column detail. read_csv is where usecols, dtype and parse_dates live, and applying the levers there rather than afterwards is the difference between a frame you shrank and a frame you never built. The Pandas user guide's chapter on scaling to large datasets picks up where all of this leaves off.

More Pandas videos on Python and Pandas with Reuven Lerner.

Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.

See it on real data

Below are the 11 Bamboo Weekly exercises that use memory_usage on real-world data — try each one, then study the worked solution.

Part of the Pandas Methods Index. See also practice by skill.