Write a data frame to disk in a format that remembers what the columns were.
What happens to your dtypes when you call to_csv? They evaporate. You spent the morning turning a column of text into a category, another into a nullable Int16, and a third into a time-zone-aware timestamp — and a CSV file records none of it, because a CSV file is characters and commas and nothing else. Tomorrow morning you do it all again.
to_parquet is the fix. Parquet stores each column separately, along with its type, and compresses it. The file is binary, so you cannot read it in an editor, and that is the only real cost.
Official documentation: pandas.DataFrame.to_parquet
The arguments that earn their keep
df.to_parquet(path,
compression='zstd', # or 'snappy' (default), 'gzip', 'brotli', None
index=False, # leave the index out of the file
engine='pyarrow', # or 'fastparquet'
partition_cols=['State']) # write a directory, split by column value
Pandas does not ship a Parquet engine, so install one first with uv add pyarrow.
The round trip, measured
The NTSB publishes every aviation accident it has investigated, and Bamboo Weekly #104 used it. Here are seven of its columns, loaded from CSV and then given the dtypes they deserve:
import pandas as pd
cols = ['EventDate', 'State', 'Make', 'HighestInjuryLevel',
'FatalInjuryCount', 'HasSafetyRec', 'Latitude']
df = (
pd.read_csv('https://files.lerner.co.il/bw-104-ntsb.csv', usecols=cols)[cols]
.assign(EventDate=lambda df_: pd.to_datetime(df_['EventDate'],
format='ISO8601', utc=True),
State=pd.col('State').astype('category'),
Make=pd.col('Make').astype('category'),
HighestInjuryLevel=pd.col('HighestInjuryLevel').astype('category'),
FatalInjuryCount=pd.col('FatalInjuryCount').astype('Int16'))
)
df.dtypes
EventDate datetime64[us, UTC]
State category
Make category
HighestInjuryLevel category
FatalInjuryCount Int16
HasSafetyRec bool
Latitude float64
dtype: object
That is 176,884 rows. format='ISO8601' lets to_datetime handle each ISO variant it meets; if your dates need explicit directives instead, strfti.me is the sandbox I use to work them out.
Now write it both ways, on Pandas 3.0.5 with pyarrow 25.0.1:
df.to_parquet('ntsb.parquet')
df.to_csv('ntsb.csv', index=False)
ntsb.csv 10.7 MB 484 ms
ntsb.parquet 2.2 MB 25 ms
A fifth of the space, and nineteen times faster to write. But the size is the least interesting number here. Read each file back and look at what survived:
pd.read_parquet('ntsb.parquet').dtypes
EventDate datetime64[us, UTC]
State category
Make category
HighestInjuryLevel category
FatalInjuryCount Int16
HasSafetyRec bool
Latitude float64
dtype: object
pd.read_csv('ntsb.csv').dtypes
EventDate str
State str
Make str
HighestInjuryLevel str
FatalInjuryCount int64
HasSafetyRec bool
Latitude float64
dtype: object
Every one of the seven came back from Parquet unchanged. From CSV, three did not: the three categories are plain text again, the time-zone-aware timestamp is a string, and Int16 has become int64 — sixteen times the memory, and no longer able to hold a missing value as <NA>. Pandas is not at fault; the information was never written down.
compression=
snappy is the default, and it is the right default. The alternatives trade time for bytes:
None 2.52 MB 22 ms
snappy 2.15 MB 24 ms
zstd 1.98 MB 25 ms
gzip 1.65 MB 595 ms
brotli 1.50 MB 96 ms
zstd is essentially free, so I reach for it on anything I write more than once. gzip costs twenty-five times the write time to save half a megabyte, which is a bargain only if the file crosses a network many times.
index=
By default the index goes into the file and comes back out of it, which is what you want:
top = df['Make'].value_counts().head(3).to_frame()
top.to_parquet('makes.parquet')
pd.read_parquet('makes.parquet')
count
Make
CESSNA 47510
PIPER 31082
BEECH 9547
Pass index=False and those labels are gone for good, replaced by a RangeIndex — silently, with no warning at write time and no clue at read time.
Four mistakes people make
Duplicate column names. to_csv writes them without comment; Parquet refuses, with ValueError: Duplicate column names found: ['a', 'a']. This usually turns up after a merge or a concat that produced two columns with the same name.
A column holding more than one type. An object column with 1, 'two' and 3.0 in it has no Parquet type, so the write fails: ArrowInvalid: ("Could not convert 'two' with type str: tried to convert to int64", ...). Cast it to str, or clean it, before writing.
Expecting to append. There is no append mode. Writing to a path that already exists replaces the file, quietly and completely. To grow a data set, either pd.concat in memory and rewrite, or use partition_cols and add new directories alongside the old ones.
Calling it on a Series. to_parquet is a data frame method only, so counts.to_parquet(...) raises AttributeError. Call .to_frame() first, as above.
Where it shows up in Bamboo Weekly
Only four solutions call to_parquet, and in every one of them the reason is the same: something slow or awkward happened once, and the result was saved so it would not have to happen again.
Bamboo Weekly #151: PyPI in 2025 pulls a year of download counts out of BigQuery — a query you do not want to run twice — and lands it with df.to_parquet('bw-151-daily-downloads.parquet').
Bamboo Weekly #86: FEMA converts four columns of disaster declarations to real datetimes, then writes the frame back out. The file shrinks from 6.3 MB to 5.7 MB, because a date stored as 64 bits beats a date stored as characters.
Bamboo Weekly #112: Programming jobs times the formats head to head with %timeit df.to_parquet('/tmp/data.parquet'), against Feather and against the original file, which takes over a minute to load.
Bamboo Weekly #128: Extreme heat uses the round trip as a conversion tool: to_parquet followed by read_parquet(..., dtype_backend='pyarrow') is a quick way to move an entire frame onto Arrow-backed dtypes and see what it does to memory.
Practice it
Work through a to_parquet() exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/to-parquet/
Go deeper
read_parquet is the other half of this page, and it is where columns= and filters= live — the two arguments that make a well-written Parquet file pay you back. read_csv covers the arguments you stop needing once your data is Parquet, and astype is how the dtypes worth preserving get set in the first place. The Apache Parquet project documents the format itself.
Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.
Related methods
pd.read_parquet()— for reading back what you just wrote.memory_usage()— which shows you what the dtypes Parquet preserves are actually costing
See it on real data
Below are the 3 Bamboo Weekly exercises that use to_parquet on real-world data — try each one, then study the worked solution.
- Bamboo Weekly #151: PyPI in 2025
- Bamboo Weekly #128: Extreme heat
- Bamboo Weekly #112: Programming jobs
Part of the Pandas Methods Index. See also practice by skill.