pandas read_parquet

Load a Parquet file — columnar, typed, and compressed — into a data frame.

Have you ever waited for a large CSV file to load, and then used four of its forty columns? That is the moment Parquet was designed for.

A CSV file is a wall of text. Every number, date, and boolean is written out as characters, in row order, and the file records nothing at all about what any of it means. Pandas has to read every byte, guess at every column's type, and hand you back the thirty-six columns you did not want along with the four you did.

A Parquet file is the opposite on all three counts. It is columnar, so each column's values are stored together and a reader can pick up one column without touching the others. It is typed, so the file itself knows that this column holds 16-bit integers and that one holds dates. And it is compressed column by column, which works far better than compressing text does, because values of the same kind end up sitting next to each other.

read_parquet is how you get one of these files into Pandas. In its simplest form there is nothing to configure — no dtype, no parse_dates, no na_values — because the file is already carrying that information.

Official documentation: pandas.read_parquet

The arguments that earn their keep

pd.read_parquet(source,
                columns=['state', 'incidentType'],         # read only these
                filters=[('incidentType', '==', 'Fire')],  # skip rows inside the file
                engine='pyarrow',                          # or 'fastparquet'
                dtype_backend='pyarrow',                   # Arrow dtypes the whole way
                storage_options={'User-Agent': '...'})     # headers for remote files

That is a short list next to read_csv, and the shortness is the point: most of what read_csv asks you to specify is already in the file. source can be a path, a URL, or a directory of Parquet files that Pandas will read as a single frame.

The same data, both ways

FEMA publishes every US disaster declaration since 1953, and, conveniently for this comparison, publishes it as both CSV and Parquet. Bamboo Weekly #86 used this data set, and reached for the Parquet version. Here is why.

Same rows, same columns, two files:

DisasterDeclarationsSummaries.csv       23.7 MB
DisasterDeclarationsSummaries.parquet    7.0 MB

And two very different loading times, on Pandas 3.0.5 with pyarrow 25.0.1:

%timeit pd.read_csv('DisasterDeclarationsSummaries.csv')
%timeit pd.read_parquet('DisasterDeclarationsSummaries.parquet')

206 ms ± 1.47 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
15 ms ± 1.54 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Under a third of the disk space, and nearly fourteen times faster, for 70,248 rows across 29 columns. That gap widens as files grow, which is why Parquet has quietly become the default way to hand data around.

Speed is only half of it. Look at what each reader produces for the same four columns:

cols = ['declarationDate', 'fyDeclared', 'ihProgramDeclared', 'fipsCountyCode']

pd.read_csv('DisasterDeclarationsSummaries.csv', usecols=cols)[cols].dtypes

declarationDate        str
fyDeclared           int64
ihProgramDeclared    int64
fipsCountyCode       int64
dtype: object

pd.read_parquet('DisasterDeclarationsSummaries.parquet', columns=cols).dtypes

declarationDate      object
fyDeclared            int16
ihProgramDeclared      bool
fipsCountyCode          str
dtype: object

Three of those four are wrong from the CSV, through no fault of Pandas — the information simply is not in the file. Dates arrive as text. A true/false flag arrives as an integer. And the county FIPS code, which is a label made of digits, arrives as a number:

pd.read_csv('DisasterDeclarationsSummaries.csv', usecols=cols)[cols].head(3)

            declarationDate  fyDeclared  ihProgramDeclared  fipsCountyCode
0  2026-08-18T00:00:00.000Z        2026                  0             240
1  2026-08-14T00:00:00.000Z        2026                  0              29
2  2026-08-14T00:00:00.000Z        2026                  0              33

pd.read_parquet('DisasterDeclarationsSummaries.parquet', columns=cols).head(3)

  declarationDate  fyDeclared  ihProgramDeclared fipsCountyCode
0      2026-08-18        2026              False            240
1      2026-08-14        2026              False            029
2      2026-08-14        2026              False            033

County 029 became 29. That is the classic read_csv mistake, and against Parquet it cannot happen, because the file says the column is text.

columns=, and the whole reason for columnar storage

Because each column lives in its own region of the file, asking for three of the twenty-nine means the reader fetches those three regions and leaves the rest of the file on disk:

%timeit pd.read_parquet('DisasterDeclarationsSummaries.parquet',
                        columns=['state', 'incidentType', 'fyDeclared'])

2.96 ms ± 37.2 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Fifteen milliseconds down to three, and 34.9 MB of memory down to 2.0. read_csv has usecols, which saves you the memory, but the parser still has to walk past every character of every column it is discarding. Here the reader genuinely skips them.

filters=, or making the file do the work

filters= goes one step further and drops rows before they ever become a data frame. You pass a list of (column, operator, value) tuples, which are combined with AND; a list of such lists is combined with OR.

(
    pd
    .read_parquet('DisasterDeclarationsSummaries.parquet',
                  columns=['state', 'fyDeclared', 'declarationTitle'],
                  filters=[('incidentType', '==', 'Fire'),
                           ('fyDeclared', '>=', 2020)])
    .shape
)

(644, 3)

Six hundred and forty-four rows out of seventy thousand, and Pandas never built the other 69,604. Note that incidentType does not appear in columns — you can filter on a column you have no intention of keeping.

Put the two together and a real question fits in one expression. Which states lead for each of the three most destructive incident types?

(
    pd
    .read_parquet('DisasterDeclarationsSummaries.parquet',
                  columns=['state', 'incidentType'],
                  filters=[('incidentType', 'in', ['Fire', 'Flood', 'Hurricane'])])
    .value_counts(['incidentType', 'state'])
    .groupby('incidentType')
    .nlargest(3)
    .droplevel(0)
)

incidentType  state
Fire          TX       1243
              CA        483
              FL        280
Flood         IA        729
              MN        719
              ND        674
Hurricane     LA       1450
              FL       1432
              TX       1370
Name: count, dtype: int64

Going the other way: to_parquet

The round trip is the reason to care about all this even when your data did not arrive as Parquet. Read the awkward CSV once, write Parquet, and work from that afterwards:

counts = (
    pd
    .read_parquet('DisasterDeclarationsSummaries.parquet',
                  columns=['incidentType'])
    .value_counts('incidentType')
    .head()
    .to_frame()
)

counts.to_parquet('/tmp/incidents.parquet')
pd.read_parquet('/tmp/incidents.parquet')

              count
incidentType
Severe Storm  19380
Hurricane     13726
Flood         11437
Biological     7857
Fire           3944

The index survived, dtypes and all, which a CSV round trip cannot promise. Pass index=False and it does not:

counts.to_parquet('/tmp/incidents-flat.parquet', index=False)

pd.read_parquet('/tmp/incidents-flat.parquet')

   count
0  19380
1  13726
2  11437
3   7857
4   3944

to_parquet also takes partition_cols, which writes a directory of files split by the values of a column. Point read_parquet at that directory with a matching filters= and whole files get skipped unopened.

Five mistakes people make

Forgetting that Pandas does not ship a Parquet engine. read_parquet needs pyarrow or fastparquet installed, and without either you get an ImportError before anything is read: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'. Install one with uv add pyarrow. Since engine defaults to 'auto', Pandas tries pyarrow first and falls back to fastparquet.

Expecting to look at the file. Parquet is binary. The first bytes are b'PAR1' and it goes downhill from there. You cannot head it, you cannot open it in a text editor, and — the one that really bites — you cannot usefully diff it or review it in a pull request. Git will store it as an opaque blob and rewrite the whole thing on every change. Keep the CSV under version control if you need the history, and treat the Parquet file as a build artifact.

Assuming filters= behaves the same on every engine. It does not, and it fails quietly. Asking pyarrow for fires gives you fires; asking fastparquet for fires gives you most of the file:

pd.read_parquet('DisasterDeclarationsSummaries.parquet',
                filters=[('incidentType', '==', 'Fire')]).shape

(3944, 29)

pd.read_parquet('DisasterDeclarationsSummaries.parquet', engine='fastparquet',
                filters=[('incidentType', '==', 'Fire')]).shape

(66152, 29)

This is documented rather than broken: with any engine other than pyarrow, filtering happens only at the level of whole row groups and files, so a row group containing even one fire is loaded in its entirety and never filtered further. Sixty-six thousand rows came back where 3,944 were asked for. If you rely on filters= for correctness and not merely for speed, say engine='pyarrow' explicitly.

Losing the index by accident. to_parquet keeps the index by default, which is the sensible choice, but index=False throws it away silently — no warning, and you only find out when the labels are gone. Watch for the mirror image too: to_parquet is a data frame method, not a Series method, so counts.to_parquet(...) on a Series raises AttributeError. Call .to_frame() first, as above.

Believing Parquet is a neutral container. It is a good deal more faithful than CSV, but it is not type-agnostic. Dates come back as object, holding Python datetime.date values rather than datetime64, until you ask for dtype_backend='pyarrow' and get a proper date32[day][pyarrow]. And a file written by another tool can carry a dtype Pandas has never heard of — see Bamboo Weekly #151 below for one that stops read_parquet dead.

Where it shows up in Bamboo Weekly

Bamboo Weekly #86: FEMA is the data set above. The solution loads the Parquet file, discovers that the date columns did not arrive as dates, converts four of them with assign and pd.to_datetime, and writes the result back out with to_parquet — at which point the file shrinks from 6.3 MB to 5.7 MB, because dates as 64-bit values beat dates as strings.

Bamboo Weekly #112: Programming jobs times the formats head to head on a data set that takes more than a minute to load from its original form. Writing Parquet took about 133 ms and reading it back about 43.7 ms, against 64.7 ms and 35.1 ms for Feather. The conclusion there is the one I would repeat here: if you keep going back to the same big CSV or Excel file, read it once, write it out in an Arrow format, and load from that.

Bamboo Weekly #151: PyPI in 2025 pulls a year of PyPI download counts out of BigQuery, saves them with to_parquet, and then hits TypeError: data type 'dbdate' not understood on the way back in — BigQuery's own date dtype, written into the file and meaningless to Pandas. Adding dtype_backend='pyarrow' fixed it, and every subsequent query in that issue reads with that argument in place.

Bamboo Weekly #128: Extreme heat uses the round trip deliberately as a conversion tool: to_parquet followed by read_parquet(..., dtype_backend='pyarrow') is a quick way to get a whole data frame onto Arrow-backed dtypes and see what that does to memory.

Practice it

Work through a read_parquet() exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/read-parquet/

Go deeper

read_csv is the page to read alongside this one, because most Parquet files in your life will start as CSV files and the arguments you need there are exactly the ones you stop needing here. If your data is in a spreadsheet, that is read_excel; if it is a table on a web page, read_html. The Pandas user guide's IO chapter covers Parquet alongside every other format, and the Apache Parquet and Apache Arrow projects explain the format itself.

Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.

Related methods

• pd.read_csv() — when the file is text and its types have to be inferred or declared
• .memory_usage() — when you want to see what the typed columns actually cost you
• .astype() — when the stored dtype is not the one you want to work in

See it on real data

Below are the 4 Bamboo Weekly exercises that use read_parquet on real-world data — try each one, then study the worked solution.

• Bamboo Weekly #151: PyPI in 2025
• Bamboo Weekly #128: Extreme heat
• Bamboo Weekly #112: Programming jobs
• Bamboo Weekly #86: FEMA

Part of the Pandas Methods Index. See also practice by skill.