Skip to content
Bamboo Weekly

pandas read_html

Pull every HTML table off a web page into a list of data frames.

What read_html actually gives you

What if the data you want is already sitting in a table on a web page, and nobody has bothered to publish it as a CSV file? That is the situation read_html was built for. You hand it a URL, and it hands you back every <table> on that page, already parsed into data frames.

Note the word "every." This surprises almost everyone the first time, so let me say it plainly: read_html does not return a data frame. It returns a list of data frames, even when the page contains exactly one table. If your first line is df = pd.read_html(url) followed by df.head(), you get an AttributeError, because lists have no head method. That single misunderstanding accounts for most of the confused questions I get about this function.

Official documentation: pandas.read_html

The arguments that earn their keep

pd.read_html(source,
             match='Largest city',   # only tables whose text matches this regex
             header=1,               # which row holds the column names
             index_col='Name',       # which column becomes the index
             flavor='lxml',          # which parser to use
             na_values=['-', 'n/a'])  # what counts as missing

Like read_csv and read_excel, source can be a path or a URL, so the example below runs without downloading anything first.

A worked example, on real data

Let me use the Wikipedia page on NATO membership, which Bamboo Weekly #58 built a puzzle around. Start by asking what is on the page rather than assuming:

import pandas as pd

url = 'https://en.wikipedia.org/wiki/Member_states_of_NATO'
headers = {'User-Agent': 'Mozilla/5.0'}

tables = pd.read_html(url, storage_options=headers)

type(tables), len(tables)

(<class 'list'>, 5)

Five tables, and only some of them are data. Their shapes tell the story:

[t.shape for t in tables]

[(32, 11), (33, 8), (33, 7), (1, 2), (7, 2)]

The last two are page furniture — a footer and a navigation box. Counting brackets until you find the right index is a miserable way to work, and it breaks the moment Wikipedia adds a table above yours. Describe the table instead, with match, which takes a regular expression and keeps only tables whose text matches it:

len(pd.read_html(url, match='Largest city', storage_options=headers))

1

One table, selected by what it contains rather than where it sits. Now I can take element zero honestly, and set a sensible index while I am at it:

df = pd.read_html(url, match='Largest city',
                  index_col='Name', storage_options=headers)[0]

df[['Capital', 'Largest city', 'Accession[7]']].head()

                   Capital             Largest city       Accession[7]
Name
Albania             Tirana                   Tirana       1 April 2009
Belgium   City of Brussels  Brussels Capital Region  24 August 1949[a]
Bulgaria             Sofia                    Sofia      29 March 2004
Canada              Ottawa                  Toronto  24 August 1949[a]
Croatia             Zagreb                   Zagreb       1 April 2009

Real data, and visibly dirty data. Wikipedia's footnote markers came along for the ride — in the column name Accession[7], in the values, and worst of all in the index:

[name for name in df.index if '[' in name]

['Czech Republic[b]', 'Denmark[c]', 'France[e]', 'Germany[f]', 'Netherlands[g]',
 'Norway[j]', 'Spain[l]', 'Turkey[m]', 'United Kingdom[n]', 'United States[o]']

Ten countries whose names will silently fail to match anything when you join this against another data set. Strip the markers, and the frame becomes usable:

(
    df
    .rename(index=lambda name: name.split('[')[0])
    ['Accession[7]']
    .str.extract(r'(\d{4})', expand=False)
    .value_counts()
    .sort_index()
)

Accession[7]
1949    12
1952     2
1955     1
1982     1
1999     3
2004     7
2009     2
2017     1
2020     1
2023     1
2024     1

Twelve founding members in 1949, the post-Cold War expansion of seven countries in 2004, then Finland and Sweden at the end — a real historical shape, pulled out of a web page in a handful of lines.

One more argument worth meeting. The defense-spending table on that same page has a header cell spanning three columns, so Pandas gives you a two-level MultiIndex:

spend = pd.read_html(url, match='Defence expenditure', storage_options=headers)[0]
spend.columns.nlevels

2

If you would rather have flat column names, tell header which row to use:

flat = pd.read_html(url, match='Defence expenditure', header=1,
                    storage_options=headers)[0]
list(flat.columns)

['Member state', 'Population[s]', 'GDP (nominal) ($billions)[t]',
 'Total ($millions)', '% real GDP', 'Per capita', 'Personnel[t]']

The flavor argument picks the parser: lxml is the default and fastest, while bs4 and html5lib are more forgiving of malformed markup. If a page returns nothing but you can see a table in your browser, flavor='html5lib' is a reasonable second guess.

Where it shows up in Bamboo Weekly

Scraping a table off the web is a recurring Bamboo Weekly move — 20 issues use read_html on live pages.

Bamboo Weekly #96 analyzed the dates, cities, and venues of the Taylor Swift Eras Tour. The tour's Wikipedia page is thick with tables, so the solution uses a regex to grab exactly the per-year show listings: pd.read_html(url, match='List of 202. shows'). That is match doing precisely what it is for.

Bamboo Weekly #159 asked which presidents delivered the State of the Union as a speech and which sent a written message. The source table has a two-row header, so the solution passes header=[1,2] and works with the resulting MultiIndex columns.

Bamboo Weekly #60 compared Iceland's population density with every other country, joining two Wikipedia tables. It illustrates both positional selection and footnote cleanup, using .str.replace(r'\[\w+\]', '', regex=True) before the join.

Bamboo Weekly #40 looked at sovereign bond ratings, and is worth reading for the part that went wrong: the site returned HTTP 403, so the solution fetches the page with requests and passes the result to read_html through StringIO.

Five mistakes people make

Treating the result as a data frame. It is a list. Either index into it, unpack it, or loop over it — but do not call data frame methods on it and expect anything but an AttributeError.

Assuming the table you want is [0]. Element zero is frequently an infobox, a navigation banner, or a layout table. Even when it happens to be right today, it is a positional guess that breaks when the page is edited. Prefer match= with a distinctive string from the table you actually want.

Being surprised by MultiIndex columns. Any header cell that spans several columns produces a two-level column index, and df['Total ($millions)'] will then raise a KeyError. Either address the columns as tuples, or pass header= to pick a single row.

Forgetting that you need a parser installed. read_html does not ship its own; it needs lxml, or beautifulsoup4 together with html5lib. Without one you get an ImportError before any parsing happens. Install with uv add lxml or uv add beautifulsoup4 html5lib.

Letting footnote markers poison your data. As above, [1] and [a] markers ride along inside values and index labels. A column that should be numeric comes back as text, astype(int) fails, and joins miss silently. Strip them with .str.replace(r'\[\w+\]', '', regex=True) as a matter of routine. Wikipedia also sprinkles non-breaking spaces and soft hyphens through its headers, so a column name that looks like Population[s] may not compare equal to the string you typed.

Practice it

Work through a read_html() exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/read-html/

Go deeper

Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.

If your data is in a spreadsheet rather than a web page, see read_excel.

See it on real data

Below are the 20 Bamboo Weekly exercises that use read_html on real-world data — try each one, then study the worked solution.

Part of the Pandas Methods Index. See also practice by skill.