Skip to content
Bamboo Weekly

pandas str.split

Break each string into pieces on a delimiter.

What it does

How do you get at the part of a string you actually want? Real data sets hand you compound text constantly: 2013q2 for a quarter, Rat Islands, Aleutian Islands, Alaska for a location, 242 km SE of Sarangani, Philippines for an earthquake. The value you need is in there, but it is glued to two or three others, and no amount of filtering will separate them.

str.split is the tool: the vectorized version of Python's own str.split, applied to every element of a series at once. Give it a delimiter and it breaks each string exactly as Python would. What comes back depends on one argument. By default you get a series of Python lists, one list per row; pass expand=True and you get a data frame, one column per piece.

That default surprises people. str.split does not give you columns unless you ask for columns, and a column of lists is a dead end for most of Pandas — the string accessor and the arithmetic and comparison operators all stop working on it. Which of the two you want is the first decision to make, and most of this page is about that choice.

Official documentation: Series.str.split and Series.str.rsplit

The arguments that earn their keep

There are four, and in Pandas 3 only the first can be passed positionally:

s.str.split()                       # split on runs of whitespace
s.str.split(', ')                   # split on a literal delimiter
s.str.split(', ', expand=True)      # a data frame of columns, not a column of lists
s.str.split(', ', n=1)              # at most one split, so at most two pieces
s.str.split(r'\s*,\s*', regex=True) # treat the pattern as a regular expression
s.str.rsplit(', ', n=1)             # count the splits from the right instead

pat is the delimiter. Leave it out and you split on runs of whitespace, with leading and trailing whitespace discarded — ' a b c ' becomes ['a', 'b', 'c'], three pieces rather than the nine that split(' ') would give you.

expand is the one that decides the shape of the answer. False, the default, gives one list per row. True gives a data frame whose column count is set by the longest string in the column, padding every shorter row with NaN.

n caps the number of splits, not the number of pieces: n=1 performs one split and so yields at most two pieces, leaving the rest of the string intact in the second. That is how you take a string apart at its first delimiter and keep everything after it whole.

regex decides how pat is read. Leave it alone and Pandas guesses: a one-character delimiter is a literal, anything longer is compiled as a regular expression. That guess is the source of a mistake I will come back to.

.str.rsplit is the same method counting from the other end. It takes pat, n and expand, but no regex — passing one raises TypeError. It earns its keep only alongside n, where it means "split off the last field and leave the rest alone."

A worked example, on real data

The USGS publishes every recorded earthquake through a query API that hands back CSV directly — the data set behind Bamboo Weekly #3. Every quake of magnitude 2.5 or greater with a loss estimate attached:

import pandas as pd

url = ('https://earthquake.usgs.gov/fdsnws/event/1/query.csv'
       '?starttime=2000-01-01&endtime=2023-02-15'
       '&minmagnitude=2.5&orderby=time&producttype=losspager')

df = pd.read_csv(url, usecols=['time', 'mag', 'place'])

df['place'].iloc[1:7]

1       44 km ENE of Luganville, Vanuatu
2          Kermadec Islands, New Zealand
3            12 km ESE of P?hala, Hawaii
4                Tristan da Cunha region
5    242 km SE of Sarangani, Philippines
6                        Carlsberg Ridge
Name: place, dtype: str

The question mark in P?hala is in the USGS file itself, not something Pandas did on the way in.

I want the region — Vanuatu, New Zealand, Hawaii — so I can count quakes by country. Split on the comma-space:

df['place'].str.split(', ').iloc[2]

['Kermadec Islands', 'New Zealand']

A Python list, one row's worth. I pulled out a single row on purpose: printing the whole column here is misleading, because Pandas renders a two-string list as [Kermadec Islands, New Zealand], byte for byte what it would print for a one-element list holding the unsplit string. Use iloc when you need to know whether the split actually happened.

Now, how many pieces per row?

df['place'].str.split(', ').str.len().value_counts()

place
2    6282
1    1640
3     138
5       1
Name: count, dtype: int64

There is the shape of the problem. Most rows split into two, 1,640 have no comma at all, and a handful have three or five. That rules out reaching for the second piece by position:

df['place'].str.split(', ').str[1].value_counts().sum()

6421

.str[1] quietly dropped 1,640 rows — every place name with no comma became NaN. Counting backward instead keeps all of them:

df['place'].str.split(', ').str.get(-1).value_counts().head(6)

place
Alaska              915
Indonesia           524
CA                  397
Papua New Guinea    368
Chile               275
Japan               271
Name: count, dtype: int64

8,061 rows, 8,061 counted. This is where rsplit matters. Suppose I want two columns — the locality and the region — for the rows with more than one comma:

sub = df.loc[pd.col('place').str.count(',') > 1, 'place']

sub.str.split(', ', expand=True).head(4)

                         0                 1          2    3    4
122            Rat Islands  Aleutian Islands     Alaska  NaN  NaN
216  84km ESE of Maneadero              B.C.         MX  NaN  NaN
365       6km NNW of Delta              B.C.         MX  NaN  NaN
383               Minahasa          Sulawesi  Indonesia  NaN  NaN

The region has landed in column 1 on one row and column 2 on the next. There is no column that means "region." Split from the right, once:

sub.str.rsplit(', ', n=1, expand=True).head(4)

                                 0          1
122  Rat Islands, Aleutian Islands     Alaska
216    84km ESE of Maneadero, B.C.         MX
365         6km NNW of Delta, B.C.         MX
383             Minahasa, Sulawesi  Indonesia

Two columns, and column 1 is the region on every row. This is the "City, State" problem in general form: the field you want is the last one, the field you do not care about may contain the delimiter, and rsplit with n=1 is the fix. Attach it to the frame and the analysis falls out:

(
    df
    .assign(region=pd.col('place').str.rsplit(', ', n=1).str.get(-1))
    .groupby('region')['mag'].agg(['count', 'max'])
    .sort_values('count', ascending=False)
    .head(5)
)

                  count  max
region
Alaska              915  7.9
Indonesia           524  7.8
CA                  397  6.4
Papua New Guinea    368  7.9
Chile               275  8.3

The pairing with explode

Splitting into lists looks useless until you meet explode, which turns a series of lists into one row per element. Split, then explode, then count: that is one of the most-used moves in Bamboo Weekly, and it is what the list-returning default is actually for.

(
    df['place']
    .str.split()
    .explode()
    .loc[lambda s_: s_.str.len() >= 5]
    .value_counts()
    .head(6)
)

place
Alaska       942
Islands      910
region       735
Indonesia    524
Guinea       370
Papua        368
Name: count, dtype: int64

8,061 place descriptions became 46,102 words, which then count like any other series. Use expand=True when each string holds the same fields in the same order and you want columns; use the list default plus explode when each string holds a variable number of the same kind of thing and you want rows.

Five mistakes people make

Forgetting expand=True and then trying to use the lists. The string accessor does not raise on a column of lists — it returns NaN for every row:

df['place'].str.split(', ').str.upper().head(3)

0   NaN
1   NaN
2   NaN
Name: place, dtype: float64

A float column of NaN where you expected text. Assigning to two columns at once at least fails loudly:

df[['locality', 'region']] = df['place'].str.split(', ')

ValueError: Columns must be same length as key

Using expand=True when the rows have different numbers of pieces. Pandas sizes the frame to the longest string, so one outlier sets the width for everybody. On this data, df['place'].str.split(', ', expand=True) returns five columns, and columns 3 and 4 hold exactly one non-null value each — the row 41 km ENE of Villa Presidente Frei, Ñuñoa, Santiago, Chile, Chile. Check .str.len().value_counts() before you expand, and cap the width with n= if the tail is junk.

Reaching for .str[0] or .str[1] after the split. A row where the delimiter never appeared still splits, into a one-element list holding the whole original string. So .str[0] gives you back the unsplit text with no warning, and .str[1] gives you NaN — 1,640 of them here. When the piece you want is the last one, .str.get(-1) is both shorter and correct.

Splitting on a regex metacharacter without escaping it. Pandas decides whether pat is a regular expression by its length, so '|' and '.' are literals while ' | ' is a pattern. That produces two different answers for what looks like the same delimiter:

s = pd.Series(['Rat Islands|Alaska', 'Kermadec|New Zealand'])

s.str.split('|').iloc[0]        # one character, so a literal

['Rat Islands', 'Alaska']

s.str.split(' | ').iloc[0]      # three characters, so a regular expression

['Rat', 'Islands|Alaska']

The second one split on a space or a pipe or a space, which is not what anyone meant. Escape it — s.str.split(r' \| ') — and you get ['Rat Islands|Alaska'], correctly finding no delimiter at all. The reverse trap is forcing the issue: pd.Series(['us.6000.jnqz']).str.split('.', regex=True) reads the dot as "any character," splits between every pair of them, and hands back a list of thirteen empty strings.

Assuming rsplit does something different from split. Without n it does not. On all 8,061 rows above, df['place'].str.rsplit(', ') and df['place'].str.split(', ') return identical results, because both run out of delimiters before they run out of splits. rsplit changes the answer only when the number of splits is capped, which is exactly the n=1 case above.

Where it shows up in Bamboo Weekly

24 Bamboo Weekly solutions use str.split. Only two string methods appear in more of them: str.replace and str.contains, at 32 apiece.

Bamboo Weekly #3: Earthquakes is the data set above, and the solution uses .str.split(',').str.get(-1).str.strip() to pull the region out of place before counting — the last-field pattern, solved with get(-1) and a strip rather than with rsplit.

Bamboo Weekly #53: Airport animals is the clearest expand=True example on the site. A column arrives holding two numbers in one cell, and df['2023 Jan - Nov'].str.split(expand=True) turns it into the two columns the rest of the puzzle needs.

Bamboo Weekly #79: Cyber attacks is the split-and-explode pairing at full length. Incident descriptions get split on whitespace, exploded to one word per row, stripped of punctuation, filtered, and counted — a five-method chain that starts with str.split.

Bamboo Weekly #65: Microplastics is the one to read for regex=. NOAA separates keywords with either a slash or a semicolon, so the split is .str.split('[/;]', regex=True) — a character class, made explicit rather than left to the length guess.

Practice it

Work through a str.split exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/str-split/

Go deeper

str.split is the start of a chain, never the end. explode is what you call when the split produced lists, str.get is how you pick one piece out of them, str.strip cleans up the whitespace a sloppy delimiter left behind, and value_counts is usually what you were after in the first place. If you only need to know whether a delimiter is there, str.contains answers that without splitting anything.

More Pandas videos on Python and Pandas with Reuven Lerner.

Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.

See it on real data

Below are the 24 Bamboo Weekly exercises that use str.split on real-world data — try each one, then study the worked solution.

Part of the Pandas Methods Index. See also practice by skill.