Skip to content
Bamboo Weekly

pandas str.replace

Find-and-replace inside every string in a column — literally, or with a regular expression.

How much of your data-analysis time goes into getting numbers out of strings? For me it is an embarrassing amount, and str.replace is the tool I reach for first. The web hands us $1,234.50 and 17,098,246 (6,601,667) and 43%, and none of those is a number until the decoration comes off. str.replace takes the decoration off.

It lives on the .str accessor, which means it works element-wise on a column of strings and hands you back a new series. Nothing is modified in place, so it slots into a method chain exactly where you need it — usually right before astype.

Official documentation: Series.str.replace

The arguments that earn their keep

The two positional arguments are pat, what to look for, and repl, what to put in its place:

s.str.replace(',', '')                    # delete every comma
s.str.replace('-Q', ' Q')                 # swap one literal for another

Then there is regex, and this is where I need you to pay attention, because Pandas is inconsistent with itself here.

In modern Pandas, regex defaults to False. I checked the signature on Pandas 3.0.5 rather than trusting my memory, and there it is:

import inspect
import pandas as pd

inspect.signature(pd.Series.str.replace)

(self, pat: 'str | re.Pattern | dict', repl: 'str | Callable | None' = None, n: 'int' = -1, case: 'bool | None' = None, flags: 'int' = 0, regex: 'bool' = False)

Compare that with str.contains, whose regex defaults to True. Same accessor, same-looking pat argument, opposite default. That is a real trap, and it is worth seeing it bite:

t = pd.Series(['a.b', 'axb'])

t.str.contains('a.b')       # the dot is a wildcard here
t.str.replace('a.b', 'MATCH')   # the dot is a literal dot here

[True, True]
['MATCH', 'axb']

Both calls received the identical pattern. contains matched both strings; replace matched only the one containing an actual period. My habit is to pass regex= explicitly on every str.replace call, even when the value matches the default, so that the next reader does not have to remember which way it goes.

n limits how many replacements happen in each string, counting from the left. The default of -1 means all of them:

df['Total in km2 (mi2)'].str.replace(',', '', n=1).head(3)

0    510072,000 (196,940,000)
1       17098,246 (6,601,667)
2       14200,000 (5,480,000)
Name: Total in km2 (mi2), dtype: str

With regex=True, repl can contain backreferences — \1 for the first capture group, \2 for the second — which is how you rearrange a string rather than merely delete part of it. And repl can be a callable, which receives the match object and returns the replacement string. That is for replacements you have to compute:

(
    df['Total in km2 (mi2)']
    .str.replace(r'^([\d,]+) .*$',
                 lambda m: f'{int(m.group(1).replace(",", "")) / 1e6:.1f}M',
                 regex=True)
    .head(3)
)

0    510.1M
1     17.1M
2     14.2M
Name: Total in km2 (mi2), dtype: str

A callable requires the regex engine, so regex=True is mandatory there. Forget it and Pandas says ValueError: Cannot use a callable replacement when regex=False.

Two smaller arguments round it out. flags takes the standard re module flags, most usefully re.IGNORECASE, and case=False is the shorthand for the same thing. And in Pandas 3 you may pass a dictionary as pat, mapping each string you want gone to its replacement — in which case repl must be left alone.

A worked example, on real data

Bamboo Weekly #60 pulled country sizes off Wikipedia, and that page is a fine specimen of the problem. Note the user agent — Wikipedia turns away Pandas' default one:

import pandas as pd

url = ('https://en.wikipedia.org/wiki/'
       'List_of_countries_and_dependencies_by_area')

df = pd.read_html(url, storage_options={'User-Agent': 'Mozilla/5.0'})[1]

That is 277 rows, and the area column looks like this:

0    510,072,000 (196,940,000)
1       17,098,246 (6,601,667)
2       14,200,000 (5,480,000)
3        9,984,670 (3,855,100)
4        9,596,960 (3,705,410)
Name: Total in km2 (mi2), dtype: str

Square kilometers, then the same figure in square miles inside parentheses, with thousands separators throughout. Two kinds of junk, so two calls — the parenthetical needs the regex engine, the commas do not:

(
    df['Total in km2 (mi2)']
    .str.replace(r' \(.*\)', '', regex=True)
    .str.replace(',', '')
    .astype(float)
    .head()
)

0    510072000.0
1     17098246.0
2     14200000.0
3      9984670.0
4      9596960.0
Name: Total in km2 (mi2), dtype: float64

Once the column is numeric it behaves like any other, so the whole cleanup drops into a chain. Row 0 is Earth itself and row 2 is Antarctica, neither of which is a country, so I drop them before asking who is biggest:

(
    df
    .assign(area=pd.col('Total in km2 (mi2)')
                   .str.replace(r' \(.*\)|,', '', regex=True)
                   .astype(float))
    .loc[lambda df_: ~df_['Country / dependency'].isin(['Earth', 'Antarctica'])]
    .nlargest(6, 'area')
    .set_index('Country / dependency')
    ['area']
)

Country / dependency
Russia           17098246.0
Canada            9984670.0
China             9596960.0
United States     9525067.0
Brazil            8510346.0
Australia         7741220.0
Name: area, dtype: float64

Notice that the two cleanup steps collapsed into one. The alternation r' \(.*\)|,' means "a parenthetical, or a comma," and one pass removes both.

Backreferences let you keep part of what you matched instead of throwing it all away. Suppose I wanted the square miles rather than the square kilometers — the figure hiding inside the parentheses:

df['Total in km2 (mi2)'].str.replace(r'.*\((.*)\)', r'\1', regex=True).head(3)

0    196,940,000
1      6,601,667
2      5,480,000
Name: Total in km2 (mi2), dtype: str

Four mistakes people make

The regex default is not the same as on str.contains. This is the big one, and the reason I labor the point above. str.contains treats your pattern as a regular expression unless told otherwise; str.replace treats it as a literal unless told otherwise. Nothing warns you. Your call simply matches nothing, or matches too much, and the result flows quietly down the chain.

A literal . or $ stops being literal once regex=True. Turn the regex engine on and every metacharacter wakes up. The dot is the classic:

s = pd.Series(['3.5', '12.25'])

s.str.replace('.', '', regex=True)     # ['', '']
s.str.replace('.', '', regex=False)    # ['35', '1225']

The first call matched every character in every string and deleted the lot. The dollar sign fails in the other direction, silently doing nothing, because $ anchors to the end of the string rather than matching a currency symbol:

u = pd.Series(['$1,234.50', '$99.00'])

u.str.replace('$', '', regex=True)     # ['$1,234.50', '$99.00']
u.str.replace('$', '', regex=False)    # ['1,234.50', '99.00']

If you need those characters as themselves under regex=True, escape them: r'\.' and r'\$'.

Chaining four replaces where one alternation would do. Stripping a dollar sign, then commas, then a percent sign, then whitespace means four passes over the column and four lines to read. r'[$,%\s]' with regex=True says the same thing once. I do not think this is a crime — the chained version is easy to write and easy to debug — but past two or three links it is worth collapsing.

Cleaning almost enough, then calling astype. This is the classic, and it is what the parenthetical above is for. Remove the commas, forget the parentheses, and:

df['Total in km2 (mi2)'].str.replace(',', '').astype(float)

ValueError: could not convert string to float: '510072000 (196940000)'

The error is at least loud, and it tells you exactly which string defeated it. More on that failure mode at astype.

One more distinction worth holding onto: str.replace is not replace. The plain method works on whole values — swap every -999 for NaN, or every 'USA' for 'United States' — and it looks at each cell as a unit. str.replace reaches inside each string and edits part of it. Confusing the two produces a no-op rather than an error, which is why the mix-up survives so long.

Where it shows up in Bamboo Weekly

#60: Iceland is the source of the example above. Two Wikipedia tables needed cleaning before they could be joined: r'\(\S+\)' for the square-mile parentheticals, then a comma strip and astype(float), and r'\[\w+\]' to lift the footnote markers off the country names.

#78: Stock markets loaded seven index CSVs whose prices arrived as strings with thousands separators. Every one of them needed .str.replace(',', '').astype(float), which is exactly the shape this method usually takes — and rather than write that three times, the solution builds the transformations in a dict comprehension and hands the whole dict to assign.

#51: Academy Awards had to cope with the early Oscars, whose ceremonies covered two years and were recorded as 1927/28. A capture group and a backreference fix it in one pass: .str.replace(r'19\d\d/(\d\d)', r'19\1', regex=True).astype(int). That is the argument for backreferences in a sentence — a plain deletion could not have built 1928 out of those characters.

#71: Holidays needed holiday names to line up across countries, which meant dropping parenthetical notes and a trailing "holiday" that some countries append and others do not. Two chained calls, the second with regex=True, flags=re.IGNORECASE, and suddenly the same festival groups together no matter who is celebrating it.

Practice it

Work through a .str.replace() exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/str-replace/

Go deeper

Nearly every hard str.replace call is really a regular-expression question wearing a Pandas costume. If patterns are still a mystery, my free 14-part crash course is at RegexpCrashCourse.com.

Three pages sit next to this one: str.contains, whose opposite regex default started this whole discussion; replace, for whole values rather than parts of strings; and astype, which is where a str.replace chain almost always ends.

More Pandas videos on Python and Pandas with Reuven Lerner.

Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.

See it on real data

Below are the 32 Bamboo Weekly exercises that use str.replace on real-world data — try each one, then study the worked solution.

Part of the Pandas Methods Index. See also practice by skill.