Skip to content
Bamboo Weekly

pandas astype

Convert a column to another dtype — and find out what your data was really hiding.

Have you ever loaded a table of numbers, tried to convert the column to integers, and been told Cannot convert non-finite values (NA or inf) to integer? Or worse, had the conversion succeed and quietly give you the wrong answer? Both happen constantly, and both come from the same misunderstanding.

astype is a cast, not a parser. It takes the values you already have and demands that they become the type you named. It will not strip a dollar sign, it will not skip a bad row, and it will not warn you when a number no longer fits. Everything it does is your instruction carried out literally. Once you expect that, the errors stop being mysterious and start being useful — because an astype that raises is usually telling you something true about your data that you had not noticed yet.

Official documentation: DataFrame.astype

The arguments that earn their keep

There are three, and really only two matter.

dtype is the one you always pass. Give it a single type and every column is converted the same way, or give it a dict to convert several columns differently in one call:

df['year'].astype(int)                  # one column, one type
df.astype({'year': 'int64',             # several columns, several types
           'price': 'float',
           'region': 'category'})

The dict form is worth the habit. It reads as a declaration of what the table should look like, it happens in a single pass, and columns you do not name are left alone.

errors accepts 'raise', the default, or 'ignore', which returns the original values untouched when the conversion fails. On a data frame that decision is made column by column, so the columns that can convert do and the rest stay as they were. Older tutorials will tell you this argument is on its way out — that is a real deprecation, but it belongs to to_numeric and to_datetime, where errors='ignore' is now gone and passing it raises ValueError: invalid error value specified. On astype it is alive and unwarned in Pandas 3.

copy is deprecated. Copy-on-Write has been the rule since Pandas 3, so passing it now earns you a DeprecationWarning and nothing else. Drop it.

The types you can ask for are worth a moment too. Alongside int, float, str and bool there are the nullable extension types, spelled with a capital letter — Int64, Float64, boolean — which look the same but can hold missing values. And category, which stores each distinct value once and gives you the memory back.

One Pandas 3 change to expect: strings are their own str dtype now, backed by PyArrow when it is installed, and df.dtypes reports str rather than the object that every older tutorial shows. .astype(str) still does what it always did; only the label changed.

A worked example, on real data

Wikipedia's list of countries by population is a good specimen, because every column arrives as text or as the wrong number:

import pandas as pd

url = ('https://en.wikipedia.org/wiki/'
       'List_of_countries_and_dependencies_by_population')

df = pd.read_html(url, storage_options={'User-Agent': 'Mozilla/5.0'})[0]

df.dtypes

Location                                            str
Population                                      float64
% of world                                          str
Date                                                str
Source (official or from the United Nations)        str
Notes                                               str
dtype: object

Population is a float, which is odd for a count of people:

        Location    Population % of world
0          World  8.232000e+09       100%
1          India  1.429404e+09      17.3%
2          China  1.404890e+09      17.0%
3  United States  3.417849e+08       4.1%
4      Indonesia  2.883151e+08       3.5%

So I ask for integers, and Pandas refuses:

df['Population'].astype(int)

IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer.

That error is doing me a favor. Somewhere in 241 rows there is a missing population — it turns out to be a blank separator row in the Wikipedia table — and a single NaN is enough to force the whole column to float. The capital-I Int64 accepts it:

df['Population'].astype('Int64').tail(3)

238    882
239    593
240     40
Name: Population, dtype: Int64

Now everything in one pass. The percentages need their sign removed before they can be floats, and the source column repeats a small number of phrases behind a litter of footnote markers:

clean = (
    df
    .rename(columns={'Source (official or from the United Nations)': 'source'})
    .assign(source=lambda df_: df_['source'].str.replace(r'\[.*?\]', '', regex=True),
            pct=lambda df_: df_['% of world'].str.rstrip('%'))
    .astype({'Population': 'Int64', 'pct': 'float', 'source': 'category'})
    [['Location', 'Population', 'pct', 'source']]
)

clean.head()

        Location  Population    pct                      source
0          World  8232000000  100.0               UN projection
1          India  1429404000   17.3         Official projection
2          China  1404890000   17.0           Official estimate
3  United States   341784857    4.1           Official estimate
4      Indonesia   288315089    3.5  National annual projection

clean.dtypes

Location           str
Population       Int64
pct            float64
source        category
dtype: object

Note the shape of that: the cleaning happens in assign, and astype runs last, on values that are finally ready to be converted. That ordering is most of the skill.

The category conversion was not decoration. Those 241 rows hold only 31 distinct source phrases, and storing each one once is a large saving:

clean['source'].astype(str).memory_usage(deep=True, index=False)   # 6709
clean['source'].memory_usage(deep=True, index=False)               # 1053

Six times smaller, for a column of 241 values. On a few million rows this is the difference between a data frame that fits in memory and one that does not, which is the subject of memory_usage.

Five mistakes people make

Converting to int when the column has missing values. This is the most common astype error there is, and the fix is a capital letter. Lowercase int and int64 are NumPy types, which have no way to represent a missing value; capital Int64 is the Pandas nullable type, which holds <NA> alongside your integers. If you find yourself dropping rows to make .astype(int) work, ask first whether Int64 would have let you keep them.

Cleaning the string, but missing one symbol. Wikipedia's area table gives totals as '17,098,246 (6,601,667)' — square kilometers, then square miles in parentheses. Strip the commas, feel finished, and the conversion still fails:

area['Total in km2 (mi2)'].str.replace(',', '').astype(float)
# ValueError: could not convert string to float: '510072000 (196940000)'

Read the error rather than skimming it. It quotes the offending value, and the value shows you exactly what you forgot. Deleting the parenthetical first fixes it, and the cleaning half of this pattern lives at str.replace.

Assuming .astype(float) validates your data. It does not, and the difference from pd.to_numeric matters. astype is all-or-nothing: one bad value and nothing converts. pd.to_numeric(errors='coerce') converts what it can and marks the rest as NaN:

s = pd.Series(['1.5', 'nan', 'inf', 'N/A'])

s.astype(float)                     # ValueError: could not convert string to float: 'N/A'
pd.to_numeric(s, errors='coerce')   # [1.5, nan, inf, nan]

Notice that both let 'nan' and 'inf' through as real floating-point values, because Python's float() accepts those spellings. Neither function is checking whether your data makes sense. Which one is honest depends on what you know: astype when every value ought to be a number and a failure means something is wrong, to_numeric(errors='coerce') when you expect junk and want it labeled as missing rather than hidden.

Forgetting that a category is a closed set. Convert a column to category and Pandas records the values it saw. Assign one it did not see, and it objects:

clean.loc[0, 'source'] = 'My own guess'
# TypeError: Cannot setitem on a Categorical with a new category
#            (My own guess), set the categories first

This surprises people weeks later, long after the conversion. cat.add_categories is the answer when you meant it, and the error is a genuine safeguard when you did not — it catches typos that a plain string column would have swallowed.

Trusting a cast that cannot warn you. Two conversions will corrupt your data in silence. Undersized integers wrap around:

pd.Series([300, 40000]).astype('int16')   # [300, -25536]

And bool asks only whether a value is empty or zero, never what it says:

pd.Series(['True', 'False', 'no', '']).astype(bool)   # [True, True, True, False]

The string 'False' is a non-empty string, so it is True. If you are reading in a column of "yes"/"no" or "True"/"False" text, you want map with an explicit dictionary, so that anything unexpected shows up as missing instead of as True.

Where it shows up in Bamboo Weekly

#60: Iceland is where the area table above comes from. To compare population density across countries, two Wikipedia tables had to be cleaned and joined, which meant lifting the parenthetical square-mile figures, stripping the commas, and only then calling .astype(float).

#30: Uncertainty worked with the World Uncertainty Index, whose periods are recorded as strings like 1990q1. Splitting on the q and calling .astype(np.int16) on the year and .astype(np.int8) on the quarter turns that into two numeric columns — and shows the other reason to name a specific width, which is that a quarter never needs more than one byte.

#72: City travel classified how people commute in cities around the world as A for active, B for bus, or C for car. Three distinct values repeated across thousands of rows is precisely the case category exists for, and .astype('category') is how the solution stores it.

#42: Plant hardiness compared the USDA's plant hardiness zones against the previous map. Each zone carries its temperature range as a single string, so .str.split().str.get(0).astype(int) and its get(-1) twin split one text column into the two numbers the comparison actually needed.

Practice it

Work through an .astype() exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/astype/

Go deeper

astype is almost never the whole job. It is the last step of a chain that begins somewhere else, which is why the pages worth reading next are the ones about getting the values ready: str.replace for stripping the characters that block a conversion, str.split for pulling numbers out of a combined field, and memory_usage for measuring what a category conversion bought you.

More Pandas videos on Python and Pandas with Reuven Lerner.

Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.

See it on real data

Below are the 74 Bamboo Weekly exercises that use astype on real-world data — try each one, then study the worked solution.

Part of the Pandas Methods Index. See also practice by skill.