Skip to content
Bamboo Weekly

pandas corr

Measure how strongly numeric columns move together — and learn how easily that measure is fooled.

Do these two columns move together? That is one of the first questions I ask of any data set, and corr is how Pandas answers it. Call it on a data frame and you get a square matrix: every numeric column against every other, each cell between -1 and +1. Call it on a series, handing it a second, and you get one float.

The scale is the easy part. A +1 means the columns rise and fall in lockstep, -1 means one rises exactly as the other falls, and 0 means knowing one tells you nothing about the other. The diagonal is always 1.0, because a column is perfectly correlated with itself.

The hard part is that a number this compact hides which rows it used, which columns it left out, and the shape of the relationship. Most of this page is about that hiding.

Official documentation: DataFrame.corr and Series.corr

The arguments that earn their keep

df.corr(method='pearson',   # 'pearson', 'spearman' or 'kendall'
        min_periods=1,      # ignore pairs with fewer overlapping rows than this
        numeric_only=False) # True quietly skips non-numeric columns

s.corr(other,               # the series to compare against
       method='pearson',
       min_periods=None)

method chooses the statistic. Pearson, the default, measures how close the points lie to a straight line. Spearman and Kendall discard the values and keep only their ranks, so they ask whether the relationship runs consistently upward or downward without caring whether it is straight. When your data is skewed, or has a couple of enormous members, or is ordinal rather than interval, rank correlation is the honest choice.

min_periods sets a floor on how many overlapping non-null rows a pair must have before Pandas reports a number at all. Below that floor you get NaN instead of a value computed from six rows that happen to agree.

numeric_only decides what happens to your text columns. The default is False, which means a string column raises rather than being ignored. Pass True and it is skipped in silence.

A worked example, on real data

Let me use the NATO membership page on Wikipedia — the source behind Bamboo Weekly #58. Its defense-expenditure table gives population, GDP, military spending, spending as a share of GDP, and active personnel.

import pandas as pd

url = 'https://en.wikipedia.org/wiki/Member_states_of_NATO'

nato = (
    pd.read_html(url, match='Defence expenditure', header=1,
                 storage_options={'User-Agent': 'Mozilla/5.0'})[0]
    .set_axis(['country', 'population', 'gdp', 'defense_total',
               'defense_pct_gdp', 'defense_per_capita', 'personnel'],
              axis='columns')
    .loc[lambda df_: df_['country'] != 'NATO']
    .drop(columns='defense_per_capita')
    .set_index('country')
    .replace('—N/a', None)
    .astype('Float64')
)

nato.head()

          population     gdp  defense_total  defense_pct_gdp  personnel
country
Albania    2872849.0   29.89          598.0              2.0     6600.0
Belgium   12055794.0   725.0        14465.0              2.0    22100.0
Bulgaria   6670476.0  128.97         2755.0             2.14    26600.0
Canada    40086114.0  2282.5        45562.0              2.0    70900.0
Croatia    4026339.0  104.04         2182.0              2.1    13600.0

That .replace('—N/a', None) is doing real work. Wikipedia writes an em dash where Iceland's figures should be, because Iceland has no standing army, and that one string makes four numeric columns arrive as text.

Now the matrix:

nato.corr().round(3)

                 population    gdp  defense_total  defense_pct_gdp  personnel
population            1.000  0.969          0.953            0.108      0.981
gdp                   0.969  1.000          0.994            0.145      0.957
defense_total         0.953  0.994          1.000            0.167      0.952
defense_pct_gdp       0.108  0.145          0.167            1.000      0.199
personnel             0.981  0.957          0.952            0.199      1.000

The top-left block looks obvious: big countries have big economies, big budgets, and big armies. The defense_pct_gdp row is where it stops. What share of its economy a country devotes to defense has almost nothing to do with how large that economy is — the interesting finding, and the one the matrix nearly buries.

Usually I want one column of it, sorted, rather than the whole grid:

nato.corr()['defense_total'].sort_values(ascending=False)

defense_total      1.000000
gdp                0.994287
population         0.952905
personnel          0.952333
defense_pct_gdp    0.166608
Name: defense_total, dtype: float64

For a single pair, use the series form, which takes the other series as its first argument:

nato['gdp'].corr(nato['defense_total'])

0.994286939208244

Same number, no wasted work.

Now method. Put both countries and budgets on a per-person footing, and ask whether richer members spend more per citizen on defense:

per_capita = nato.assign(
    gdp_per_capita=lambda df_: df_['gdp'] * 1e9 / df_['population'],
    defense_per_capita=lambda df_: df_['defense_total'] * 1e6 / df_['population'],
)

for method in ['pearson', 'spearman', 'kendall']:
    print(method,
          per_capita['gdp_per_capita'].corr(per_capita['defense_per_capita'],
                                            method=method))

pearson 0.44709910404899544
spearman 0.8673387096774194
kendall 0.7548387096774194

Pearson says the link is moderate; Spearman says it is strong. When two methods disagree that badly, something in the data is worth looking at:

per_capita[['gdp_per_capita', 'defense_per_capita']].dropna().nlargest(3, 'defense_per_capita').round(0)

                gdp_per_capita  defense_per_capita
country
Czech Republic         36111.0              7277.0
Norway                 88784.0              2840.0
United States          88667.0              2833.0

The Czech Republic is listed as outspending the United States per person, on an income two-thirds lower. It is not: the table says $78,194 million, while the same row's 2.01% of a $388 billion economy is about $7,800 million. Wikipedia has a stray zero. What that one digit costs:

trimmed = per_capita.drop(index='Czech Republic')

trimmed['gdp_per_capita'].corr(trimmed['defense_per_capita'])
trimmed['gdp_per_capita'].corr(trimmed['defense_per_capita'], method='spearman')

0.9151643855055478
0.9252502780867631

One row out of 31 moves Pearson from 0.447 to 0.915, and Spearman from 0.867 to 0.925. That is the case for rank correlation in two numbers: Pearson works with the values, so one wrong magnitude dominates; Spearman works with the ordering, where a wrong value can only be one place out of line.

Nor does the problem need a typo. Drop the largest legitimate member and nato['population'].corr(nato['gdp']) falls from 0.969 to 0.887, while Spearman moves only from 0.944 to 0.938 — one perfectly correct row was carrying eight points of Pearson correlation.

Finally, min_periods. Iceland is missing three of the five columns, so corr had 32 countries for some pairs and 31 for the rest. Demand all 32 and Pandas says which cells it cannot fill:

nato.corr(min_periods=32).round(3)

                 population    gdp  defense_total  defense_pct_gdp  personnel
population            1.000  0.969            NaN              NaN        NaN
gdp                   0.969  1.000            NaN              NaN        NaN
defense_total           NaN    NaN            NaN              NaN        NaN
defense_pct_gdp         NaN    NaN            NaN              NaN        NaN
personnel               NaN    NaN            NaN              NaN        NaN

Four mistakes people make

Letting numeric_only=True shrink the matrix in silence. Run corr on the raw Wikipedia table and Pandas raises ValueError: could not convert string to float: 'Albania'. That error is a gift. Silence it with numeric_only=True and you get a 2×2 matrix out of seven columns, with no hint that the em dashes turned five of them into text. Fix the dtypes instead, as Bamboo Weekly #25 does with select_dtypes('float64').

Assuming every cell rests on the same rows. Missing values are dropped pairwise, not row-wise. Above, one cell comes from 32 countries and the rest from 31, and the output does not say so. On messier data you can compare a correlation built from 900 rows against one built from 40. Check with .notna().sum(), or set min_periods.

Reading a near-zero correlation as "no relationship." Pearson measures straightness and Spearman measures monotonicity, so a relationship that goes up and then comes back down defeats both. Here is a year of daily mean temperatures for Washington, DC, against the day of the year:

url = ('https://archive-api.open-meteo.com/v1/archive'
       '?latitude=38.91&longitude=-77.04'
       '&start_date=2024-01-01&end_date=2024-12-31'
       '&daily=temperature_2m_mean&timezone=UTC')

dc = (
    pd.DataFrame(pd.read_json(url)['daily'].to_dict())
    .assign(time=lambda df_: pd.to_datetime(df_['time']))
    .assign(day_of_year=lambda df_: df_['time'].dt.day_of_year)
)

dc['day_of_year'].corr(dc['temperature_2m_mean'])
dc['day_of_year'].corr(dc['temperature_2m_mean'], method='spearman')

0.20100453612373748
0.1917628501649174

Two numbers near zero, describing one of the most reliable relationships on Earth:

dc.groupby(dc['time'].dt.month)['temperature_2m_mean'].mean().round(1)

time
1      2.6
2      4.8
3      8.9
4     14.0
5     18.9
6     24.5
7     26.0
8     23.7
9     20.3
10    14.7
11    10.6
12     3.6
Name: temperature_2m_mean, dtype: float64

From 2.6°C in January to 26.0°C in July and back to 3.6°C in December. The correlation is near zero because the curve comes back down, not because the seasons are a coincidence. Plot before you correlate.

Reading correlation as causation. This deserves saying plainly, because Bamboo Weekly works with current-events data and the temptation is real. That +0.994 between GDP and defense spending does not say which drives which, or rule out a third thing driving both. Bamboo Weekly #14 puts it well on job openings and the quits rate: they move together at 86 percent, which says nothing about what led to what.

Where it shows up in Bamboo Weekly

corr appears in 39 Bamboo Weekly solutions, which makes it one of the workhorses of the series.

Bamboo Weekly #78: Stock markets is the full matrix earning its place. Daily percentage changes across seven world indexes go into corr, and the 7×7 grid is hard to read — so the solution pipes it into .style.highlight_between() to color the strong and weak cells differently. The S&P and NASDAQ land at 0.944; Shanghai barely relates to anything.

Bamboo Weekly #68: Dangerously hot weather shows the pattern I use most: df.corr()['Heat Fatalities'].drop([...]).sort_values(ascending=False), turning the matrix into one ranked series. Rip current deaths top the list at 0.44 — a number the solution is careful to call moderate rather than meaningful.

Bamboo Weekly #25: Entrepreneurship runs corr across dozens of Global Entrepreneurship Monitor indicators, and is worth reading for the dtype problem it walks into and then solves with select_dtypes.

Bamboo Weekly #58: NATO uses the data above to ask whether founding members spend more of their GDP on defense than recent joiners. The answer is 0.279 — something, but not much, and a good exercise in resisting a tidy conclusion.

Practice it

Work through a corr exercise, with instant feedback and no signup required: practice.lernerpython.com/bamboo-weekly/corr/

Go deeper

Correlation is the second thing you do, not the first. describe shows whether your columns are skewed enough to mislead Pearson, astype fixes the dtypes that make corr raise, and sort_values turns a column of the matrix into a ranking. For time series, correlate pct_change rather than levels — two rising series correlate highly no matter what. To compare one data frame's columns against another's, use DataFrame.corrwith.

Bamboo Weekly is the practice. If you want the structured version — full courses with downloadable Jupyter notebooks, plus live Pandas office hours when you get stuck — that is LernerPython+Data. A paid Bamboo Weekly subscription is included with it.

See it on real data

Below are the 39 Bamboo Weekly exercises that use corr on real-world data — try each one, then study the worked solution.

Part of the Pandas Methods Index. See also practice by skill.