Two administrative notes:
- We'll have office hours today (Thursday, October 15th), at 17:00 in Israel (aka 10 a.m. Eastern). This is open to paid Bamboo Weekly subscribers and members of my LernerPython+data program. Come and ask any Pandas, Polars, and data-related questions you have! You should have received Zoom info in a separate message.
- The 10th cohort of my Python, Data, and AI bootcamp (PythonDAB) starts on November 5th. If you want to get fluent with these technologies in a small, collaborative cohort, then this might be the program for you. Get more information at https://lernerpython.com/bootcamp/ or join the free info session I'm running on Sunday, registering here: https://us02web.zoom.us/webinar/register/WN_4rLcX1GYShmBvdQDQlM3ZA
The Nobel Prizes for 2026 were awarded earlier this month; you can read about the recipients at https://www.nytimes.com/article/nobel-prizes-2026.html?unlocked_article_code=1.IlE.fisy.lQXkws2dYxyt&smid=url-share. This week, we used Polars to explore some data about Nobel laureates, giving us some more experience playing with a popular data-analysis library.
Data and five questions
This week's data comes from the API for the Nobel Prize, which is as official as things can get. We'll retrieve information in JSON format about the prize winners ("laureates") via the API provided at https://api.nobelprize.org, and documented at https://www.nobelprize.org/about/developer-zone-2/ . Once we have put the data into a Polars data frame, we'll then try to answer some questions.
Two notes about the API:
- By default, it restricted my retrieval of information, because of concern from bots. (Although who else is reading from a JSON API?) So you might need to pass a
User-Agentheader in your remote request. - By default, you get 50 responses per request. You can change that by passing a
limitargument to the URL, along with an integer describing the maximum number of records to return.
Paid subscribers, both to Bamboo Weekly and to my LernerPython+data membership program (https://LernerPython.com) get all of the questions and answers, as well as downloadable data files, downloadable versions of my notebooks, one-click access to my notebooks, and invitations to monthly office hours.
Learning goals for this week include: Polars, working with APIs, working with JSON, cleaning data, grouping, and plotting with Plotly.
Here are my solutions and explanations for this week's five questions:
Retrieve information from the Nobel laureates API, and turn that data into a Polars data frame. You'll want columns with the English-language values for id, fullName, orgName, gender, birth_date, now_country, death_date, prize, and award_year. If the person received more than one prize, then have two rows, one for each prize.
First, I loaded up Polars and Plotly. I also loaded requests, since (spoiler alert!) we'll also need that in order to retrieve data from the API:
import polars as pl
from plotly import express as px
import requestsMy first instinct was to take the URL for the Nobel Prize API, which returns JSON, then to invoke pl.read_json , getting a data frame back. But of course, that just reflected my Pandas bias. Pandas is very flexible on this front – too flexible, Polars advocates would say! – in that it lets you pass a filename, a file-like object, or a URL to its file-opening methods, and it'll (try to) do the right thing.
Polars, by contrast, only accepts a string (a filename) or a file-like object. That's it. What if you have a URL? Then you're responsible for retrieving the data from that URL, and then handing its content to an appropriate Polars method.
That's why I imported requests above: I'll retrieve the JSON response from the Nobel Prize API, and then hand that to Polars.
Here's the code I can use:
- I invoke
requests.get, handing it the URL - I pass along the
limitargument, giving it a value of 100,000 - The response is assigned to
resp, on which I invokeraise_for_status– which basically means that if we got an error code from the server, we want to raise an exception. But if things went well, then there's no need. - We then turn the JSON data into a bunch of Python objects with the
jsonmethod - Finally, we create a new Polars data frame by invoking
pl.DataFrameon the contents of thedata['laureates']data:
resp = requests.get(
"https://api.nobelprize.org/2.1/laureates",
params={"limit": 100_000}
)
resp.raise_for_status()
data = resp.json()
df = pl.DataFrame(data["laureates"])This is great, except for one little problem, namely that it doesn't work: CloudFlare, which provides a front-end proxying, caching, and security for a huge number of Web sites, saw my request as an illegal bot, and gave me an error.
The solution was to identify my program by setting some information in the User-Agent header. I then passed headers to requests.get:
headers = {"User-Agent": "Mozilla/5.0 (compatible; bamboo-weekly-192/1.0; reuven@lernerpython.com)"}
resp = requests.get(
"https://api.nobelprize.org/2.1/laureates",
params={"limit": 100_000},
headers=headers,
timeout=30,
)
resp.raise_for_status()
data = resp.json()
df = (
pl
.DataFrame(data["laureates"])
['id', 'fullName', 'orgName', 'gender', 'birth', 'nobelPrizes', 'death']
)
As you can see, after creating the data frame, I then used [] to retrieve a subset of the columns that came with the data set.
The result was a Polars data frame with 1,028 rows and 7 columns – but a closer examination of the data frame indicated that it wasn't exactly ready to be used. Several of the columns have a dtype of struct, what we in the Python world might call a dictionary. For example, I can retrieve the first row (index 0) and the first column (named firstName) with:
df[0, 'fullName']The result is the winner's name in English, Norwegian (sometimes) and Swedish:
{
"en": "A. Michael Spence",
"no": None,
"se": "A. Michael Spence"
}How can I replace the fullName column with the contents of the en key in the struct? I'll start with the with_columns method, which is similar to assign in Pandas, letting us create a new column. (If the new column has the same name as an existing column, the old one is overwritten.) I then indicate that I want to set fullName to the result of retrieving the 'en' key from the struct in fullname by saying struct.field('en').
I similarly set birthdate by retrieving from pl.col('birth').struct.field('date'). But I want to get this value as a datetime dtype. I decided to use str.to_date to perform this conversion from a string to a date value – but to avoid problems if there are any illegal dates, I passed strict=False, meaning that if there are issues with any dates, I'll get NaN instead. I did the same thing for death, since we'll need it in a later question.
What about now_country? This comes from birth, but not directly. We need to retrieve the place key from the struct (i.e., .struct.field('place')), and then the countryNow key from within that (i.e., .struct.field('countryNow')), and then from the English-language place name within that (i.e., .struct.field('en')). That's a lot of deep digging inside of the struct, but it works.
When I'm done with these actions, I don't need the birth column any more, and invoke drop on it:
df = (
raw_df
.with_columns(
fullName = pl.col('fullName').struct.field('en'),
orgName = pl.col('orgName').struct.field('en'),
birthdate = pl.col('birth').struct.field('date').str.to_date(strict=False),
now_country = pl.col('birth').struct.field('place').struct.field('countryNow').struct.field('en'),
death = pl.col('death').struct.field('date').str.to_date(strict=False)
)
.drop('birth')
)However, I also wanted information about each laureate's prize. The thing is, some people have won more than one Nobel Prize, which means that they have more than one entry within the nobelPrizes field. I thus invoked explode, getting one row for each element in that list. The data frame was unchanged except that the nobelPrizes column became a dtype of struct, rather than a list of structs. Any person with more than one prize appeared in multiple rows:
df = (
raw_df
.with_columns(
fullName = pl.col('fullName').struct.field('en'),
orgName = pl.col('orgName').struct.field('en'),
birthdate = pl.col('birth').struct.field('date').str.to_date(strict=False),
now_country = pl.col('birth').struct.field('place').struct.field('countryNow').struct.field('en'),
death = pl.col('death').struct.field('date').str.to_date(strict=False)
)
.drop('birth')
.explode('nobelPrizes')
With this in place, I thus picked apart the nobelPrizes column, creating awardYear and prize columns based on elements within the struct. In the case of awardYear, I not only retrieved the value for the awardYear key, but also used cast to turn it into an integer.
In the case of the prize category, I needed to retrieve from two levels down – first the category key, and then the en key for the English name.
The full query was thus:
df = (
raw_df
.with_columns(
fullName = pl.col('fullName').struct.field('en'),
orgName = pl.col('orgName').struct.field('en'),
birthdate = pl.col('birth').struct.field('date').str.to_date(strict=False),
now_country = pl.col('birth').struct.field('place').struct.field('countryNow').struct.field('en'),
death = pl.col('death').struct.field('date').str.to_date(strict=False)
)
.drop('birth')
.explode('nobelPrizes')
.with_columns(
awardYear = pl.col('nobelPrizes').struct.field('awardYear').cast(int),
prize = pl.col('nobelPrizes').struct.field('category').struct.field('en')
)
.drop('nobelPrizes')
)
dfThe result was a data frame with 1,036 rows and 9 columns. I got the column names and dtypes:
for key, value in dict(zip(df.columns, df.dtypes)).items():
print(f'{key:.<14}{value}')The result:
id............String
fullName......String
orgName.......String
gender........String
death.........Date
birthdate.....Date
now_country...String
awardYear.....Int64
prize.........String
How many people and organizations have won more than one prize? Show them, in what years they won, and which prizes they got.
Polars has a method, is_duplicated, which returns True for any row whose value is duplicated in another row. We can thus invoke filter, which keeps any rows that return True for a boolean expression, passing it pl.col('id').is_duplicated(), given that the id column represents the person or organization's unique identifier:
(
df
.filter(pl.col('id').is_duplicated())
)The good news is that this worked. The bad news (well, not that bad) is that because we have both people and organizations, we have two separate columns for the names. I decided to combine them into a single name column that would take either fullName or orgName, whichever isn't null.
Because I wanted to create a new column, I again invoked with_columns. I then used pl.coalesce, which takes any number of columns, returning the first non-null value.
Then, to avoid confusion and a bit of messiness, I again used drop to remove the original fullName and orgName columns, along with others that we don't need:
(
df
.filter(pl.col('id').is_duplicated())
.with_columns(
name = pl.coalesce(pl.col('fullName'), pl.col('orgName'))
)
.drop('fullName', 'orgName', 'id', 'gender', 'birthdate', 'now_country')
)Finally, I decided to sort them by award year. In Pandas, we have sort_index and sort_values – but in Polars, data frames don't have an index, which explains the method sort that we use. Just name the column by which the data frame should be sorted, and you'll get a new one back sorted in ascending order:
(
df
.filter(pl.col('id').is_duplicated())
.with_columns(
name = pl.coalesce(pl.col('fullName'), pl.col('orgName'))
)
.drop('fullName', 'orgName', 'id', 'gender', 'birthdate', 'now_country')
.sort('awardYear')
)The results:
awardYear prize name
1903 Physics Marie Curie, née Skłodowska
1911 Chemistry Marie Curie, née Skłodowska
1917 Peace International Committee of the Red Cross
1944 Peace International Committee of the Red Cross
1954 Chemistry Linus Carl Pauling
1954 Peace Office of the United Nations High Commissioner for Refugees
1956 Physics John Bardeen
1958 Chemistry Frederick Sanger
1962 Peace Linus Carl Pauling
1963 Peace International Committee of the Red Cross
1972 Physics John Bardeen
1980 Chemistry Frederick Sanger
1981 Peace Office of the United Nations High Commissioner for Refugees
2001 Chemistry K. Barry Sharpless
2022 Chemistry K. Barry Sharpless
Honestly, I was sure that it had been many years since anyone had won two Nobels. I mean Marie Curie and Linus Pauling, sure, but they won quite some time ago. But it turns out that K. Barry Sharpless not only won recently, but won twice the last 25 years.