This week’s topic: Happiness
This week, we looked at data from the World Happiness Report for 2023, which was released on March 20th — the International Day of Happiness.
The data set came from an Excel spreadsheet provided by the researchers. Much of their data seems to come from Gallup surveys, in which people are asked to indicate whether they’re living their best possible life (10), worst possible life (1), or somewhere in the middle.
The survey has taken place for several years, and I thought that it would be fun to look at this data — especially after we’ve been looking at depressing data for the last few weeks, including Ukrainian grain exports and failed banks.
Discussion
I started with my standard setup for working with Pandas:
import numpy as np
import pandas as pd
from pandas import Series, DataFrameI then needed to download the data file, which was available in Excel format at:
https://happiness-report.s3.amazonaws.com/2023/DataForTable2.1WHR2023.xlsFirst thing, I used the URL to download the Excel file into a data frame:
url = 'https://happiness-report.s3.amazonaws.com/2023/DataForTable2.1WHR2023.xls'
df = pd.read_excel(url)Remember that “read_excel”, the method that allows us to take input from an Excel spreadsheet and import it into Pandas as a data frame, works just like similar “read_” methods in Python. As such, its first argument can be:
- A filename (string)
- A file-like object, open for reading
- A URL (string)
In the third case, Pandas retrieves the file and returns a new data frame. The file might be cached somewhere on your local system, but I’m not sure where that would happen, if at all. I love downloading data in this way, but if the file is large and you’ll be retrieving it frequently, then you should probably just put it onto your computer a single time and open it via a file path.
I didn’t ask you to do this, because I didn’t think it would make much of a difference on a relatively small data set — but it’s usually a good idea to load only those columns you truly need into a data set. When loading via “read_excel”, you can pass the “usecols” keyword argument, giving it a list of strings, the names of columns you want to load. My code thus looks like:
df = pd.read_excel(url,
usecols=['Country name', 'year',
'Life Ladder', 'Freedom to make life choices'])With this in place, we’re all set to answer my questions for this week!
The main measure in the World Happiness Report is known as "life ladder," where people are asked where they currently are, on a scale from 1 to 10 — where 10 is the happiest possible life. According to this measure in 2022, which 10 countries are happiest?
The data is a bit unusual looking, in that we have one row for each time the survey was performed in a country. This means that we have 14 rows for Afghanistan (for years 2008-2022), 15 rows for Albania (2007-2022), 10 for Algeria (2010 - 2021), and so on.
To answer this question, it won’t be enough to get the “Life ladder” value for each country, because we have several such values for each country, one per year in which the survey was performed.
Rather, we’ll need to find all of the rows from the year 2022, and then compare the countries’ scores.
I start off by comparing the “year” column with 2022, thus creating a boolean series:
df['year'] == 2022I can then apply that boolean series as a mask index to “df.loc”:
df.loc[df['year'] == 2022]This returns all of the rows from df with a “year” value of 2022. But we’re only interested in two columns, “Country name” and “Life Ladder”. I’ll add a 2-element list containing those two column names as the column selector in “.loc”:
df.loc[df['year'] == 2022, ['Country name', 'Life Ladder']]That returns a data frame containing two columns (“Country name” and “Life Ladder”), with all rows from the year 2022 in df.
If I’m interested in finding the happiest countries, then I’ll want to find those with the highest value for “Life Ladder”. I can get that by sorting our data frame by the values of “Life Ladder”, in descending order, using sort_values:
df.loc[df['year'] == 2022, ['Country name', 'Life Ladder']].sort_values('Life Ladder', ascending=False)There isn’t anything wrong with this, but given that I’m only interested in the 10 highest-scoring countries, I’ll use “head” to get the top 10 elements:
df.loc[df['year'] == 2022, ['Country name', 'Life Ladder']].sort_values('Life Ladder', ascending=False).head(10)According to the most recent year’s results, the happiest countries are (in descending order) Finland, Israel, Denmark, Iceland, and Sweden.
Another way to get the same results would be to take our data frame, and set the index to be the “year” column, with the “set_index” method. Then we don’t need to make the comparison with 2022; we can just retrieve all rows with that index using .loc. The rest of our query will look the same, though:
df.set_index('year').loc[2022, ['Country name', 'Life Ladder']].sort_values('Life Ladder', ascending=False).head(10)The WHR actually calculates happiness by averaging the results from the three most recent years (i.e., 2020, 2021, and 2022). Given that measure, what are the 10 happiest countries in the world?
The above calculation is good, but the researchers who put together the annual report don’t rely on the most recent year’s survey alone. Rather, they take the mean from the three most recent surveys to give a country its happiness score. This is done to smooth over variations in the data from year to year.
I thus asked you to find the 10 happiest countries in the world using that methodology.
I started off by asking the data frame for those rows where the year was 2020, 2021, or 2022. The easiest way to do this is to use the “isin” method, which checks for membership in a list, returning True or False for each row:
df.loc[df['year'].isin([2020, 2021, 2022]Using the returned boolean series, I extended my use of “.loc” to retrieve only two columns, “Country name” and “Life Ladder”:
df.loc[df['year'].isin([2020, 2021, 2022]),
['Country name', 'Life Ladder']]Here, the boolean series is acting as my row selector, while the list of names (strings) acts as a column selector. The result is a data frame, a subset of df in which the only years are 2020-2022, and the only columns are “Country name” and “Life Ladder”.
With this in hand, I’m now ready to perform the calculation. Simply put, I want to get the mean for “Life Ladder” for each value of “Country name” — a classic example of using “groupby”.
In theory, I could use the following syntax:
df.loc[df['year'].isin([2020, 2021, 2022]),
['Country name', 'Life Ladder']
].groupby('Country name')['Life Ladder'].mean()And there’s no doubt that this will work.
However, I know that I’ll later want to use the result of this query somewhere else, and that I’ll need a data frame, with multiple columns. Thus, rather than invoke “mean” directly, I’ll use the “agg” method, which allows me to invoke any aggregate method I want on a series.:
df.loc[df['year'].isin([2020, 2021, 2022]),
['Country name', 'Life Ladder']
].groupby('Country name')['Life Ladder'].agg(['mean'])By passing a one-element list to “agg”, I ensure that I get a data frame back. This will make my life easier a bit down the road.
But wait, don’t I want to know which of the countries have the highest mean score from 2020-2022? Yes, and in order to get that, I’ll need to sort the results in descending order:
df.loc[df['year'].isin([2020, 2021, 2022]),
['Country name', 'Life Ladder']
].groupby('Country name')['Life Ladder'].agg(['mean']
).sort_values('mean', ascending=False)This will give the correct results, which is great. However, rather than print them (or the 10 top scorers) right away, I’m going to stick them into a variable:
happiness_2023 = df.loc[df['year'].isin([2020, 2021, 2022]),
['Country name', 'Life Ladder']
].groupby('Country name')['Life Ladder'].agg(['mean']
).sort_values('mean', ascending=False)Why a variable? Because I’ll use it later on, and having it accessible in this way will be useful.
Finally, let’s take a look at the 10 happiest countries in the world, according to this year’s survey:
happiness_2023.head(10)The answer, in case you’re wondering, is close (but not identical) to what we saw from the raw scores: Finland, Denmark, Iceland, Israel, Netherlands, Sweden, Norway, Switzerland, Luxembourg, and New Zealand.
As someone who lives in Israel, I can assure you that this made local headlines. It ws repeatedly pointed out that the survey was taken last year, before our government made proposals that have unleashed the largest and most widespread protests we’ve seen in a long time. I’m definitely curious to hear how our score does in 2024!
I also found some articles asking how Finland is consistently ranked as the happiest country in the world. Here are two good ones:
- https://bigthink.com/neuropsych/happiest-country-in-the-world/
- https://www.washingtonpost.com/travel/2022/03/31/finland-happiest-country/
In any event, we now have a variable, happiness_2023, containing each country and its 2023 happiness score.
Calculate how countries' ranks shifted between last year's survey and this year's survey. Which countries had the greatest positive change, and which had the greatest negative change?
In order to find how countries changed since last year, we’ll need to calculate last year’s happiness ranking. The first step will be to perform the three-year mean for each country — as we did before, but from 2019, 2020, and 2021:
happiness_2022 = df.loc[df['year'].isin([2019, 2020, 2021]),
['Country name', 'Life Ladder']
].groupby('Country name')['Life Ladder'].agg(['mean']
).sort_values('mean', ascending=False)Once again, I’m using .agg and a one-element list of aggregation functions (“mean”) to get a one-column data frame back.
I now have two variables (happiness_2022 and happiness_2023). They’re structured identically, with an index containing country names and values containing the happiness score for that year.
How can I calculate the amount by which countries changed their rankings from one year to the next?
I decided that it would be easiest to add a new “rank” column to each of our data frames. The country with the highest happiness ranking would be assigned 1, the next happiest 2, and so forth.
We can add a new column to a data frame by assigning to it. If we assign a series or list, then the number of elements must match the number of existing rows in the data frame. We’ll thus want an iterable data structure that starts at 1 and goes up to… well, the number of rows that we have.
I decided that the easiest solution would be to use Python’s builtin “range”, which is meant for precisely these purposes. We can pass “range” two arguments — the number at which we want to start, and one past the number we want to end. I can thus say:
happiness_2023['rank'] = range(1, len(happiness_2023)+1)
happiness_2022['rank'] = range(1, len(happiness_2022)+1)Notice how I used Python’s “len” builtin to get the length of the data frame, which returns the number of rows it contains. I then added 1 to make sure that we didn’t cut things off prematurely.
I now have two identically structured data frames, one containing happiness data (including rank) for 2022, and a second containing similar data from 2023. How can I use these to calculate how far a country has moved up or down in the ranks?
The answer? Subtraction! Whenever you subtract (or perform any other arithmetic operation) on two series that share an index, you get a new series back, one with the same index and with the result of running the operation on each row.
To get the change in rank for Afghanistan from 2022 to 2023, we subtract:
happiness_2022.loc['Afghanistan'] - happiness_2023.loc['Afghanistan']In 2022, Afghanistan was ranked 145th in happiness. In 2023, it improved by 8 points, to 137. Sure enough, the above calculation produces a result of 8.
We can do this to all of the rows with:
happiness_2022['rank'] - happiness_2023['rank']In order to see which countries improved the most, we can run:
(happiness_2022['rank'] - happiness_2023['rank']).sort_values(ascending=False).head(10)Note that I used parentheses in order to allow me to capture the series that was returned by the subtraction operation. I then ran “sort_values” on that series, showing it in descending order, and looking at the top 10 results.
I should note that these countries aren’t necessarily happy. However, they have demonstrated an improvement in happiness since last year’s survey. And that’s certainly good news for them! The five most-improved countries are Mauritania, Palestine, Venezuela, Namibia, and Chad.
Of course, some countries also went down in the rankings. Who declined the most? We can find out with a similar query to the above one; the only difference is that I don’t say “ascending=False”:
(happiness_2022['rank'] - happiness_2023['rank']).sort_values().head(10)A few countries really fell in the rankings: Liberia, Gambia, Bahrain, and Comoros. Yikes.
Show the change (positive or negative) in this year's 10 happiest countries, vs. last year's survey.
How about this year’s top 10 happy countries — can we calculate how much they improved (or not) since last year?
In order to answer this, I performed the same subtraction as we did before, to get the change in rank:
happiness_2022['rank'] - happiness_2023['rank']That gave me a series — a series whose index contains country names. I decided to perform a join between our “happiness_2023” data frame, thus combining the data frame’s columns with the rank series.
However, the column names in a data frame must be unique. And because we have a “rank” column in happiness_2023, the series is also called “rank”, we end up with a name conflict.
We could solve this conflict in a few different ways, but I decided to make it easy on myself, and I kept only the “mean” column from the happiness_2023 data frame before joining:
happiness_2023[['mean']].join(happiness_2022['rank'] - happiness_2023['rank']).head(10)Notice that I used double square brackets when selecting the “mean” column. This ensured that I ended up with a one-column data frame, as opposed to a series — important, because data frames have a “join” method, but series don’t.
The result of this join was a data frame with country names as the index, and two columns — the “mean” score from the 2023 survey, and “rank”, which now actually gave us an integer indicating how many slots forward or backward a country moved in the last year.
Bottom line: Israel advanced 4 places, Sweden advanced 1, Norway advanced 2, Switzerland declined 4, and Luxembourg declined 3.
What countries, if any, are in top half of 2022 happiness score and in the bottom half of 2022's "freedom to make life choices"?
Finally, the survey includes a number of additional factors that they say correlate, when used together, with happiness. However, I decided to ask another sort of question, mostly out of interest rather than true research value: Are there any countries where the people feel like they don’t have the freedom to make life choices,” but where they are still happy?
I thus asked you to find countries with below-average freedom scores, but above-average happiness scores. The freedom score should just come from 2022; you don’t need to calculate the mean from the last three years.
I wanted to find the rows from 2022 where the freedom score was less than the mean. This meant performing two different queries, and using & to combine them:
((df['Freedom to make life choices'] <
df['Freedom to make life choices'].mean()) &
(df['year'] == 2022))Notice that I put parentheses around the individual clauses, and also around the entire thing. That allows me to split the lines across multiple lines.
Notice that each of the comparisons here produces a boolean series. The & combines those boolean series, giving us a True value only when the corresponding element in the two series are both True.
Applying this to “df.loc” gave me a data frame:
df.loc[(df['Freedom to make life choices']
< df['Freedom to make life choices'].mean()) &
(df['year'] == 2022),
'Country name']Notice that I set “Country name” to be my column selector.
Finally, I assigned this data frame to a variable:
not_free_choices = df.loc[(df['Freedom to make life choices']
< df['Freedom to make life choices'].mean()) &
(df['year'] == 2022),
'Country name']But now what? I know the countries in which people have below-average senses of free choice, but I want to know how many of these have above-average happiness ranks. (Note: An above-average happiness rank means a lower number!)
I can query “df” to find those rows with ranks below the mean:
happiness_2023.loc[happiness_2023['rank'] < happiness_2023['rank'].mean()]That’s great, but… now what? How can I find the countries that are in this subset of happiness_2023, and also in not_free_choices?
Turns out that Pandas has a nice trick up its sleeve: On Index objects — not on a series or data frame! — you can run the “intersection” method, which finds common elements in two indexes and returns them.
In other words, I can run:
happiness_2023.loc[happiness_2023['rank'] < happiness_2023['rank'].mean()].index.intersection(not_free_choices)And I get back a few countries:
Index(['United States', 'Lithuania', 'Italy', 'Cyprus', 'Croatia',
'South Korea', 'Greece', 'Mongolia'],
dtype='object', name='Country name')Wow! That was definitely a different set of outcomes than I was expecting.
So, how are you feeling now? Happy?
Let me know what you thought about this topic, data set, and explanation, including if I made any mistakes!
Meanwhile, here’s my Jupyter notebook for this week: https://drive.google.com/file/d/1OfH0dqcDi78MVSVpOKm1oY5o6n9cTuLu/view?usp=drive_link
I’ll be back next Wednesday with another set of questions.
Reuven