We keep hearing and talking about climate change as something that will happen, or that will affect us in the future — but things are already changing, all over the world. One way to reduce and slow climate change is by reducing the degree to which we generate energy using fossil fuels.
Just this week, representatives from the Group of Seven (“G7”) nations met in Turin, Italy, announced a plan to reduce the use of coal, a particularly popular (and thus problematic) fossil fuel. Their plan would largely eliminate the use of coal by the year 2035. (An Associated Press report is here: https://www.msn.com/en-us/money/companies/g7-nations-commit-to-phasing-out-coal-by-2035-but-give-japan-some-flexibility/ar-AA1nVWIO )
Upon reading this news, I started to wonder just how much coal is used, and what countries are using it. I soon discovered Global Energy Monitor (https://globalenergymonitor.org/), whose Global Coal Plant Tracker contains lots of interesting and useful data about precisely this subject. That's the data we looked at this week, exploring data about the history and future of coal-powered plants, we'll be looking at this week, trying to understand which countries are still running coal plants, what kind of coal and process they're using, and how much emissions they're creating.
Data and seven questions
The Global Energy Monitor's Global Coal Plant Tracker has its home page here:
https://globalenergymonitor.org/projects/global-coal-plant-tracker/
To download the data, you'll need to go to the "download data" link:
https://globalenergymonitor.org/projects/global-coal-plant-tracker/download-data/
If you prefer (and you probably will), I downloaded the data already via their e-mail link; it’s available here:
/content/files/wp-content/uploads/2024/02/global-coal-plant-tracker-january-2024.xlsx
It takes 6-8 hours to research and write each edition of Bamboo Weekly. Thanks to those of you who support my work with a paid subscription! Even if you can’t, it would mean a lot if you would share BW with your Python- and Pandas-using colleagues. Thanks!
Download the Excel spreadsheet, and load the "Units" sheet from that document into a data frame. We'll only want the following columns: "Country", "Capacity (MW)", "Status", "Start year", "Combustion technology", "Coal type", "Region", and "Annual CO2 (million tonnes / annum)".
Before doing anything else, I loaded up Pandas:
import pandas as pdI then wanted to load data from the Excel file. The file, as is often the case with Excel, several different “sheets,” which you can think of as sub-documents in one spreadsheet or tabs in a browser window. We can specify which sheet we want to load by passing the “sheet_name” keyword argument, either passing the name of the sheet as a string or as an integer. Here, the “Units” sheet has a simple enough name, so I passed it as a string to “read_excel”:
filename = 'Global-Coal-Plant-Tracker-January-2024.xlsx'
df = pd.read_excel(filename, sheet_name='Units')The above worked just fine. Moreover, “read_excel” is smart enough to use the first row of the spreadsheet as the column names. (If it doesn’t, then you can always pass the “header” keyword argument, to indicate which row should be used for the column names.)
We aren’t going to be using all of the columns in the spreadsheet in our analysis, however, For that reason, I thought it might be a good idea to limit which columns we use. We can do that by passing the “usecols” keyword argument, with a list of strings indicating which columns we want to keep:
filename = 'Global-Coal-Plant-Tracker-January-2024.xlsx'
df = pd.read_excel(filename, sheet_name='Units',
usecols=["Country", "Capacity (MW)", "Status",
"Start year", "Combustion technology",
"Coal type", "Region",
"Annual CO2 (million tonnes / annum)"])With this in place, we’re now set to do our analysis.
How much memory does the data frame take up? How much memory do you save by turning columns into categories? Which columns are most (and least) likely to save us memory in this way? Are there any columns that we *could* turn into categories, but shouldn't?
How big is our data frame? The easiest way to find out is to invoke the “memory_usage” method. It’ll return one result for each column in the data frame. I invoke:
df.memory_usage()And here’s what I get back:
Index 132
Country 111248
Capacity (MW) 111248
Status 111248
Start year 111248
Combustion technology 111248
Coal type 111248
Region 111248
Annual CO2 (million tonnes / annum) 111248
dtype: int64Does this seem a bit suspicious to you? It should; I find it a bit hard to believe that every single column (except for the index) uses the same amount of memory. I mean, the “Start year” column contains integers, whereas the “Country” column contains strings. Could it be that they use precisely the same amount of memory?
Moreover, the data frame contains 13,906 rows. If we divide 111,248 by 13,906, we get 8, meaning that each of these is an 8-byte (i.e., 64-bit) value. Something seems weird here.
And indeed, something is weird here, namely the way that we’re calculating memory usage. When Pandas uses NumPy as a backend (which is standard), it doesn’t want to use NumPy’s strings. So it instead uses Python strings, storing pointers to those Python objects in NumPy. That’s what it means for the dtype to be “object” in Pandas; we’re referring to Python objects.
When we call “memory_usage”, Pandas doesn’t by default calculate the usage of each individual Python object, because that’ll take far longer. So it returns the size of the NumPy-allocated memory, instead. If we want “memory_usage” to truly gather the size of each Python string, we need to tell it to go deep with the “deep=True” keyword argument:
df.memory_usage(deep=True)The result, this time, is a bit different:
Index 132
Country 778963
Capacity (MW) 111248
Status 801876
Start year 111248
Combustion technology 839171
Coal type 800189
Region 747644
Annual CO2 (million tonnes / annum) 111248
dtype: int64Sure enough, we now have very different amounts of memory. Because “memory_usage” returns its results as a series, we can get the total memory usage by running “sum” on the output:
before_memory_usage = df.memory_usage(deep=True).sum()
print(f'Before:\t{before_memory_usage:>12,}')I asked you to calculate the total memory usage, and this is the best / easiest way to do so. The result, on my computer:
Before: 4,301,719I asked you to consider using categories to shrink down this memory usage. You can think of a category in Pandas as similar to an “enum” in a programming language, allowing us to store integers (which are small) instead of strings (which are large). A strings column with many repeated values will benefit greatly from being turned into a category, because we’ll store the string once and the integers referring to it many times.
Can we turn non-string columns into categories in Pandas? The answer is “yes,” but you have to be careful. That’s because if you turn an integer column into a category, you won’t be able to perform mathematical operations on them any more. (By contrast, you can definitely perform string operations on category columns.) So while it’s theoretically a good idea to crunch down numeric columns using categories, you should be a bit careful about doing so.
Let’s thus turn all non-numeric columns in our data frame into categories. We’ll first select all of the columns without “float64” dtypes using “select_dtypes”. This method lets us select all columns with a particular dtype (using “include”) or without a particular dtype (using “exclude”). Here, I’m going to ask for all columns that are not npfloat64. I’ll then run “astype(‘category’)” on that column, assigning the result back to the original column:
for one_column in df.select_dtypes(exclude='float64').columns:
df[one_column] = df[one_column].astype('category')After this transformation, I can then gain run “memory_usage” and sum up the size of the data frame:
after_memory_usage = df.memory_usage(deep=True).sum()
print(f'After:\t{after_memory_usage:>12,}')The result:
After: 416,504That looks like quite a savings of memory! How much did we save, in total?
print(f'Saved:\t{(before_memory_usage - after_memory_usage):>12,}')Which gives us output of:
Saved: 3,885,215In other words, we saved more than 3.8 MB of memory by switching to categories for those columns. It took almost no time, and doesn’t affect the functionality or performance. That’s why I often point to categories as one of the easiest and fastest optimizations you can do in Pandas.
By the way, I decided to try specifying the use of PyArrow for the dtype backend, rather than NumPy, to see what sort of memory savings we might enjoy:
dfa = pd.read_excel(filename, sheet_name='Units',
usecols=["Country", "Capacity (MW)", "Status",
"Start year", "Combustion technology",
"Coal type", "Region",
"Annual CO2 (million tonnes / annum)"],
dtype_backend='pyarrow')I then calculated the memory usage. Note that because PyArrow stores all of the values right in Apache Arrow, and not in Python, there’s no need to say “deep=True”. Then let’s see how much we save:
arrow_memory_usage = dfa.memory_usage().sum()
print(f'Arrow:\t{(before_memory_usage - arrow_memory_usage):>12,}')The result:
Arrow: 3,130,846So Arrow still saved us a lot of memory, and we didn’t have to use categories. However, it didn’t save quite as much memory as categories. I was, however, able to run “astype(‘category’)” on my Arrow columns, and I got the same memory usage as before.
How many millions of tonnes of CO2 do coal plants produce? How many are produced per country? Where do the G7 countries stand among emitters?
Now that we have the data in a useful, efficient form, we can start to ask some questions of it. For starters, I wanted to know how much CO2 each country emits from its coal plants.
We can find the total number of tonnes (i.e., metric tons, aka 1,000 kg, or slightly more than a US “ton”) by summing the values in that column:
df['Annual CO2 (million tonnes / annum)'].sum()The answer I got is 22,769 metric tons of carbon dioxide, which certainly seems like a lot to me. Of this annual output, how much does each country contribute? We have a “Country” column in our data frame, and we’ll want to sum our CO2 output column’s values for each separate country. We can use “groupby” to sum up each country’s contribution to CO2 output:
(
df
.groupby('Country', observed=True)
['Annual CO2 (million tonnes / annum)']
.sum()
)The results we get are interesting, but they’re sorted alphabetically by country. I would rather find out who the greatest polluters are, which means using “sort_values” to sort by value in descending order:
(
df
.groupby('Country', observed=True)
['Annual CO2 (million tonnes / annum)']
.sum()
.sort_values(ascending=False)
)Finally, I’ll use “head” to find the 15 most-polluting countries (with coal):
(
df
.groupby('Country', observed=True)
['Annual CO2 (million tonnes / annum)']
.sum()
.sort_values(ascending=False)
.head(15)
)The result:
Country
China 10091.0
India 3941.4
United States 1999.9
Indonesia 540.5
Türkiye 515.5
Germany 415.2
Vietnam 405.2
Russia 347.8
Japan 328.8
South Africa 324.0
Poland 291.5
South Korea 228.4
Australia 223.1
United Kingdom 205.0
Bangladesh 182.6
Name: Annual CO2 (million tonnes / annum), dtype: float64This list isn’t a huge surprise. And yet, I’d like to know where all of the G7 countries are in this ranking. Here’s how I calculated that:
(
df
.groupby('Country', observed=True)
['Annual CO2 (million tonnes / annum)']
.sum()
.sort_values(ascending=False)
.rank(ascending=False)
.loc[['United States', 'United Kingdom', 'France',
'Italy', 'Germany', 'Japan', 'Canada']]
.sort_values()
)The first few lines are the same as I did before, through the call to “sort_values”. But then, I invoked “rank”, which returns a series of integers indicating where each index value was relative to the others in the series.
Finally, I used “.loc” and square brackets to retrieve only those values from G7 countries, and then sorted those values. The result:
Country
United States 3.0
Germany 6.0
Japan 9.0
United Kingdom 14.0
Canada 20.0
Italy 23.0
France 32.5
Name: Annual CO2 (million tonnes / annum), dtype: float64As we saw, the US is the third-greatest emitter of CO2 from coal plants, Germany, Japan, and the UK follow not too far away. The G7 country with the lowest degree of CO2 emissions from coal is France, which makes sense given their very high reliance on nuclear power.
Which combination of coal type and combustion technology, on average, produce the least CO2? What do we see in common among the least-polluting combinations?
When I started to look through the data set, I noticed that there were columns for “coal type” and “combustion technology.” I read up on these a bit, to better understand why a coal plant would want to use a particular type of coal and/or a particular combustion technology — and there are obviously many factors, including price and availability. But certain types of coal seem to burn more cleanly, and certain types of combustion seem to give off less CO2.
I asked you to find out which combination of coal and combustion reduces the least CO2, and if we see any commonality among the least-polluting combinations.
This is most easily solved via another grouping operation. But this time, we’ll group on two columns, both “coal type” and “combustion technology.” We can do that by putting the column names as a list when invoking “groupby”:
(
df
.groupby(['Coal type', 'Combustion technology'])
['Annual CO2 (million tonnes / annum)']
.mean()
)In the above query, we ask Pandas to show, for every combination of coal type and combustion technology, the mean CO2 output. We get a result, but we also get a warning:
ipykernel_9614/3876730893.py:6: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
.groupby(['Coal type', 'Combustion technology'])What does this mean? Normally, when we run “groupby” on a column, Pandas perform the calculation for each distinct value in the column. But when we run “groupby” on a column that is categorical, Pandas performs the calculation for every value in the categorical, regardless of whether there are any rows matching that value. In our case, this means that Pandas would produce output for every combination of coal and combustion, even if that combination doesn’t exist in reality.
We can say that this OK, getting results with NaN values, by passing the “observed=False” keyword argument to “groupby”. But normally, we’ll want to ignore any combination that doesn’t actually exist, which we can accomplish by passing “observed=True”. The current default is observed=False, but that’ll be changing soon— so we should really be explicit about passing that keyword argument.
(
df
.groupby(['Coal type', 'Combustion technology'], observed=True)
['Annual CO2 (million tonnes / annum)']
.mean()
)Because we grouped on two columns, the result is a series with a two-level multi-index. However, “groupby” normally sorts the index of its output. We want to sort by the values, so we invoke “sort_values”:
(
df
.groupby(['Coal type', 'Combustion technology'], observed=True)
['Annual CO2 (million tonnes / annum)']
.mean()
.sort_values(ascending=False)
)We see that there is a large difference in pollution output from the most-polluting to the least-polluting combination:
Coal type Combustion technology
lignite ultra-supercritical 3.290909
unknown ultra-supercritical 3.277816
bituminous ultra-supercritical 3.269977
subbituminous ultra-supercritical 3.172000
anthracite ultra-supercritical 2.905263
lignite supercritical 2.739370
bituminous supercritical 2.706009
subbituminous supercritical 2.667213
unknown supercritical 2.476105
anthracite supercritical 2.429688
waste coal ultra-supercritical 2.415000
bituminous unknown 2.162774
IGCC 2.083333
unknown unknown 1.765078
IGCC 1.684615
waste coal supercritical 1.677660
lignite unknown 1.523256
subbituminous IGCC 1.500000
waste coal CFB 1.468421
subbituminous subcritical 1.468072
unknown 1.464000
anthracite unknown 1.295652
bituminous subcritical 1.246323
anthracite subcritical 1.235586
lignite IGCC 1.200000
bituminous CFB 1.177778
anthracite CFB 1.086667
lignite subcritical 1.061953
CFB 1.011429
waste coal subcritical 0.903788
bituminous supercritical/ccs 0.900000
subbituminous CFB 0.880000
waste coal unknown 0.870968
unknown subcritical 0.827184
lignite unknown/ccs 0.700000
unknown ultra-supercritical/ccs 0.600000
bituminous ultra-supercritical/ccs 0.600000
subcritical/ccs 0.600000
unknown CFB 0.567742
subbituminous subcritical/ccs 0.500000
bituminous IGCC/ccs 0.366667
subbituminous supercritical/ccs 0.300000
unknown IGCC/ccs 0.200000
subbituminous IGCC/ccs 0.200000
bituminous unknown/ccs 0.200000
lignite subcritical/ccs 0.100000
IGCC/ccs 0.100000
unknown subcritical/ccs 0.000000
Name: Annual CO2 (million tonnes / annum), dtype: float64The least-polluting coal plants, all seem to use Integrated Gasification Combined Cycle (IGCC), Carbon Capture and Storage (CCS), or both. These are, as I’ve learned, two technologies that capture quite a bit of the output from the process, reusing and/or burying it in order to avoid putting it into the air.
How many operating plants use either IGCC or CCS, two technologies that make coal-fired plants more efficient?
Given that these two technologies pollute so much less, I thought it reasonable to ask how many currently operating plants actually use them.
I first wanted to find all plants that use either IGCC or CCS. To do that, I’ll need to find all of the rows in which “Combustion technology” contains either “IGCC” or “CCS” in the string. By using a regular expression (regex=True) and also the case-insensitive flag for regular expressions (re.IGNORECASE), I was able to do this, getting all rows with one or both of these strings in them.
import re
(
df
.loc[lambda df_: df_['Combustion technology'].str.contains('igcc|ccs', regex=True, flags=re.IGNORECASE)]
['Status']
.value_counts()
)I then grabbed the “Status” column, indicating the current status of each plant. Finally, I invoked “value_counts” to learn how often each of those statuses applies. The results:
Status
cancelled 40
operating 13
retired 5
construction 4
mothballed 1
pre-permit 1
announced 0
permitted 0
shelved 0
Name: count, dtype: int64First, notice that two of these statuses have a count of 0. Why would “value_counts” include these two labels if no rows use them? The answer is that we have categorical data, and that all of the categories will be counted, even if that count is 0.
But to me, the biggest surprise was that the most common category was “cancelled”! Given that they pollute less, I figured that they would be more popular — but I’m also guessing that they are more expensive. I decided to at least find out in which countries such projects had been cancelled:
(
df
.loc[lambda df_: df_['Combustion technology'].str.contains('igcc|ccs', regex=True, flags=re.IGNORECASE)]
.loc[lambda df_: df_['Status'] == 'cancelled']
['Country']
.value_counts()
.head(11)
)Here’s what I got:
Country
China 9
United States 9
Australia 6
United Kingdom 6
India 2
Germany 2
Canada 2
Poland 1
Vietnam 1
Italy 1
Netherlands 1
Name: count, dtype: int64The data set explicitly indicates that data from China isn’t that reliable or up to date. Nevertheless, the data indicates that China, the US, and Australia have all canceled nine such projects. Where, then, are they currently operating?
Country
United States 4
South Korea 3
Japan 3
China 2
Canada 1I should add that even if IGCC and CCS reduce the CO2 output significantly, they still cause many more environment problems than renewables, which have come down in price quite a bit in recent years. I’d speculate that this is also a contributing factor to the cancellation of such less-polluting coal plants
Create a scatter plot comparing capacity with annual CO2 output, colorizing each dot based on the year in which the plant was opened. Do newer plants appear to be getting more efficient than old ones?
We know, for each plant in our data set, how much electricity it produces, and also how much CO2 it puts out. Is there a clear correlation between the two? I asked you to create a scatterplot comparing them.
We can do that in Pandas with “plot.scatter”:
df.plot.scatter(x='Capacity (MW)',
y='Annual CO2 (million tonnes / annum)')The above query produces a plot, and we can see that there is a pretty clear correlation:
However, I asked you to colorize each dot based on the year in which the plant was opened. We can do that by passing a column to the “c” keyword argument, and also by specifying a colormap. (If you don’t specify a colormap, you get white, gray, and black, and it’s hard to really understand the data.)
Here’s what I used:
df.plot.scatter(x='Capacity (MW)',
y='Annual CO2 (million tonnes / annum)',
c='Start year', colormap='Spectral')The result looks like this:
We can see that older plants appear in red, orange, and yellow. Newer plants appear in green and blue. Thanks to the colors, we can see that newer plants are much larger in capacity than older ones.
But most interesting to me was the fact that older plants, for the same capacity, seemed to pollute a bit more than the newer ones. Notice that the red-orange-yellow diagonal line is above the blue-green diagonal line. There is still a correlation between capacity and CO2 output, but plants seem to be doing better overall. And of course, there are a bunch of plants along the bottom, which I have to assume are using IGCC and CCS technologies to reduce pollution.
Create a line plot showing, for each region of the world, how many coal-fired plants started in each year, with a separate line for each region and the x axis representing years. What region stands out?
Finally, we know not just the country for each coal plant, but also its region in the world. I asked you to produce a line plot showing, year by year, how many new plants came online in each region.
In order to create such a plot, we’ll need a data frame in which the years form the index (row labels), the region names form the column names, and the cells indicate how many plants were opened in that year. We can create such a data frame as a pivot table, using the “pivot_table” method:
- The rows will contain the unique values from the “Start year” column
- The columns will contain the unique values from the “Region” column
- We’ll use the “count” aggregation function
- Our values will come from the “Status” column.
Here’s the query:
(
df
.pivot_table(index='Start year',
columns='Region',
aggfunc='count',
values='Status',
observed=True)
.plot.line()
)Notice that we pass “observed=True” to avoid the problems that we had before with “groupby” and categorical data. The result of our call to “pivot_table” is a data frame. We can then invoke “plot.line” on the data frame, getting the following:
We can see that from 1980, there was absolutely massive growth in the number of coal plants in Asia. That number looks like it’ll drop significantly in the coming years, although these are projections, so it’s hard to know if that’s part of the fight against climate change or a lack of data. We can also see, albeit less obviously, a slow but steady decline in the number of new coal-based plants built every year.
That’s it for this week! Here’s the link to my Jupyter notebook: https://drive.google.com/file/d/1F1BauXbP8t_SVkFjXMEQstj-o105djti/view?usp=sharing
I’ll be back next week with more Pandas puzzles based on current events.
Reuven