> ## Content Index
> Fetch the complete content index at: https://www.bambooweekly.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Bamboo Weekly #42: Plant hardiness (solutions)
- URL: https://www.bambooweekly.com/bw-42-plant-hardiness-solution/
- Published: 2023-11-30T16:00:58.000Z
- Updated: 2026-08-23T09:37:01.000Z
- Description: Get better at: CSV files, index operations, string operations, multi-indexes, cleaning, and plotting.
- Author: Reuven M. Lerner
- Tags: csv, index-operations, strings, multi-index, cleaning, plotting

This week, we looked at the latest [plant hardiness zone map](https://www.ars.usda.gov/news-events/news/research-news/2023/usda-unveils-updated-plant-hardiness-zone-map/?ref=bambooweekly.com) distributed by the US Department of Agriculture (USDA), and how it differs from the previous map, which came out in 2012.

The map is meant to help gardeners, and people interested in gardening and plants, know which plants will survive and thrive in different parts of the United States. The main data point used by the map is the mean low temperature recorded (in Fahrenheit) for a given location. Locations with similar mean low temperatures, within a 10-degree range, are categorized as being in the same zone. Each zone is then further divided into two sub-zones, with a 5-degree range for minimum temperatures.

The latest zone map showed that for many locations, the minimum temperature had risen somewhat. This doesn’t come as a massive surprise, given trends in climate change, but I thought that it would be interesting to explore the data and understand just where things had changed.

The data was collected by the PRISM research group at Oregon State University ([https://prism.oregonstate.edu/projects/plant\_hardiness\_zones.php](https://prism.oregonstate.edu/projects/plant%5Fhardiness%5Fzones.php?ref=bambooweekly.com)). A new data set was just released on November 15th by the US Department of Agriculture, at [https://www.ars.usda.gov/news-events/news/research-news/2023/usda-unveils-updated-plant-hardiness-zone-map/](https://www.ars.usda.gov/news-events/news/research-news/2023/usda-unveils-updated-plant-hardiness-zone-map/?ref=bambooweekly.com).

The data is available in a variety of formats, but I decided that it would probably be easiest to work with it via zip codes, because they are spread out through the entire country. In order to figure out just where the zip codes are located, we downloaded an additional data set, a map of zip codes along with location information for each one, joining it together with our other data.

### Data and eight questions

This week's data comes in several parts:

First, we'll use the latest (2023) Plant Hardiness Zone report from [https://prism.oregonstate.edu/projects/plant\_hardiness\_zones.php](https://prism.oregonstate.edu/projects/plant%5Fhardiness%5Fzones.php?ref=bambooweekly.com) . The data comes in several formats and parts; we'll use the CSV file that provides us with data per US zip code:

[https://prism.oregonstate.edu/projects/phm\_data/phzm\_us\_zipcode\_2023.csv](https://prism.oregonstate.edu/projects/phm%5Fdata/phzm%5Fus%5Fzipcode%5F2023.csv?ref=bambooweekly.com)

Next, we'll download data from the previous survey in 2012, described at [https://prism.oregonstate.edu/projects/plant\_hardiness\_zones\_2012.php](https://prism.oregonstate.edu/projects/plant%5Fhardiness%5Fzones%5F2012.php?ref=bambooweekly.com) :

[https://prism.oregonstate.edu/projects/public/phm/2012/phm\_us\_zipcode\_2012.csv](https://prism.oregonstate.edu/projects/public/phm/2012/phm%5Fus%5Fzipcode%5F2012.csv?ref=bambooweekly.com)

Finally, we'll download and work with a CSV file containing US zip codes:

[http://uszipcodelist.com/zip\_code\_database.csv](http://uszipcodelist.com/zip%5Fcode%5Fdatabase.csv?ref=bambooweekly.com)

Here are this week’s eight tasks and questions, along with my solutions. A link to the Jupyter notebook I used to solve these problems follows the final answer.

### Create data frames from the 2012 and 2023 plant hardiness zone data. Each data frame should have a 5-character "zipcode" column, as well as a "zone" column with the zone's name and a "trange" column with the range it includes. Make the "zipcode" column the index.

To start off, let’s load Pandas:

```
import pandas as pd
from pandas import Series, DataFrame
```

I normally use these two lines whenever I work with Pandas. The first loads it up, along with the standard alias used throughout the Pandas world. The second makes it a bit more convenient to create a series or data frame.

Once we’ve done that, I want to create two different data frames, one from the 2012 data and another from the 2023 data. Fortunately, the CSV files are identical except for the URLs, so the moment we have the query correct for one, we’ll have it right for the other, as well.

I could download the CSV file onto my computer and then use “[read\_csv](https://www.bambooweekly.com/pandas-read-csv/)” in order to import it into a data frame. But if the file isn’t too big, and if the server doesn’t refuse my requests, I prefer to just pass the URL as an argument to read\_csv:

```
df_2023 = pd.read_csv('https://prism.oregonstate.edu/projects/phm_data/phzm_us_zipcode_2023.csv')
```

This works, but there are a few things that we want to change from the defaults:

First: We only need three of the columns: zone, trange, and zipcode. We can use the “usecols” keyword argument to specify those.

Next: We want the “zipcode” column to be our index. We can use the “index\_col” keyword argument to specify that.

A more subtle problem is that the “zipcode” column contains only digits, so when Pandas reads that data in, it sets the dtype to be an integer type. However, it then removes leading zeroes from the zip codes. This means that zip code 02134 (of “send it to Zoom!” fame) would be stored as the integer 2314.

The solution is for us to set the dtype for this column by passing the “dtype” keyword argument to read\_csv. The value for this keyword argument is a dict, whose keys are the columns we want to specify and whose values are the dtypes we want to use. In this particular case, we just want to set the “zipcode” column to be a string.

Our final query is thus:

```
df_2012 = pd.read_csv('https://prism.oregonstate.edu/projects/public/phm/2012/phm_us_zipcode_2012.csv',
                      usecols=['zone', 'trange', 'zipcode'],
                      dtype={'zipcode':str},
                      index_col='zipcode')
```

We can repeat the same query for 2023, using the URL for the 2023 data:

```
df_2023 = pd.read_csv('https://prism.oregonstate.edu/projects/phm_data/phzm_us_zipcode_2023.csv',
                      usecols=['zone', 'trange', 'zipcode'],
                     dtype={'zipcode':str},
                     index_col='zipcode')
```

Here’s what the start of the 2023 data frame looks like:

![](https://storage.ghost.io/c/06/ba/06ba0cc0-be6f-4de7-af2f-5c20165279b9/content/images/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https-3a-2f-2fsubstack-post-media.s3.amazonaws.com-2fpublic-2fimages-2ffaae7588-d02d-457c-b1a8-83bb1f1068d1_668x796.png)

### Create a data frame from the zip code database, in which the 5-character "zip" column is the index.

Next, I asked you to create a separate data frame from the zip code database that I found online. Once again, I passed a URL to read\_csv, and used the “dtype” and “index\_col” keyword arguments to ensure that zip codes are treated as five-character strings, and that the “zip” column is set to be our data frame’s index:

```
zip_code_csv = pd.read_csv('http://uszipcodelist.com/zip_code_database.csv',
                          dtype={'zip':str},
                          index_col='zip')
```

I should note that specifying the dtypes of columns in the data frame you’re creating from a CSV file can speed up the loading and creation of the data frame. That’s because Pandas doesn’t need to analyze and guess the type, but will just go along with whatever you’ve said it should do.

Now that we have our three data frames set up, let’s start to analyze the data!

### How many zip codes are in the 2012 report, but not in the 2013 report? According to our zip-code database, how many zip codes from each state went missing in those 11 years?

I was surprised to find that the shapes of our two USDA reports weren’t identical:

```
(df_2012.shape, df_2023.shape)
```

That gave me the following:

```
((40534, 2), (39921, 2))
```

In other words, the 2012 data had 40,534 rows (i.e., zip codes), whereas the 2023 data had 39,921 rows. How can we find not only how many zip codes were dropped, but also *which* ones were dropped?

I decided to use a Python set for this purpose. (And yes, Pandas index objects have some set-like qualities, including the ability to [calculate the intersection between two indexes](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Index.intersection.html?ref=bambooweekly.com). However, you cannot subtract one from the other. I thus decided to create two sets, one from the index of each data frame, and then subtract one from the other:

```
set(df_2012.index)  - set(df_2023.index)
```

The above returns a new set containing all of the zip codes that were in the 2012 report, but not in the 2023 report. (We could also calculate it in the other direction, or even use [symmetric\_difference](https://docs.python.org/3/library/stdtypes.html?ref=bambooweekly.com#set.symmetric%5Fdifference) to do it in both directions.) But for now, we’ll stick with this direction.

How many zip codes were in 2012’s report but not in 2023’s report? We can run “len” on the resulting set:

```
len(set(df_2012.index)  - set(df_2023.index))
```

I got a result of 635\. That struck me as a rather high number, and while my research showed that yes, some zip codes are retired each year, that wasn’t the case for many of them here. I’m not sure what happened, or why they weren’t in the 2023 data.

But I was curious to find out how many zip codes went missing from each state.

In order to do that, I had to use zip\_code\_df, the data frame we created with zip-code information. I can use “[isin](https://www.bambooweekly.com/pandas-isin/)” to tell me whether each element of the index is or isn’t in this set of missing zip codes:

```
zip_code_csv.index.isin(set(df_2012.index)  - set(df_2023.index))
```

I can then use “[loc](https://www.bambooweekly.com/pandas-loc/)” to turn that boolean series into a mask index, retrieving only those rows from zip\_code\_csv that match the missing zip codes. I can also say that I only want to get the “state” column back:

```
(
    zip_code_csv
    .loc[
        zip_code_csv.index.isin(set(df_2012.index)  - set(df_2023.index)), 
        'state']
)
```

The result is a series of states (i.e., strings) with an index of missing zip codes. That’s nice, but I really wanted to know how many times each state had a missing zip code. I can calculate that with “[value\_counts](https://www.bambooweekly.com/pandas-value-counts/)”:

```
(
    zip_code_csv
    .loc[
        zip_code_csv.index.isin(set(df_2012.index)  - set(df_2023.index)), 
        'state']
    .value_counts()
)
```

The result is a series whose index contains state abbreviations, and whose values are integers — the number of times that each state had a zip code missing in 2023:

```
state
PA    68
MN    44
IN    38
CA    37
IL    30
TX    27
NY    25
OH    24
WI    21
CO    20
FL    19
VA    17
GA    14
KY    13
AR    13
IA    13
MO    12
TN    11
NE     9
LA     9
UT     9
CT     9
NM     8
OR     8
MA     8
WA     8
SC     8
NC     8
WV     8
ID     7
ME     7
AL     7
KS     7
MS     7
NJ     7
MD     6
DC     6
AZ     6
MT     5
OK     4
NV     4
MI     4
WY     3
DE     3
VT     3
SD     3
ND     2
RI     2
NH     1
Name: count, dtype: int64
```

Again, I’m at a loss to explain why these zip codes weren’t in the 2023 report. (Maybe it was documented somewhere, and I failed to notice it?) We can see, though, that there were 68 (!) zip codes in Pennsylvania that no longer appear in the list, followed by 44 in Minnesota and 38 in Indiana.

If you have a good guess as to what happened to these zip codes, please do let me know!

### In each of the 2012 and 2023 data frames, create two new columns, "trange\_min" and "trange\_max", containing the min and max temperatures of the "trange" column. Remove the original "trange" column.

The “trange” column contains a string indicating the minimum-temperature range for a given zone. It’s easy for people to read, but because it’s a string, we can’t perform calculations.

I thus asked that you modify both of the data frames, replacing the “trange” column with “trange\_min” and “trange\_max”, both integers.

We can do this by using the “[str](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.str.html?ref=bambooweekly.com)” accessor, which lets us run string methods on every element of a series. Many of these methods are taken from pure Python, with others coming from other languages and libraries.

If I want to get the minimum value from “trange”, I can do the following:

- Invoke “[str.split](https://www.bambooweekly.com/pandas-str-split/)” on the column, splitting on whitespace. This returns a list.
- Invoke “[str.get](https://www.bambooweekly.com/pandas-str-get/)” on the column, retrieving the item at index 0\. This returns a string.
- Invoke “[astype](https://www.bambooweekly.com/pandas-astype/)” on the column, returning a new integer column.

We can do the asme thing for the maximum value, but ask for index -1 (i.e., the final item) rather than index 0.

We could do this with standard Python assignment, but I thought it made sense to use method chaining, including the “[assign](https://www.bambooweekly.com/pandas-assign/)” method:

```
df_2012 = (
    df_2012
    .assign(trange_min=lambda df_: df_['trange'].str.split().str.get(0).astype(int),
            trange_max=lambda df_:
df_['trange'].str.split().str.get(-1).astype(int)
           )
    .drop('trange', axis='columns')
)
```

In the above code, I use “lambda” in the value to “assign” to run an inline function across all of our rows. I do this twice, once for the min value and once for the max value. The resulting data frame has two additional columns, “trange\_min” and “trange\_max”.

Once we’re done creating those, I can get rid of “trange”, which I do by using “[drop](https://www.bambooweekly.com/pandas-drop/)”. Note that I have to specify that I want to drop a column, rather than a row, the default value.

I did this not only for the 2012 data, but also for the 2013 data:

```
df_2023 = (
    df_2023
    .assign(trange_min=lambda df_: df_['trange'].str.split().str.get(0).astype(int),
            trange_max=lambda df_: df_['trange'].str.split().str.get(-1).astype(int)
           )
    .drop('trange', axis='columns')
)
```

Notice that in both cases, I needed to assign the resulting data frame back to the original variable. That’s because “assign” and “drop”, along with many other methods in Pandas, don’t modify the data frame, but rather return a new one with the modified attributes and values.

### Create a new data frame, combining the 2012 and 2023 data frames. The index should contain the combination of all zip codes from both reports. The columns should be a multi-index, with the top level being the year (2012 or 2023), and the column names remaining from the original data frames.

Now that we have the data from 2012 and 2023 in separate data frames, I want to combine them into a single data frame. However, I still want them to retain their original identities. We’ll do that by using a multi-index in the columns, meaning that the columns will have a hierarchy.

There are several ways to pull this off, but what we’ll do here is change the columns in each of the individual data frames to have multi-indexed column names. Then we can just combine the two data frames together.

We could assign multi-indexed column names to each data frame. But Pandas comes with a function, [MultiIndex.from\_product](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.MultiIndex.from%5Fproduct.html?ref=bambooweekly.com), to which we pass a list of iterables. The “from\_product” function multiplies all of the values from each iterable with each other, returning a multi-index suitable for assignment back to columns. In this case, we’re only really interested in multiplying 2012 by each of the column names in df\_2012, and 2013 by each of the column names in df\_2023\. But it still works just fine:

```
df_2012.columns = pd.MultiIndex.from_product([[2012], df_2012.columns])
df_2023.columns = pd.MultiIndex.from_product([[2023], df_2023.columns])
```

With these updated columns in place, we can now combine the two data frames into a new one. The “[concat](https://www.bambooweekly.com/pandas-concat/)” method lets us do so. However, we need to tell Pandas that we want to combine the two data frames side-by-side, rather than top-and-bottom:

```
df = pd.concat([df_2012, df_2023], axis='columns')
```

The resulting data frame looks like this:

![](https://storage.ghost.io/c/06/ba/06ba0cc0-be6f-4de7-af2f-5c20165279b9/content/images/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https-3a-2f-2fsubstack-post-media.s3.amazonaws.com-2fpublic-2fimages-2f67523010-3b80-4397-917a-eb221b8820ef_1764x1486.png)

Notice a few things:

1. Normally, column names have to be unique in Pandas. However, because each of the columns is a multi-index, it’s the *combination* of names that are unique here, which is fine.
2. There are a bunch of rows containing NaN (“not a number”) values. We can see this at the bottom of the 2012 data, meaning that those zip codes were in 2023, but not 2012\. (This is the reverse of what we looked for earlier.)

### On average, how much has the minimum temperature in a zip code changed from 2012 to 2023?

We now get to the crux of the analysis I wanted to do: For a given zip code, how much has the minimum temperature changed since 2012? We know that climate change is making the earth warmer, and that people are now able to plant flowers that were previously unthinkable in their backyards. But on average, how much have temperatures really changed?

What we’ll need to do is grab the min temperature from 2012 and from 2023 for each zip code. We’ll then need to calculate the difference between those. Finally, we’ll calculate the mean of those differences.

We can do that with this query:

```
(df[(2023, 'trange_min')] - df[(2012, 'trange_min')]).mean()
```

Normally, we use square brackets for Pandas column names. But when we want to specify the various parts of a multi-index, we use a tuple, with round parentheses. Each element of the tuple represents one level of the hierarchy. So we get the column under (2023, ‘trange\_min’) and the column under (2012, ‘trange\_min’), then we subtract one from the other and get the mean, resulting in:

```
2.8846587633775282
```

In other words, the minimum temperatures have risen by an average of 2.88 degrees Fahrenheit since 2012\. That might not sound like a lot, but remember that this is just a mean! I found that the standard deviation was 2.72, which means that about 68% of the zip codes in the US have seen a rise in temperature from 0.16 degrees Fahrenheit (i.e., 2.88 - 2.72) to 5.6 (i.e., 2.88 + 2.72) degrees Fahrenheit. In other words, a majority of US zip codes have gotten warmer by at least a bit — and in some cases, a lot! — in the last 12 years.

There is another way for us to perform this calculation. Consider that we want to grab both of the columns whose inner portion is named “trange\_min”. We can use the “[xs](https://www.bambooweekly.com/pandas-xs/)” method to do this, telling Pandas that we want all of the columns whose level-1 column name is “trange\_min”:

```
df.xs('trange_min', level=1, axis='columns')
```

This returns a new data frame, one whose columns are 2012 and 2013\. We can then find out by how much they have changed, calling “[diff](https://www.bambooweekly.com/pandas-diff/)” along the columns:

```
df.xs('trange_min', level=1, axis='columns').diff(axis='columns')
```

Finally, we can grab the 2023 column and calculate its mean:

```
df.xs('trange_min', level=1, axis='columns').diff(axis='columns')[2023].mean()
```

The result? Same as before:

```
2.8846587633775282
```

### Find zip codes in the top 10% of minimum temperature increase. What states are they in? Do you see anything odd with the number of zip codes in the top state?

Next, I asked you to find which zip codes were in the top 10% of minimum temperature increase. This will require a large number of small steps:

First, we’ll use “xs” to retrieve just the “trange\_min” values from 2012 and 2023, and will calculate our diff again:

```
(
    df
    .xs('trange_min', level=1, axis='columns')
    .diff(axis='columns')
)
```

Next, we want to grab the 2023 column (i.e., the column with the diffs. We could just ask for \[2023\], but that would give us a single column, on which you cannot run the “join” method. So we’ll be sneaky, and use double square brackets — thus returning a one-column data frame.

We can then join that with zip\_code\_csv, getting a new data frame with one row per zip code and the combined columns from our 2023-diff data frame and zip\_code\_csv:

```
(
    df
    .xs('trange_min', level=1, axis='columns')
    .diff(axis='columns')
    [[2023]]
    .join(zip_code_csv)
)
```

We don’t really need all of these columns, so we’ll select just two, 2023 and ‘state’. We’ll then drop the NaN values:

```
(
    df
    .xs('trange_min', level=1, axis='columns')
    .diff(axis='columns')
    [[2023]]
    .join(zip_code_csv)
    [[2023, 'state']]
    .dropna()
)
```

Now we’ll run a filter, keeping only those rows whose value for 2023 is greater than the 90th percentile. We can calculate that with the “[quantile](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.quantile.html?highlight=quantile&ref=bambooweekly.com#pandas.Series.quantile)” method, and then use a combination of “loc” and lambda to keep only that highest 10%. Note that quantile(0.9) finds the value that is greater than 90% of the other values — meaning, 10% from the top.

```
(
    df
    .xs('trange_min', level=1, axis='columns')
    .diff(axis='columns')
    [[2023]]
    .join(zip_code_csv)
    [[2023, 'state']]
    .dropna()
    .loc[lambda df_: df_[2023] > df_[2023].quantile(.9)]
)
```

Finally, we’ll keep just the “state” column, and count how often each state appears:

```
(
    df
    .xs('trange_min', level=1, axis='columns')
    .diff(axis='columns')
    [[2023]]
    .join(zip_code_csv)
    [[2023, 'state']]
    .dropna()
    .loc[lambda df_: df_[2023] > df_[2023].quantile(.9)]
    ['state']
    .value_counts()
)
```

The result:

```
state
DC    120
MO     34
CO     25
PA     25
NY     21
MT     19
VA     18
ME     17
ID     16
NM     16
CA     16
WY     15
VT     13
TN     12
WV     11
KY     11
NC     10
OH     10
AR      9
NV      9
UT      8
OR      8
TX      7
MI      7
WA      7
IL      6
AZ      6
MD      6
SC      5
FL      4
MA      4
WI      3
NH      3
IN      3
AL      3
NJ      2
SD      1
LA      1
OK      1
IA      1
GA      1
Name: count, dtype: int64
```

Now, I can understand why some states have a large number of zones with higher minimum temperatures. But Washington, DC is a city, not a state, and it has a *far* greater number of zip codes in the top 10% of temperature-change increases. How can that be?

My guess: Washington has a huge number of zip codes, because it houses so many government agencies. However, these agencies are all within a very small geography. So if Washington is in the top 10% of temperature increases (which it is), then that’ll be reflected in a huge number of zip codes.

### Create a scatter plot where the x axis is the longitude, the y axis is the latitude, the color is based on the minimum temperature in 2012, and we only look at longitude < -60\. How does it look?

Finally, I grabbed the 2012 data (removing the multi-index), and joined it with the zip\_code\_csv:

```
(
    df_2012[2012]
    .join(zip_code_csv)

)
```

Then I used “loc” along with lambda to keep only those rows where the longitude is less than -60:

```
(
    df_2012[2012]
    .join(zip_code_csv)
    .loc[lambda df_: df_['longitude'] < -60]
)
```

Then I kept only the longitude, latitude, and trange\_min columns, and dropped the NaN values:

```
(
    df_2012[2012]
    .join(zip_code_csv)
    .loc[lambda df_: df_['longitude'] < -60]
    [['longitude', 'latitude', 'trange_min']]
    .dropna()
)
```

Finally, I created a scatter plot, using the x axis as longitude, the y axis as latitude, and the trange\_min column for the color. I chose the Spectral\_r colormap:

```
(
    df_2012[2012]
    .join(zip_code_csv)
    .loc[lambda df_: df_['longitude'] < -60]
    [['longitude', 'latitude', 'trange_min']]
    .dropna()
    .plot.scatter(x='longitude', y='latitude', c='trange_min', colormap='Spectral_r')    
)
```

The result? We can see the minimum temperatures for each zip code in the US in 2012:

![](https://storage.ghost.io/c/06/ba/06ba0cc0-be6f-4de7-af2f-5c20165279b9/content/images/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https-3a-2f-2fsubstack-post-media.s3.amazonaws.com-2fpublic-2fimages-2f12bd80ce-9b3a-4738-b35f-118745f0aa1a_565x432.jpg)

Pretty snazzy, right?

Let me know what you think in the comments below.

Meanwhile, you can download my Jupyter notebook from here: [https://drive.google.com/file/d/1QFGq7UocFw4AjJeLh1tKAW362CHRfUyM/view?usp=sharing](https://drive.google.com/file/d/1QFGq7UocFw4AjJeLh1tKAW362CHRfUyM/view?usp=sharing&ref=bambooweekly.com)

I’ll be back next week with more Pandas questions using current events.

Reuven