> ## Content Index
> Fetch the complete content index at: https://www.bambooweekly.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Bamboo Weekly #59: Long covid (solutions)
- URL: https://www.bambooweekly.com/bw-59-long-covid-solution/
- Published: 2024-03-28T16:01:40.000Z
- Updated: 2026-09-02T14:02:30.000Z
- Description: Get better at: CSV, dates and times, multi-indexes, plotting, filtering, and grouping
- Author: Reuven M. Lerner
- Tags: csv, datetime, multi-index, plotting, filtering, grouping

This week marks four years since covid-19 was officially declared a pandemic by the World Health Organization (WHO). Of course, it had been spreading through China and other countries for the previous two months, and people had already been getting nervous about what the effects were going to be. The official declaration of a pandemic was a milestone in the madness that followed, and in the many societal changes that we continue to see to this day.

Covid-19 isn't gone, but it is far more treatable and under control than was the case four years ago. At the same time, we’re now more aware of “long covid,” a set of symptoms that affect people who were sick with covid-19 and have long-term medical issues as a result.

The [Centers for Disease Control and Prevention](https://www.cdc.gov/?ref=bambooweekly.com) (CDC) has been surveying people regularly about long covid, in an attempt to understand and treat it better. The CDC calls this a "pulse survey," because it's taking the pulse of an issue between official, formal Census Bureau surveys. It's described in full here:

[https://www.cdc.gov/nchs/covid19/pulse/long-covid.htm](https://www.cdc.gov/nchs/covid19/pulse/long-covid.htm?ref=bambooweekly.com)

This week, we looked at data about long covid, including some of the CDC's survey results.

### Data and five questions

This week, I gave you five tasks and questions. As usual, a link to the Jupyter notebook I used to solve these problems is at the bottom of this newsletter.

The questions are all be based on the data that you can download from the CDC, from this link:

[https://data.cdc.gov/NCHS/Post-COVID-Conditions/gsea-w83j/about\_data](https://data.cdc.gov/NCHS/Post-COVID-Conditions/gsea-w83j/about%5Fdata?ref=bambooweekly.com)

Click on the "export" button at the top right of the page to get a CSV file with the latest pulse survey results. The above URL also serves as a data dictionary, albeit one with limited descriptions.

### Import the CSV file into a data frame. Ensure that "Time Period Start Date" and "Time Period End Date" are both treated as datetime values. Also, the index should consist of the columns "Phase", "Group", and "Subgroup".

As usual, the first thing that I did was load Pandas:

```
import pandas as pd
```

With that in place, I wanted to load the downloaded CSV file into a data frame, using “[read\_csv](https://www.bambooweekly.com/pandas-read-csv/)”. And I could:

```
filename = 'Post-COVID_Conditions_20240320.csv'

df = pd.read_csv(filename)
```

However, I asked you to only include some of the columns from the CSV file. We can specify which ones we want with the “usecols” keyword argument:

```
df = pd.read_csv(filename,
                usecols=['Indicator', 'Group', 
                         'State', 'Subgroup', 'Phase',
                         'Time Period Start Date',
                         'Time Period End Date',
                        'Value'])
```

I also asked you to ensure that the two “Time Period” columns are of “datetime” types. We can do that by passing their names as a list of strings to the “parse\_dates” keyword argument:

```
df = pd.read_csv(filename,
                usecols=['Indicator', 'Group', 
                         'State', 'Subgroup', 'Phase',
                         'Time Period Start Date',
                         'Time Period End Date',
                        'Value'],
                parse_dates=['Time Period Start Date',
                            'Time Period End Date'])
```

Finally, I asked you to have a three-level multi-index on the rows. We can always tell “read\_csv” to choose a column as the index. If we want a multi-index, then we just have to pass a list of strings, indicating the columns we want to use:

```
df = pd.read_csv(filename,
                usecols=['Indicator', 'Group', 
                         'State', 'Subgroup', 'Phase',
                         'Time Period Start Date',
                         'Time Period End Date',
                        'Value'],
                parse_dates=['Time Period Start Date',
                            'Time Period End Date'],
                index_col=['Phase', 'Group', 'Subgroup'])
```

With this in place, our data frame now has a three-way multi-index on the rows. The data frame has 13,023 rows and five columns that aren’t in the multi-index. While the data frame contains information from a single survey, it combines information from numerous, different cross sections, which makes it a bit difficult to understand and work with at first.

### Create a line graph showing, at the national ("United States") level, the percentage of all adults who had each indicator. The x axis should reflect the phases of the study, and the y axis should reflect the percentage reporting each indicator. Each line should reflect a different indicator.

To answer this, we first need to retrieve only those rows where “United States” is in the “Subgroup” portion of the multi-index. We can do this with “[xs](https://www.bambooweekly.com/pandas-xs/)”, an extremely powerful Pandas method that lets us retrieve based on values at different parts of a multi-index. Here, we say that the “Subgroup” level (and yes, you can use a name instead of a number) should equal “United States”:

```
(
    df
    .xs('United States', level='Subgroup')
)
```

Having selected those rows, I decided that the easiest way to create such a comparison data frame would be to create a pivot table. Pivot tables are basically 2D grouping operations, letting us have all unique values of one categorical column in the rows, all unique values of a second categorical column in the columns, and then the mean of the values.

Because my pivot table will include the “Phase” values, and because those are currently in the multi-index, I use “[reset\_index](https://www.bambooweekly.com/pandas-reset-index/)” to get the column back:

```
(
    df
    .xs('United States', level='Subgroup')
    .reset_index()
)
```

Now I can build my pivot table:

- The rows should be from each unique value of “Phase”
- The columns should be from each unique value of “Indicator”
- The values should come from the “Value” column

Here’s how we can do that, using “[pivot\_table](https://www.bambooweekly.com/pandas-pivot-table/)”:

```
(
    df
    .xs('United States', level='Subgroup')
    .reset_index()
    .pivot_table(index='Phase', columns='Indicator', values='Value')
)
```

This is great, except that we now have *all* of the values for “Indicator”. We only want those columns that have the phrase “of all adults” in them. We can grab all of those columns with the “[filter](https://www.bambooweekly.com/pandas-filter/)” method:

```
(
    df
    .xs('United States', level='Subgroup')
    .reset_index()
    .pivot_table(index='Phase', columns='Indicator', values='Value')
    .filter(like='of all adults', axis='columns')
)
```

Now that we only have those columns we want, we can create a line plot with “[plot.line](https://www.bambooweekly.com/pandas-plot-line/)”:

```
(
    df
    .xs('United States', level='Subgroup')
    .reset_index()
    .pivot_table(index='Phase', columns='Indicator', values='Value')
    .filter(like='of all adults', axis='columns')
    .plot.line()
)
```

The good news? We get a plot! The bad news? The legend takes up nearly everything, because the “Indicator” columns’ values are too long. I decided to use the “[rename](https://www.bambooweekly.com/pandas-rename/)” method to modify the column names, removing “long COVID, as a percentage of all adults” and also “from” before plotting:

```
(
    df
    .xs('United States', level='Subgroup')
    .reset_index()
    .pivot_table(index='Phase', columns='Indicator', values='Value')
    .filter(like='of all adults', axis='columns')
    .rename(mapper=lambda x: x.removesuffix('long COVID, as a percentage of all adults'), axis='columns')
    .rename(mapper=lambda x: x.removesuffix('from '), axis='columns')
    .plot.line()
)
```

Here’s the result:

![](https://storage.ghost.io/c/06/ba/06ba0cc0-be6f-4de7-af2f-5c20165279b9/content/images/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https-3a-2f-2fsubstack-post-media.s3.amazonaws.com-2fpublic-2fimages-2ff53b433f-05c2-4a9e-ab2b-31c53197d21d_543x432.jpg)

We can see that the percentage of people who say that they have *ever* experienced long covid has gone up over time. Perhaps that shouldn’t be a surprise, as more and more people are exposed to covid-19 multiple times, and each occurrence adds to their chances of having long covid.

The good news is that while the number of people who have ever had long covid continues to rise, the percentage that has significant activity limitations has gone down.

### In phase 4 (i.e., the latest phase), create a bar graph showing the proportion of each age group that reported each indicator for all adults. There should be one cluster of bars for each age group, and a separate bar within the cluster for each indicator.

Next, I asked you to only look at data from phase 4, to compare the indicators reported by various age groups, and then to create a bar graph. Remember, when you create a bar graph (with “[plot.bar](https://www.bambooweekly.com/pandas-plot-bar/)”) on a series, you’ll get one line. But if you do it on a data frame with multiple columns, you’ll get one line (each in a different color) per column.

I started by using “[loc](https://www.bambooweekly.com/pandas-loc/)” to retrieve rows from phase 4 and “By Age”:

```
(
    df
    .loc[(4, 'By Age')]
)
```

The thing is, I got a warning from Pandas when I did this, because the index wasn’t already sorted. I thus tried again, first sorting it with “[sort\_index](https://www.bambooweekly.com/pandas-sort-index/)”:

```
(
    df
    .sort_index()
    .loc[(4, 'By Age')]
)
```

Excellent! The warning is gone.

By the way, I could have used “xs” again in this example. But because I was retrieving values from the outer layers of the multi-index, I could just pass “loc” a tuple (not a list!) and peel off those outer two layers, in order. I usually reserve “xs” for when I want to select parts of the multi-index that aren’t on the outside.

Now what? Well, I’ll again create a pivot table, as I did in the previous exercise. So I’ll again invoke “reset\_index”, and then I’ll invoke “pivot\_table”, followed by filtering and renaming the columns.

In the end, I’ll then invoke “plot.bar”:

```
(
    df
    .sort_index()
    .loc[(4, 'By Age')]
    .reset_index()
    .pivot_table(index='Subgroup', columns='Indicator', values='Value')
    .filter(like='of all adults', axis='columns')
    .rename(mapper=lambda x: x.removesuffix('long COVID, as a percentage of all adults'), axis='columns')
    .rename(mapper=lambda x: x.removesuffix('from '), axis='columns')
    .rename(mapper=lambda x: x.removesuffix(' years'))
    .plot.bar()
)
```

Here’s the result of the above query:

![](https://storage.ghost.io/c/06/ba/06ba0cc0-be6f-4de7-af2f-5c20165279b9/content/images/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https-3a-2f-2fsubstack-post-media.s3.amazonaws.com-2fpublic-2fimages-2f3851ac05-699b-4598-bc2a-60f7146ba116_556x556.png)

We can see that the number of people who have ever experienced long covid is largest in the middle-aged rankings. That might mean that people above the age of 70 don’t get long covid. Or it could be a more dire indication, namely that the elderly are less likely to survive, and are thus less likely to get long covid. But we do see that roughly 20 percent of people have experienced some form of long covid, even if relatively minor and no longer a factor.

### In phase 4, which 10 states had the greatest percentage of people reporting "Significant activity limitations from long COVID, as a percentage of all adults"?

Next, I asked you to find results for phase 4 and by state. Once again, because we’re picking out the first two parts of the multi-index, we can use “loc” and a tuple:

```
(
    df
    .sort_index()
    .loc[(4.0, 'By State')]
)
```

Next, I wanted to keep only those rows where “Indicator” was “Significant activity … of all adults”. There are several ways to filter, but I’ve increasingly become a fan of using “lambda” along with the condition that I want. Any rows for which this condition returns True are kept, while the others are thrown out:

```
(
    df
    .sort_index()
    .loc[(4.0, 'By State')]
    .loc[lambda df_: df_['Indicator'] == 'Significant activity limitations from long COVID, as a percentage of all adults']
)
```

Next, I used “[nlargest](https://www.bambooweekly.com/pandas-nlargest/)” to find the rows in which the 10 largest values of the “Value” column were located. Note that this sorts the rows and takes the 10 largest ones; if there are any tie scores, we’ll still only get 10 results.

Then, since we’re only interested in the state (which is the only part remaining of the original three-part multi-index) and the value, we can just select “Value” with square brackets:

```
(
    df
    .sort_index()
    .loc[(4.0, 'By State')]
    .loc[lambda df_: df_['Indicator'] == 'Significant activity limitations from long COVID, as a percentage of all adults']
    .nlargest(10, columns='Value')
    ['Value']
)
```

Here’s what I got:

```
Subgroup
Alaska           2.6
West Virginia    2.6
Kansas           2.5
Alabama          2.4
Kentucky         2.4
New Jersey       2.4
Oregon           2.4
Vermont          2.4
Georgia          2.2
Maine            2.2
Name: Value, dtype: float64
```

In other words, we’re seeing that 2.6 percent of all people in Alaska and West Virginia have reported significant limitations from long covid. It could be worse, of course, but that still seems like a very large number. Multiplied out, if the entire US population of 334 million were to get covid, 2.6 percent would work out to 8,684,000 people. Wow.

### For each year surveyed (based on the "Time Period Start Date"), and for the entire United States, create a line plot showing the mean percentage of people reporting "ever experienced long COVID, as a percentage of all adults".

Finally, I asked you to take each year (not phase) for the entire US, and plot the mean percentage of people who ever experienced long covid.

First, I used “xs” to grab those rows for which “United States” was the subgroup:

```
(
    df
    .xs('United States', level='Subgroup')
)
```

Then I once again used “loc” to keep only those rows with an “Indicator” value for what we want:

```
(
    df
    .xs('United States', level='Subgroup')
    .loc[lambda df_: df_['Indicator'] == 'Ever experienced long COVID, as a percentage of all adults']
)
```

But then… I want to group by the year in a column of the data frame. How can I do that? Sure, I can sometimes run “[groupby](https://www.bambooweekly.com/pandas-groupby/)” on the result of calling “dt” on a datetime column, but this seems different and harder.

Fortunately, Pandas provides the “[Grouper](https://www.bambooweekly.com/pandas-grouper/)” object. If we pass “pd.Grouper” to “groupby”, we can then tell it which columns, and which part of the datetime, shoudl be used. Here, I pass pd.Grouper the key (i.e., which column I want to use) and the frequency (“1YE” which means “every 1 year end”). The rest of the “groupby” runs normally, calculating on the “Value” column and getting the “mean”.

And then, finally, a line plot:

```
(
    df
    .xs('United States', level='Subgroup')
    .loc[lambda df_: df_['Indicator'] == 'Ever experienced long COVID, as a percentage of all adults']
    .groupby(pd.Grouper(key='Time Period Start Date', freq='1YE'))['Value'].mean()
    .plot.line()
)
```

The result:

![](https://storage.ghost.io/c/06/ba/06ba0cc0-be6f-4de7-af2f-5c20165279b9/content/images/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https-3a-2f-2fsubstack-post-media.s3.amazonaws.com-2fpublic-2fimages-2fca3f5566-02c8-4432-b582-7fed68975fee_574x432.png)

This is another way of visualizing what we’ve seen elsewhere already, namely that the proportion of people who have ever had long covid seems to be on the rise. That is both to be expected as time goes on, and a bit worrisome.

What do you think?

Here’s my Jupyter notebook from this week: [https://drive.google.com/file/d/13fKesWYLdJh\_lc6rFm5cNw\_8l3AnPB5r/view?usp=sharing](https://drive.google.com/file/d/13fKesWYLdJh%5Flc6rFm5cNw%5F8l3AnPB5r/view?usp=sharing&ref=bambooweekly.com)

I’ll be back next week with more questions about Pandas, Python, and current events!

Reuven