Bamboo Weekly #6: End of the humanities? (solutions)

This week’s topic: End of the humanities?

This week, we looked at which majors have become more popular over the last few decades at American universities, and which have become less popular.

Our data set set comes from the Digest of Education Statistics, with a table showing the total number of students majoring in each of 30 different areas of study. You can view the table at https://nces.ed.gov/programs/digest/d21/tables/dt21_322.10.asp. Better yet, you can download the file in Excel format from:

https://nces.ed.gov/programs/digest/d21/tables/xls/tabn322.10.xls

Our questions for this week are:

1. Retrieve the Excel file with selected years, and turn it into a data frame.
2. Remove the line numbering the surveys. Remove the total. And remove the lines at the bottom, after "Other and not classified".
3. Remove the \n\ markings (for footnotes) from the "Field of study" column
4. Remove newlines and other extraneous whitespace from the "Field of study" column
5. Set the "Field of study" column to be the index.
6. Which majors have had the greatest *increase* since the survey began in 1970-1971?
7. Display the numbers with a comma before every three digits -- so that instead of showing "1000" it shows "1,000"
8. Which majors have had the greatest *decrease* since the survey began in 1970-1971?
9. If we only look at the last 10 years of the study (i.e., starting in 2010-11), do we see similar changes in majors?
10. What percentage decline do we see in all fields containing the word "art," "language," "history," or "culture" in their names in the last 10 years?

Discussion

First, before anything else, I did my standard setup for working with Pandas:

import numpy as np
import pandas as pd
from pandas import Series, DataFrame

I spent some time looking through the Department of Education site, and found the page at https://nces.ed.gov/programs/digest/d21/tables/dt21_322.10.asp with a visual version of the data. Fortunately, there was also a button on that page labeled “download Excel,” making it possible to download the data into an Excel file.

I could have downloaded the file onto my computer and loaded it from there. If the file were huge, or if I would need to load it multiple times, I might have done that. But it’s a small file, and I only downloaded it a handful of times in preparing this week’s newsletter — so I took of the fact that read_excel, as well as all of the other read_* methods in Pandas, can take a URL, as well as a filename. I thus downloaded the file as follows:

url = 'https://nces.ed.gov/programs/digest/d21/tables/xls/tabn322.10.xls'
df = pd.read_excel(url)

However, this wasn’t quite enough, because the Excel file was formatted a bit strangely. Pandas normally assumes that the first line contains column names, and that subsequent rows contain data. Sometimes, though, the first line might contain titles or descriptions. In such cases, we need to tell Pandas to ignore one or more lines, skipping down to where the headers are located.

We can do that by passing the header=n keyword argument to read_excel, where n is the line number (starting at 0) on which the headers are located.

I thus ended up downloading the file with:

df = pd.read_excel(url, header=1)

Remove the line numbering the surveys. Remove the total. And remove the lines at the bottom, after "Other and not classified".

However, that wasn’t quite enough to get our data frame into shape. That’s because we had a few other rows that would throw off any analysis we tried to do.

For starters, once we loaded the file into a data frame, the first two non-header rows (indexes 0 and 1) contained information that we didn’t really need — one with the serial number of the survey in that column, and another with the total number of people enrolled in post-secondary (i.e., college or vocational) programs in that year.

In other words, we want to remove the rows at indexes 0 and 1. The easiest way to do this is with the “drop” method. Note that drop can be used to drop either rows or columns, and you can specify it explicitly by passing the “axis” keyword argument, with a value of either “rows” or “columns”. (Yes, you can use numbers instead, but I never remember which is 0 and which is 1, so I prefer the names.) The default is to drop rows.

We can drop one row by passing a single index value. To drop more than one row, we can pass a list of indexes.

By default, “drop” returns a new data frame, identical to the original one but without the rows that were specified. There is the option of passing the inplace=True keyword argument, which returns None and modifies the original data frame. However, the core Pandas developers have made it clear that there is no benefit to passing inplace=True, and that it will soon be deprecated.

In the end, we can remove the first two lines with:

df = df.drop([0, 1])

What about the final lines, after “Other and not classified”? We could again use the “drop” method, but here I think it’ll just be easiest to define a slice on our data frame, up to and not including the final five lines:

df = df[:-5]

Remove the \n\ markings (for footnotes) from the "Field of study" column

Those final five rows contained numbered footnotes. And while we’ve removed those footnotes, the references to them are still in the first column, labeled “Field of study.” It wouldn’t be terrible for us to keep things the way they are, but they look pretty ugly. For example, the major

Agriculture and natural resources

is currently written as

Agriculture and natural resources\1\

I asked you to remove all of the footnote references in that column. How can we do that?

Remember that every Pandas column is a series. On a series of strings, we can apply the “.str” accessor. This gives us access to a wide variety of string methods — some from standard Python, and others that Pandas has added to our arsenal. For example, if we have a series s containing strings, we can invoke

s.str.len()

We will get back a new series with the same index as s, but with values reflecting the length of each string in s.

The “replace” method lets us replace one string with another. For example, if I say

s.str.replace(‘a’, ‘b’)

then we’ll get back a new series with the same index as s, but with any “a” in s converted into “b”.

This is similar to what we want, but not quite good enough. After all, I don’t want to replace a particular character, or even a set of characters. I want to replace a pattern, one which I could describe as “a backslash, a digit, and another backslash.”

Whenever you find yourself describing text in such a way, you almost certainly want to reach for regular expressions, aka “regexp” or “regex”. Regular expressions have a long history, and are well known for being hard to learn and/or hard to use. I can assure you that they’re not bad — in fact, my free “Regexp crash course” will teach them to you in just 14 short e-mail lessons.

How can I use regular expressions here? It turns out that the Pandas version of replace actually supports them! You should pass the regex=True keyword argument, as follows:

s.str.replace(‘a’, ‘b’, regex=True)

What regular expression will we want to provide? Well, we want to find a backslash, followed by a digit, followed by another backslash.

This provides us with three different issues!

Remember that backslashes are used in Python strings for all sorts of special characters, such as \t and \n. If we want our string to include an actual backslash character, then we need to double it, as \.

But the Python string that we create then needs to be passed along to the regular expression engine. And that engine has all sorts of internal needs for backslashes, too! So we’ll actually need to escape all of our backslashes, doubling them to get them into our regexp engine.

That’s already ugly, but we can be saved (somewhat) by Python’s raw strings. A raw string has the letter “r” before the opening quote, and basically tells Python to double the backslashes in the string. The idea is that you get to write things the way you want them to be, and Python takes care of the backslash doubling internally. The two primary use cases for raw strings are, in my mind, (a) Windows paths and (b) regular expressions.

My regular expression is thus looking like this:

r’\\\\’

That is: An initial “r”, followed by two backslashes (which will be turned into one), followed by two more backslashes (which will be turned into one).

We’re missing something important, though — the digit in the middle. Fortunately, if you put \d in a regular expression, it’ll match any single digit. We can thus modify our regexp to read:

r’\\\d\\’

Can you believe that someone would call this unreadable? Crazy, I know.

Anyway, I want to remove any thing that matches this pattern in the “Field of study” column. To do that, I’ll look for my regexp, and replace it with an empty string:

df['Field of study'].str.replace(r'\\\d\\', '', regex=True)

This returns a new series, without changing the original one. I still need to assign it back to the column in df:

df['Field of study'] = df['Field of study'].str.replace(r'\\\d\\', '', regex=True)

After executing this code, the footnotes with their weird backslash syntax are gone.

Remove newlines and other extraneous whitespace from the "Field of study" column

However, the “Field of study” column has another problem, namely that it contains newline characters (\n) and bunches of whitepace (mostly space characters). I’d like to make things a little nicer, such that one or more whitespace characters will be replaced by a space.

“Whitespace,” by the way, is the term for the characters that take up space on the screen, but aren’t seen. They are: Space, tab (\t), newline (\n), carriage return (\r), and vertical tab (\v).

We’ll once again use “str.replace” — this time, looking for one or more whitespace characters. Any one whitespace character can be represented in regular expressions as \s. And we can indicate to the regexp engine that a character can occur one or more times with +. Thus, we want to replace any occurrence of \s+ with a single space.

Here’s how that will look:

df['Field of study'] = df['Field of study'].str.replace('\s+', ' ', regex=True)

Once again, the result of invoking str.replace is a new Pandas series, which we assign back to the “Field of study” column.

If you’re wondering why I didn’t use a raw string here, it’s because I didn’t have to do so — Python doesn’t see \s as a special character, and thus passes along the \ and the s as is. There’s nothing wrong with using a raw string, but it’s not needed.

Set the "Field of study" column to be the index.

At this point, we have successfully cleaned up the “Field of study” column. Given that these are the majors, and that the rest of the columns indicate the number of students in each major, it seems reasonable to make “Field of study” into our data frame’s index.

We can do that by invoking “set_index” on our data frame, passing the column name “Field of index” as an argument. As with the “drop” method we used earlier, you can pass inplace=True as keyword arguments to set_index, which will modify the existing data frame and return None. However, that’s generally seen as a bad idea; it’s better to get a new data frame back, and assign it to a variable:

df = df.set_index('Field of study')

Which majors have had the greatest increase since the survey began in 1970-1971?

Each column in the data frame now represents one of the times the survey was run. The earliest is in the column “1970-71”. The most recent is in the column “2019-20”.

Because they’re part of the same data frame, we know that these columns (series) share an index. And when we have two series with a shared index, we can perform mathematical operations on them. The operation will be performed on each of the elements, according to the index.

I can thus say

df['2019-20'] - df['1970-71']

and I’ll get back the difference between the latest survey (2019-20) and the first one (1970-71). The result is a series with the same index as our original data frame.

However, I didn’t ask to see all of the majors. Rather, I just asked to see the five that had the greatest increase. I thus needed to sort the results by value, using the “sort_values” method. In order to have the biggest changes at the top, I passed the keyword argument ascending=False:

(df['2019-20'] - df['1970-71']).sort_values(ascending=False)

Notice that in order to invoke “sort_values” on the difference between our two columns, I had to use an extra set of parentheses. Without the parentheses, sort_values would be invoked on the column from 1970-71, which is not what we wanted.

Finally, I wanted to know which five majors increased the most. Having sorted the values in descending order, I can invoke “head” on the result:

(df['2019-20'] - df['1970-71']).sort_values(ascending=False).head(5)

We can thus see that business has increased the most, followed by health professions, computer-related professions, biology and biomedical, and engineering. Which makes a lot of sense, given how our modern economy differs from what was happening in the early 1970s.

Display the numbers with a comma before every three digits

There isn’t anything particularly wrong with the results that we have displayed. However, it’s often nice to show large numbers with commas every three digits. How can we do this to the results that we just found?

The easiest way is to use the “apply” method, which invokes a function on each of the values in a series. We could write a function and name it, but if it’s a one-time thing, then it’s often easier to just use a “lambda”, an anonymous function.

What function can I invoke on a number that’ll return a string formatted with commas? Once again, Python is way ahead of us: If the format code (i.e., the thing after the “:”) in an f-string is , (comma), and if the value is a number, then we’ll get a string back with commas in it. That is:

>>> n = 1234567890
>>> print(f'{n:,}')
1,234,567,890

I can thus take the series that was returned by “head”, and apply a short lambda on each of the elements in it:

(df['2019-20'] - df['1970-71']).sort_values(ascending=False).head(5).apply(lambda x: f'{x:,}')

Sure enough, we now see the results in a much nicer way:

Field of study
Business                                                  272,455.0
Health professions and related programs                   232,061.0
Computer and information sciences and support services     94,659.0
Biological and biomedical sciences                         90,885.0
Engineering                                                83,298.0
dtype: object

Which majors have had the greatest decrease since the survey began in 1970-1971?

Now that we’ve seen what majors have increased, we’ll look at the other side of things, namely which have decreased. This query is exactly the same as what we’ve already done, but we’ll sort our values in ascending order:

(df['2019-20'] - df['1970-71']).sort_values(ascending=True).head(5).apply(lambda x: f'{x:,}')

Once again, I applied our lambda to the code, in order to get things formatted nicely. The winners?

Field of study
Education                                          -91,250.0
English language and literature/letters            -25,878.0
Foreign languages, literatures, and linguistics     -4,683.0
Library science                                       -895.0
Other and not classified                                 0.0
dtype: object

The biggest drop (by far) was in education, but we see languages and literature have declined a fair amount, too.

If we only look at the last 10 years of the study (i.e., starting in 2010-11), do we see similar changes in majors?

The thing is, 1970 was quite some time ago. (I was born in that year, so I know!) Maybe things changed so much since then that we shouldn’t really compare with today’s majors. Let’s instead take a look at what has changed in just the last 10 years.

To do that, we’ll compare the numbers in 2010-11 with those from the most recent survey:

(df['2019-20'] - df['2010-11']).sort_values(ascending=True).head(5).apply(lambda x: f'{x:,}')

The results are not quite as dramatic, and point to precisely the trend that was discussed in the New Yorker and Axios articles:

Field of study
Education                                                      -18,951.0
Social sciences and history                                    -16,005.0
English language and literature/letters                        -14,718.0
Foreign languages, literatures, and linguistics                 -5,400.0
Liberal arts and sciences, general studies, and humanities      -3,814.0
dtype: object

In other words, we see major declines in humanities, arts, and social sciences. Also in education, but I don’t know how many education schools are closing, if only because there is always going to be some demand for them. (Someone has to train teachers before they’re all replaced by chatbots, after all.)

What percentage decline do we see in all fields containing the word "art," "language," "history," or "culture" in their names in the last 10 years?

Let’s find all of the majors that have these words in their names, and then find not by how many people they have declined, but rather by what percentage they have declined. After all, the percentage decline might be more telling than the absolute numbers.

Rather than look at all of the majors, I asked you to look at those with four specific words in their names. In order to do that, we’ll need to search in our “Fields of study” column, which means turning it from the data frame’s index back into a normal column.

We can do that with reset_index, a method that (like set_index) returns a new data frame:

df = df.reset_index()

But wait — how are we going to find those fields of study with names containing one of four different words?

You might have already guessed that the answer is: Regular expressions! I mean, we could have set up a query with a whole bunch of logical “or” clauses strung together, but why work hard when we can ask the regexp engine to do it for us?

The way that we can ask regexps to match one of several alternatives is with the vertical bar. And while we can think of the “str.contains” method as being analogous to the Python “in” keyword, returning True or False after checking membership in something. But actually, str.contains allows us to use regular expressions to search, as well.

I could thus do something like this:

df['Field of study'].str.contains('art|language|history|culture')

This returns a boolean series, which we can then apply to the original data frame via “.loc”. Using “.loc” means applying both a row selector (the boolean series we got back from “str.contains”) and also a column selector — in this case, a list of strings, the names of the columns we want back. We can thus say:

df.loc[df['Field of study'].str.contains('art|language|history|culture'), ['Field of study', '1970-71', '2019-20']]

The good news? We get results! The bad news? One of the rows we get back is “Agriculture and natural resources,” not one of the values I expected or wanted.

That’s because the word “agriculture” contains the word “culture,” and was thus a match for our call to “str.contains”. How can we look for “culture” as a word by itself, but not at the end of the word “agriculture”?

The answer, once again, is with regular expressions: The special “\b” metacharacter in a regular expression means “word boundary,” meaning that the position cannot be filled with a letter or number. If we put “\b” before the word “culture”, it’ll match “culture” in a word by itself, at the start of a string, sentence, or word, but not if it’s part of another word.

However, “\b” is interpreted by Python strings to be the backspace character. In order to ensure that our “\b” is passed along to the regexp engine intact, we need to double its backslash. Or, if we want to retain our sanity, we can use a raw string:

df.loc[df['Field of study'].str.contains(r'art|language|history|\bculture'), ['Field of study', '2010-11', '2019-20']]

This returns a five-row data frame. Let’s put “Field of study” back as an index:

df.loc[df['Field of study'].str.contains(r'art|language|history|\bculture'), ['Field of study', '2010-11', '2019-20']].set_index('Field of study')

We’re now left with two numeric columns, “2010-11” and “2019-20”. I asked you to calculate the percentage change. The good news is that we can use the “pct_change” method to get the result. The bad news is that pct_change works on the rows. I don’t want to calculate the percentage increase between “English language and literature” and “Foreign languages, literatures, and linguistics.” Rather, I want to know the percentage change in each distinct major.

One solution would be to turn the rows into columns, and the columns into rows, using the “transpose” method (or its T alias). Then we could run pct_change.

But most aggregate methods in Pandas, including pct_change, take an “axis” keyword argument, allowing us to specify that the calculation should be performed across columns, rather than across rows. If I do that, I get the result I wanted:

df.loc[df['Field of study'].str.contains(r'art|language|history|\bculture'), ['Field of study', '2010-11', '2019-20']].set_index('Field of study').pct_change(axis='columns')

The result? We get NaN in the 2010-11 column, as is usual with pct_change. But in the second column, we can see that the declines have been pretty big:

Field of study
English language and literature/letters                       -0.278993
Foreign languages, literatures, and linguistics               -0.248791
Liberal arts and sciences, general studies, and humanities    -0.081641
Social sciences and history                                   -0.090337
Visual and performing arts                                    -0.017107
Name: 2019-20, dtype: float64

Or if we translate it into percentages:

Field of study
English language and literature/letters                       -27.90
Foreign languages, literatures, and linguistics               -24.88
Liberal arts and sciences, general studies, and humanities     -8.16
Social sciences and history                                    -9.03
Visual and performing arts                                     -1.71
Name: 2019-20, dtype: float64

Sure enough, we see quite a large drop in many of the humanities, arts, and social sciences. Less than we might have expected; I thought that “liberal arts and sciences” would drop along with the rest of them.

I was also surprised by how small the drop was in visual and performing arts — but that might have to do with the growing opportunities online, the inclusion of such topics as video and audio, or the small number of people in such fields to begin with.

That’s it for this week’s analysis! As always, I welcome your comments, corrections, and suggestions about the format, topics, and my solutions.

Meanwhile, here’s my Jupyter notebook for this week: https://drive.google.com/file/d/1C3_lrf8gS9K_6bwh4Y8r_jjgMRHBRlz4/view?usp=drive_link

I’ll be back next Wednesday with another topical data-science discussion.

Until then,

Reuven