If you live outside of the United States, then traveling there – and more specifically, driving there – likely strikes you as surprisingly cheap. That's because gasoline in the US is cheaper than in most of the developed world. You fill up your tank and ask, "Really? That's it?"
But Americans aren't comparing their gas prices to the rest of the world. They're comparing the prices with what they remember paying in the past, including as recently as last month. And in the wake of the Iran war, the price of gas has skyrocketed. This affects not only people (i.e., just about everyone) who is buying gas for their cars, but also taxi, truck, and bus companies.
Economists say that there is "inelastic demand" for gasoline, which is a fancy way of saying that people buy roughly the same amount of it, regardless of the price. And if the price soars unexpectedly? It means that people will have less to spend on other things. Or they'll withdraw more from their savings and retirement funds (which has been happening), in order to pay for gasoline, food, and other inelastic purchases. (Although the CEO of Pizza Hut recently reported that they have had a huge drop in sales over the last few months, as people are trying to cut down on spending.)
There is also a major political angle to gas prices: As Kai Ryssdal from Marketplace often says, people see gas prices, on huge signs, everywhere they go. And they fill up their tanks on a regular basis, which means they know how much they're paying – and how much they used to pay. The Republican party, which was already worried about its chances in November's mid-term elections, are particularly nervous now, with rising gas prices directly attributable to President Trump's decision to attack Iran.
This week, I thus decided to look into the price of gasoline in the United States. (I had originally planned to compare US gas prices with the rest of the world, but the data I found was interesting and rich enough on its own. I expect to cover that in a future issue.)
We'll get our data from AAA (the American Automobile Association), known for its towing services and (in the pre-GPS era) for its maps. We'll see how much prices have changed, what types of fuel has changed the most, and where the biggest changes have taken place.
Paid subscribers, both to Bamboo Weekly and to my LernerPython+data membership program (https://LernerPython.com) get all of the questions and answers, as well as downloadable data files, downloadable versions of my notebooks, one-click access to my notebooks, and invitations to monthly office hours.
Learning goals for this week include web scraping, combining data, cleaning data, multi-indexes, grouping, window functions, and plotting with Plotly.
Data and five questions
This week's data comes from AAA's "gas prices" web site, at https://gasprices.aaa.com/ . Specifically, we'll be looking at data from each of the 50 states, at URLs of the form https://gasprices.aaa.com/?state=XX, where XX is a two-letter state abbreviation.
To add to the challenge, you cannot download the data directly from the site. Nor can you use the read_html method directly on the site's URL's you will get a 403 ("forbidden") error message. Nor, it seems, can you even use the requests library to retrieve the data.
Instead, you'll have to use the curl_cffi package from PyPI, which has enough smarts to get around the bot-blocking software that AAA and others use. (Please download the data as few times as possible, to avoid angering them or abusing their site.) You'll then combine the 50 states' data into a single data frame, and start to perform calculations on it.
Here are my five questions for this week. I'll be back tomorrow with solutions and explanations:
- Using
curl_cffi, create a Python dictionary with 50 key-value pairs. The keys will be the 50 US state abbreviations, and the values will be thecurl_cffiresponse object that you got back from each state page. - Create a single data frame from the dict that you have created, using the first HTML table on each page (i.e., the one listing different types of gas prices, and at different times). The resulting data frame should have a two-level multi-index, with the outer layer being the two-letter state abbreviation, and the inner layer being the description. The data frame itself should have four float columns reflecting average gas prices in a given state.