A Polars solution to BW #184 — the first contest winner

Happy Wednesday! I'll be back later today with new questions. But I first wanted to thank Omar Malik for entering our first-ever community contest, sharing his solutions (in Polars) to BW #184.

You can see his entire solution in Molab.

I certainly learned something from his solutions, and I want to walk you through a few things that he wrote, which I found particularly interesting.

First, I'm always struck by the similarities between Pandas and Polars. Granted, there are differences between the two, but for the most part, being comfortable with Pandas (especially if you use method chaining) makes for an easy entry to Polars.

For example, here's how Omar read all of the files into a single Polars data frame:

df = (
    pl.concat(
        [
            pl.read_csv(one_file, skip_rows=3)
            .with_columns(
                pl.lit(
                    one_file.stem.split("-", 2)[-1]
                ).alias("city")
            )
            for one_file in directory.glob("*.csv")
        ]
    )
    .with_columns(
        pl.col("time").str.to_datetime()
    )
)

This looks quite similar to how I solved it in Pandas, down to the use of pl.concat (where I used pd.concat). The with_columns method in Polars is similar to assign in Pandas, adding a new column to the data frame. Here, Omar added a city column, based on the filename. The use of one_file.stem.split was particularly nice.

You can also see how each file's data frame got a city column. Then, after all of the files were loaded into a data frame and concatenated together, Omar used a second with_columns to convert the time column into datetime values.

I'll just point to one more piece of code, namely the line plot that compared mean annual temperatures for each of the three Italian cities we examined:

(
    df
    .sort('time', descending=False)
    .group_by_dynamic(
        'time',
        every='1y',
        group_by='city'
    )
    .agg(
        pl.col('temperature_2m_max (°C)').mean().round(2).alias('mean_temp')
    )
    .pipe(px.line, x='time', y= 'mean_temp', color='city', labels={'time': 'Year', 'mean_temp': 'Max Mean Temp', 'city': 'City'},
         title="Maximum Mean Temperature: Parma, Reggio Emilia and Reggio Calabria",
         template='plotly_white')
)

In Pandas, we have resample, which performs a kind of groupby on the index, assuming that the index contains datetime values. But Polars doesn't have an index, which means that you need to approach that same problem a bit differently.

Omar used group_by_dynamic, which works on any column. As you can see, you indicate via keyword arguments the period (every='1y') and the column on which to group (group_by='city'). As for the aggregation function, that was specified by the agg method, which specified a number of actions to perform on the max-temperature column.

Also, notice the use of pl.col to specify columns? It's not an accident that Pandas 3 introduced this syntax.

I always enjoy learning from other people's code. Thanks to Omar for entering — and I look forward to seeing many more entries when we do the next contest in just a few weeks.

Reuven