Exploratory Analysis with Pandas¶

Overview¶

1. Data Access and Reading
2. Understanding the Pandas Data Structure
3. Exploratory Data Analysis

Prerequisites¶

• Time to learn: 30 minutes

Imports¶

Data Access and Reading¶

The data for this example/notebook is available from the CSIRO Kennaook/Cape Grim page https://www.csiro.au/en/research/natural-environment/atmosphere/latest-greenhouse-gas-data

More details on the science motivation, data overview, and data citation practices are available on that website. We staged data for you locally in preparation for this workshop - which would be the same as going to the page, downloading, and uploading to this Jupyterhub Space.

Try Reading the Data with Pandas¶

Troubleshooting the Data Reading¶

By default, we cannot read the data! We need to pass more information to pandas to be able to read the data. For example, if we open the file with a text reader, we notice there is a good amount of metadata at the top of the file

Metadata: Data about our data. This provides helpful links, when data was processed, and units. All of this data is critical to understanding our data before we perform analysis.

Understanding the Pandas Data Structure¶

The pandas DataFrame...¶

...is a labeled, two-dimensional columnar structure, similar to a table, spreadsheet, or the R data.frame.

The columns that make up our DataFrame can be lists, dictionaries, NumPy arrays, pandas Series, or many other data types not mentioned here. Within these columns, you can have data values of many different data types used in Python and NumPy, including text, numbers, and dates/times. The first column of a DataFrame, shown in the image above in dark gray, is uniquely referred to as an index; this column contains information characterizing each row of our DataFrame. Similar to any other column, the index can label rows by text, numbers, datetime objects, and many other data types. Datetime objects are a quite popular way to label rows.

For our first example using Pandas DataFrames, we start by reading in some data in comma-separated value (.csv) format. We retrieve this dataset from our local data directory; however, the dataset was originally contained within the CSIRO data page. This dataset contains many types of greenhouse gas data, including gas measurements at Kennaook/Cape Grim. For more information on this dataset, review the description here.

The DataFrame index, as described above, contains information characterizing rows; each row has a unique ID value, which is displayed in the index column. By default, the IDs for rows in a DataFrame are represented as sequential integers, which start at 0.

At the moment, the index column of our DataFrame is not very helpful for humans. However, Pandas has clever ways to make index columns more human-readable. The next example demonstrates how to use optional keyword arguments to convert DataFrame index IDs to a human-friendly datetime format.

Each of our data rows is now helpfully labeled by a datetime-object-like index value; this means that we can now easily identify data values not only by named columns, but also by date labels on rows. This is a sneak preview of the DatetimeIndex functionality of Pandas; this functionality enables a large portion of Pandas’ timeseries-related usage. Don’t worry; DatetimeIndex will be discussed in full detail later on this page. In the meantime, let’s look at the columns of data read in from the .csv file:

The pandas Series...¶

...is essentially any one of the columns of our DataFrame. A Series also includes the index column from the source DataFrame, in order to provide a label for each value in the Series.

The pandas Series is a fast and capable 1-dimensional array of nearly any data type we could want, and it can behave very similarly to a NumPy ndarray or a Python dict. You can take a look at any of the Series that make up your DataFrame, either by using its column name and the Python dict notation, or by using dot-shorthand with the column name:

Slicing and Dicing the DataFrame and Series¶

In this section, we will expand on topics covered in the previous sections on this page. One of the most important concepts to learn about Pandas is that it allows you to access anything by its associated label, regardless of data organization structure.

Indexing a Series¶

As a review of previous examples, we’ll start our next example by pulling a Series out of our DataFrame using its column label.

You can use syntax similar to that of NumPy ndarrays to index, select, and subset with Pandas Series, as shown in this example:

This is an example of label-based slicing. With label-based slicing, Pandas will automatically find a range of values based on the labels you specify.

If you already have some knowledge of xarray (used with ACT), you will quite likely know how to create slice objects by hand. This can also be used in pandas, as shown below. If you are completely unfamiliar with xarray, it will be covered on a later Pythia tutorial page.

Using .iloc and .loc to index¶

In this section, we introduce ways to access data that are preferred by Pandas over the methods listed above. When accessing by label, it is preferred to use the .loc method, and when accessing by index, the .iloc method is preferred. These methods behave similarly to the notation introduced above, but provide more speed, security, and rigor in your value selection. Using these methods can also help you avoid chained assignment warnings generated by pandas.

Extending to the DataFrame¶

These subsetting capabilities can also be used in a full DataFrame; however, if you use the same syntax, there are issues, as shown below:

When indexing a DataFrame, pandas will not assume as readily the intention of your code. In this case, using a row label by itself will not work; with DataFrames, labels are used for identifying columns.

As shown below, you also cannot subset columns in a DataFrame using integer indices:

From earlier examples, we know that we can use an index or label with a DataFrame to pull out a column as a Series, and we know that we can use an index or label with a Series to pull out a single value. Therefore, by chaining brackets, we can pull any individual data value out of the DataFrame.

However, subsetting data using this chained-bracket technique is not preferred by Pandas. As described above, Pandas prefers us to use the .loc and .iloc methods for subsetting. In addition, these methods provide a clearer, more efficient way to extract specific data from a DataFrame, as illustrated below:

The .loc and .iloc methods also allow us to pull entire rows out of a DataFrame, as shown in these examples:

In the next example, we illustrate how you can use slices of rows and lists of columns to create a smaller DataFrame out of an existing DataFrame:

Exploratory Data Analysis¶

Get a Quick Look at the Beginning/End of your DataFrame¶

Pandas also gives you a few shortcuts to quickly investigate entire DataFrames. The head method shows the first five rows of a DataFrame, and the tail method shows the last five rows of a DataFrame.

Quick Plots of Your Data¶

A good way to explore your data is by making a simple plot. Pandas contains its own plot method; this allows us to plot Pandas series without needing matplotlib. In this example, we plot the Nino34 series of our df DataFrame in this way:

Before, we called .plot(), which generated a single line plot. Line plots can be helpful for understanding some types of data, but there are other types of data that can be better understood with different plot types. For example, if your data values form a distribution, you can better understand them using a histogram plot.

The code for plotting histogram data differs in two ways from the code above for the line plot. First, two series are being used from the DataFrame instead of one. Second, after calling the plot method, we call an additional method called hist, which converts the plot into a histogram.

The histogram plot helped us better understand our data; there are clear differences in the distributions. To even better understand this type of data, it may also be helpful to create a box plot. This can be done using the same line of code, with one change: we call the box method instead of hist.

Just like the histogram plot, this box plot indicates a clear difference in the distributions. Using multiple types of plot in this way can be useful for verifying large datasets. The pandas plotting methods are capable of creating many different types of plots. To see how to use the plotting methods to generate each type of plot, please review the pandas plot documentation.

Customize your Plot¶

The pandas plotting methods are, in fact, wrappers for similar methods in matplotlib. This means that you can customize pandas plots by including keyword arguments to the plotting methods. These keyword arguments, for the most part, are equivalent to their matplotlib counterparts.

Although plotting data can provide a clear visual picture of data values, sometimes a more quantitative look at data is warranted. As elaborated on in the next section, this can be achieved using the describe method. The describe method is called on the entire DataFrame, and returns various summarized statistics for each column in the DataFrame.

Basic Statistics¶

We can garner statistics for a DataFrame by using the describe method. When this method is called on a DataFrame, a set of statistics is returned in tabular format. The columns match those of the DataFrame, and the rows indicate different statistics, such as minimum.

You can also view specific statistics using corresponding methods. In this example, we look at the mean values in the entire DataFrame, using the mean method. When such methods are called on the entire DataFrame, a Series is returned. The indices of this Series are the column names in the DataFrame, and the values are the calculated values (in this case, mean values) for the DataFrame columns.

Subsetting Using the Datetime Column¶

Slicing is a useful technique for subsetting a DataFrame, but there are also other options that can be equally useful. In this section, some of these additional techniques are covered.

If your DataFrame uses datetime values for indices, you can select data from only one month using df.index.month. In this example, we specify the number 1, which only selects data from January.

This example shows how to create a new column containing the month portion of the datetime index for each data row. The value returned by df.index.month is used to obtain the data for this new column:

This next example illustrates how to use the new month column to calculate average monthly values over the other data columns. First, we use the groupby method to group the other columns by the month. Second, we take the average (mean) to obtain the monthly averages. Finally, we plot the resulting data as a line plot by simply calling plot().

Investigating Extreme Values¶

If you need to search for rows that meet a specific criterion, you can use conditional indexing. In this example, we search for rows where the $CO^{2}$ values exceed the 95th percentile across all the data from the beginning of the record:

This example shows how to use the sort_values method on a DataFrame. This method sorts values in a DataFrame by the column specified as an argument.

You can also reverse the ordering of the sort by specifying the ascending keyword argument as False:

Resampling¶

In these examples, we illustrate a process known as resampling. Using resampling, you can change the frequency of index data values, reducing so-called ‘noise’ in a data plot. This is especially useful when working with timeseries data; plots can be equally effective with resampled data in these cases. The resampling performed in these examples converts monthly values to yearly averages. This is performed by passing the value ‘1YE’ to the resample method.

Applying operations to a DataFrame¶

One of the most commonly used features in Pandas is the performing of calculations to multiple data values in a DataFrame simultaneously. Let’s first look at a familiar concept: a function that converts single values. The following example uses such a function to convert concentrations from parts ber million (ppm) to parts per billion (ppb):

The following examples instead illustrate a new concept: using such functions with DataFrames and Series. For the first example, we start by creating a Series; in order to do so, we subset the DataFrame by the CO2(ppm) column. This has already been done earlier in this page; we do not need to create this Series again. We are using this particular Series for a reason: the data values are in parts per million.

This example illustrates how to use the temperature-conversion function defined above on a Series object. Just as calling the function with a single value returns a single value, calling the function on a Series object returns another Series object. The function performs the temperature conversion on each data value in the Series, and returns a Series with all values converted.

If we call the .values method on the Series passed to the function, the Series is converted to a numpy array, as described above. The function then converts each value in the numpy array, and returns a new numpy array with all values sorted.

As described above, when our temperature-conversion function accepts a Series as an argument, it returns a Series. We can directly assign this returned Series to a new column in our DataFrame, as shown below:

In this final example, we demonstrate the use of the to_csv method to save a DataFrame as a .csv file. This example also demonstrates the read_csv method, which reads .csv files into Pandas DataFrames.

Summary¶

• Pandas is a very powerful tool for working with tabular (i.e., spreadsheet-style) data
• There are multiple ways of subsetting your pandas dataframe or series
• Pandas allows you to refer to subsets of data by label, which generally makes code more readable and more robust
• Pandas can be helpful for exploratory data analysis, including plotting and basic statistics
• One can apply calculations to pandas dataframes and save the output via csv files

What’s Next?¶

In the next notebook, we will take a look at how to work with cloud radar/remote sensing data.

Resources and References¶

1. CSIRO Data Page
2. Getting Started with Pandas
3. Pandas User Guide