Introduction to Exploratory Analysis with Xarray¶

Overview¶

This notebook will introduce the basics of gridded, labeled data with Xarray. Since Xarray introduces additional abstractions on top of plain arrays of data, our goal is to show why these abstractions are useful and how they frequently lead to simpler, more robust code.

We’ll cover these topics:

1. Create a DataArray, one of the core object types in Xarray
2. Understand how to use named coordinates and metadata in a DataArray
3. Combine individual DataArrays into a Dataset, the other core object type in Xarray
4. Subset, slice, and interpolate the data using named coordinates
5. Open netCDF data using XArray
6. Basic subsetting and aggregation of a Dataset
7. Brief introduction to plotting with Xarray

Prerequisites¶

• Time to learn: 30 minutes

Imports¶

Simmilar to numpy, np; pandas, pd; you may often encounter xarray imported within a shortened namespace as xr.

Introducing the DataArray and Dataset¶

Xarray expands on the capabilities on NumPy arrays, providing a lot of streamlined data manipulation. It is similar in that respect to Pandas, but whereas Pandas excels at working with tabular data, Xarray is focused on N-dimensional arrays of data (i.e. grids). Its interface is based largely on the netCDF data model (variables, attributes, and dimensions), but it goes beyond the traditional netCDF interfaces to provide functionality similar to netCDF-java’s Common Data Model (CDM).

Creation of a DataArray object¶

The DataArray is one of the basic building blocks of Xarray (see docs here). It provides a numpy.ndarray-like object that expands to provide two critical pieces of functionality:

1. Coordinate names and values are stored with the data, making slicing and indexing much more powerful
2. It has a built-in container for attributes

Here we’ll initialize a DataArray object by wrapping a plain NumPy array, and explore a few of its properties.

Generate a random numpy array¶

For our first example, we’ll just create a random array of “temperature” data in units of Kelvin:

Wrap the array: first attempt¶

Now we create a basic DataArray just by passing our plain data as input:

Note two things:

1. Xarray generates some basic dimension names for us (dim_0, dim_1, dim_2). We’ll improve this with better names in the next example.
2. Wrapping the numpy array in a DataArray gives us a rich display in the notebook! (Try clicking the array symbol to expand or collapse the view)

Assign dimension names¶

Much of the power of Xarray comes from making use of named dimensions. So let’s add some more useful names! We can do that by passing an ordered list of names using the keyword argument dims:

This is already improved upon from a NumPy array, because we have names for each of the dimensions (or axes in NumPy parlance). Even better, we can take arrays representing the values for the coordinates for each of these dimensions and associate them with the data when we create the DataArray. We’ll see this in the next example.

Create a DataArray with named Coordinates¶

Make time and space coordinates¶

Here we will use Pandas to create an array of datetime data, which we will then use to create a DataArray with a named coordinate time.

We’ll also create arrays to represent sample longitude and latitude:

Initialize the DataArray with complete coordinate info¶

When we create the DataArray instance, we pass in the arrays we just created:

Set useful attributes¶

...and while we’re at it, we can also set some attribute metadata:

Attributes are not preserved by default!¶

Notice what happens if we perform a mathematical operaton with the DataArray: the coordinate values persist, but the attributes are lost. This is done because it is very challenging to know if the attribute metadata is still correct or appropriate after arbitrary arithmetic operations.

To illustrate this, we’ll do a simple unit conversion from Kelvin to Celsius:

For an in-depth discussion of how Xarray handles metadata, start in the Xarray docs here.

The Dataset: a container for DataArrays with shared coordinates¶

Along with DataArray, the other key object type in Xarray is the Dataset: a dictionary-like container that holds one or more DataArrays, which can also optionally share coordinates (see docs here).

The most common way to create a Dataset object is to load data from a file (see below). Here, instead, we will create another DataArray and combine it with our temp data.

This will illustrate how the information about common coordinate axes is used.

Create a pressure DataArray using the same coordinates¶

This code mirrors how we created the temp object above.

Create a Dataset object¶

Each DataArray in our Dataset needs a name!

The most straightforward way to create a Dataset with our temp and pressure arrays is to pass a dictionary using the keyword argument data_vars:

Notice that the Dataset object ds is aware that both data arrays sit on the same coordinate axes.

Access Data variables and Coordinates in a Dataset¶

We can pull out any of the individual DataArray objects in a few different ways.

Using the “dot” notation:

... or using dictionary access like this:

We’ll return to the Dataset object when we start loading data from files.

Subsetting and selection by coordinate values¶

Much of the power of labeled coordinates comes from the ability to select data based on coordinate names and values, rather than array indices. We’ll explore this briefly here.

NumPy-like selection¶

Suppose we want to extract all the spatial data for one single date: January 2, 2018. It’s possible to achieve that with NumPy-like index selection:

HOWEVER, notice that this requires us (the user / programmer) to have detailed knowledge of the order of the axes and the meaning of the indices along those axes!

Named coordinates free us from this burden...

Selecting with .sel()¶

We can instead select data based on coordinate values using the .sel() method, which takes one or more named coordinate(s) as keyword argument:

We got the same result, but

• we didn’t have to know anything about how the array was created or stored
• our code is agnostic about how many dimensions we are dealing with
• the intended meaning of our code is much clearer!

Approximate selection and interpolation¶

With time and space data, we frequently want to sample “near” the coordinate points in our dataset. Here are a few simple ways to achieve that.

Nearest-neighbor sampling¶

Suppose we want to sample the nearest datapoint within 2 days of date 2018-01-07. Since the last day on our time axis is 2018-01-05, this is well-posed.

.sel has the flexibility to perform nearest neighbor sampling, taking an optional tolerance:

where we see that .sel indeed pulled out the data for date 2018-01-05.

Interpolation¶

Suppose we want to extract a timeseries for Boulder (40°N, 105°W). Since lon=-105 is not a point on our longitude axis, this requires interpolation between data points.

The .interp() method (see the docs here) works similarly to .sel(). Using .interp(), we can interpolate to any latitude/longitude location:

Slicing along coordinates¶

Frequently we want to select a range (or slice) along one or more coordinate(s). We can achieve this by passing a Python slice object to .sel(), as follows:

Notice how the length of each coordinate axis has changed due to our slicing.

One more selection method: .loc¶

All of these operations can also be done within square brackets on the .loc attribute of the DataArray:

This is sort of in between the NumPy-style selection

temp[1,:,:]

and the fully label-based selection using .sel()

With .loc, we make use of the coordinate values, but lose the ability to specify the names of the various dimensions. Instead, the slicing must be done in the correct order:

One advantage of using .loc is that we can use NumPy-style slice notation like 25:45, rather than the more verbose slice(25,45). But of course that also works:

What doesn’t work is passing the slices in a different order:

Opening netCDF data¶

With its close ties to the netCDF data model, Xarray also supports netCDF as a first-class file format. This means it has easy support for opening netCDF datasets, so long as they conform to some of Xarray’s limitations (such as 1-dimensional coordinates).

Access netCDF data with xr.open_dataset¶

Once we have a valid path to a data file that Xarray knows how to read, we can open it like this:

Subsetting the Dataset¶

Our call to xr.open_dataset() above returned a Dataset object that we’ve decided to call ds. We can then pull out individual fields:

(recall that we can also use dictionary syntax like ds['isobaric1'] to do the same thing)

Datasets also support much of the same subsetting operations as DataArray, but will perform the operation on all data:

And further subsetting to a single DataArray:

Aggregation operations¶

Not only can you use the named dimensions for manual slicing and indexing of data, but you can also use it to control aggregation operations, like std (standard deviation):

Using the sample dataset, we can calculate the temperature profile across our time period!

Plotting with Xarray¶

Another major benefit of using labeled data structures is that they enable automated plotting with sensible axis labels.

Simple visualization with .plot()¶

Much like we saw in Pandas, Xarray includes an interface to Matplotlib that we can access through the .plot() method of every DataArray.

For quick and easy data exploration, we can just call .plot() without any modifiers:

Here Xarray has generated a line plot of the temperature data against the coordinate variable isobaric. Also the metadata are used to auto-generate axis labels and units.

Customizing the plot¶

As in Pandas, the .plot() method is mostly just a wrapper to Matplotlib, so we can customize our plot in familiar ways.

In this air temperature profile example, we would like to make two changes:

• swap the axes so that we have isobaric levels on the y (vertical) axis of the figure
• make pressure decrease upward in the figure, so that up is up

A few keyword arguments to our .plot() call will take care of this:

Plotting 2D data¶

In the example above, the .plot() method produced a line plot.

What if we call .plot() on a 2D array?

We can also make this interactive!

Customize our Interactive Plots¶

Our time axis doesn’t tell us much... we can change that! Also note that we add additional parameters to customize our view of the field.

And the same for velocity...

Combine our Plots¶

Now that we have our interactive plots, we can combine them using +

Or stacked on top of each other...

Xarray has recognized that the DataArray object calling the plot method has two coordinate variables, and generates a 2D plot using the pcolormesh method from Matplotlib.

In this case, we are looking at air temperatures on the 1000 hPa isobaric surface over North America. We could of course improve this figure by using Cartopy to handle the map projection and geographic features!

Summary¶

Xarray brings the joy of Pandas-style labeled data operations to N-dimensional data. As such, it has become a central workhorse in the geoscience community for the analysis of gridded datasets. Xarray allows us to open self-describing NetCDF files and make full use of the coordinate axes, labels, units, and other metadata. By making use of labeled coordinates, our code is often easier to write, easier to read, and more robust.

We also covered some interactive plots using xarray and hvPlot!

What’s next?¶

Additional notebooks to appear in this section will go into more detail about

• arithemtic and broadcasting with Xarray data structures
• using “group by” operations
• remote data access with OpenDAP
• more advanced visualization including map integration with Cartopy

Resources and references¶

This notebook was adapated from material in Unidata’s Python Training.

The best resource for Xarray is the Xarray documentation. See in particular

• Why Xarray
• Quick overview
• Example gallery

Another excellent resource is this Xarray Tutorial collection.