{ "cells": [ { "cell_type": "markdown", "id": "950099de-bfc3-4e1d-85a4-0184030e85a8", "metadata": {}, "source": [ "# ACT Basics with BNF\n", "\n", "\n", " \n", " \n", " \n", " \n", "
\n", " \"BNF\n", " \n", "

Atmospheric Radiation Measurement user facility (ARM)

\n", "

Bankhead National Forest (BNF)

\n", " Notebook for learning the basics of ACT with BNF data
\n", " Corresponding Author: Adam Theisen (atheisen@anl.gov)\n", "
" ] }, { "cell_type": "markdown", "id": "830e2e64-3205-41cd-8247-2fa3905f6106", "metadata": {}, "source": [ "## Overview\n", "The third ARM Mobile Facility (AMF3) has deployed to Bankhead National Forest in northern Alabama, opening October 1, 2024. For at least five years, the BNF will investigate the complex interactions among clouds, vegetation, and aerosols suspended in the atmosphere. This will be done using the instrumentation at the main site, 43-meter tower site, and three supplemental sites in the region. This tutorial will provide an introduction to ACT using the BNF data.\n", "\n", "1. Intro to ACT\n", "1. Instrument Overview\n", "1. Downloading and Reading in Data\n", "1. Quality Controlling Data\n", "1. Visualizing Data" ] }, { "cell_type": "markdown", "id": "74e672b9-ef14-4f01-be19-a222f5ea1573", "metadata": { "tags": [] }, "source": [ "## Prerequisites\n", "This notebook will rely heavily on Python and the [Atmospheric data Community Toolkit (ACT)](https://github.com/ARM-DOE/ACT). Don't worry if you don't have experience with either, this notebook will walk you though what you need to know.\n", "\n", "You will also need an account and token to download data using the ARM Live webservice. Navigate to the [webservice information page](https://adc.arm.gov/armlive/) and log in to get your token. Your account username will be your ARM username.\n", "\n", "| Concepts | Importance | Notes |\n", "| --- | --- | --- |\n", "| [ACT](https://github.com/ARM-DOE/ACT) | Helpful | |\n", "\n", "- **Time to learn**: 15 Minutes\n", "- **System requirements**:\n", " - Python 3.11 or latest\n", " - ACT v2.0.0 or latest\n", " - numpy\n", " - xarray\n", " - matplotlib" ] }, { "cell_type": "markdown", "id": "f89921ee-514a-4e11-859a-4858ed7354ab", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "id": "355585e9-451c-4bfa-82d3-e2199431c707", "metadata": {}, "source": [ "## Intro to ACT\n", " \n", "The [Atmospheric data Community Toolkit (ACT)](https://github.com/ARM-DOE/ACT) is an open-source Python toolkit for exploring and analyzing atmospheric time-series datasets. Examples can be found in the [ACT Example Gallery](https://arm-doe.github.io/ACT/source/auto_examples/index.html). The toolkit has modules for many different parts of the scientific process, including:\n", "\n", "\n", " \n", " \n", " \n", " \n", "
\n", " \n", "#### Data Discovery (act.discovery)\n", "The [discovery](https://arm-doe.github.io/ACT/API/generated/act.discovery.html) module houses functions to download or access data from different groups. Currently it includes function to get data for ARM, NOAA, EPA, NEON, and more!\n", "\n", "#### Input/Output (act.io)\n", "[io](https://arm-doe.github.io/ACT/API/generated/act.io.html) contains functions for reading and writing data from various sources and formats.\n", "\n", "#### Visualization (act.plotting)\n", "[plotting](https://arm-doe.github.io/ACT/API/generated/act.plotting.html) contains various routines, built on matplotlib, to help visualize and explore data. These include \n", "1. Time-series plots\n", "1. Distribution plots like histograms and heatmaps\n", "1. Geographic plots for moving systems like radiosondes or aircraft\n", "1. Skew-T plots for radiosonde data, built off MetPy\n", "1. Wind rose plots for wind and data roses\n", "1. Cross-section plots for working with 3-dimensional data\n", "\n", "#### Corrections (act.corrections)\n", "[corrections](https://arm-doe.github.io/ACT/API/generated/act.corrections.html) apply different corrections to data based on need. A majority of the existing corrections are for lidar data.\n", "\n", "#### Quality Control (act.qc)\n", "The [qc](https://arm-doe.github.io/ACT/API/generated/act.qc.html) module has a lot of functions for working with quality control information, apply new tests, or filtering data based on existing tests. We will explore some of that functionality in this notebook.\n", "\n", "#### Retrievals (act.retrievals)\n", "There are many cases in which some additional calculations are necessary to get more value from the instrument data. The [retrievals](https://arm-doe.github.io/ACT/API/generated/act.retrievals.html) module houses some functions for performing these advanced calculations. \n", "\n", "#### Utilities (act.utils)\n", "The [utils](https://arm-doe.github.io/ACT/API/generated/act.utils.html) module has a lot of general utilities to help with the data. Some of these include adding in a solar variable to indicate day/night (useful in filtering data), unit conversions, decoding WMO weather codes, performing weighted averaging, etc...\n", " \n", "
\n", " \"NEON-ARM\"\n", "
ARM and NEON data from Utquivaik, AK
\n", " \"SONDE\"\n", "
Enhanced Skew-T plot from ARM's Southern Great Plains Site (SGP)
\n", "
\n", "
" ] }, { "cell_type": "markdown", "id": "7cfa5fcb-2220-4e50-91a3-ca74931a099f", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "id": "eb1d0ec1-0d93-4ffd-ace0-97252e69cde1", "metadata": {}, "source": [ "### Preparing for the Future\n", "Please take the survey to help us develop the third roadmap for ACT which helps to prioritize features and activities in ACT.\n", "\n", "\"BNF" ] }, { "cell_type": "markdown", "id": "37b2d1ab-9030-416e-9612-3395a164f93b", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "id": "b353659a-3401-492f-bcca-24963e5e6664", "metadata": {}, "source": [ "## Instrument Overview\n", "\n", " \n", " \n", " \n", " \n", "
\n", " \n", "### Surface Meteorological Instrumentation (MET)\n", "The ARM Surface Meteorology Systems (MET) use mainly conventional in situ sensors to obtain 1-minute statistics of surface wind speed, wind direction, air temperature, relative humidity, barometric pressure, and rain-rate. [Learn more](https://arm.gov/capabilities/instruments/met)\n", "\n", "### Scanning mobility particle sizer (SMPS)\n", "The scanning mobility particle sizer (SMPS) is a particle size spectrometer that measures the aerosol number size distribution by sizing particles based on their electrical mobility diameter using a differential mobility analyzer (DMA) and by counting particles using a condensation particle counter (CPC). It measures aerosol concentration and aerosol particle size distribution. [Learn more](https://arm.gov/capabilities/instruments/smps)\n", " \n", "
\n", " \"MET\"\n", "
Surface Meteorology Station
\n", "
\n", "
" ] }, { "cell_type": "markdown", "id": "75b8b2a7-ef24-4836-b488-d985647da664", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "id": "0de3c05a-6e49-4859-850b-32c989f2cc33", "metadata": {}, "source": [ "## Imports\n", "Let's get started with some data! But first, we need to import some libraries." ] }, { "cell_type": "code", "execution_count": null, "id": "a5b00201-8c78-4ef5-b02f-57e55021f5d6", "metadata": { "tags": [] }, "outputs": [], "source": [ "import act\n", "import numpy as np\n", "import xarray as xr\n", "import matplotlib.pyplot as plt\n", "import matplotlib.colors as colors" ] }, { "cell_type": "markdown", "id": "3647028a-cf78-436a-be33-6a0148474bf6", "metadata": {}, "source": [ "## Downloading and Reading ARM's NetCDF Data\n", "ARM's standard file format is NetCDF (network Common Data Form) which makes it very easy to work with in Python! ARM data are available through a data portal called [Data Discovery](https://adc.arm.gov/discovery/#/) or through a webservice. If you didn't get your username and token earlier, please go back and see the Prerequisites!\n", "\n", "Let's download some of the MET data first!" ] }, { "cell_type": "code", "execution_count": null, "id": "6dc01deb-8e97-4e45-9efb-196b37ca7386", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Set your username and token here!\n", "username = 'YourUserName'\n", "token = 'YourToken'\n", "\n", "# Set the datastream and start/enddates\n", "datastream = 'bnfmetM1.b1'\n", "startdate = '2024-12-16'\n", "enddate = '2024-12-19T23:59:59'\n", "\n", "# Use ACT to easily download the data. Watch for the data citation! Show some support\n", "# for ARM's instrument experts and cite their data if you use it in a publication\n", "result_met = act.discovery.download_arm_data(username, token, datastream, startdate, enddate)\n", "\n", "datastream = 'bnfaossmpsM1.b1'\n", "result_smps = act.discovery.download_arm_data(username, token, datastream, startdate, enddate)" ] }, { "cell_type": "markdown", "id": "1f0a2335-1835-40f7-95b3-4be2bc2e2017", "metadata": {}, "source": [ "### Note: Did you notice the citation and DOI?" ] }, { "cell_type": "code", "execution_count": null, "id": "acdaa696-ebb5-4bbe-8d1e-a4c66f524964", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Let's read in the data using ACT and check out the data\n", "ds_met = act.io.read_arm_netcdf(result_met)\n", "ds_smps = act.io.read_arm_netcdf(result_smps)\n", "\n", "ds_met" ] }, { "cell_type": "markdown", "id": "a93b725a-3216-4eca-8ad0-d9e9e31b1372", "metadata": {}, "source": [ "## Quality Controlling Data\n", "\n", "ARM has multiple methods that it uses to communicate data quality information out to the users. One of these methods is through \"embedded QC\" variables. These are variables within the file that have information on automated tests that have been applied. Many times, they include Min, Max, and Delta tests but as is the case with the AOS instruments, there can be more complicated tests that are applied.\n", "\n", "The results from all these different tests are stored in a single variable using bit-packed QC. We won't get into the full details here, but it's a way to communicate the results of multiple tests in a single integer value by utilizing binary and bits! You can learn more about bit-packed QC [here](https://code.arm.gov/docs/QC-flag-examples/-/wikis/home) but ACT also has many of the tools for working with ARM QC.\n", "\n", "#### Other Sources of Quality Control\n", "ARM also communicates problems with the data quality through Data Quality Reports (DQR). These reports are normally submitted by the instrument mentor when there's been a problem with the instrument. The categories include:\n", "\n", "- **Data Quality Report Categories**\n", " - Missing: Data are not available or set to -9999\n", " - Suspect: The data are not fully incorrect but there are problems that increases the uncertainty of the values. Data should be used with caution.\n", " - Bad: The data are incorrect and should not be used.\n", " - Note: Data notes are a way to communicate information that would be useful to the end user but does not rise to the level of suspect or bad data\n", " \n", "Additionally, data quality information can be found in the Instrument Handbooks, which are included on most instrument pages. Here is an [example](https://www.arm.gov/publications/tech_reports/handbooks/met_handbook.pdf) of the MET handbook." ] }, { "cell_type": "code", "execution_count": null, "id": "a1dd0dc1-7ae2-4c81-812e-cad0a525ecfe", "metadata": { "tags": [] }, "outputs": [], "source": [ "# We can see that there's some missing data in the plot above so let's take a look at the embedded QC!\n", "# First, for many of the ACT QC features, we need to get the dataset more to CF standard and that\n", "# involves cleaning up some of the attributes and ways that ARM has historically handled QC\n", "ds_smps.clean.cleanup()\n", "\n", "# Query the ARM DQR Webservice\n", "variable = 'total_N_conc'\n", "ds_smps = act.qc.add_dqr_to_qc(ds_smps, variable=variable)\n", "\n", "# Next, let's take a look at visualizing the quality control information\n", "# Create a plotting display object with 2 plots\n", "display = act.plotting.TimeSeriesDisplay(ds_smps, figsize=(15, 10), subplot_shape=(2,))\n", "\n", "# Plot up the variable in the first plot\n", "display.plot(variable, subplot_index=(0,))\n", "\n", "# Plot up a day/night background\n", "display.day_night_background(subplot_index=(0,))\n", "\n", "# Plot up the QC variable in the second plot\n", "display.qc_flag_block_plot(variable, subplot_index=(1,))\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "dcef61c0-becf-4f59-a92a-7707997877ae", "metadata": {}, "source": [ "## Visualizing Data\n", "We've already worked with visualizing the data in basic ways but what other options are there in ACT? This section will show you how to create a variety of different plots. More plotting examples can be found in [ACT's Documentation](https://arm-doe.github.io/ACT/source/auto_examples/index.html#plotting-examples)." ] }, { "cell_type": "markdown", "id": "215d8efc-9939-43c8-a23a-260ac94af769", "metadata": {}, "source": [ "### Multi-Panel Plot\n", "This example shows how to plot the MET and SMPS together on a multi-panel plot." ] }, { "cell_type": "code", "execution_count": null, "id": "0058fb57-2950-419a-8583-c2dc43989c63", "metadata": {}, "outputs": [], "source": [ "#And plot again!\n", "# Create a plotting display object with 2 plots\n", "# Note we have to create a dictionary of datasets to pass in\n", "display = act.plotting.TimeSeriesDisplay({'SMPS': ds_smps, 'MET': ds_met}, figsize=(15, 10), subplot_shape=(2,))\n", "\n", "# Plot up the variable in the first plot\n", "# Need to specify the dsname so it knows which dataset\n", "# to use for this data. This is helpful when datasets \n", "# have similar variable names\n", "display.plot('dN_dlogDp', dsname='SMPS', subplot_index=(0,))\n", "\n", "# Plot up the MET btemperature and precipitation\n", "display.plot('temp_mean', dsname='MET', subplot_index=(1,))\n", "ax2 = display.axes[1].twinx()\n", "ax2.plot(ds_met.time, ds_met.tbrg_precip_total, color='orange')\n", "\n", "# Plot up a day/night background\n", "display.day_night_background(dsname='MET', subplot_index=(1,))" ] }, { "cell_type": "markdown", "id": "71df9bf3-b38d-4374-8e6a-bf0b37251516", "metadata": {}, "source": [ "### Adding some customization\n", "Let's make this plot a little nicer and easier to read. We can use a mix of ACT and Matplotlib to help" ] }, { "cell_type": "code", "execution_count": null, "id": "8157d53e-9faf-4489-abd5-c737b8126fcf", "metadata": {}, "outputs": [], "source": [ "#And plot again!\n", "# Create a plotting display object with 2 plots\n", "# Note we have to create a dictionary of datasets to pass in\n", "display = act.plotting.TimeSeriesDisplay({'SMPS': ds_smps, 'MET': ds_met}, figsize=(15, 10), subplot_shape=(2,))\n", "\n", "# Let's add in titles and more to the SMPS plot\n", "title = 'Scanning Mobility Particle Sizer (SMPS) Number Size Distribution'\n", "cbar_title = 'dN/dlogD$_p$ (1/cm$^{3}$)'\n", "display.plot('dN_dlogDp', dsname='SMPS', subplot_index=(0,), cvd_friendly=True,\n", " norm=colors.LogNorm(vmin=100., vmax=10000.), set_title=title, cbar_label=cbar_title,\n", " ylabel='Pariticle Diameter (nm)')\n", "display.axes[0].set_yscale('log')\n", "display.set_yrng([10, 1000], subplot_index=(0,))\n", "\n", "# Let's accumulate the precipitation using ACT to make this easier to read\n", "ds_met = act.utils.data_utils.accumulate_precip(ds_met, 'tbrg_precip_total')\n", "\n", "# Plot up the MET btemperature and precipitation\n", "title = 'MET Temperature and Cumulative Precipitation'\n", "display.plot('temp_mean', dsname='MET', subplot_index=(1,), set_title=title, color='black')\n", "ax2 = display.axes[1].twinx()\n", "ax2.plot(ds_met.time, ds_met.tbrg_precip_total_accumulated, color='blue')\n", "ax2.set_ylabel('Precipitation (mm)')\n", "\n", "# Plot up a day/night background\n", "display.day_night_background(dsname='MET', subplot_index=(1,))" ] }, { "cell_type": "markdown", "id": "68b3e3ea-9f8a-4bd3-ba86-9bae0d9b9609", "metadata": {}, "source": [ "### Distribution Display\n", "\n", "For the example, we will go over some functions within ACT's distribution display. Functions such as, the stacked bar plot, scatter and groupby." ] }, { "cell_type": "code", "execution_count": null, "id": "53326c0d-df46-4ce6-b0a1-a87e6bd90641", "metadata": { "tags": [] }, "outputs": [], "source": [ "# First, let's plot up a histogram of the data\n", "# All the ACT plotting is very similar to what we\n", "# did earlier, first we create a display object\n", "display = act.plotting.DistributionDisplay(ds_met)\n", "\n", "# And then we can plot the data! Note that we are passing a range into the\n", "# histogram function to set the min/max range of the data\n", "display.plot_stacked_bar('temp_mean', hist_kwargs={'range': [4, 20]})\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "fa58bc20-1158-4784-b98f-cc9cf9ac9a75", "metadata": { "tags": [] }, "outputs": [], "source": [ "# First we need to combine data and to do that, we need to get it on the same time grid\n", "ds_combined = xr.merge([ds_met.resample(time='1min').nearest(), ds_smps.resample(time='1min').nearest()], compat='override')\n", "\n", "\n", "# Next up, let's do some scatter plots to compare some variables\n", "# Scatter plots are also found in the DistributionDisplay module\n", "display = act.plotting.DistributionDisplay(ds_combined)\n", "\n", "# And then we can plot the data!\n", "display.plot_scatter('rh_mean', 'total_N_conc', m_field='time')\n", "\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "e96b4681-80e7-4dc4-9554-1c40c9c61b08", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Sometimes these scatter plots hide the number of points there actually\n", "# are in some areas so let's try a heatmap as well\n", "display = act.plotting.DistributionDisplay(ds_combined, figsize=(12, 5), subplot_shape=(1, 2))\n", "\n", "# And then we can plot the data!\n", "display.plot_scatter('rh_mean', 'total_N_conc', m_field='time', subplot_index=(0, 0))\n", "# This can be used to adjust the axes limits\n", "# display.set_xrng([0, 20], subplot_index=(0, 0))\n", "\n", "# we can also pass in an array of values for the bins using np.arange(start, stop, step)\n", "display.plot_heatmap('rh_mean', 'total_N_conc', subplot_index=(0, 1), x_bins=25, y_bins=25)\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "a0b0b8e6-4bc8-495c-9adf-dabde740a2c8", "metadata": {}, "source": [ "### Data Rose Plots\n", "\n", "If your data may be wind speed or direction dependent, these plots can help inform where the influences may be coming from. For this example, let's see if the total number concentration from the SMPS is dependent on wind direction." ] }, { "cell_type": "code", "execution_count": null, "id": "970a06cc-8fa4-407f-98a3-4be4c0d49018", "metadata": {}, "outputs": [], "source": [ "# Plot out the data rose using the WindRose display object\n", "display = act.plotting.WindRoseDisplay(ds_combined)\n", "display.plot_data('wdir_vec_mean', 'wspd_vec_mean', 'total_N_conc', num_dirs=15, plot_type='line', line_plot_calc='mean')\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "1433bbe8-061c-4323-af47-ed00e35a56a8", "metadata": {}, "outputs": [], "source": [ "# Plot out the data rose using the WindRose display object\n", "display = act.plotting.WindRoseDisplay(ds_combined)\n", "\n", "# Let's try a different type of data rose type that is using contours amd the polar y axis is wind speed\n", "# depending on wind direction and speed\n", "display.plot_data('wdir_vec_mean', 'wspd_vec_mean', 'total_N_conc', num_dirs=15, plot_type='contour', contour_type='mean')\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "4178c3dd-4a15-4e05-839c-b5fdf96c0efd", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "id": "0eaee516-805e-46ed-94e9-1d115bcebb6c", "metadata": {}, "source": [ "### Data Used in this Notebook\n", "Kuang, C., Singh, A., Howie, J., Salwen, C., & Hayes, C. Scanning mobility particle sizer (AOSSMPS), 2024-12-16 to 2024-12-19, Bankhead National Forest, AL, USA; Long-term Mobile Facility (BNF), Bankhead National Forest, AL, AMF3 (Main Site) (M1). Atmospheric Radiation Measurement (ARM) User Facility. https://doi.org/10.5439/1476898\n", "\n", "Kyrouac, J., Shi, Y., & Tuftedal, M. Surface Meteorological Instrumentation (MET), 2024-12-16 to 2024-12-19, Bankhead National Forest, AL, USA; Long-term Mobile Facility (BNF), Bankhead National Forest, AL, AMF3 (Main Site) (M1). Atmospheric Radiation Measurement (ARM) User Facility. https://doi.org/10.5439/1786358" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.4" } }, "nbformat": 4, "nbformat_minor": 5 }