Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Chapter 4: NOAA Multi-Radar / Multi-Sensor System (MRMS) at the BNF Field Site(s)

Authors
Affiliations
City College of New York and NOAA/OAR National Severe Storms Laboratory
NSF National Center for Atmospheric Research
University at Albany (State University of New York)
Metropolitan State University of Denver
Jackson State University
Argonne National Laboratory
ARM Logo

Chapter 4: NOAA Multi-Radar / Multi-Sensor System (MRMS) at the BNF Field Site(s)

Overview

We’ll go through the steps of:

  1. Define our region of verification sites

  2. Query and Load Data from MRMS Buckets on AWS

  3. Create a Multi-Panel Display of QPE for the different sites

Prerequisites

ConceptsImportanceNotes
Intro to CartopyNecessaryMapping and Tiles
Intro to XarrayNecessaryFamiliarity with metadata structure
  • Time to learn: 30 minutes

  • System requirements:

    • Any Operating System

    • At least 8 GB of RAM

Imports

import glob
import gzip
import tempfile
import warnings

import cartopy.crs as ccrs
import cfgrib
import cmweather
import fsspec
import geopandas as gpd
import numpy as np
import pandas as pd
import xarray as xr
from cartopy import feature as cfeature
from cartopy.io.img_tiles import OSM
from matplotlib import pyplot as plt
from matplotlib.transforms import offset_copy
from metpy.plots import USCOUNTIES

# To ignore all RuntimeWarnings globally
warnings.filterwarnings("ignore", category=RuntimeWarning)

Hourly QPE BNF Mosaic

The NOAA Multi-Radar / Multi-Sensor System (MRMS) was created to produce products of preciptiation impacts on transportation and aviation.

Using the NOAA MRMS AWS Bucket, this notebook details creation of quicklooks to investigate a Quantitative Preciptiation Estimates (QPE) for the U.S. Department of Energy Atmospheric Radiation Measurement (ARM) AMF-3 Deployment to Bankhead National Forest.

We start first with a dictionary containing our sites of interest, these are located Southwest of Decatur, Alabama.

More about the BNF Site can be found on the ARM Website.

global_sites = {
    "M1": [34.34525, -87.33842],
    "S4": [34.46451, -87.23598],
    "S3": [34.63080, -87.13311],
    "S20": [34.65401, -87.29264],
    "S30": [34.38501, -86.92757],
    "S40": [34.17932, -87.45349],
}


# Define a domain to set the extent of the figures
bnf_domain = [272.0, 274.0, 34.1, 35.1]

Visualize the Site Locations Using Cartopy

fig = plt.figure(figsize=(12, 8))
ax = plt.subplot(projection=ccrs.PlateCarree())

tiler = OSM()
ax.add_image(tiler, 12, zorder=0, alpha=0.7)
for site in global_sites:
    ax.scatter(global_sites[site][1], global_sites[site][0], label=site)

ax.set_extent(bnf_domain)

plt.legend(loc="upper right")
plt.title("ARM Bankhead National Forest Sites", fontsize=16);
<Figure size 1200x800 with 1 Axes>

Query and Load Data from MRMS Buckets on AWS

Note the Multi-Sensor (i.e. gauge adjusted) QPE product is split into two categories (Pass 1 and Pass 2), which defines the gauge latency used to adjust radar dervied QPE.

# Define a Date for Analysis [YYYYMMDD format]
DATE = "20250524"
HOUR = "000000"
## Setup the AWS S3 filesystem
fs = fsspec.filesystem("s3", anon=True)
s3_multi_bucket = [
    f"s3://noaa-mrms-pds/CONUS/MultiSensor_QPE_01H_Pass1_00.00/{DATE}/*.gz"
]
s3_pass2_bucket = [
    f"s3://noaa-mrms-pds/CONUS/MultiSensor_QPE_01H_Pass2_00.00/{DATE}/*.gz"
]
s3_radar_bucket = [
    f"s3://noaa-mrms-pds/CONUS/RadarOnly_QPE_01H_00.00/{DATE}/*[0-9]0000.grib2.gz"
]

Loop through and Create Lists of Datasets

Our next step is to search, access, and load our data into merged datasets, adding some additional metadata such as units. We apply this for each our our multipass, pass2, and radar datasets.

ds_multi_list = []
for scan in s3_multi_bucket:
    file_path = sorted(fs.glob(scan))
    for mrms in file_path:
        with fs.open(mrms, "rb") as gzip_file:
            with tempfile.NamedTemporaryFile(suffix=".grib2") as f:
                # Uncompress and read the file
                f.write(gzip.decompress(gzip_file.read()))
                ds = xr.load_dataset(f.name, decode_timedelta=False)
                # Parameters are stored as 'unknown'; meta data in filename
                ds = ds.rename({"unknown": "multisensor_qpe_1hr"})
                ds["multisensor_qpe_1hr"].attrs["units"] = "mm"
                ds["multisensor_qpe_1hr"].attrs[
                    "long_name"
                ] = "Precipitation Accumulation (1-Hr latency)"
                # Subset for the desired bounding box and take out all missing values
                ds = ds.sel(
                    latitude=slice(bnf_domain[3], bnf_domain[2]),
                    longitude=slice(bnf_domain[0], bnf_domain[1]),
                ).where(ds.multisensor_qpe_1hr > 0)
                ds_multi_list.append(ds)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[7], line 3
      1 ds_multi_list = []
      2 for scan in s3_multi_bucket:
----> 3     file_path = sorted(fs.glob(scan))
      4     for mrms in file_path:
      5         with fs.open(mrms, "rb") as gzip_file:
      6             with tempfile.NamedTemporaryFile(suffix=".grib2") as f:

File ~/micromamba/envs/mrms-cookbook-dev/lib/python3.12/site-packages/fsspec/asyn.py:118, in sync_wrapper.<locals>.wrapper(*args, **kwargs)
    115 @functools.wraps(func)
    116 def wrapper(*args, **kwargs):
    117     self = obj or args[0]
--> 118     return sync(self.loop, func, *args, **kwargs)

File ~/micromamba/envs/mrms-cookbook-dev/lib/python3.12/site-packages/fsspec/asyn.py:103, in sync(loop, func, timeout, *args, **kwargs)
    101     raise FSTimeoutError from return_result
    102 elif isinstance(return_result, BaseException):
--> 103     raise return_result
    104 else:
    105     return return_result

File ~/micromamba/envs/mrms-cookbook-dev/lib/python3.12/site-packages/fsspec/asyn.py:56, in _runner(event, coro, result, timeout)
     54     coro = asyncio.wait_for(coro, timeout=timeout)
     55 try:
---> 56     result[0] = await coro
     57 except Exception as ex:
     58     result[0] = ex

File ~/micromamba/envs/mrms-cookbook-dev/lib/python3.12/site-packages/fsspec/asyn.py:874, in AsyncFileSystem._glob(self, path, maxdepth, **kwargs)
    872 if prefix:
    873     kwargs["prefix"] = prefix
--> 874 allpaths = await self._find(
    875     root, maxdepth=depth, withdirs=withdirs, detail=True, **kwargs
    876 )
    878 pattern = glob_translate(path + ("/" if ends_with_sep else ""))
    879 pattern = re.compile(pattern)

File ~/micromamba/envs/mrms-cookbook-dev/lib/python3.12/site-packages/s3fs/core.py:558, in S3FileSystem._find(self, path, maxdepth, withdirs, detail)
    556     raise ValueError("Cannot traverse all of S3")
    557 if maxdepth:
--> 558     return await super()._find(
    559         bucket + "/" + key, maxdepth=maxdepth, withdirs=withdirs, detail=detail
    560     )
    561 # TODO: implement find from dircache, if all listings are present
    562 # if refresh is False:
    563 #     out = incomplete_tree_dirs(self.dircache, path)
   (...)    568 #         return super().find(path)
    569 #     # else: we refresh anyway, having at least two missing trees
    570 out = await self._lsdir(path, delimiter="")

File ~/micromamba/envs/mrms-cookbook-dev/lib/python3.12/site-packages/fsspec/asyn.py:914, in AsyncFileSystem._find(self, path, maxdepth, withdirs, **kwargs)
    910 detail = kwargs.pop("detail", False)
    912 # Add the root directory if withdirs is requested
    913 # This is needed for posix glob compliance
--> 914 if withdirs and path != "" and await self._isdir(path):
    915     out[path] = await self._info(path)
    917 # async for?

File ~/micromamba/envs/mrms-cookbook-dev/lib/python3.12/site-packages/s3fs/core.py:1021, in S3FileSystem._isdir(self, path)
   1018     return False
   1020 # This only returns things within the path and NOT the path object itself
-> 1021 return bool(await self._lsdir(path))

File ~/micromamba/envs/mrms-cookbook-dev/lib/python3.12/site-packages/s3fs/core.py:509, in S3FileSystem._lsdir(self, path, refresh, max_items, delimiter)
    507 try:
    508     logger.debug("Get directory listing page for %s" % path)
--> 509     await self._connect()
    510     pag = self.s3.get_paginator("list_objects_v2")
    511     config = {}

File ~/micromamba/envs/mrms-cookbook-dev/lib/python3.12/site-packages/s3fs/core.py:365, in S3FileSystem._connect(self, refresh, kwargs)
    363     config_kwargs["signature_version"] = UNSIGNED
    364 conf = AioConfig(**config_kwargs)
--> 365 self.session = aiobotocore.AioSession(**self.kwargs)
    366 s3creator = self.session.create_client(
    367     "s3", config=conf, **init_kwargs, **client_kwargs
    368 )
    369 self._s3creator = s3creator

AttributeError: module 'aiobotocore' has no attribute 'AioSession'
ds_radar_list = []
for scan in s3_radar_bucket:
    file_path = sorted(fs.glob(scan))
    for mrms in file_path:
        with fs.open(mrms, "rb") as gzip_file:
            with tempfile.NamedTemporaryFile(suffix=".grib2") as f:
                # Uncompress and read the file
                f.write(gzip.decompress(gzip_file.read()))
                ds = xr.load_dataset(f.name, decode_timedelta=False)
                ds = ds.rename({"unknown": "radar_qpe_1hr"})
                ds["radar_qpe_1hr"].attrs["units"] = "mm"
                ds["radar_qpe_1hr"].attrs["long_name"] = "Precipitation Accumulation"
                # Subset for the desired bounding box and take out all missing values
                ds = ds.sel(
                    latitude=slice(bnf_domain[3], bnf_domain[2]),
                    longitude=slice(bnf_domain[0], bnf_domain[1]),
                ).where(ds.radar_qpe_1hr > 0)
                ds_radar_list.append(ds)
ds_pass2_list = []
for scan in s3_pass2_bucket:
    file_path = sorted(fs.glob(scan))
    for mrms in file_path:
        with fs.open(mrms, "rb") as gzip_file:
            with tempfile.NamedTemporaryFile(suffix=".grib2") as f:
                # Uncompress and read the file
                f.write(gzip.decompress(gzip_file.read()))
                ds = xr.load_dataset(f.name, decode_timedelta=False)
                ds = ds.rename({"unknown": "multisensor_qpe_pass2"})
                ds["multisensor_qpe_pass2"].attrs["units"] = "mm"
                ds["multisensor_qpe_pass2"].attrs[
                    "long_name"
                ] = "Precipitation Accumulation (2-Hr latency)"
                # Subset for the desired bounding box and take out all missing values
                ds = ds.sel(
                    latitude=slice(bnf_domain[3], bnf_domain[2]),
                    longitude=slice(bnf_domain[0], bnf_domain[1]),
                ).where(ds.multisensor_qpe_pass2 > 0)
                ds_pass2_list.append(ds)

Merge our Files Together

Once we have lists of files, we can merge based on the time dimension.

# Concatenate all hourly files into xarray datasets
ds_radar_merged = xr.concat(ds_radar_list, dim="time")
ds_multi_merged = xr.concat(ds_multi_list, dim="time")
ds_pass2_merged = xr.concat(ds_pass2_list, dim="time")

And finally merge our various passes and QPE data into one single dataset.

# Merge Radar, Multi-Sensor Pass 1 and Multi-Sensor Pass 2 QPE into single dataset
ds_merged = xr.merge([ds_radar_merged, ds_multi_merged, ds_pass2_merged])

Calculate Precipitation Accumulation

Our last step is to calculate our preciptiation accumulation as observed from radar data. We do this by using the xarray cumulative sum cumsum function.

# Calculate the Cumulative Distribution
radar_cumulative = ds_merged["radar_qpe_1hr"].cumsum(dim="time")
multisensor = ds_merged["multisensor_qpe_1hr"].cumsum(dim="time")
multisensor_pass2 = ds_merged["multisensor_qpe_pass2"].cumsum(dim="time")

ds_merged["cumulative_radar_qpe"] = radar_cumulative
ds_merged["cumulative_radar_qpe"].attrs["units"] = "mm"
ds_merged["cumulative_radar_qpe"].attrs["long_name"] = "Precipitation Accumulation"

ds_merged["cumulative_multisensor"] = multisensor
ds_merged["cumulative_multisensor"].attrs["units"] = "mm"
ds_merged["cumulative_multisensor"].attrs[
    "long_name"
] = "Precipitation Accumulation (1-Hr latency)"

ds_merged["cumulative_ms_pass2"] = multisensor_pass2
ds_merged["cumulative_ms_pass2"].attrs["units"] = "mm"
ds_merged["cumulative_ms_pass2"].attrs[
    "long_name"
] = "Precipitation Accumulation (2-Hr latency)"
ds_merged

Create a Multi-Panel QPE Display

Now that we have our merged, cleaned data, we can create a single graphic summarizing the cumulative precipitation at our different sites. We use nearest neighbor here to subset from the broader region.

# ---------------------------------------------------
# Define the Figure for Detailed Subplot Placement
# ---------------------------------------------------
fig = plt.figure(figsize=(24, 10))
tiler = OSM()
mercator = tiler.crs
ax = fig.add_subplot(1, 3, 1, projection=ccrs.PlateCarree())

# adjust the subplot widths
plt.subplots_adjust(wspace=0.3)

# Find the maximum value at each position
da_max = ds_merged.isel(time=-1).radar_qpe_1hr.max()

# Find the minimum value at each position
da_min = 0

# ---------------------------------------------
# Display the Radar Precipitation Accumulation
# ---------------------------------------------

## subset the data
ds_merged.isel(time=15).radar_qpe_1hr.plot(
    transform=ccrs.PlateCarree(),
    ax=ax,
    cmap="ChaseSpectral",
    vmin=da_min,
    vmax=da_max,
    cbar_kwargs={"location": "bottom"},
)

# Add some various map elements to the plot to make it recognizable.
ax.add_feature(cfeature.LAND)
ax.add_feature(cfeature.OCEAN)
ax.add_feature(cfeature.BORDERS)
ax.add_image(tiler, 12, zorder=1, alpha=0.55)
ax.gridlines(draw_labels=True)

# Set plot bounds
ax.set_extent(bnf_domain)

# add in crosshairs to indicate the lat/lon slices
ax.axhline(y=global_sites["M1"][0], color="black", linestyle="--")
ax.axvline(x=global_sites["M1"][1], color="red", linestyle="--")

# Display the location of the BNF supplementarly sites
for key in global_sites:
    # Add a marker for the BNF sites.
    ax.plot(
        global_sites[key][1],
        global_sites[key][0],
        marker="o",
        color="black",
        markersize=10,
        alpha=0.7,
        transform=ccrs.PlateCarree(),
    )

    # Use the cartopy interface to create a matplotlib transform object
    # for the Geodetic coordinate system. We will use this along with
    # matplotlib's offset_copy function to define a coordinate system which
    # translates the text by 25 pixels to the left.
    geodetic_transform = ccrs.PlateCarree()._as_mpl_transform(ax)
    text_transform = offset_copy(geodetic_transform, units="dots", x=+50, y=+15)

    # Add text to the right of the symbol.
    ax.text(
        global_sites[key][1] - 0.1,
        global_sites[key][0],
        key,
        verticalalignment="center",
        horizontalalignment="right",
        transform=text_transform,
        bbox=dict(facecolor="sandybrown", alpha=0.5, boxstyle="round"),
    )

# update the title of the display
ax.set_title(
    np.datetime_as_string(ds_merged["valid_time"].isel(time=15).data, unit="s").replace(
        "T", " - "
    )
    + "Z\n"
    + "Radar Derived 1-Hr QPE - MRMS"
)

# ----------------------------
# Display the Multisensor QPE
# ----------------------------
## subset the data
ax1 = fig.add_subplot(1, 3, 2, projection=ccrs.PlateCarree())
ds_merged.isel(time=15).multisensor_qpe_1hr.plot(
    transform=ccrs.PlateCarree(),
    ax=ax1,
    cmap="ChaseSpectral",
    vmin=da_min,
    vmax=da_max,
    cbar_kwargs={"location": "bottom"},
)

# Add some various map elements to the plot to make it recognizable.
ax1.add_feature(cfeature.LAND)
ax1.add_feature(cfeature.OCEAN)
ax1.add_feature(cfeature.BORDERS)
ax1.add_image(tiler, 12, zorder=1, alpha=0.55)
ax1.gridlines(draw_labels=True)

# Set plot bounds
ax1.set_extent(bnf_domain)

# add in crosshairs to indicate the lat/lon slices
ax1.axhline(y=global_sites["M1"][0], color="black", linestyle="--")
ax1.axvline(x=global_sites["M1"][1], color="red", linestyle="--")

# Display the location of the BNF Supplementary Site
for key in global_sites:
    # Add a marker for the BNF sites.
    ax1.plot(
        global_sites[key][1],
        global_sites[key][0],
        marker="o",
        color="black",
        markersize=10,
        alpha=0.7,
        transform=ccrs.PlateCarree(),
    )

    # Use the cartopy interface to create a matplotlib transform object
    # for the Geodetic coordinate system. We will use this along with
    # matplotlib's offset_copy function to define a coordinate system which
    # translates the text by 25 pixels to the left.
    geodetic_transform = ccrs.PlateCarree()._as_mpl_transform(ax1)
    text_transform = offset_copy(geodetic_transform, units="dots", x=+50, y=+15)

    # Add text to the right of the site marker.
    ax1.text(
        global_sites[key][1] - 0.1,
        global_sites[key][0],
        key,
        verticalalignment="center",
        horizontalalignment="right",
        transform=text_transform,
        bbox=dict(facecolor="sandybrown", alpha=0.5, boxstyle="round"),
    )

# update the title of the display
ax1.set_title(
    np.datetime_as_string(ds_merged["valid_time"].isel(time=15).data, unit="s").replace(
        "T", " - "
    )
    + "Z\n"
    + "Multisensor 1-Hr QPE - Pass 1"
)

# ----------------------------
# Display the QPE Difference
# ----------------------------
## subset the data
ax3 = fig.add_subplot(1, 3, 3, projection=ccrs.PlateCarree())
ds_merged.isel(time=15).multisensor_qpe_pass2.plot(
    transform=ccrs.PlateCarree(),
    ax=ax3,
    cmap="ChaseSpectral",
    vmin=da_min,
    vmax=da_max,
    cbar_kwargs={"location": "bottom"},
)

# Add some various map elements to the plot to make it recognizable.
ax3.add_feature(cfeature.LAND)
ax3.add_feature(cfeature.OCEAN)
ax3.add_feature(cfeature.BORDERS)
ax3.add_image(tiler, 12, zorder=1, alpha=0.55)
ax3.gridlines(draw_labels=True)

# Set plot bounds
ax3.set_extent(bnf_domain)

# add in crosshairs to indicate the lat/lon slices
ax3.axhline(y=global_sites["M1"][0], color="black", linestyle="--")
ax3.axvline(x=global_sites["M1"][1], color="red", linestyle="--")

# Display the location of the BNF Supplementary Sites
for key in global_sites:
    # Add a marker for the BNF sites.
    ax3.plot(
        global_sites[key][1],
        global_sites[key][0],
        marker="o",
        color="black",
        markersize=10,
        alpha=0.7,
        transform=ccrs.PlateCarree(),
    )

    # Use the cartopy interface to create a matplotlib transform object
    # for the Geodetic coordinate system. We will use this along with
    # matplotlib's offset_copy function to define a coordinate system which
    # translates the text by 25 pixels to the left.
    geodetic_transform = ccrs.PlateCarree()._as_mpl_transform(ax3)
    text_transform = offset_copy(geodetic_transform, units="dots", x=+50, y=+15)

    # Add text to the right of the site marker.
    ax3.text(
        global_sites[key][1] - 0.1,
        global_sites[key][0],
        key,
        verticalalignment="center",
        horizontalalignment="right",
        transform=text_transform,
        bbox=dict(facecolor="sandybrown", alpha=0.5, boxstyle="round"),
    )

# update the title of the display
ax3.set_title(
    np.datetime_as_string(ds_merged["valid_time"].isel(time=15).data, unit="s").replace(
        "T", " - "
    )
    + "Z\n"
    + "MultiSensor 24-Hr QPE - Pass 2"
);

Summary

Within this notebook, we explored plotting a set of a field sites, accessing MRMS data, and visualizing a case over the ARM DOE Bankhead National Forest field site. We hope this serves as a framework for verification and understanding precipitation values in specific regions of interest.

What’s Next

We can extend this workflow by investigating timeseries for the various sites and looking into more robust verification techniques.

References