Enhanced Intake-ESM Catalog Demo - CESM LENS on AWS Cookbook

Overview¶

This notebook compares one Intake-ESM catalog with an enhanced version that includes additional attributes. Both catalogs are an inventory of the NCAR Community Earth System Model (CESM) Large Ensemble (LENS) data hosted on AWS S3.

Prerequisites¶

Concepts	Importance	Notes
Intro to Pandas	Necessary

Time to learn: 10 minutes

Imports¶

import intake
import pandas as pd

Original Intake-ESM Catalog¶

At import time, the intake-esm plugin is available in intake’s registry as esm_datastore and can be accessed with intake.open_esm_datastore() function.

cat_url_orig = 'https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le.json'
coll_orig = intake.open_esm_datastore(cat_url_orig)
coll_orig

Here’s a summary representation:

print(coll_orig)

<aws-cesm1-le catalog with 56 dataset(s) from 442 asset(s)>

In an Intake-ESM catalog object, the esmcat class provides many useful attributes and functions. For example, we can get the collection’s description:

coll_orig.esmcat.description

'This is an ESM collection for CESM1 Large Ensemble Zarr dataset publicly available on Amazon S3 (us-west-2 region)'

We can also get the URL pointing to the catalog’s underlying tabular representation:

coll_orig.esmcat.catalog_file

'https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le.csv'

That’s a CSV file ... let’s take a peek.

df_orig = pd.read_csv(coll_orig.esmcat.catalog_file)
df_orig

However, we can save a step since an ESM catalog object provides a df instance which returns a dataframe too:

df_orig = coll_orig.df
df_orig

Print out a sorted list of the unique values of selected columns

for col in ['component', 'frequency', 'experiment', 'variable']:
    unique_vals = coll_orig.unique()[col]
    unique_vals.sort()
    count = len(unique_vals)
    print (col + ': ' ,unique_vals, " count: ", count, '\n')

component:  ['atm', 'ice_nh', 'ice_sh', 'lnd', 'ocn']  count:  5 

frequency:  ['daily', 'hourly6-1990-2005', 'hourly6-2026-2035', 'hourly6-2071-2080', 'monthly', 'static']  count:  6 

experiment:  ['20C', 'CTRL', 'HIST', 'RCP85']  count:  4 

variable:  ['DIC', 'DOC', 'FLNS', 'FLNSC', 'FLUT', 'FSNO', 'FSNS', 'FSNSC', 'FSNTOA', 'FW', 'H2OSNO', 'HMXL', 'ICEFRAC', 'LHFLX', 'O2', 'PD', 'PRECC', 'PRECL', 'PRECSC', 'PRECSL', 'PRECT', 'PRECTMX', 'PS', 'PSL', 'Q', 'Q850', 'QBOT', 'QFLUX', 'QREFHT', 'QRUNOFF', 'QSW_HBL', 'QSW_HTP', 'RAIN', 'RESID_S', 'SALT', 'SFWF', 'SFWF_WRST', 'SHF', 'SHFLX', 'SHF_QSW', 'SNOW', 'SOILLIQ', 'SOILWATER_10CM', 'SSH', 'SST', 'T', 'TAUX', 'TAUX2', 'TAUY', 'TAUY2', 'TEMP', 'TMQ', 'TREFHT', 'TREFHTMN', 'TREFHTMX', 'TREFMNAV_U', 'TREFMXAV_U', 'TS', 'U', 'UBOT', 'UES', 'UET', 'UVEL', 'V', 'VBOT', 'VNS', 'VNT', 'VVEL', 'WSPDSRFAV', 'WTS', 'WTT', 'WVEL', 'Z3', 'Z500', 'aice', 'aice_d', 'hi', 'hi_d']  count:  78

Finding Data¶

If you happen to know the meaning of the variable names, you can find what data are available for that variable. For example:

df = coll_orig.search(variable='FLNS').df
df

We can narrow the filter to specific frequency and experiment:

df = coll_orig.search(variable='FLNS', frequency='daily', experiment='RCP85').df
df

The problem: Do all potential users know that `FLNS` is a CESM-specific abbreviation for "net longwave flux at surface”? How would a novice user find out, other than by finding separate documentation, or by opening a Zarr store in the hopes that the long name might be recorded there? How do we address the fact that every climate model code seems to have a different, non-standard name for all the variables, thus making multi-source research needlessly difficult?

The solution:

Enhanced Intake-ESM Catalog!¶

By adding additional columns to the Intake-ESM catalog, we should be able to improve semantic interoperability and provide potentially useful information to the users. Let’s now open the enhanced collection description file:

Note: The URL for the enhanced catalog differs from the original only in that it has -enhanced appended to aws-cesm1-le

cat_url = 'https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le-enhanced.json'
coll = intake.open_esm_datastore(cat_url)
coll

As we did for the first catalog, let’s obtain the description and catalog_file attributes.

print(coll.esmcat.description) # Description of collection
print("Catalog file:", coll.esmcat.catalog_file)
print(coll) # Summary of collection structure

This is an inventory of the Community Earth System Model (CESM) Large Ensemble (LENS) dataset in Zarr format publicly available on Amazon S3 (https://doi.org/10.26024/wt24-5j82)
Catalog file: https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le-enhanced.csv
<aws-cesm1-le catalog with 27 dataset(s) from 365 asset(s)>

Long names¶

In the catalog’s representation above, note the addition of additional elements: long_name, start, end, and dim. Here are the first/last few lines of the enhanced catalog:

df_enh = coll.df
df_enh

Warning

The long_names are not CF Standard Names, but rather are those documented at the NCAR LENS website. For interoperability, the long_name column should be replaced by a cf_name column and possibly an attribute column to disambiguate if needed.

List all available variables by long name, sorted alphabetically:

nameList = coll.unique()['long_name']
nameList.sort()
print(*nameList, sep='\n')

Clearsky net longwave flux at surface
Clearsky net solar flux at surface
Convective precipitation rate (liq + ice)
Convective snow rate (water equivalent)
Dissolved Inorganic Carbon
Dissolved Organic Carbon
Dissolved Oxygen
Fraction of sfc area covered by sea-ice
Free-Surface Residual Heat Flux
Free-Surface Residual Salt Flux
Freshwater Flux
Geopotential Height (above sea level)
Geopotential Z at 500 mbar pressure surface
Heat Flux across top face
Heat Flux in grid-x direction
Heat Flux in grid-y direction
Horizontal total wind speed average at the surface
Internal Ocean Heat Flux Due to Ice Formation
Large-scale (stable) precipitation rate (liq + ice)
Large-scale (stable) snow rate (water equivalent)
Lowest model level zonal wind
Maximum (convective and large-scale) precipitation rate (liq+ice)
Maximum reference height temperature over output period
Meridional wind
Minimum reference height temperature over output period
Mixed-Layer Depth
Net longwave flux at surface
Net solar flux at surface
Net solar flux at top of atmosphere
Potential Density Ref to Surface
Potential Temperature
Reference height humidity
Reference height temperature
Salinity
Salt Flux across top face
Salt Flux in grid-x direction
Salt Flux in grid-y direction
Sea Surface Height
Sea level pressure
Solar Short-Wave Heat Flux
Solar Short-Wave Heat Flux in boundary layer
Solar Short-Wave Heat Flux in top layer
Specific Humidity at 850 mbar pressure surface
Specific humidity
Surface latent heat flux
Surface pressure
Surface sensible heat flux
Surface temperature (radiative)
Temperature
Total (convective and large-scale) precipitation rate (liq + ice)
Total (vertically integrated) precipitable water
Total Surface Heat Flux including short-wave
Upwelling longwave flux at top of model
Velocity in grid-x direction
Velocity in grid-y direction
Vertical Velocity
Virtual Salt Flux due to weak restoring
Virtual Salt Flux in FW Flux formulation
Wind stress (squared) in grid-x direction
Wind stress (squared) in grid-y direction
Wind stress in grid-x direction
Wind stress in grid-y direction
Zonal wind
atmospheric rain
atmospheric snow
fraction of ground covered by snow
grid cellmean ice thickness
grid cellmean ice thickness (daily)
grid cellmean ice thickness (monthly)
ice area, aggregate (daily)
ice area, aggregate (monthly)
snow depth (liquid water)
soil liquid water (vegetated landunits only)
soil liquid water + ice in top 10cm of soil (veg landunits only)
total liquid runoff (does not include QSNWCPICE)

Search capabilities¶

We can use an intake-esm catalog object’s search function in several ways:

Show all available data for a specific variable based on long name:

myName = 'Salinity'
df = coll.search(long_name=myName).df
df

Search based on multiple criteria:

df = coll.search(experiment=['20C','RCP85'], dim='3D', variable=['T','Q']).df
df

Substring matches¶

In some cases, you may not know the exact term to look for. For such cases, inkake-esm supports searching for substring matches. With use of wildcards and/or regular expressions, we can find all items with a particular substring in a given column. Let’s search for:

entries from experiment = ‘20C’
all entries whose variable long name contains wind

coll_subset = coll.search(experiment="20C", long_name="Wind*")

coll_subset.df

If we wanted to search for Wind and wind, we can take advantage of regular expression syntax to do so:

coll_subset = coll.search(experiment="20C" , long_name="[Ww]ind*")

coll_subset.df

Other attributes¶

Other columns in the enhanced catalog may be useful. For example, the dimensionality column enables us to list all data from the ocean component that is 3D.

df = coll.search(dim="3D",component="ocn").df
df

Spatiotemporal filtering¶

If there were both regional and global data available (e.g., LENS and NA-CORDEX data for the same variable, both listed in same catalog), some type of coverage indicator (or columns for bounding box edges) could be listed.

Temporal extent in LENS is conveyed by the experiment (HIST, 20C, etc) but this is imprecise and requires external documentation. We have added start/end columns to the catalog, but Intake-ESM currently does not have built-in functionality to filter based on time.

We can do a simple search that exactly matches a temporal value:

df = coll.search(dim="3D",component="ocn", end='2100-12').df
df

Summary¶

In this notebook, we used Intake-ESM to explore a catalog of CESM LENS data. We then worked through some helpful features of the enhanced catalog.

What’s next?¶

We will use this data to recreate some figures from a paper published in BAMS that describes the CESM LENS project: Kay et al. (2015)

References¶

de la Beaujardière, J., Banihirwe, A., Shi, C.-F., Paul, K., & Hamman, J. (2019). NCAR CESM LENS Cloud-Optimized Subset. UCAR/NCAR - Computational. 10.26024/WT24-5J82
Kay, J. E., Deser, C., Phillips, A., Mai, A., Hannay, C., Strand, G., Arblaster, J. M., Bates, S. C., Danabasoglu, G., Edwards, J., Holland, M., Kushner, P., Lamarque, J.-F., Lawrence, D., Lindsday, K., Middleton, A., Munoz, E., Neale, R., Oleson, K., … Vertenstein, M. (2015). The Community Earth System Model (CESM) Large Ensemble Project. Bull. Amer. Meteor. Soc. 10.1175/BAMS-D-13-00255.1

Preamble

How to Cite This Cookbook

Example workflows

Reproducing Key Figures from Kay et al. (2015)