Enhanced Intake-ESM Catalog Demo
Overview
This notebook compares one Intake-ESM catalog with an enhanced version that includes additional attributes. Both catalogs are an inventory of the NCAR Community Earth System Model (CESM) Large Ensemble (LENS) data hosted on AWS S3 (doi:10.26024/wt24-5j82).
Prerequisites
Concepts |
Importance |
Notes |
|---|---|---|
Necessary |
Time to learn: 10 minutes
Imports
import intake
import pandas as pd
Original Intake-ESM Catalog
At import time, the intake-esm plugin is available in intake’s registry as esm_datastore and can be accessed with intake.open_esm_datastore() function.
cat_url_orig = 'https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le.json'
coll_orig = intake.open_esm_datastore(cat_url_orig)
coll_orig
aws-cesm1-le catalog with 56 dataset(s) from 442 asset(s):
| unique | |
|---|---|
| variable | 78 |
| long_name | 75 |
| component | 5 |
| experiment | 4 |
| frequency | 6 |
| vertical_levels | 3 |
| spatial_domain | 5 |
| units | 25 |
| start_time | 12 |
| end_time | 13 |
| path | 427 |
| derived_variable | 0 |
Here’s a summary representation:
print(coll_orig)
<aws-cesm1-le catalog with 56 dataset(s) from 442 asset(s)>
In an Intake-ESM catalog object, the esmcat class provides many useful attributes and functions. For example, we can get the collection’s description:
coll_orig.esmcat.description
'This is an ESM collection for CESM1 Large Ensemble Zarr dataset publicly available on Amazon S3 (us-west-2 region)'
We can also get the URL pointing to the catalog’s underlying tabular representation:
coll_orig.esmcat.catalog_file
'https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le.csv'
That’s a CSV file … let’s take a peek.
df_orig = pd.read_csv(coll_orig.esmcat.catalog_file)
df_orig
| variable | long_name | component | experiment | frequency | vertical_levels | spatial_domain | units | start_time | end_time | path | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | FLNS | net longwave flux at surface | atm | 20C | daily | 1.0 | global | W/m2 | 1920-01-01 12:00:00 | 2005-12-31 12:00:00 | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLNS.... |
| 1 | FLNSC | clearsky net longwave flux at surface | atm | 20C | daily | 1.0 | global | W/m2 | 1920-01-01 12:00:00 | 2005-12-31 12:00:00 | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLNSC... |
| 2 | FLUT | upwelling longwave flux at top of model | atm | 20C | daily | 1.0 | global | W/m2 | 1920-01-01 12:00:00 | 2005-12-31 12:00:00 | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLUT.... |
| 3 | FSNS | net solar flux at surface | atm | 20C | daily | 1.0 | global | W/m2 | 1920-01-01 12:00:00 | 2005-12-31 12:00:00 | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FSNS.... |
| 4 | FSNSC | clearsky net solar flux at surface | atm | 20C | daily | 1.0 | global | W/m2 | 1920-01-01 12:00:00 | 2005-12-31 12:00:00 | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FSNSC... |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 437 | WVEL | vertical velocity | ocn | RCP85 | monthly | 60.0 | global_ocean | centimeter/s | 2006-01-16 12:00:00 | 2100-12-16 12:00:00 | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-W... |
| 438 | NaN | NaN | ocn | CTRL | static | NaN | global_ocean | NaN | NaN | NaN | s3://ncar-cesm-lens/ocn/static/grid.zarr |
| 439 | NaN | NaN | ocn | HIST | static | NaN | global_ocean | NaN | NaN | NaN | s3://ncar-cesm-lens/ocn/static/grid.zarr |
| 440 | NaN | NaN | ocn | RCP85 | static | NaN | global_ocean | NaN | NaN | NaN | s3://ncar-cesm-lens/ocn/static/grid.zarr |
| 441 | NaN | NaN | ocn | 20C | static | NaN | global_ocean | NaN | NaN | NaN | s3://ncar-cesm-lens/ocn/static/grid.zarr |
442 rows × 11 columns
However, we can save a step since an ESM catalog object provides a df instance which returns a dataframe too:
df_orig = coll_orig.df
df_orig
| variable | long_name | component | experiment | frequency | vertical_levels | spatial_domain | units | start_time | end_time | path | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | FLNS | net longwave flux at surface | atm | 20C | daily | 1.0 | global | W/m2 | 1920-01-01 12:00:00 | 2005-12-31 12:00:00 | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLNS.... |
| 1 | FLNSC | clearsky net longwave flux at surface | atm | 20C | daily | 1.0 | global | W/m2 | 1920-01-01 12:00:00 | 2005-12-31 12:00:00 | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLNSC... |
| 2 | FLUT | upwelling longwave flux at top of model | atm | 20C | daily | 1.0 | global | W/m2 | 1920-01-01 12:00:00 | 2005-12-31 12:00:00 | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLUT.... |
| 3 | FSNS | net solar flux at surface | atm | 20C | daily | 1.0 | global | W/m2 | 1920-01-01 12:00:00 | 2005-12-31 12:00:00 | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FSNS.... |
| 4 | FSNSC | clearsky net solar flux at surface | atm | 20C | daily | 1.0 | global | W/m2 | 1920-01-01 12:00:00 | 2005-12-31 12:00:00 | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FSNSC... |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 437 | WVEL | vertical velocity | ocn | RCP85 | monthly | 60.0 | global_ocean | centimeter/s | 2006-01-16 12:00:00 | 2100-12-16 12:00:00 | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-W... |
| 438 | NaN | NaN | ocn | CTRL | static | NaN | global_ocean | NaN | NaN | NaN | s3://ncar-cesm-lens/ocn/static/grid.zarr |
| 439 | NaN | NaN | ocn | HIST | static | NaN | global_ocean | NaN | NaN | NaN | s3://ncar-cesm-lens/ocn/static/grid.zarr |
| 440 | NaN | NaN | ocn | RCP85 | static | NaN | global_ocean | NaN | NaN | NaN | s3://ncar-cesm-lens/ocn/static/grid.zarr |
| 441 | NaN | NaN | ocn | 20C | static | NaN | global_ocean | NaN | NaN | NaN | s3://ncar-cesm-lens/ocn/static/grid.zarr |
442 rows × 11 columns
Print out a sorted list of the unique values of selected columns
for col in ['component', 'frequency', 'experiment', 'variable']:
unique_vals = coll_orig.unique()[col]
unique_vals.sort()
count = len(unique_vals)
print (col + ': ' ,unique_vals, " count: ", count, '\n')
component: ['atm', 'ice_nh', 'ice_sh', 'lnd', 'ocn'] count: 5
frequency: ['daily', 'hourly6-1990-2005', 'hourly6-2026-2035', 'hourly6-2071-2080', 'monthly', 'static'] count: 6
experiment: ['20C', 'CTRL', 'HIST', 'RCP85'] count: 4
variable: ['DIC', 'DOC', 'FLNS', 'FLNSC', 'FLUT', 'FSNO', 'FSNS', 'FSNSC', 'FSNTOA', 'FW', 'H2OSNO', 'HMXL', 'ICEFRAC', 'LHFLX', 'O2', 'PD', 'PRECC', 'PRECL', 'PRECSC', 'PRECSL', 'PRECT', 'PRECTMX', 'PS', 'PSL', 'Q', 'Q850', 'QBOT', 'QFLUX', 'QREFHT', 'QRUNOFF', 'QSW_HBL', 'QSW_HTP', 'RAIN', 'RESID_S', 'SALT', 'SFWF', 'SFWF_WRST', 'SHF', 'SHFLX', 'SHF_QSW', 'SNOW', 'SOILLIQ', 'SOILWATER_10CM', 'SSH', 'SST', 'T', 'TAUX', 'TAUX2', 'TAUY', 'TAUY2', 'TEMP', 'TMQ', 'TREFHT', 'TREFHTMN', 'TREFHTMX', 'TREFMNAV_U', 'TREFMXAV_U', 'TS', 'U', 'UBOT', 'UES', 'UET', 'UVEL', 'V', 'VBOT', 'VNS', 'VNT', 'VVEL', 'WSPDSRFAV', 'WTS', 'WTT', 'WVEL', 'Z3', 'Z500', 'aice', 'aice_d', 'hi', 'hi_d'] count: 78
Finding Data
If you happen to know the meaning of the variable names, you can find what data are available for that variable. For example:
df = coll_orig.search(variable='FLNS').df
df
| variable | long_name | component | experiment | frequency | vertical_levels | spatial_domain | units | start_time | end_time | path | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | FLNS | net longwave flux at surface | atm | 20C | daily | 1.0 | global | W/m2 | 1920-01-01 12:00:00 | 2005-12-31 12:00:00 | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLNS.... |
| 1 | FLNS | net longwave flux at surface | atm | CTRL | daily | 1.0 | global | W/m2 | 0402-01-01 12:00:00 | 2200-12-31 12:00:00 | s3://ncar-cesm-lens/atm/daily/cesmLE-CTRL-FLNS... |
| 2 | FLNS | net longwave flux at surface | atm | HIST | daily | 1.0 | global | W/m2 | 1850-01-01 12:00:00 | 1919-12-31 12:00:00 | s3://ncar-cesm-lens/atm/daily/cesmLE-HIST-FLNS... |
| 3 | FLNS | net longwave flux at surface | atm | RCP85 | daily | 1.0 | global | W/m2 | 2006-01-01 12:00:00 | 2100-12-31 12:00:00 | s3://ncar-cesm-lens/atm/daily/cesmLE-RCP85-FLN... |
| 4 | FLNS | net longwave flux at surface | atm | 20C | hourly6-1990-2005 | 1.0 | global | W/m2 | 1990-01-01 00:00:00 | 2006-01-01 00:00:00 | s3://ncar-cesm-lens/atm/hourly6-1990-2005/cesm... |
| 5 | FLNS | net longwave flux at surface | atm | RCP85 | hourly6-2026-2035 | 1.0 | global | W/m2 | 2026-01-01 00:00:00 | 2036-01-01 00:00:00 | s3://ncar-cesm-lens/atm/hourly6-2026-2035/cesm... |
| 6 | FLNS | net longwave flux at surface | atm | RCP85 | hourly6-2071-2080 | 1.0 | global | W/m2 | 2071-01-01 00:00:00 | 2081-01-01 00:00:00 | s3://ncar-cesm-lens/atm/hourly6-2071-2080/cesm... |
| 7 | FLNS | net longwave flux at surface | atm | 20C | monthly | 1.0 | global | W/m2 | 1920-01-16 12:00:00 | 2005-12-16 12:00:00 | s3://ncar-cesm-lens/atm/monthly/cesmLE-20C-FLN... |
| 8 | FLNS | net longwave flux at surface | atm | CTRL | monthly | 1.0 | global | W/m2 | 0400-01-16 12:00:00 | 2200-12-16 12:00:00 | s3://ncar-cesm-lens/atm/monthly/cesmLE-CTRL-FL... |
| 9 | FLNS | net longwave flux at surface | atm | HIST | monthly | 1.0 | global | W/m2 | 1850-01-16 12:00:00 | 1919-12-16 12:00:00 | s3://ncar-cesm-lens/atm/monthly/cesmLE-HIST-FL... |
| 10 | FLNS | net longwave flux at surface | atm | RCP85 | monthly | 1.0 | global | W/m2 | 2006-01-16 12:00:00 | 2100-12-16 12:00:00 | s3://ncar-cesm-lens/atm/monthly/cesmLE-RCP85-F... |
We can narrow the filter to specific frequency and experiment:
df = coll_orig.search(variable='FLNS', frequency='daily', experiment='RCP85').df
df
| variable | long_name | component | experiment | frequency | vertical_levels | spatial_domain | units | start_time | end_time | path | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | FLNS | net longwave flux at surface | atm | RCP85 | daily | 1.0 | global | W/m2 | 2006-01-01 12:00:00 | 2100-12-31 12:00:00 | s3://ncar-cesm-lens/atm/daily/cesmLE-RCP85-FLN... |
The problem: Do all potential users know that `FLNS` is a CESM-specific abbreviation for "net longwave flux at surface”? How would a novice user find out, other than by finding separate documentation, or by opening a Zarr store in the hopes that the long name might be recorded there? How do we address the fact that every climate model code seems to have a different, non-standard name for all the variables, thus making multi-source research needlessly difficult?
The solution:
Enhanced Intake-ESM Catalog!
By adding additional columns to the Intake-ESM catalog, we should be able to improve semantic interoperability and provide potentially useful information to the users. Let’s now open the enhanced collection description file:
Note: The URL for the enhanced catalog differs from the original only in that it has
-enhanced appended to aws-cesm1-lecat_url = 'https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le-enhanced.json'
coll = intake.open_esm_datastore(cat_url)
coll
aws-cesm1-le catalog with 27 dataset(s) from 365 asset(s):
| unique | |
|---|---|
| component | 5 |
| dim | 2 |
| frequency | 5 |
| experiment | 6 |
| start | 10 |
| end | 11 |
| variable | 75 |
| long_name | 75 |
| path | 365 |
| derived_variable | 0 |
As we did for the first catalog, let’s obtain the description and catalog_file attributes.
print(coll.esmcat.description) # Description of collection
print("Catalog file:", coll.esmcat.catalog_file)
print(coll) # Summary of collection structure
This is an inventory of the Community Earth System Model (CESM) Large Ensemble (LENS) dataset in Zarr format publicly available on Amazon S3 (https://doi.org/10.26024/wt24-5j82)
Catalog file: https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le-enhanced.csv
<aws-cesm1-le catalog with 27 dataset(s) from 365 asset(s)>
Long names
In the catalog’s representation above, note the addition of additional elements: long_name, start, end, and dim. Here are the first/last few lines of the enhanced catalog:
df_enh = coll.df
df_enh
| component | dim | frequency | experiment | start | end | variable | long_name | path | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | atm | 2D | monthly | HIST | 1850-01 | 1919-12 | FLNS | Net longwave flux at surface | s3://ncar-cesm-lens/atm/monthly/cesmLE-HIST-FL... |
| 1 | atm | 2D | monthly | 20C | 1920-01 | 2005-12 | FLNS | Net longwave flux at surface | s3://ncar-cesm-lens/atm/monthly/cesmLE-20C-FLN... |
| 2 | atm | 2D | daily | 20C | 1920-01-01 | 2005-12-31 | FLNS | Net longwave flux at surface | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLNS.... |
| 3 | atm | 2D | monthly | RCP85 | 2006-01 | 2100-12 | FLNS | Net longwave flux at surface | s3://ncar-cesm-lens/atm/monthly/cesmLE-RCP85-F... |
| 4 | atm | 2D | daily | RCP85 | 2006-01-01 | 2100-12-31 | FLNS | Net longwave flux at surface | s3://ncar-cesm-lens/atm/daily/cesmLE-RCP85-FLN... |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 360 | ice_sh | 2D | daily | RCP85 | 2006-01-01 | 2100-12-31 | aice_d | ice area, aggregate (daily) | s3://ncar-cesm-lens/ice_sh/daily/cesmLE-RCP85-... |
| 361 | ice_sh | 2D | monthly | 20C | 1920-01 | 2005-12 | hi | grid cellmean ice thickness | s3://ncar-cesm-lens/ice_sh/monthly/cesmLE-20C-... |
| 362 | ice_sh | 2D | monthly | RCP85 | 2006-01 | 2100-12 | hi | grid cellmean ice thickness | s3://ncar-cesm-lens/ice_sh/monthly/cesmLE-RCP8... |
| 363 | ice_sh | 2D | daily | 20C | 1920-01-01 | 2005-12-31 | hi_d | grid cellmean ice thickness | s3://ncar-cesm-lens/ice_sh/daily/cesmLE-20C-hi... |
| 364 | ice_sh | 2D | daily | RCP85 | 2006-01-01 | 2100-12-31 | hi_d | grid cellmean ice thickness | s3://ncar-cesm-lens/ice_sh/daily/cesmLE-RCP85-... |
365 rows × 9 columns
Warning
The long_names are notCF Standard Names, but rather are those documented at
the NCAR LENS website. For interoperability, the long_name column should be replaced by a cf_name column and possibly an attribute column to disambiguate if needed.
List all available variables by long name, sorted alphabetically:
nameList = coll.unique()['long_name']
nameList.sort()
print(*nameList, sep='\n')
Clearsky net longwave flux at surface
Clearsky net solar flux at surface
Convective precipitation rate (liq + ice)
Convective snow rate (water equivalent)
Dissolved Inorganic Carbon
Dissolved Organic Carbon
Dissolved Oxygen
Fraction of sfc area covered by sea-ice
Free-Surface Residual Heat Flux
Free-Surface Residual Salt Flux
Freshwater Flux
Geopotential Height (above sea level)
Geopotential Z at 500 mbar pressure surface
Heat Flux across top face
Heat Flux in grid-x direction
Heat Flux in grid-y direction
Horizontal total wind speed average at the surface
Internal Ocean Heat Flux Due to Ice Formation
Large-scale (stable) precipitation rate (liq + ice)
Large-scale (stable) snow rate (water equivalent)
Lowest model level zonal wind
Maximum (convective and large-scale) precipitation rate (liq+ice)
Maximum reference height temperature over output period
Meridional wind
Minimum reference height temperature over output period
Mixed-Layer Depth
Net longwave flux at surface
Net solar flux at surface
Net solar flux at top of atmosphere
Potential Density Ref to Surface
Potential Temperature
Reference height humidity
Reference height temperature
Salinity
Salt Flux across top face
Salt Flux in grid-x direction
Salt Flux in grid-y direction
Sea Surface Height
Sea level pressure
Solar Short-Wave Heat Flux
Solar Short-Wave Heat Flux in boundary layer
Solar Short-Wave Heat Flux in top layer
Specific Humidity at 850 mbar pressure surface
Specific humidity
Surface latent heat flux
Surface pressure
Surface sensible heat flux
Surface temperature (radiative)
Temperature
Total (convective and large-scale) precipitation rate (liq + ice)
Total (vertically integrated) precipitable water
Total Surface Heat Flux including short-wave
Upwelling longwave flux at top of model
Velocity in grid-x direction
Velocity in grid-y direction
Vertical Velocity
Virtual Salt Flux due to weak restoring
Virtual Salt Flux in FW Flux formulation
Wind stress (squared) in grid-x direction
Wind stress (squared) in grid-y direction
Wind stress in grid-x direction
Wind stress in grid-y direction
Zonal wind
atmospheric rain
atmospheric snow
fraction of ground covered by snow
grid cellmean ice thickness
grid cellmean ice thickness (daily)
grid cellmean ice thickness (monthly)
ice area, aggregate (daily)
ice area, aggregate (monthly)
snow depth (liquid water)
soil liquid water (vegetated landunits only)
soil liquid water + ice in top 10cm of soil (veg landunits only)
total liquid runoff (does not include QSNWCPICE)
Search capabilities
We can use an intake-esm catalog object’s search function in several ways:
Show all available data for a specific variable based on long name:
myName = 'Salinity'
df = coll.search(long_name=myName).df
df
| component | dim | frequency | experiment | start | end | variable | long_name | path | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | SALT | Salinity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-SAL... |
| 1 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | SALT | Salinity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-S... |
| 2 | ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | SALT | Salinity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-SA... |
Search based on multiple criteria:
df = coll.search(experiment=['20C','RCP85'], dim='3D', variable=['T','Q']).df
df
| component | dim | frequency | experiment | start | end | variable | long_name | path | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | atm | 3D | monthly | 20C | 1920-01 | 2005-12 | Q | Specific humidity | s3://ncar-cesm-lens/atm/monthly/cesmLE-20C-Q.zarr |
| 1 | atm | 3D | hourly6-1990-2005 | 20C | 1990-01-01T00 | 2005-12-31T18 | Q | Specific humidity | s3://ncar-cesm-lens/atm/hourly6-1990-2005/cesm... |
| 2 | atm | 3D | monthly | RCP85 | 2006-01 | 2100-12 | Q | Specific humidity | s3://ncar-cesm-lens/atm/monthly/cesmLE-RCP85-Q... |
| 3 | atm | 3D | hourly6-2026-2035 | RCP85 | 2026-01-01T00 | 2035-12-31T18 | Q | Specific humidity | s3://ncar-cesm-lens/atm/hourly6-2026-2035/cesm... |
| 4 | atm | 3D | hourly6-2071-2080 | RCP85 | 2071-01-01T00 | 2080-12-31T18 | Q | Specific humidity | s3://ncar-cesm-lens/atm/hourly6-2071-2080/cesm... |
| 5 | atm | 3D | monthly | 20C | 1920-01 | 2005-12 | T | Temperature | s3://ncar-cesm-lens/atm/monthly/cesmLE-20C-T.zarr |
| 6 | atm | 3D | hourly6-1990-2005 | 20C | 1990-01-01T00 | 2005-12-31T18 | T | Temperature | s3://ncar-cesm-lens/atm/hourly6-1990-2005/cesm... |
| 7 | atm | 3D | monthly | RCP85 | 2006-01 | 2100-12 | T | Temperature | s3://ncar-cesm-lens/atm/monthly/cesmLE-RCP85-T... |
| 8 | atm | 3D | hourly6-2026-2035 | RCP85 | 2026-01-01T00 | 2035-12-31T18 | T | Temperature | s3://ncar-cesm-lens/atm/hourly6-2026-2035/cesm... |
| 9 | atm | 3D | hourly6-2071-2080 | RCP85 | 2071-01-01T00 | 2080-12-31T18 | T | Temperature | s3://ncar-cesm-lens/atm/hourly6-2071-2080/cesm... |
Substring matches
In some cases, you may not know the exact term to look for. For such cases, inkake-esm supports searching for substring matches. With use of wildcards and/or regular expressions, we can find all items with a particular substring in a given column. Let’s search for:
entries from experiment = ‘20C’
all entries whose variable long name contains wind
coll_subset = coll.search(experiment="20C", long_name="Wind*")
coll_subset.df
| component | dim | frequency | experiment | start | end | variable | long_name | path | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | ocn | 2D | monthly | 20C | 1920-01 | 2005-12 | TAUX | Wind stress in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-TAU... |
| 1 | ocn | 2D | monthly | 20C | 1920-01 | 2005-12 | TAUX2 | Wind stress (squared) in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-TAU... |
| 2 | ocn | 2D | monthly | 20C | 1920-01 | 2005-12 | TAUY | Wind stress in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-TAU... |
| 3 | ocn | 2D | monthly | 20C | 1920-01 | 2005-12 | TAUY2 | Wind stress (squared) in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-TAU... |
If we wanted to search for Wind and wind, we can take advantage of regular expression syntax to do so:
coll_subset = coll.search(experiment="20C" , long_name="[Ww]ind*")
coll_subset.df
| component | dim | frequency | experiment | start | end | variable | long_name | path | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | atm | 3D | monthly | 20C | 1920-01 | 2005-12 | U | Zonal wind | s3://ncar-cesm-lens/atm/monthly/cesmLE-20C-U.zarr |
| 1 | atm | 3D | hourly6-1990-2005 | 20C | 1990-01-01T00 | 2005-12-31T18 | U | Zonal wind | s3://ncar-cesm-lens/atm/hourly6-1990-2005/cesm... |
| 2 | atm | 2D | daily | 20C | 1920-01-01 | 2005-12-31 | UBOT | Lowest model level zonal wind | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-UBOT.... |
| 3 | atm | 3D | monthly | 20C | 1920-01 | 2005-12 | V | Meridional wind | s3://ncar-cesm-lens/atm/monthly/cesmLE-20C-V.zarr |
| 4 | atm | 3D | hourly6-1990-2005 | 20C | 1990-01-01T00 | 2005-12-31T18 | V | Meridional wind | s3://ncar-cesm-lens/atm/hourly6-1990-2005/cesm... |
| 5 | atm | 2D | daily | 20C | 1920-01-01 | 2005-12-31 | WSPDSRFAV | Horizontal total wind speed average at the sur... | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-WSPDS... |
| 6 | ocn | 2D | monthly | 20C | 1920-01 | 2005-12 | TAUX | Wind stress in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-TAU... |
| 7 | ocn | 2D | monthly | 20C | 1920-01 | 2005-12 | TAUX2 | Wind stress (squared) in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-TAU... |
| 8 | ocn | 2D | monthly | 20C | 1920-01 | 2005-12 | TAUY | Wind stress in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-TAU... |
| 9 | ocn | 2D | monthly | 20C | 1920-01 | 2005-12 | TAUY2 | Wind stress (squared) in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-TAU... |
Other attributes
Other columns in the enhanced catalog may be useful. For example, the dimensionality column enables us to list all data from the ocean component that is 3D.
df = coll.search(dim="3D",component="ocn").df
df
| component | dim | frequency | experiment | start | end | variable | long_name | path | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | DIC | Dissolved Inorganic Carbon | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-DIC... |
| 1 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | DIC | Dissolved Inorganic Carbon | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-D... |
| 2 | ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | DIC | Dissolved Inorganic Carbon | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-DI... |
| 3 | ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | DOC | Dissolved Organic Carbon | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-DOC... |
| 4 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | DOC | Dissolved Organic Carbon | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-D... |
| 5 | ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | DOC | Dissolved Organic Carbon | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-DO... |
| 6 | ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | O2 | Dissolved Oxygen | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-O2.... |
| 7 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | O2 | Dissolved Oxygen | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-O... |
| 8 | ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | O2 | Dissolved Oxygen | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-O2... |
| 9 | ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | PD | Potential Density Ref to Surface | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-PD.... |
| 10 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | PD | Potential Density Ref to Surface | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-P... |
| 11 | ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | PD | Potential Density Ref to Surface | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-PD... |
| 12 | ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | SALT | Salinity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-SAL... |
| 13 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | SALT | Salinity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-S... |
| 14 | ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | SALT | Salinity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-SA... |
| 15 | ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | TEMP | Potential Temperature | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-TEM... |
| 16 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | TEMP | Potential Temperature | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-T... |
| 17 | ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | TEMP | Potential Temperature | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-TE... |
| 18 | ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | UES | Salt Flux in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-UES... |
| 19 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | UES | Salt Flux in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-U... |
| 20 | ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | UES | Salt Flux in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-UE... |
| 21 | ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | UET | Heat Flux in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-UET... |
| 22 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | UET | Heat Flux in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-U... |
| 23 | ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | UET | Heat Flux in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-UE... |
| 24 | ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | UVEL | Velocity in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-UVE... |
| 25 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | UVEL | Velocity in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-U... |
| 26 | ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | UVEL | Velocity in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-UV... |
| 27 | ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | VNS | Salt Flux in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-VNS... |
| 28 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | VNS | Salt Flux in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-V... |
| 29 | ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | VNS | Salt Flux in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-VN... |
| 30 | ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | VNT | Heat Flux in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-VNT... |
| 31 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | VNT | Heat Flux in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-V... |
| 32 | ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | VNT | Heat Flux in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-VN... |
| 33 | ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | VVEL | Velocity in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-VVE... |
| 34 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | VVEL | Velocity in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-V... |
| 35 | ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | VVEL | Velocity in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-VV... |
| 36 | ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | WTS | Salt Flux across top face | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-WTS... |
| 37 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | WTS | Salt Flux across top face | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-W... |
| 38 | ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | WTS | Salt Flux across top face | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-WT... |
| 39 | ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | WTT | Heat Flux across top face | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-WTT... |
| 40 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | WTT | Heat Flux across top face | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-W... |
| 41 | ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | WTT | Heat Flux across top face | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-WT... |
| 42 | ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | WVEL | Vertical Velocity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-WVE... |
| 43 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | WVEL | Vertical Velocity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-W... |
| 44 | ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | WVEL | Vertical Velocity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-WV... |
Spatiotemporal filtering
If there were both regional and global data available (e.g., LENS and NA-CORDEX data for the same variable, both listed in same catalog), some type of coverage indicator (or columns for bounding box edges) could be listed.
Temporal extent in LENS is conveyed by the experiment (HIST, 20C, etc) but this is imprecise and requires external documentation. We have added start/end columns to the catalog, but Intake-ESM currently does not have built-in functionality to filter based on time.
We can do a simple search that exactly matches a temporal value:
df = coll.search(dim="3D",component="ocn", end='2100-12').df
df
| component | dim | frequency | experiment | start | end | variable | long_name | path | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | DIC | Dissolved Inorganic Carbon | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-D... |
| 1 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | DOC | Dissolved Organic Carbon | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-D... |
| 2 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | O2 | Dissolved Oxygen | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-O... |
| 3 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | PD | Potential Density Ref to Surface | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-P... |
| 4 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | SALT | Salinity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-S... |
| 5 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | TEMP | Potential Temperature | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-T... |
| 6 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | UES | Salt Flux in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-U... |
| 7 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | UET | Heat Flux in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-U... |
| 8 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | UVEL | Velocity in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-U... |
| 9 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | VNS | Salt Flux in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-V... |
| 10 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | VNT | Heat Flux in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-V... |
| 11 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | VVEL | Velocity in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-V... |
| 12 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | WTS | Salt Flux across top face | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-W... |
| 13 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | WTT | Heat Flux across top face | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-W... |
| 14 | ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | WVEL | Vertical Velocity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-W... |
Summary
In this notebook, we used Intake-ESM to explore a catalog of CESM LENS data. We then worked through some helpful features of the enhanced catalog.
What’s next?
We will use this data to recreate some figures from a paper published in BAMS that describes the CESM LENS project.
