.ipynb .pdf

repository open issue suggest edit

Binder JupyterHub

On this page

---

Using intake-esgf with rooki

Here we dig into using intake-esgf to search for data, then rooki to do server-side computing!

---

Overview

If you have an introductory paragraph, lead with it here! Keep it short and tied to your material, then be sure to continue into the required list of topics below,

1. Search and find data using intake-esgf, returning the dataset ids

2. Feed the dataset ids to rooki to subset and average the data remotely

3. Visualize the results on the end-user side

Prerequisites

Concepts	Importance	Notes
Intro to Intake-ESGF	Necessary	How to configure a search and use output
Intro to Rooki	Helpful	How to initialize and run rooki
Intro to hvPlot	Necessary	How to plot interactive visualizations

• Time to learn: 30 minutes

---

Imports

import os

# Set the rooki client to use the oak ridge national lab (ORNL) WPS deployment
url = 'https://esgf-wps.apps.onyx.ccs.ornl.gov/wps'
os.environ['ROOK_URL'] = url

from rooki import rooki
from rooki import operators as ops
import intake_esgf
from intake_esgf import ESGFCatalog
import xarray as xr
import hvplot.xarray
import holoviews as hv
import panel as pn
hv.extension("bokeh")

Search and Find Data for Surface Temperature on DKRZ Node

Let’s start with refining which index we would like to search from. For this analysis, we are remotely computing on the ORNL node since this is where rooki is running. We know this from checking the ._url method of rooki!

rooki._url

'https://esgf-wps.apps.onyx.ccs.ornl.gov/wps'

Set the Index Node and Search

We need to ensure only the Oak Ridge National Lab (ORNL) index node is active, not the Argonne National Laboratory (ANL) node. We can accomplish this using the configuration settings for intake-esgf.

intake_esgf.conf.set(indices={"anl-dev":False,
                              "ornl-dev":True,
                              }
                    )

cat = ESGFCatalog()

cat = ESGFCatalog()
cat.search(
        activity_id='CMIP',
        experiment_id=["historical",],
        variable_id=["tas"],
        member_id='r1i1p1f1',
        grid_label='gn',
        table_id="Amon",
        institution_id=["MIROC", "NCAR", "NASA-GISS", "CMCC"]
    )
cat.df

	mip_era	experiment_id	variable_id	source_id	version	grid_label	member_id	table_id	institution_id	activity_drs	project	id
0	CMIP6	historical	tas	GISS-E2-2-H	20191120	gn	r1i1p1f1	Amon	NASA-GISS	CMIP	CMIP6	[CMIP6.CMIP.NASA-GISS.GISS-E2-2-H.historical.r...
1	CMIP6	historical	tas	CMCC-ESM2	20210114	gn	r1i1p1f1	Amon	CMCC	CMIP	CMIP6	[CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1...
2	CMIP6	historical	tas	CESM2-WACCM	20190227	gn	r1i1p1f1	Amon	NCAR	CMIP	CMIP6	[CMIP6.CMIP.NCAR.CESM2-WACCM.historical.r1i1p1...
3	CMIP6	historical	tas	CMCC-CM2-SR5	20200616	gn	r1i1p1f1	Amon	CMCC	CMIP	CMIP6	[CMIP6.CMIP.CMCC.CMCC-CM2-SR5.historical.r1i1p...
4	CMIP6	historical	tas	GISS-E2-1-G	20180827	gn	r1i1p1f1	Amon	NASA-GISS	CMIP	CMIP6	[CMIP6.CMIP.NASA-GISS.GISS-E2-1-G.historical.r...
5	CMIP6	historical	tas	GISS-E2-2-G	20191120	gn	r1i1p1f1	Amon	NASA-GISS	CMIP	CMIP6	[CMIP6.CMIP.NASA-GISS.GISS-E2-2-G.historical.r...
6	CMIP6	historical	tas	CESM2-WACCM-FV2	20191120	gn	r1i1p1f1	Amon	NCAR	CMIP	CMIP6	[CMIP6.CMIP.NCAR.CESM2-WACCM-FV2.historical.r1...
8	CMIP6	historical	tas	CESM2-FV2	20191120	gn	r1i1p1f1	Amon	NCAR	CMIP	CMIP6	[CMIP6.CMIP.NCAR.CESM2-FV2.historical.r1i1p1f1...
9	CMIP6	historical	tas	CMCC-CM2-HR4	20200904	gn	r1i1p1f1	Amon	CMCC	CMIP	CMIP6	[CMIP6.CMIP.CMCC.CMCC-CM2-HR4.historical.r1i1p...
10	CMIP6	historical	tas	GISS-E2-1-H	20190403	gn	r1i1p1f1	Amon	NASA-GISS	CMIP	CMIP6	[CMIP6.CMIP.NASA-GISS.GISS-E2-1-H.historical.r...
15	CMIP6	historical	tas	CESM2	20190308	gn	r1i1p1f1	Amon	NCAR	CMIP	CMIP6	[CMIP6.CMIP.NCAR.CESM2.historical.r1i1p1f1.Amo...
16	CMIP6	historical	tas	GISS-E2-1-G-CC	20190815	gn	r1i1p1f1	Amon	NASA-GISS	CMIP	CMIP6	[CMIP6.CMIP.NASA-GISS.GISS-E2-1-G-CC.historica...
22	CMIP6	historical	tas	MIROC6	20181212	gn	r1i1p1f1	Amon	MIROC	CMIP	CMIP6	[CMIP6.CMIP.MIROC.MIROC6.historical.r1i1p1f1.A...

Extract the Dataset ID and Pass to Rooki

Now that we have set of datasets, we need to extract the dataset_id, which is the unique identifier for the dataset. We can pull this from the id column from intake-esgf

Separate the Dataset ID

cat.df.id.values[0]

['CMIP6.CMIP.NASA-GISS.GISS-E2-2-H.historical.r1i1p1f1.Amon.tas.gn.v20191120|esgf-data04.diasjp.net',
 'CMIP6.CMIP.NASA-GISS.GISS-E2-2-H.historical.r1i1p1f1.Amon.tas.gn.v20191120|dpesgf03.nccs.nasa.gov',
 'CMIP6.CMIP.NASA-GISS.GISS-E2-2-H.historical.r1i1p1f1.Amon.tas.gn.v20191120|esgf-node.ornl.gov']

Notice how the node information is added onto end of the file id. We need to “chop off” that last bit, leaving everything before the | character. We put this into a function to make it easier to generalize and apply.

def separate_dataset_id(full_dataset):
    """
    Create the path to the data, adding an additional css03_data to indicate the CMIP6 archive at ORNL
    """
    return f"css03_data.{full_dataset[0].split('|')[0]}"

separate_dataset_id(cat.df.id.values[0])

'css03_data.CMIP6.CMIP.NASA-GISS.GISS-E2-2-H.historical.r1i1p1f1.Amon.tas.gn.v20191120'

Now, we can apply this to the entire list within our dataframe using the following

dsets = [separate_dataset_id(dataset) for dataset in list(cat.df.id.values)]
dsets

['css03_data.CMIP6.CMIP.NASA-GISS.GISS-E2-2-H.historical.r1i1p1f1.Amon.tas.gn.v20191120',
 'css03_data.CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.Amon.tas.gn.v20210114',
 'css03_data.CMIP6.CMIP.NCAR.CESM2-WACCM.historical.r1i1p1f1.Amon.tas.gn.v20190227',
 'css03_data.CMIP6.CMIP.CMCC.CMCC-CM2-SR5.historical.r1i1p1f1.Amon.tas.gn.v20200616',
 'css03_data.CMIP6.CMIP.NASA-GISS.GISS-E2-1-G.historical.r1i1p1f1.Amon.tas.gn.v20180827',
 'css03_data.CMIP6.CMIP.NASA-GISS.GISS-E2-2-G.historical.r1i1p1f1.Amon.tas.gn.v20191120',
 'css03_data.CMIP6.CMIP.NCAR.CESM2-WACCM-FV2.historical.r1i1p1f1.Amon.tas.gn.v20191120',
 'css03_data.CMIP6.CMIP.NCAR.CESM2-FV2.historical.r1i1p1f1.Amon.tas.gn.v20191120',
 'css03_data.CMIP6.CMIP.CMCC.CMCC-CM2-HR4.historical.r1i1p1f1.Amon.tas.gn.v20200904',
 'css03_data.CMIP6.CMIP.NASA-GISS.GISS-E2-1-H.historical.r1i1p1f1.Amon.tas.gn.v20190403',
 'css03_data.CMIP6.CMIP.NCAR.CESM2.historical.r1i1p1f1.Amon.tas.gn.v20190308',
 'css03_data.CMIP6.CMIP.NASA-GISS.GISS-E2-1-G-CC.historical.r1i1p1f1.Amon.tas.gn.v20190815',
 'css03_data.CMIP6.CMIP.MIROC.MIROC6.historical.r1i1p1f1.Amon.tas.gn.v20181212']

Compute with Rooki

Now that we have a list of IDs to pass to rooki, let’s compute!

In this case, we are:

• Subsetting from the year 1900 to 2000

• Subsetting near India using the bounds 65,0,100,35

• Computing the yealy average

We then check to make sure the response is okay, and if it is, return that to the user!

def compute_annual_mean_subset(dset_id):
    # Subset by area then time
    wf = ops.AverageByTime(
        ops.Subset(
            ops.Input(
                'tas', [dsets[0]]
            ),
            time='1900-01-01/2000-12-31',
            area='65,0,100,35',
        ),
    freq="year"
    )
    
    resp = wf.orchestrate()
    
    if resp.ok:
        ds = resp.datasets()[0]
    else:
        ds = xr.Dataset()
    return ds

compute_annual_mean_subset(dsets[0])

Metalink content-type detected.
Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_p9cwdemi/tas_Amon_GISS-E2-2-H_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.

<xarray.Dataset> Size: 156kB
Dimensions:    (lat: 18, time: 101, bnds: 2, lon: 14)
Coordinates:
  * lat        (lat) float64 144B 1.0 3.0 5.0 7.0 9.0 ... 29.0 31.0 33.0 35.0
  * lon        (lon) float64 112B 66.25 68.75 71.25 73.75 ... 93.75 96.25 98.75
    height     float64 8B ...
  * time       (time) object 808B 1900-01-01 00:00:00 ... 2000-01-01 00:00:00
Dimensions without coordinates: bnds
Data variables:
    lat_bnds   (time, lat, bnds) float64 29kB ...
    lon_bnds   (time, lon, bnds) float64 23kB ...
    tas        (time, lat, lon) float32 102kB ...
    time_bnds  (time, bnds) object 2kB ...
Attributes: (12/48)
    Conventions:            CF-1.7 CMIP-6.2
    activity_id:            CMIP
    branch_method:          standard
    branch_time_in_child:   0.0
    branch_time_in_parent:  0.0
    contact:                Kenneth Lo (cdkkl@giss.nasa.gov)
    ...                     ...
    title:                  GISS-E2-2-H output prepared for CMIP6
    tracking_id:            hdl:21.14100/09d7bd73-f74e-4f9a-a14b-205fa5078217
    variable_id:            tas
    variant_label:          r1i1p1f1
    license:                CMIP6 model data produced by NASA Goddard Institu...
    cmor_version:           3.3.2

xarray.Dataset

• Dimensions:
  • lat: 18
  • time: 101
  • bnds: 2
  • lon: 14
• Coordinates: (4)
  • lat
    (lat)
    float64
    1.0 3.0 5.0 7.0 ... 31.0 33.0 35.0

    bounds :

    lat_bnds

    units :

    degrees_north

    axis :

    Y

    long_name :

    latitude

    standard_name :

    latitude

    array([ 1.,  3.,  5.,  7.,  9., 11., 13., 15., 17., 19., 21., 23., 25., 27.,
           29., 31., 33., 35.])

  • lon
    (lon)
    float64
    66.25 68.75 71.25 ... 96.25 98.75

    bounds :

    lon_bnds

    units :

    degrees_east

    axis :

    X

    long_name :

    longitude

    standard_name :

    longitude

    array([66.25, 68.75, 71.25, 73.75, 76.25, 78.75, 81.25, 83.75, 86.25, 88.75,
           91.25, 93.75, 96.25, 98.75])

  • height
    ()
    float64
    ...

    units :

    m

    axis :

    Z

    positive :

    up

    long_name :

    height

    standard_name :

    height

    [1 values with dtype=float64]

  • time
    (time)
    object
    1900-01-01 00:00:00 ... 2000-01-...

    array([cftime.DatetimeNoLeap(1900, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1901, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1902, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1903, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1904, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1905, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1906, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1907, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1908, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1909, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1910, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1911, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1912, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1913, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1914, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1915, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1916, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1917, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1918, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1919, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1920, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1921, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1922, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1923, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1924, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1925, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1926, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1927, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1928, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1929, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1930, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1931, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1932, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1933, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1934, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1935, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1936, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1937, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1938, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1939, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1940, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1941, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1942, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1943, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1944, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1945, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1946, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1947, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1948, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1949, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1950, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1951, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1952, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1953, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1954, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1955, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1956, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1957, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1958, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1959, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1960, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1961, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1962, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1963, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1964, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1965, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1966, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1967, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1968, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1969, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1970, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1971, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1972, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1973, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1974, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1975, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1976, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1977, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1978, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1979, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1980, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1981, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1982, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1983, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1984, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1985, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1986, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1987, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1988, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1989, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1990, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1991, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1992, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1993, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1994, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1995, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1996, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1997, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1998, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1999, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2000, 1, 1, 0, 0, 0, 0, has_year_zero=True)],
          dtype=object)

• Data variables: (4)
  • lat_bnds
    (time, lat, bnds)
    float64
    ...

    [3636 values with dtype=float64]

  • lon_bnds
    (time, lon, bnds)
    float64
    ...

    [2828 values with dtype=float64]

  • tas
    (time, lat, lon)
    float32
    ...

    standard_name :

    air_temperature

    long_name :

    Near-Surface Air Temperature

    comment :

    near-surface (usually, 2 meter) air temperature

    units :

    K

    cell_methods :

    area: time: mean

    cell_measures :

    area: areacella

    history :

    2021-02-22T23:27:40Z altered by CMOR: Treated scalar dimension: 'height'. 2021-02-22T23:27:40Z altered by CMOR: replaced missing value flag (-1e+30) with standard missing value (1e+20).

    [25452 values with dtype=float32]

  • time_bnds
    (time, bnds)
    object
    ...

    [202 values with dtype=object]

• Indexes: (3)
  • lat
    PandasIndex

    PandasIndex(Index([ 1.0,  3.0,  5.0,  7.0,  9.0, 11.0, 13.0, 15.0, 17.0, 19.0, 21.0, 23.0,
           25.0, 27.0, 29.0, 31.0, 33.0, 35.0],
          dtype='float64', name='lat'))

  • lon
    PandasIndex

    PandasIndex(Index([66.25, 68.75, 71.25, 73.75, 76.25, 78.75, 81.25, 83.75, 86.25, 88.75,
           91.25, 93.75, 96.25, 98.75],
          dtype='float64', name='lon'))

  • time
    PandasIndex

    PandasIndex(CFTimeIndex([1900-01-01 00:00:00, 1901-01-01 00:00:00, 1902-01-01 00:00:00,
                 1903-01-01 00:00:00, 1904-01-01 00:00:00, 1905-01-01 00:00:00,
                 1906-01-01 00:00:00, 1907-01-01 00:00:00, 1908-01-01 00:00:00,
                 1909-01-01 00:00:00,
                 ...
                 1991-01-01 00:00:00, 1992-01-01 00:00:00, 1993-01-01 00:00:00,
                 1994-01-01 00:00:00, 1995-01-01 00:00:00, 1996-01-01 00:00:00,
                 1997-01-01 00:00:00, 1998-01-01 00:00:00, 1999-01-01 00:00:00,
                 2000-01-01 00:00:00],
                dtype='object', length=101, calendar='noleap', freq='YS-JAN'))

• Attributes: (48)

  Conventions :

  CF-1.7 CMIP-6.2

  activity_id :

  CMIP

  branch_method :

  standard

  branch_time_in_child :

  0.0

  branch_time_in_parent :

  0.0

  contact :

  Kenneth Lo (cdkkl@giss.nasa.gov)

  creation_date :

  2021-02-22T23:27:40Z

  data_specs_version :

  01.00.23

  experiment :

  all-forcing simulation of the recent past

  experiment_id :

  historical

  external_variables :

  areacella

  forcing_index :

  1

  frequency :

  mon

  further_info_url :

  https://furtherinfo.es-doc.org/CMIP6.NASA-GISS.GISS-E2-2-H.historical.none.r1i1p1f1

  grid :

  atmospheric grid: 144x90, ocean grid: 360x180

  grid_label :

  gn

  history :

  2021-02-22T23:27:40Z ; CMOR rewrote data to be consistent with CMIP6, CF-1.7 CMIP-6.2 and CF standards.

  initialization_index :

  1

  institution :

  Goddard Institute for Space Studies, New York, NY 10025, USA

  institution_id :

  NASA-GISS

  mip_era :

  CMIP6

  model_id :

  Eh06f10aF102

  nominal_resolution :

  250 km

  parent_activity_id :

  CMIP

  parent_experiment_id :

  piControl

  parent_experiment_rip :

  r1i1p1

  parent_mip_era :

  CMIP6

  parent_source_id :

  GISS-E2-2-H

  parent_time_units :

  days since 2000-1-1

  parent_variant_label :

  r1i1p1f1

  physics_index :

  1

  product :

  model-output

  realization_index :

  1

  realm :

  atmos

  references :

  https://data.giss.nasa.gov/modelE/cmip6

  source :

  GISS-E2-2-H (2019): aerosol: varies with physics-version (p==1 none, p==3 OMA, p==4 TOMAS, p==5 MATRIX) atmos: GISS-E2.2 (High-top, 2 x 2.5 degrees; 144 x 90 longitude/latitude; 102 levels; top level 0.002 hPa) atmosChem: varies with physics-version (p==1 Non-interactive, p>1 GPUCCINI) land: GISS LSM landIce: Fixed ocean: GISS Ocean (GO1, 1 degree; 360 x 180 longitude/latitude; 40 levels; top grid cell 0-10m) ocnBgchem: none seaIce: GISS SI

  source_id :

  GISS-E2-2-H

  source_type :

  AOGCM

  sub_experiment :

  none

  sub_experiment_id :

  none

  table_id :

  Amon

  table_info :

  Creation Date:(21 March 2018) MD5:a2d322762b918858e63dee163d841246

  title :

  GISS-E2-2-H output prepared for CMIP6

  tracking_id :

  hdl:21.14100/09d7bd73-f74e-4f9a-a14b-205fa5078217

  variable_id :

  tas

  variant_label :

  r1i1p1f1

  license :

  CMIP6 model data produced by NASA Goddard Institute for Space Studies is licensed under a Creative Commons Attribution ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and at https:///pcmdi.llnl.gov/. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.

  cmor_version :

  3.3.2

Now that it works with a single dataset, let’s do this for all the datasets and put them into a dictionary with the dataset ids as the keys.

dset_dict = {}
for dset in dsets:
    dset_dict[dset] = compute_annual_mean_subset(dset)

Metalink content-type detected.
Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_4tki31ka/tas_Amon_GISS-E2-2-H_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.
Metalink content-type detected.
Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_r2f02hvl/tas_Amon_GISS-E2-2-H_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.
Metalink content-type detected.
Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_lu8c8qa1/tas_Amon_GISS-E2-2-H_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.
Metalink content-type detected.
Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_uguvw4ac/tas_Amon_GISS-E2-2-H_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.
Metalink content-type detected.
Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_277dxih2/tas_Amon_GISS-E2-2-H_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.
Metalink content-type detected.
Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_8orbkvmb/tas_Amon_GISS-E2-2-H_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.
Metalink content-type detected.
Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_nzhf2ws7/tas_Amon_GISS-E2-2-H_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.
Metalink content-type detected.
Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_lv3psy2l/tas_Amon_GISS-E2-2-H_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.
Metalink content-type detected.
Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_9nop9w3l/tas_Amon_GISS-E2-2-H_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.
Metalink content-type detected.
Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_1eb7837q/tas_Amon_GISS-E2-2-H_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.
Metalink content-type detected.
Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_g_5ggv2j/tas_Amon_GISS-E2-2-H_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.
Metalink content-type detected.
Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_2s_1j9tu/tas_Amon_GISS-E2-2-H_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.
Metalink content-type detected.
Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_sb406lac/tas_Amon_GISS-E2-2-H_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.

Visualize the Output

Let’s use hvPlot to visualize. The datasets are stored in a dictionary of datasets, we need to:

• Extract a single key

• Plot a contour filled visualization, with some geographic features

dset_dict[list(dset_dict.keys())[-1]].tas.hvplot.contourf(x='lon',
                                                          y='lat',
                                                          cmap='Reds',
                                                          levels=20,
                                                          clim=(250, 320),
                                                          features=["land", "ocean"],
                                                          alpha=0.7,
                                                          widget_location='bottom',
                                                          clabel="Yearly Average Temperature (K)",
                                                          geo=True)

---

Summary

Within this notebook, we learned how to specify a specific index node to search from, pass discovered datasets to rooki, and chain remote-compute with several operations using rooki. We then visualized the output using hvPlot, leading to an interactive plot!

What’s next?

More adaptations of the intake-esgf + rooki to remotely compute on ESGF data.

Resources and references

• intake-esgf documentation

• rooki documentation

• Working with geographic data with hvPlot

previous

Compute Demo: Use Rooki to access CMIP6 data

next

ENSO Calculations using Globus Compute