.ipynb .pdf

repository open issue suggest edit

Binder JupyterHub

On this page

Compare Data from ESGF and ARM

Overview

This notebook details how to compare CMIP6 data hosted through the Earth System Grid Federation (ESGF) to observations collected and hosted through the Department of Energy’s Atmospheric Radiation Measurement (ARM) user facility.

The measurement of focus is 2 meter air temperature, collected at the Southern Great Plains (SGP) site in Northern Oklahoma. This climate observatory has collected state-of-the-art observations since 1993.

Prerequisites

Concepts	Importance	Notes
Intro to Xarray	Necessary
Search and Load CMIP6 Data via ESGF/OPeNDAP	Necessary	Familiarity with data access patterns
Understanding of NetCDF	Helpful	Familiarity with metadata structure
Dask Arrays with Xarray	Helpful	Familiarity with lazy-loading

• Time to learn: 25 minutes

Imports

import os
import warnings

import act
from distributed import Client
import holoviews as hv
import hvplot.xarray
from matplotlib import pyplot as plt
import numpy as np
import pandas as pd
import cf_xarray
import metpy
from pyesgf.search import SearchConnection
import xarray as xr

xr.set_options(display_style='html')
warnings.filterwarnings("ignore")
hv.extension('bokeh')

Spin up a Dask Cluster

We will use a Dask Local Cluster to compute in parellel and distribute our data, enabling us to work with these large datasets.

client = Client()
client

Client

Client-4b621a6d-87ed-11ee-8506-4eeb28ce1cac

Connection method: Cluster object	Cluster type: distributed.LocalCluster
Dashboard: http://127.0.0.1:8787/status

Cluster Info

LocalCluster

eb993f71

Dashboard: http://127.0.0.1:8787/status	Workers: 8
Total threads: 32	Total memory: 122.83 GiB
Status: running	Using processes: True

Scheduler Info

Scheduler

Scheduler-21cebf0d-97f8-4e26-a219-fa2ac4df23f4

Comm: tcp://127.0.0.1:38437	Workers: 8
Dashboard: http://127.0.0.1:8787/status	Total threads: 32
Started: Just now	Total memory: 122.83 GiB

Workers

Worker: 0

Comm: tcp://127.0.0.1:35235	Total threads: 4
Dashboard: http://127.0.0.1:34503/status	Memory: 15.35 GiB
Nanny: tcp://127.0.0.1:35145
Local directory: /tmp/dask-scratch-space/worker-payxyzi0

Worker: 1

Comm: tcp://127.0.0.1:38603	Total threads: 4
Dashboard: http://127.0.0.1:36903/status	Memory: 15.35 GiB
Nanny: tcp://127.0.0.1:41195
Local directory: /tmp/dask-scratch-space/worker-67obd1st

Worker: 2

Comm: tcp://127.0.0.1:33325	Total threads: 4
Dashboard: http://127.0.0.1:33625/status	Memory: 15.35 GiB
Nanny: tcp://127.0.0.1:40799
Local directory: /tmp/dask-scratch-space/worker-ln6zrmrv

Worker: 3

Comm: tcp://127.0.0.1:40361	Total threads: 4
Dashboard: http://127.0.0.1:38293/status	Memory: 15.35 GiB
Nanny: tcp://127.0.0.1:39359
Local directory: /tmp/dask-scratch-space/worker-l3c_kmm0

Worker: 4

Comm: tcp://127.0.0.1:34233	Total threads: 4
Dashboard: http://127.0.0.1:43769/status	Memory: 15.35 GiB
Nanny: tcp://127.0.0.1:43539
Local directory: /tmp/dask-scratch-space/worker-yli0x_h7

Worker: 5

Comm: tcp://127.0.0.1:46675	Total threads: 4
Dashboard: http://127.0.0.1:33783/status	Memory: 15.35 GiB
Nanny: tcp://127.0.0.1:37899
Local directory: /tmp/dask-scratch-space/worker-p4rpvzzr

Worker: 6

Comm: tcp://127.0.0.1:42801	Total threads: 4
Dashboard: http://127.0.0.1:45579/status	Memory: 15.35 GiB
Nanny: tcp://127.0.0.1:40771
Local directory: /tmp/dask-scratch-space/worker-nnxl63mu

Worker: 7

Comm: tcp://127.0.0.1:43771	Total threads: 4
Dashboard: http://127.0.0.1:32879/status	Memory: 15.35 GiB
Nanny: tcp://127.0.0.1:36761
Local directory: /tmp/dask-scratch-space/worker-4ew2q96_

Access Data

Our first step is to access data from the ESGF data servers, and the Atmospheric Radiation Measurement (ARM) user facility, which has a long term site in Northern Oklahoma.

Access ESGF Data

A tutorial on how to access ESGF-hosted CMIP6 data is included in the Foundations section of this cookbook:

• ESGF OpenDAP Tutorial

We use the following block of code to search for a single earth system model simulation, the Energe Exascale Earth System Model (E3SM), which is the Department of Energy’s flagship coupled Earth System Model.

conn = SearchConnection('https://esgf-node.llnl.gov/esg-search',
                        distrib=False)
ctx = conn.new_context(
    facets='project,experiment_id',
    project='CMIP6',
    table_id='Amon',
    institution_id = 'E3SM-Project',
    experiment_id='historical',
    source_id='E3SM-1-0',
    variable='tas',
    variant_label='r1i1p1f1',
)
result = ctx.search()[1]
files = result.file_context().search()
opendap_urls = [file.opendap_url for file in files]

esgf_ds = xr.open_mfdataset(opendap_urls,
                       combine='by_coords',
                       chunks={'time':480})
esgf_ds

<xarray.Dataset>
Dimensions:    (time: 1980, bnds: 2, lat: 180, lon: 360)
Coordinates:
  * time       (time) object 1850-01-16 12:00:00 ... 2014-12-16 12:00:00
  * lat        (lat) float64 -89.5 -88.5 -87.5 -86.5 ... 86.5 87.5 88.5 89.5
  * lon        (lon) float64 0.5 1.5 2.5 3.5 4.5 ... 356.5 357.5 358.5 359.5
    height     float64 2.0
Dimensions without coordinates: bnds
Data variables:
    time_bnds  (time, bnds) object dask.array<chunksize=(300, 2), meta=np.ndarray>
    lat_bnds   (time, lat, bnds) float64 dask.array<chunksize=(300, 180, 2), meta=np.ndarray>
    lon_bnds   (time, lon, bnds) float64 dask.array<chunksize=(300, 360, 2), meta=np.ndarray>
    tas        (time, lat, lon) float32 dask.array<chunksize=(300, 180, 360), meta=np.ndarray>
Attributes: (12/54)
    Conventions:                     CF-1.7 CMIP-6.2
    activity_id:                     CMIP
    branch_method:                   standard
    branch_time_in_child:            0.0
    branch_time_in_parent:           36500.0
    contact:                         Dave Bader (bader2@llnl.gov)
    ...                              ...
    e3sm_source_code_reference:      https://github.com/E3SM-Project/E3SM/rel...
    doe_acknowledgement:             This research was supported as part of t...
    computational_acknowledgement:   The data were produced using resources o...
    ncclimo_generation_command:      ncclimo --var=${var} -7 --dfl_lvl=1 --no...
    ncclimo_version:                 4.8.1-alpha04
    DODS_EXTRA.Unlimited_Dimension:  time

xarray.Dataset

• Dimensions:
  • time: 1980
  • bnds: 2
  • lat: 180
  • lon: 360
• Coordinates: (4)
  • time
    (time)
    object
    1850-01-16 12:00:00 ... 2014-12-...

    bounds :

    time_bnds

    axis :

    T

    long_name :

    time

    standard_name :

    time

    _ChunkSizes :

    1

    array([cftime.DatetimeNoLeap(1850, 1, 16, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1850, 2, 15, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1850, 3, 16, 12, 0, 0, 0, has_year_zero=True),
           ...,
           cftime.DatetimeNoLeap(2014, 10, 16, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2014, 11, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2014, 12, 16, 12, 0, 0, 0, has_year_zero=True)],
          dtype=object)

  • lat
    (lat)
    float64
    -89.5 -88.5 -87.5 ... 88.5 89.5

    bounds :

    lat_bnds

    units :

    degrees_north

    axis :

    Y

    long_name :

    Latitude

    standard_name :

    latitude

    array([-89.5, -88.5, -87.5, -86.5, -85.5, -84.5, -83.5, -82.5, -81.5, -80.5,
           -79.5, -78.5, -77.5, -76.5, -75.5, -74.5, -73.5, -72.5, -71.5, -70.5,
           -69.5, -68.5, -67.5, -66.5, -65.5, -64.5, -63.5, -62.5, -61.5, -60.5,
           -59.5, -58.5, -57.5, -56.5, -55.5, -54.5, -53.5, -52.5, -51.5, -50.5,
           -49.5, -48.5, -47.5, -46.5, -45.5, -44.5, -43.5, -42.5, -41.5, -40.5,
           -39.5, -38.5, -37.5, -36.5, -35.5, -34.5, -33.5, -32.5, -31.5, -30.5,
           -29.5, -28.5, -27.5, -26.5, -25.5, -24.5, -23.5, -22.5, -21.5, -20.5,
           -19.5, -18.5, -17.5, -16.5, -15.5, -14.5, -13.5, -12.5, -11.5, -10.5,
            -9.5,  -8.5,  -7.5,  -6.5,  -5.5,  -4.5,  -3.5,  -2.5,  -1.5,  -0.5,
             0.5,   1.5,   2.5,   3.5,   4.5,   5.5,   6.5,   7.5,   8.5,   9.5,
            10.5,  11.5,  12.5,  13.5,  14.5,  15.5,  16.5,  17.5,  18.5,  19.5,
            20.5,  21.5,  22.5,  23.5,  24.5,  25.5,  26.5,  27.5,  28.5,  29.5,
            30.5,  31.5,  32.5,  33.5,  34.5,  35.5,  36.5,  37.5,  38.5,  39.5,
            40.5,  41.5,  42.5,  43.5,  44.5,  45.5,  46.5,  47.5,  48.5,  49.5,
            50.5,  51.5,  52.5,  53.5,  54.5,  55.5,  56.5,  57.5,  58.5,  59.5,
            60.5,  61.5,  62.5,  63.5,  64.5,  65.5,  66.5,  67.5,  68.5,  69.5,
            70.5,  71.5,  72.5,  73.5,  74.5,  75.5,  76.5,  77.5,  78.5,  79.5,
            80.5,  81.5,  82.5,  83.5,  84.5,  85.5,  86.5,  87.5,  88.5,  89.5])

  • lon
    (lon)
    float64
    0.5 1.5 2.5 ... 357.5 358.5 359.5

    bounds :

    lon_bnds

    units :

    degrees_east

    axis :

    X

    long_name :

    Longitude

    standard_name :

    longitude

    array([  0.5,   1.5,   2.5, ..., 357.5, 358.5, 359.5])

  • height
    ()
    float64
    2.0

    units :

    m

    axis :

    Z

    positive :

    up

    long_name :

    height

    standard_name :

    height

    array(2.)

• Data variables: (4)
  • time_bnds
    (time, bnds)
    object
    dask.array<chunksize=(300, 2), meta=np.ndarray>

    _ChunkSizes :

    [1 2]

    Array Chunk Bytes 30.94 kiB 4.69 kiB Shape (1980, 2) (300, 2) Dask graph 7 chunks in 15 graph layers Data type object numpy.ndarray

  • lat_bnds
    (time, lat, bnds)
    float64
    dask.array<chunksize=(300, 180, 2), meta=np.ndarray>

    _ChunkSizes :

    [180 2]

    Array Chunk Bytes 5.44 MiB 843.75 kiB Shape (1980, 180, 2) (300, 180, 2) Dask graph 7 chunks in 22 graph layers Data type float64 numpy.ndarray

  • lon_bnds
    (time, lon, bnds)
    float64
    dask.array<chunksize=(300, 360, 2), meta=np.ndarray>

    _ChunkSizes :

    [360 2]

    Array Chunk Bytes 10.88 MiB 1.65 MiB Shape (1980, 360, 2) (300, 360, 2) Dask graph 7 chunks in 22 graph layers Data type float64 numpy.ndarray

  • tas
    (time, lat, lon)
    float32
    dask.array<chunksize=(300, 180, 360), meta=np.ndarray>

    standard_name :

    air_temperature

    long_name :

    Near-Surface Air Temperature

    comment :

    near-surface (usually, 2 meter) air temperature

    units :

    K

    cell_methods :

    area: time: mean

    cell_measures :

    area: areacella

    history :

    2019-07-24T22:40:35Z altered by CMOR: Treated scalar dimension: 'height'.

    _ChunkSizes :

    [ 1 180 360]

    Array Chunk Bytes 489.44 MiB 74.16 MiB Shape (1980, 180, 360) (300, 180, 360) Dask graph 7 chunks in 15 graph layers Data type float32 numpy.ndarray

• Indexes: (3)
  • time
    PandasIndex

    PandasIndex(CFTimeIndex([1850-01-16 12:00:00, 1850-02-15 00:00:00, 1850-03-16 12:00:00,
                 1850-04-16 00:00:00, 1850-05-16 12:00:00, 1850-06-16 00:00:00,
                 1850-07-16 12:00:00, 1850-08-16 12:00:00, 1850-09-16 00:00:00,
                 1850-10-16 12:00:00,
                 ...
                 2014-03-16 12:00:00, 2014-04-16 00:00:00, 2014-05-16 12:00:00,
                 2014-06-16 00:00:00, 2014-07-16 12:00:00, 2014-08-16 12:00:00,
                 2014-09-16 00:00:00, 2014-10-16 12:00:00, 2014-11-16 00:00:00,
                 2014-12-16 12:00:00],
                dtype='object', length=1980, calendar='noleap', freq='None'))

  • lat
    PandasIndex

    PandasIndex(Index([-89.5, -88.5, -87.5, -86.5, -85.5, -84.5, -83.5, -82.5, -81.5, -80.5,
           ...
            80.5,  81.5,  82.5,  83.5,  84.5,  85.5,  86.5,  87.5,  88.5,  89.5],
          dtype='float64', name='lat', length=180))

  • lon
    PandasIndex

    PandasIndex(Index([  0.5,   1.5,   2.5,   3.5,   4.5,   5.5,   6.5,   7.5,   8.5,   9.5,
           ...
           350.5, 351.5, 352.5, 353.5, 354.5, 355.5, 356.5, 357.5, 358.5, 359.5],
          dtype='float64', name='lon', length=360))

• Attributes: (54)

  Conventions :

  CF-1.7 CMIP-6.2

  activity_id :

  CMIP

  branch_method :

  standard

  branch_time_in_child :

  0.0

  branch_time_in_parent :

  36500.0

  contact :

  Dave Bader (bader2@llnl.gov)

  creation_date :

  2019-07-24T22:40:35Z

  data_specs_version :

  01.00.29

  experiment :

  all-forcing simulation of the recent past

  experiment_id :

  historical

  external_variables :

  areacella

  forcing_index :

  1

  frequency :

  mon

  further_info_url :

  https://furtherinfo.es-doc.org/CMIP6.E3SM-Project.E3SM-1-0.historical.none.r1i1p1f1

  grid :

  data regridded to a CMIP6 standard 1x1 degree lonxlat grid from the native grid using an area-average preserving method.

  grid_label :

  gr

  history :

  2019-07-24T22:40:35Z ;rewrote data to be consistent with CMIP for variable tas found in table Amon.; Output from 20180129.DECKv1b_piControl.ne30_oEC.edison

  initialization_index :

  1

  institution :

  LLNL (Lawrence Livermore National Laboratory, Livermore, CA 94550, USA); ANL (Argonne National Laboratory, Argonne, IL 60439, USA); BNL (Brookhaven National Laboratory, Upton, NY 11973, USA); LANL (Los Alamos National Laboratory, Los Alamos, NM 87545, USA); LBNL (Lawrence Berkeley National Laboratory, Berkeley, CA 94720, USA); ORNL (Oak Ridge National Laboratory, Oak Ridge, TN 37831, USA); PNNL (Pacific Northwest National Laboratory, Richland, WA 99352, USA); SNL (Sandia National Laboratories, Albuquerque, NM 87185, USA). Mailing address: LLNL Climate Program, c/o David C. Bader, Principal Investigator, L-103, 7000 East Avenue, Livermore, CA 94550, USA

  institution_id :

  E3SM-Project

  mip_era :

  CMIP6

  nominal_resolution :

  100 km

  parent_activity_id :

  CMIP

  parent_experiment_id :

  piControl

  parent_mip_era :

  CMIP6

  parent_source_id :

  E3SM-1-0

  parent_time_units :

  days since 0001-01-01

  parent_variant_label :

  r1i1p1f1

  physics_index :

  1

  product :

  model-output

  realization_index :

  1

  realm :

  atmos

  references :

  Golaz, J.-C., P. M. Caldwell, L. P. Van Roekel and co-authors, 2019: The DOE E3SM coupled model version 1: Overview and evaluation at standard resolution. JAMES, doi: 10.1029/2018MS001603; http://e3sm.org'

  source :

  E3SM 1.0 (2018): aerosol: MAM4 with resuspension, marine organics, and secondary organics (same grid as atmos) atmos: EAM (v1.0, cubed sphere spectral-element grid; 5400 elements with p=3; 1 deg average grid spacing; 90 x 90 x 6 longitude/latitude/cubeface; 72 levels; top level 0.1 hPa) atmosChem: Troposphere specified oxidants for aerosols. Stratosphere linearized interactive ozone (LINOZ v2) (same grid as atmos) land: ELM (v1.0, cubed sphere spectral-element grid; 5400 elements with p=3; 1 deg average grid spacing; 90 x 90 x 6 longitude/latitude/cubeface; satellite phenology mode), MOSART (v1.0, 0.5 degree latitude/longitude grid) landIce: none ocean: MPAS-Ocean (v6.0, oEC60to30 unstructured SVTs mesh with 235160 cells and 714274 edges, variable resolution 60 km to 30 km; 60 levels; top grid cell 0-10 m) ocnBgchem: none seaIce: MPAS-Seaice (v6.0, same grid as ocean)

  source_id :

  E3SM-1-0

  source_type :

  AOGCM AER

  sub_experiment :

  none

  sub_experiment_id :

  none

  table_id :

  Amon

  table_info :

  Creation Date:(08 March 2019) MD5:2b12b5db6db112aa8b8b0d6c1645b121

  title :

  E3SM-1-0 output prepared for CMIP6

  variable_id :

  tas

  variant_label :

  r1i1p1f1

  license :

  CMIP6 model data produced by E3SM is licensed under a Creative Commons Attribution ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and at https:///pcmdi.llnl.gov/. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.

  cmor_version :

  3.4.0

  tracking_id :

  hdl:21.14100/b0aaf6ad-4795-4cc2-8ea8-38c1a3c38a44

  e3sm_source_code_doi :

  10.11578/E3SM/dc.20180418.36

  e3sm_paper_reference :

  https://doi.org/10.1029/2018MS001603

  e3sm_source_code_reference :

  https://github.com/E3SM-Project/E3SM/releases/tag/v1.0.0

  doe_acknowledgement :

  This research was supported as part of the Energy Exascale Earth System Model (E3SM) project, funded by the U.S. Department of Energy, Office of Science, Office of Biological and Environmental Research.

  computational_acknowledgement :

  The data were produced using resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231.

  ncclimo_generation_command :

  ncclimo --var=${var} -7 --dfl_lvl=1 --no_cll_msr --no_frm_trm --no_stg_grd --yr_srt=1 --yr_end=500 --ypf=25 --map=map_ne30np4_to_cmip6_180x360_aave.20181001.nc

  ncclimo_version :

  4.8.1-alpha04

  DODS_EXTRA.Unlimited_Dimension :

  time

Clean up the dataset

We need to adjust the 0 to 360 degree longitude to be -180 to 180 - we can do this generically using the climate forecast (CF) conventions.

lon_coord = esgf_ds.cf['X'].name
esgf_ds[lon_coord] = (esgf_ds[lon_coord] + 180) % 360 - 180
esgf_ds = esgf_ds.sortby(lon_coord)

Access ARM Data

We use the ARM data API, which is included in the Atmospheric Data Community Toolkit (ACT) to access the data.

Setup the Search

Before downloading our data, we need to make sure we have an ARM Data Account, and ARM Live token. Both of these can be found using this link:

• ARM Live Signup

Once you sign up, you will see your token. Copy and replace that where we have arm_username and arm_password below.

arm_username = os.getenv("ARM_USERNAME")
arm_password = os.getenv("ARM_PASSWORD")

# Meteorological observations at the Southern Great Plains site
datastream = "sgpmetE13.b1"

start_date = "2013-01-01"
end_date = "2013-02-28"
files = act.discovery.download_data(arm_username,
                                    arm_password,
                                    datastream,
                                    start_date,
                                    end_date
                                   )

[DOWNLOADING] sgpmetE13.b1.20130101.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130102.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130103.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130104.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130105.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130106.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130107.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130108.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130109.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130110.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130111.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130112.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130113.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130114.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130115.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130116.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130117.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130118.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130119.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130120.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130121.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130122.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130123.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130124.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130125.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130126.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130127.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130128.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130129.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130130.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130131.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130201.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130202.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130203.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130204.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130205.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130206.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130207.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130208.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130209.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130210.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130211.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130212.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130213.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130214.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130215.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130216.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130217.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130218.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130218.170700.cdf

[DOWNLOADING] sgpmetE13.b1.20130219.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130220.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130221.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130222.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130223.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130224.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130225.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130226.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130227.000000.cdf

[DOWNLOADING] sgpmetE13.b1.20130228.000000.cdf

If you use these data to prepare a publication, please cite:

Kyrouac, J., Shi, Y., & Tuftedal, M. Surface Meteorological Instrumentation
(MET). Atmospheric Radiation Measurement (ARM) User Facility.
https://doi.org/10.5439/1786358

Load the Data Using Xarray

arm_ds = xr.open_mfdataset(files,
                           combine='nested',
                           concat_dim='time',
                           chunks={'time':86400})

Subset and Prepare Data to be Compared

We need to subset the climate model output for the nearest grid point, over the SGP site.

lat = arm_ds.lat.values[0]
lon = arm_ds.lon.values[0]
lat, lon

(36.605, -97.485)

Xarray offers this subsetting functionality, and we specify we want the nearest gird point to the site.

cmip6_nearest = esgf_ds.cf.sel(lat=lat,
                               lon=lon,
                               method='nearest')
cmip6_nearest

<xarray.Dataset>
Dimensions:    (time: 1980, bnds: 2)
Coordinates:
  * time       (time) object 1850-01-16 12:00:00 ... 2014-12-16 12:00:00
    lat        float64 36.5
    lon        float64 -97.5
    height     float64 2.0
Dimensions without coordinates: bnds
Data variables:
    time_bnds  (time, bnds) object dask.array<chunksize=(300, 2), meta=np.ndarray>
    lat_bnds   (time, bnds) float64 dask.array<chunksize=(300, 2), meta=np.ndarray>
    lon_bnds   (time, bnds) float64 dask.array<chunksize=(300, 2), meta=np.ndarray>
    tas        (time) float32 dask.array<chunksize=(300,), meta=np.ndarray>
Attributes: (12/54)
    Conventions:                     CF-1.7 CMIP-6.2
    activity_id:                     CMIP
    branch_method:                   standard
    branch_time_in_child:            0.0
    branch_time_in_parent:           36500.0
    contact:                         Dave Bader (bader2@llnl.gov)
    ...                              ...
    e3sm_source_code_reference:      https://github.com/E3SM-Project/E3SM/rel...
    doe_acknowledgement:             This research was supported as part of t...
    computational_acknowledgement:   The data were produced using resources o...
    ncclimo_generation_command:      ncclimo --var=${var} -7 --dfl_lvl=1 --no...
    ncclimo_version:                 4.8.1-alpha04
    DODS_EXTRA.Unlimited_Dimension:  time

xarray.Dataset

• Dimensions:
  • time: 1980
  • bnds: 2
• Coordinates: (4)
  • time
    (time)
    object
    1850-01-16 12:00:00 ... 2014-12-...

    bounds :

    time_bnds

    axis :

    T

    long_name :

    time

    standard_name :

    time

    _ChunkSizes :

    1

    array([cftime.DatetimeNoLeap(1850, 1, 16, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1850, 2, 15, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1850, 3, 16, 12, 0, 0, 0, has_year_zero=True),
           ...,
           cftime.DatetimeNoLeap(2014, 10, 16, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2014, 11, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2014, 12, 16, 12, 0, 0, 0, has_year_zero=True)],
          dtype=object)

  • lat
    ()
    float64
    36.5

    bounds :

    lat_bnds

    units :

    degrees_north

    axis :

    Y

    long_name :

    Latitude

    standard_name :

    latitude

    array(36.5)

  • lon
    ()
    float64
    -97.5

    array(-97.5)

  • height
    ()
    float64
    2.0

    units :

    m

    axis :

    Z

    positive :

    up

    long_name :

    height

    standard_name :

    height

    array(2.)

• Data variables: (4)
  • time_bnds
    (time, bnds)
    object
    dask.array<chunksize=(300, 2), meta=np.ndarray>

    _ChunkSizes :

    [1 2]

    Array Chunk Bytes 30.94 kiB 4.69 kiB Shape (1980, 2) (300, 2) Dask graph 7 chunks in 15 graph layers Data type object numpy.ndarray

  • lat_bnds
    (time, bnds)
    float64
    dask.array<chunksize=(300, 2), meta=np.ndarray>

    _ChunkSizes :

    [180 2]

    Array Chunk Bytes 30.94 kiB 4.69 kiB Shape (1980, 2) (300, 2) Dask graph 7 chunks in 23 graph layers Data type float64 numpy.ndarray

  • lon_bnds
    (time, bnds)
    float64
    dask.array<chunksize=(300, 2), meta=np.ndarray>

    _ChunkSizes :

    [360 2]

    Array Chunk Bytes 30.94 kiB 4.69 kiB Shape (1980, 2) (300, 2) Dask graph 7 chunks in 24 graph layers Data type float64 numpy.ndarray

  • tas
    (time)
    float32
    dask.array<chunksize=(300,), meta=np.ndarray>

    standard_name :

    air_temperature

    long_name :

    Near-Surface Air Temperature

    comment :

    near-surface (usually, 2 meter) air temperature

    units :

    K

    cell_methods :

    area: time: mean

    cell_measures :

    area: areacella

    history :

    2019-07-24T22:40:35Z altered by CMOR: Treated scalar dimension: 'height'.

    _ChunkSizes :

    [ 1 180 360]

    Array Chunk Bytes 7.73 kiB 1.17 kiB Shape (1980,) (300,) Dask graph 7 chunks in 17 graph layers Data type float32 numpy.ndarray

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(CFTimeIndex([1850-01-16 12:00:00, 1850-02-15 00:00:00, 1850-03-16 12:00:00,
                 1850-04-16 00:00:00, 1850-05-16 12:00:00, 1850-06-16 00:00:00,
                 1850-07-16 12:00:00, 1850-08-16 12:00:00, 1850-09-16 00:00:00,
                 1850-10-16 12:00:00,
                 ...
                 2014-03-16 12:00:00, 2014-04-16 00:00:00, 2014-05-16 12:00:00,
                 2014-06-16 00:00:00, 2014-07-16 12:00:00, 2014-08-16 12:00:00,
                 2014-09-16 00:00:00, 2014-10-16 12:00:00, 2014-11-16 00:00:00,
                 2014-12-16 12:00:00],
                dtype='object', length=1980, calendar='noleap', freq='None'))

• Attributes: (54)

  Conventions :

  CF-1.7 CMIP-6.2

  activity_id :

  CMIP

  branch_method :

  standard

  branch_time_in_child :

  0.0

  branch_time_in_parent :

  36500.0

  contact :

  Dave Bader (bader2@llnl.gov)

  creation_date :

  2019-07-24T22:40:35Z

  data_specs_version :

  01.00.29

  experiment :

  all-forcing simulation of the recent past

  experiment_id :

  historical

  external_variables :

  areacella

  forcing_index :

  1

  frequency :

  mon

  further_info_url :

  https://furtherinfo.es-doc.org/CMIP6.E3SM-Project.E3SM-1-0.historical.none.r1i1p1f1

  grid :

  data regridded to a CMIP6 standard 1x1 degree lonxlat grid from the native grid using an area-average preserving method.

  grid_label :

  gr

  history :

  2019-07-24T22:40:35Z ;rewrote data to be consistent with CMIP for variable tas found in table Amon.; Output from 20180129.DECKv1b_piControl.ne30_oEC.edison

  initialization_index :

  1

  institution :

  LLNL (Lawrence Livermore National Laboratory, Livermore, CA 94550, USA); ANL (Argonne National Laboratory, Argonne, IL 60439, USA); BNL (Brookhaven National Laboratory, Upton, NY 11973, USA); LANL (Los Alamos National Laboratory, Los Alamos, NM 87545, USA); LBNL (Lawrence Berkeley National Laboratory, Berkeley, CA 94720, USA); ORNL (Oak Ridge National Laboratory, Oak Ridge, TN 37831, USA); PNNL (Pacific Northwest National Laboratory, Richland, WA 99352, USA); SNL (Sandia National Laboratories, Albuquerque, NM 87185, USA). Mailing address: LLNL Climate Program, c/o David C. Bader, Principal Investigator, L-103, 7000 East Avenue, Livermore, CA 94550, USA

  institution_id :

  E3SM-Project

  mip_era :

  CMIP6

  nominal_resolution :

  100 km

  parent_activity_id :

  CMIP

  parent_experiment_id :

  piControl

  parent_mip_era :

  CMIP6

  parent_source_id :

  E3SM-1-0

  parent_time_units :

  days since 0001-01-01

  parent_variant_label :

  r1i1p1f1

  physics_index :

  1

  product :

  model-output

  realization_index :

  1

  realm :

  atmos

  references :

  Golaz, J.-C., P. M. Caldwell, L. P. Van Roekel and co-authors, 2019: The DOE E3SM coupled model version 1: Overview and evaluation at standard resolution. JAMES, doi: 10.1029/2018MS001603; http://e3sm.org'

  source :

  E3SM 1.0 (2018): aerosol: MAM4 with resuspension, marine organics, and secondary organics (same grid as atmos) atmos: EAM (v1.0, cubed sphere spectral-element grid; 5400 elements with p=3; 1 deg average grid spacing; 90 x 90 x 6 longitude/latitude/cubeface; 72 levels; top level 0.1 hPa) atmosChem: Troposphere specified oxidants for aerosols. Stratosphere linearized interactive ozone (LINOZ v2) (same grid as atmos) land: ELM (v1.0, cubed sphere spectral-element grid; 5400 elements with p=3; 1 deg average grid spacing; 90 x 90 x 6 longitude/latitude/cubeface; satellite phenology mode), MOSART (v1.0, 0.5 degree latitude/longitude grid) landIce: none ocean: MPAS-Ocean (v6.0, oEC60to30 unstructured SVTs mesh with 235160 cells and 714274 edges, variable resolution 60 km to 30 km; 60 levels; top grid cell 0-10 m) ocnBgchem: none seaIce: MPAS-Seaice (v6.0, same grid as ocean)

  source_id :

  E3SM-1-0

  source_type :

  AOGCM AER

  sub_experiment :

  none

  sub_experiment_id :

  none

  table_id :

  Amon

  table_info :

  Creation Date:(08 March 2019) MD5:2b12b5db6db112aa8b8b0d6c1645b121

  title :

  E3SM-1-0 output prepared for CMIP6

  variable_id :

  tas

  variant_label :

  r1i1p1f1

  license :

  CMIP6 model data produced by E3SM is licensed under a Creative Commons Attribution ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and at https:///pcmdi.llnl.gov/. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.

  cmor_version :

  3.4.0

  tracking_id :

  hdl:21.14100/b0aaf6ad-4795-4cc2-8ea8-38c1a3c38a44

  e3sm_source_code_doi :

  10.11578/E3SM/dc.20180418.36

  e3sm_paper_reference :

  https://doi.org/10.1029/2018MS001603

  e3sm_source_code_reference :

  https://github.com/E3SM-Project/E3SM/releases/tag/v1.0.0

  doe_acknowledgement :

  This research was supported as part of the Energy Exascale Earth System Model (E3SM) project, funded by the U.S. Department of Energy, Office of Science, Office of Biological and Environmental Research.

  computational_acknowledgement :

  The data were produced using resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231.

  ncclimo_generation_command :

  ncclimo --var=${var} -7 --dfl_lvl=1 --no_cll_msr --no_frm_trm --no_stg_grd --yr_srt=1 --yr_end=500 --ypf=25 --map=map_ne30np4_to_cmip6_180x360_aave.20181001.nc

  ncclimo_version :

  4.8.1-alpha04

  DODS_EXTRA.Unlimited_Dimension :

  time

We need to convert our time to datetime to make it easier to compare.

cmip6_nearest['time'] = cmip6_nearest.indexes['time'].to_datetimeindex()

Next, we select the times we have data from the SGP site, specified earlier in the notebook.

cmip6_nearest = cmip6_nearest.sel(time=slice(start_date,
                                             end_date)).resample(time='1M').mean()

Calculate Monthly Mean Temperature at SGP

We can calculate the monthly average temperature at the SGP site using the resample method in Xarray.

arm_ds = arm_ds.sortby('time')
sgp_monthly_mean_temperature = arm_ds.temp_mean.resample(time='1M').mean().compute().rename('tas (ARM)')

We need to apply some data cleaning here too - converting our units of temperature to degrees Celsius for the CMIP6 data.

cmip6_monthly_mean_temperature = cmip6_nearest.tas.compute().metpy.quantify()

cmip6_monthly_mean_temperature = cmip6_monthly_mean_temperature.metpy.convert_units('degC').rename("tas (CMIP6)")

Visaulize the Output

Once we have our comparisons ready, we can visualize using hvPlot, which produces an interactive visualization!

esgf_plot = cmip6_monthly_mean_temperature.hvplot.bar(title='Average Surface Temperature \n near the Southern Great Plains Field Site',
                                                       xlabel='Time')
arm_plot = sgp_monthly_mean_temperature.hvplot.bar(ylabel='Average Temperature (degC)',
                                                    xlabel='Time')

esgf_plot * arm_plot

Summary

In this notebook, we searched for and opened a CMIP6 E3SM dataset using the ESGF API and OPeNDAP, and compared to an ARM dataset collected at the Southern Great Plains climate observatory.

What’s next?

We will see some more advanced examples of using the CMIP6 and obsverational data.

Resources and references

• ARM Surface Meteorological Handbook

previous

Regridding with xESMF and calculating a multi-model mean