Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Load CMIP6 Data with Intake-ESM

Authors
Affiliations
University at Albany (SUNY)
Argonne National Laboratory

Overview

Intake-ESM is an experimental new package that aims to provide a higher-level interface to searching and loading Earth System Model data archives, such as CMIP6. The package is under very active development, and features may be unstable. Please report any issues or suggestions on GitHub.

Prerequisites

ConceptsImportanceNotes
Intro to XarrayNecessary
Understanding of NetCDFHelpfulFamiliarity with metadata structure

  • Time to learn: 5 minutes

Imports

import xarray as xr
xr.set_options(display_style='html')
import intake
import warnings

Loading Data

Intake ESM works by parsing an ESM Collection Spec and converting it to an Intake catalog. The collection spec is stored in a .json file. Here we open it using Intake.

cat_url = "https://storage.googleapis.com/cmip6/pangeo-cmip6.json"
col = intake.open_esm_datastore(cat_url)
col
Loading...

We can now use Intake methods to search the collection, and, if desired, export a Pandas dataframe.

cat = col.search(experiment_id=['historical', 'ssp585'], table_id='Oyr', variable_id='o2',
                 grid_label='gn')
cat.df
Loading...

Intake knows how to automatically open the Datasets using Xarray. Furthermore, Intake-ESM contains special logic to concatenate and merge the individual results of our query into larger, more high-level aggregated Xarray Datasets.

#  We are quieting a large number of warnings about inefficient chunking here
warnings.filterwarnings("ignore", message="The specified chunks separate")

time_coder = xr.coders.CFDatetimeCoder(use_cftime=True)

dset_dict = cat.to_dataset_dict(zarr_kwargs={'consolidated': True}, 
                                storage_options={'token': 'anon'},
                                xarray_open_kwargs={'decode_times': time_coder},
                                )
list(dset_dict.keys())

--> The keys in the returned dictionary of datasets are constructed as follows:
	'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'
Loading...
Loading...
/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/google/api_core/_python_package_support.py:206: FutureWarning: Package google.api_core depends on grpcio, currently installed at version 1.82.1. grpcio < 1.83.0 does not support Post-Quantum Cryptography (PQC). Support for non-PQC environments is deprecated. In October 2026, Google Cloud Python packages will raise their minimum requirements (including google-api-core, grpcio, and grpcio-status) to enforce grpcio >= 1.83.0. For more details on Google Cloud's post-quantum security migration, visit: https://cloud.google.com/security/resources/post-quantum-cryptography
  warnings.warn(
/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/google/auth/transport/grpc.py:44: FutureWarning: grpcio < 1.83.0 does not support Post-Quantum Cryptography (PQC). Support for non-PQC environments is deprecated. In October 2026, google-auth will raise its minimum requirements to enforce grpcio >= 1.83.0. For more details on Google Cloud's post-quantum security migration, visit: https://cloud.google.com/security/resources/post-quantum-cryptography
  warnings.warn(

/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/google/api_core/_python_package_support.py:206: FutureWarning: Package google.cloud._storage_v2 depends on grpcio, currently installed at version 1.82.1. grpcio < 1.83.0 does not support Post-Quantum Cryptography (PQC). Support for non-PQC environments is deprecated. In October 2026, Google Cloud Python packages will raise their minimum requirements (including google-api-core, grpcio, and grpcio-status) to enforce grpcio >= 1.83.0. For more details on Google Cloud's post-quantum security migration, visit: https://cloud.google.com/security/resources/post-quantum-cryptography
  warnings.warn(
/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/google/api_core/_python_package_support.py:206: FutureWarning: Package google.cloud.storage_control_v2 depends on grpcio, currently installed at version 1.82.1. grpcio < 1.83.0 does not support Post-Quantum Cryptography (PQC). Support for non-PQC environments is deprecated. In October 2026, Google Cloud Python packages will raise their minimum requirements (including google-api-core, grpcio, and grpcio-status) to enforce grpcio >= 1.83.0. For more details on Google Cloud's post-quantum security migration, visit: https://cloud.google.com/security/resources/post-quantum-cryptography
  warnings.warn(

/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/pyarrow/compute.py:230: FutureWarning: Specifying null_placement in RankOptions is deprecated as of 25.0.0. Specify null_placement per sort_key instead.
  return options_class(*args, **kwargs)

/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/pyarrow/compute.py:230: FutureWarning: Specifying null_placement in RankOptions is deprecated as of 25.0.0. Specify null_placement per sort_key instead.
  return options_class(*args, **kwargs)

/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/pyarrow/compute.py:230: FutureWarning: Specifying null_placement in RankOptions is deprecated as of 25.0.0. Specify null_placement per sort_key instead.
  return options_class(*args, **kwargs)

/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/pyarrow/compute.py:230: FutureWarning: Specifying null_placement in RankOptions is deprecated as of 25.0.0. Specify null_placement per sort_key instead.
  return options_class(*args, **kwargs)

/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/pyarrow/compute.py:230: FutureWarning: Specifying null_placement in RankOptions is deprecated as of 25.0.0. Specify null_placement per sort_key instead.
  return options_class(*args, **kwargs)

/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/pyarrow/compute.py:230: FutureWarning: Specifying null_placement in RankOptions is deprecated as of 25.0.0. Specify null_placement per sort_key instead.
  return options_class(*args, **kwargs)

/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/intake_esm/source.py:308: FutureWarning: In a future version of xarray the default value for coords will change from coords='different' to coords='minimal'. This is likely to lead to different results when multiple datasets have matching variables with overlapping values. To opt in to new defaults and get rid of these warnings now use `set_options(use_new_combine_kwarg_defaults=True) or set coords explicitly.
  self._ds = xr.combine_by_coords(

/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/pyarrow/compute.py:230: FutureWarning: Specifying null_placement in RankOptions is deprecated as of 25.0.0. Specify null_placement per sort_key instead.
  return options_class(*args, **kwargs)

/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/pyarrow/compute.py:230: FutureWarning: Specifying null_placement in RankOptions is deprecated as of 25.0.0. Specify null_placement per sort_key instead.
  return options_class(*args, **kwargs)

/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/pyarrow/compute.py:230: FutureWarning: Specifying null_placement in RankOptions is deprecated as of 25.0.0. Specify null_placement per sort_key instead.
  return options_class(*args, **kwargs)

/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/pyarrow/compute.py:230: FutureWarning: Specifying null_placement in RankOptions is deprecated as of 25.0.0. Specify null_placement per sort_key instead.
  return options_class(*args, **kwargs)

/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/pyarrow/compute.py:230: FutureWarning: Specifying null_placement in RankOptions is deprecated as of 25.0.0. Specify null_placement per sort_key instead.
  return options_class(*args, **kwargs)

/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/pyarrow/compute.py:230: FutureWarning: Specifying null_placement in RankOptions is deprecated as of 25.0.0. Specify null_placement per sort_key instead.
  return options_class(*args, **kwargs)

/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/pyarrow/compute.py:230: FutureWarning: Specifying null_placement in RankOptions is deprecated as of 25.0.0. Specify null_placement per sort_key instead.
  return options_class(*args, **kwargs)

/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/pyarrow/compute.py:230: FutureWarning: Specifying null_placement in RankOptions is deprecated as of 25.0.0. Specify null_placement per sort_key instead.
  return options_class(*args, **kwargs)
/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/intake_esm/source.py:308: FutureWarning: In a future version of xarray the default value for join will change from join='outer' to join='exact'. This change will result in the following ValueError: cannot be aligned with join='exact' because index/labels/sizes are not equal along these coordinates (dimensions): 'time' ('time',) The recommendation is to set join explicitly for this case.
  self._ds = xr.combine_by_coords(
/home/runner/micromamba/envs/cmip6-cookbook-dev/lib/python3.14/site-packages/pyarrow/compute.py:230: FutureWarning: Specifying null_placement in RankOptions is deprecated as of 25.0.0. Specify null_placement per sort_key instead.
  return options_class(*args, **kwargs)
['ScenarioMIP.MRI.MRI-ESM2-0.ssp585.Oyr.gn', 'ScenarioMIP.MIROC.MIROC-ES2L.ssp585.Oyr.gn', 'ScenarioMIP.DWD.MPI-ESM1-2-HR.ssp585.Oyr.gn', 'CMIP.NCC.NorESM2-LM.historical.Oyr.gn', 'ScenarioMIP.CCCma.CanESM5-CanOE.ssp585.Oyr.gn', 'ScenarioMIP.DKRZ.MPI-ESM1-2-HR.ssp585.Oyr.gn', 'ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp585.Oyr.gn', 'CMIP.MIROC.MIROC-ES2L.historical.Oyr.gn', 'CMIP.NCC.NorESM2-MM.historical.Oyr.gn', 'ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.Oyr.gn', 'CMIP.HAMMOZ-Consortium.MPI-ESM-1-2-HAM.historical.Oyr.gn', 'ScenarioMIP.NCC.NorESM2-MM.ssp585.Oyr.gn', 'ScenarioMIP.EC-Earth-Consortium.EC-Earth3-CC.ssp585.Oyr.gn', 'CMIP.IPSL.IPSL-CM5A2-INCA.historical.Oyr.gn', 'CMIP.MPI-M.MPI-ESM1-2-LR.historical.Oyr.gn', 'ScenarioMIP.NCC.NorESM2-LM.ssp585.Oyr.gn', 'CMIP.MPI-M.MPI-ESM1-2-HR.historical.Oyr.gn', 'CMIP.CCCma.CanESM5-CanOE.historical.Oyr.gn', 'CMIP.CSIRO.ACCESS-ESM1-5.historical.Oyr.gn', 'ScenarioMIP.CMCC.CMCC-ESM2.ssp585.Oyr.gn', 'ScenarioMIP.NCAR.CESM2.ssp585.Oyr.gn', 'CMIP.MRI.MRI-ESM2-0.historical.Oyr.gn', 'CMIP.CMCC.CMCC-ESM2.historical.Oyr.gn', 'CMIP.EC-Earth-Consortium.EC-Earth3-CC.historical.Oyr.gn', 'CMIP.CCCma.CanESM5.historical.Oyr.gn', 'ScenarioMIP.CCCma.CanESM5.ssp585.Oyr.gn', 'CMIP.IPSL.IPSL-CM6A-LR.historical.Oyr.gn']
ds = dset_dict['CMIP.CCCma.CanESM5.historical.Oyr.gn']
ds
Loading...

Summary

In this notebook, we used Intake-ESM to open an Xarray Dataset for one particular model and experiment.

What’s next?

We will see an example of downloading a dataset with fsspec and zarr.

Additional resources

Foundations
Google Cloud CMIP6 Public Data: Basic Python Example
Example workflows
Estimating Equilibrium Climate Sensitivity