{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Google Cloud CMIP6 Public Data: Basic Python Example"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Overview\n",
"\n",
"This notebooks shows how to query the Google Cloud CMIP6 catalog and load the data using Python."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"| Concepts | Importance | Notes |\n",
"| --- | --- | --- |\n",
"| [Intro to Xarray](https://foundations.projectpythia.org/core/xarray/xarray-intro.html) | Necessary | |\n",
"| [Understanding of NetCDF](https://foundations.projectpythia.org/core/data-formats/netcdf-cf.html) | Helpful | Familiarity with metadata structure |\n",
"\n",
"- **Time to learn**: 10 minutes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Imports"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"execution": {
"iopub.execute_input": "2023-11-20T21:33:08.125945Z",
"iopub.status.busy": "2023-11-20T21:33:08.125721Z",
"iopub.status.idle": "2023-11-20T21:33:09.109328Z",
"shell.execute_reply": "2023-11-20T21:33:09.108359Z"
}
},
"outputs": [],
"source": [
"from matplotlib import pyplot as plt\n",
"import numpy as np\n",
"import pandas as pd\n",
"import xarray as xr\n",
"import zarr\n",
"import fsspec\n",
"import nc_time_axis\n",
"\n",
"%matplotlib inline\n",
"plt.rcParams['figure.figsize'] = 12, 6"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Browse Catalog"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The data catatalog is stored as a CSV file. Here we read it with Pandas."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"execution": {
"iopub.execute_input": "2023-11-20T21:33:09.113529Z",
"iopub.status.busy": "2023-11-20T21:33:09.113042Z",
"iopub.status.idle": "2023-11-20T21:33:11.388982Z",
"shell.execute_reply": "2023-11-20T21:33:11.388281Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"
\n", " | activity_id | \n", "institution_id | \n", "source_id | \n", "experiment_id | \n", "member_id | \n", "table_id | \n", "variable_id | \n", "grid_label | \n", "zstore | \n", "dcpp_init_year | \n", "version | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|
0 | \n", "HighResMIP | \n", "CMCC | \n", "CMCC-CM2-HR4 | \n", "highresSST-present | \n", "r1i1p1f1 | \n", "Amon | \n", "ps | \n", "gn | \n", "gs://cmip6/CMIP6/HighResMIP/CMCC/CMCC-CM2-HR4/... | \n", "NaN | \n", "20170706 | \n", "
1 | \n", "HighResMIP | \n", "CMCC | \n", "CMCC-CM2-HR4 | \n", "highresSST-present | \n", "r1i1p1f1 | \n", "Amon | \n", "rsds | \n", "gn | \n", "gs://cmip6/CMIP6/HighResMIP/CMCC/CMCC-CM2-HR4/... | \n", "NaN | \n", "20170706 | \n", "
2 | \n", "HighResMIP | \n", "CMCC | \n", "CMCC-CM2-HR4 | \n", "highresSST-present | \n", "r1i1p1f1 | \n", "Amon | \n", "rlus | \n", "gn | \n", "gs://cmip6/CMIP6/HighResMIP/CMCC/CMCC-CM2-HR4/... | \n", "NaN | \n", "20170706 | \n", "
3 | \n", "HighResMIP | \n", "CMCC | \n", "CMCC-CM2-HR4 | \n", "highresSST-present | \n", "r1i1p1f1 | \n", "Amon | \n", "rlds | \n", "gn | \n", "gs://cmip6/CMIP6/HighResMIP/CMCC/CMCC-CM2-HR4/... | \n", "NaN | \n", "20170706 | \n", "
4 | \n", "HighResMIP | \n", "CMCC | \n", "CMCC-CM2-HR4 | \n", "highresSST-present | \n", "r1i1p1f1 | \n", "Amon | \n", "psl | \n", "gn | \n", "gs://cmip6/CMIP6/HighResMIP/CMCC/CMCC-CM2-HR4/... | \n", "NaN | \n", "20170706 | \n", "
\n", " | activity_id | \n", "institution_id | \n", "source_id | \n", "experiment_id | \n", "member_id | \n", "table_id | \n", "variable_id | \n", "grid_label | \n", "zstore | \n", "dcpp_init_year | \n", "version | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|
973 | \n", "CMIP | \n", "NOAA-GFDL | \n", "GFDL-ESM4 | \n", "historical | \n", "r3i1p1f1 | \n", "Amon | \n", "tas | \n", "gr1 | \n", "gs://cmip6/CMIP6/CMIP/NOAA-GFDL/GFDL-ESM4/hist... | \n", "NaN | \n", "20180701 | \n", "
1766 | \n", "CMIP | \n", "NOAA-GFDL | \n", "GFDL-ESM4 | \n", "historical | \n", "r2i1p1f1 | \n", "Amon | \n", "tas | \n", "gr1 | \n", "gs://cmip6/CMIP6/CMIP/NOAA-GFDL/GFDL-ESM4/hist... | \n", "NaN | \n", "20180701 | \n", "
8074 | \n", "CMIP | \n", "NOAA-GFDL | \n", "GFDL-CM4 | \n", "historical | \n", "r1i1p1f1 | \n", "Amon | \n", "tas | \n", "gr1 | \n", "gs://cmip6/CMIP6/CMIP/NOAA-GFDL/GFDL-CM4/histo... | \n", "NaN | \n", "20180701 | \n", "
22185 | \n", "CMIP | \n", "IPSL | \n", "IPSL-CM6A-LR | \n", "historical | \n", "r8i1p1f1 | \n", "Amon | \n", "tas | \n", "gr | \n", "gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor... | \n", "NaN | \n", "20180803 | \n", "
22298 | \n", "CMIP | \n", "IPSL | \n", "IPSL-CM6A-LR | \n", "historical | \n", "r2i1p1f1 | \n", "Amon | \n", "tas | \n", "gr | \n", "gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor... | \n", "NaN | \n", "20180803 | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
522952 | \n", "CMIP | \n", "MRI | \n", "MRI-ESM2-0 | \n", "historical | \n", "r7i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/MRI/MRI-ESM2-0/historica... | \n", "NaN | \n", "20210813 | \n", "
523274 | \n", "CMIP | \n", "MRI | \n", "MRI-ESM2-0 | \n", "historical | \n", "r6i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/MRI/MRI-ESM2-0/historica... | \n", "NaN | \n", "20210907 | \n", "
523712 | \n", "CMIP | \n", "CMCC | \n", "CMCC-CM2-SR5 | \n", "historical | \n", "r3i1p2f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/CMCC/CMCC-CM2-SR5/histor... | \n", "NaN | \n", "20211108 | \n", "
523721 | \n", "CMIP | \n", "CMCC | \n", "CMCC-CM2-SR5 | \n", "historical | \n", "r2i1p2f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/CMCC/CMCC-CM2-SR5/histor... | \n", "NaN | \n", "20211109 | \n", "
523769 | \n", "CMIP | \n", "EC-Earth-Consortium | \n", "EC-Earth3-Veg | \n", "historical | \n", "r1i1p1f1 | \n", "Amon | \n", "tas | \n", "gr | \n", "gs://cmip6/CMIP6/CMIP/EC-Earth-Consortium/EC-E... | \n", "NaN | \n", "20211207 | \n", "
635 rows × 11 columns
\n", "\n", " | activity_id | \n", "institution_id | \n", "source_id | \n", "experiment_id | \n", "member_id | \n", "table_id | \n", "variable_id | \n", "grid_label | \n", "zstore | \n", "dcpp_init_year | \n", "version | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|
56049 | \n", "CMIP | \n", "NCAR | \n", "CESM2-WACCM | \n", "historical | \n", "r2i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2-WACCM/histori... | \n", "NaN | \n", "20190227 | \n", "
56143 | \n", "CMIP | \n", "NCAR | \n", "CESM2-WACCM | \n", "historical | \n", "r3i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2-WACCM/histori... | \n", "NaN | \n", "20190227 | \n", "
56326 | \n", "CMIP | \n", "NCAR | \n", "CESM2-WACCM | \n", "historical | \n", "r1i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2-WACCM/histori... | \n", "NaN | \n", "20190227 | \n", "
59875 | \n", "CMIP | \n", "NCAR | \n", "CESM2 | \n", "historical | \n", "r1i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2/historical/r1... | \n", "NaN | \n", "20190308 | \n", "
61655 | \n", "CMIP | \n", "NCAR | \n", "CESM2 | \n", "historical | \n", "r4i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2/historical/r4... | \n", "NaN | \n", "20190308 | \n", "
61862 | \n", "CMIP | \n", "NCAR | \n", "CESM2 | \n", "historical | \n", "r5i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2/historical/r5... | \n", "NaN | \n", "20190308 | \n", "
62691 | \n", "CMIP | \n", "NCAR | \n", "CESM2 | \n", "historical | \n", "r2i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2/historical/r2... | \n", "NaN | \n", "20190308 | \n", "
63131 | \n", "CMIP | \n", "NCAR | \n", "CESM2 | \n", "historical | \n", "r3i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2/historical/r3... | \n", "NaN | \n", "20190308 | \n", "
63266 | \n", "CMIP | \n", "NCAR | \n", "CESM2 | \n", "historical | \n", "r6i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2/historical/r6... | \n", "NaN | \n", "20190308 | \n", "
64615 | \n", "CMIP | \n", "NCAR | \n", "CESM2 | \n", "historical | \n", "r8i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2/historical/r8... | \n", "NaN | \n", "20190311 | \n", "
64914 | \n", "CMIP | \n", "NCAR | \n", "CESM2 | \n", "historical | \n", "r7i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2/historical/r7... | \n", "NaN | \n", "20190311 | \n", "
64983 | \n", "CMIP | \n", "NCAR | \n", "CESM2 | \n", "historical | \n", "r9i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2/historical/r9... | \n", "NaN | \n", "20190311 | \n", "
66341 | \n", "CMIP | \n", "NCAR | \n", "CESM2 | \n", "historical | \n", "r10i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2/historical/r1... | \n", "NaN | \n", "20190313 | \n", "
200772 | \n", "CMIP | \n", "NCAR | \n", "CESM2 | \n", "historical | \n", "r11i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2/historical/r1... | \n", "NaN | \n", "20190514 | \n", "
385224 | \n", "CMIP | \n", "NCAR | \n", "CESM2-FV2 | \n", "historical | \n", "r1i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2-FV2/historica... | \n", "NaN | \n", "20191120 | \n", "
386297 | \n", "CMIP | \n", "NCAR | \n", "CESM2-WACCM-FV2 | \n", "historical | \n", "r1i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2-WACCM-FV2/his... | \n", "NaN | \n", "20191120 | \n", "
420771 | \n", "CMIP | \n", "NCAR | \n", "CESM2-WACCM-FV2 | \n", "historical | \n", "r3i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2-WACCM-FV2/his... | \n", "NaN | \n", "20200226 | \n", "
421251 | \n", "CMIP | \n", "NCAR | \n", "CESM2-WACCM-FV2 | \n", "historical | \n", "r2i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2-WACCM-FV2/his... | \n", "NaN | \n", "20200226 | \n", "
422013 | \n", "CMIP | \n", "NCAR | \n", "CESM2-FV2 | \n", "historical | \n", "r3i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2-FV2/historica... | \n", "NaN | \n", "20200226 | \n", "
422459 | \n", "CMIP | \n", "NCAR | \n", "CESM2-FV2 | \n", "historical | \n", "r2i1p1f1 | \n", "Amon | \n", "tas | \n", "gn | \n", "gs://cmip6/CMIP6/CMIP/NCAR/CESM2-FV2/historica... | \n", "NaN | \n", "20200226 | \n", "
<xarray.Dataset>\n", "Dimensions: (lat: 96, nbnd: 2, lon: 144, time: 1980)\n", "Coordinates:\n", " * lat (lat) float64 -90.0 -88.11 -86.21 -84.32 ... 86.21 88.11 90.0\n", " lat_bnds (lat, nbnd) float64 dask.array<chunksize=(96, 2), meta=np.ndarray>\n", " * lon (lon) float64 0.0 2.5 5.0 7.5 10.0 ... 350.0 352.5 355.0 357.5\n", " lon_bnds (lon, nbnd) float64 dask.array<chunksize=(144, 2), meta=np.ndarray>\n", " * time (time) object 1850-01-15 12:00:00 ... 2014-12-15 12:00:00\n", " time_bnds (time, nbnd) object dask.array<chunksize=(1980, 2), meta=np.ndarray>\n", "Dimensions without coordinates: nbnd\n", "Data variables:\n", " tas (time, lat, lon) float32 dask.array<chunksize=(990, 96, 144), meta=np.ndarray>\n", "Attributes: (12/48)\n", " Conventions: CF-1.7 CMIP-6.2\n", " DODS_EXTRA.Unlimited_Dimension: time\n", " activity_id: CMIP\n", " branch_method: standard\n", " branch_time_in_child: 674885.0\n", " branch_time_in_parent: 10950.0\n", " ... ...\n", " tracking_id: hdl:21.14100/99cdfde8-5b6d-452b-9b78-62a...\n", " variable_id: tas\n", " variant_info: CMIP6 CESM2-FV2 historical experiment (1...\n", " variant_label: r2i1p1f1\n", " netcdf_tracking_ids: hdl:21.14100/99cdfde8-5b6d-452b-9b78-62a...\n", " version_id: v20200226
<xarray.Dataset>\n", "Dimensions: (lat: 192, lon: 288, nbnd: 2)\n", "Coordinates:\n", " * lat (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 88.12 89.06 90.0\n", " lat_bnds (lat, nbnd) float32 dask.array<chunksize=(192, 2), meta=np.ndarray>\n", " * lon (lon) float64 0.0 1.25 2.5 3.75 5.0 ... 355.0 356.2 357.5 358.8\n", " lon_bnds (lon, nbnd) float32 dask.array<chunksize=(288, 2), meta=np.ndarray>\n", "Dimensions without coordinates: nbnd\n", "Data variables:\n", " areacella (lat, lon) float32 dask.array<chunksize=(192, 288), meta=np.ndarray>\n", "Attributes: (12/47)\n", " Conventions: CF-1.7 CMIP-6.2\n", " activity_id: CMIP\n", " branch_method: no parent\n", " branch_time_in_child: 711385.0\n", " branch_time_in_parent: 0.0\n", " case_id: 38\n", " ... ...\n", " variable_id: areacella\n", " variant_info: f.e21.FHIST_BGC.f09_f09_mg17.CMIP6-AMIP.001 \r", "\\n\r", "\\...\n", " variant_label: r1i1p1f1\n", " status: 2019-11-04;created;by nhn2@columbia.edu\n", " netcdf_tracking_ids: hdl:21.14100/23fa9dc3-4f8f-4943-b99b-58eb804c06f0\n", " version_id: v20190218
<xarray.DataArray (time: 1980)>\n", "dask.array<truediv, shape=(1980,), dtype=float32, chunksize=(990,), chunktype=numpy.ndarray>\n", "Coordinates:\n", " * time (time) object 1850-01-15 12:00:00 ... 2014-12-15 12:00:00
<xarray.DataArray (time: 1980)>\n", "array([0.00407822, 0.00405088, 0.0039282 , ..., 0.00411686, 0.00414686,\n", " 0.00418852], dtype=float32)\n", "Coordinates:\n", " * time (time) object 1850-01-15 12:00:00 ... 2014-12-15 12:00:00