{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![\"Project](\"images/ProjectPythia_Logo_Final-01-Blue.svg\")
\n",
"![\"Pangeo](\"images/logos/pangeo_simple_logo.svg\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Along Track Altimetry Analysis"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Overview\n",
"\n",
"1. Using CNES altimetry data\n",
"1. Visualizing data using hvplot\n",
"1. Use xhistogram to plot multidimensional data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"| Concepts | Importance | Notes |\n",
"| --- | --- | --- |\n",
"| [Intro to Pandas](https://foundations.projectpythia.org/core/pandas/pandas.html) | Helpful | |\n",
"| [Using hvplot](https://hvplot.holoviz.org) | Helpful | Matplotlib knowledge also helpful |\n",
"| [Dask](https://docs.dask.org/en/stable/) | Helpful | |\n",
"| [xhistogram](https://xhistogram.readthedocs.io/en/latest/) | Helpful | |\n",
"\n",
"- **Time to learn**: 15 minutes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Imports"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"import fsspec\n",
"import xarray as xr\n",
"import numpy as np\n",
"import hvplot\n",
"import hvplot.dask\n",
"import hvplot.pandas\n",
"import hvplot.xarray\n",
"from xhistogram.xarray import histogram\n",
"from intake import open_catalog"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load Data\n",
"\n",
"The analysis ready along-track altimetry data were prepared by CNES. They are catalogged in the Pangeo Cloud Data Catalog here: https://catalog.pangeo.io/browse/master/ocean/altimetry/\n",
"\n",
"We will work with Jason 3."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['al', 'alg', 'c2', 'e1', 'e1g', 'e2', 'en', 'enn', 'g2', 'h2', 'j1', 'j1g', 'j1n', 'j2', 'j2g', 'j2n', 'j3', 's3a', 's3b', 'tp', 'tpn']\n"
]
},
{
"ename": "TypeError",
"evalue": "Invalid variable type: value should be str, int or float, got None of type ",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m/Users/dquint/Desktop/physical-oceanography-cookbook/notebooks/02_along_track.ipynb Cell 10'\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 2\u001b[0m cat \u001b[39m=\u001b[39m open_catalog(\u001b[39m\"\u001b[39m\u001b[39mhttps://raw.githubusercontent.com/pangeo-data/pangeo-datastore/master/intake-catalogs/ocean/altimetry.yaml\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m 3\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39mlist\u001b[39m(cat))\n\u001b[0;32m----> 4\u001b[0m ds \u001b[39m=\u001b[39m cat[\u001b[39m'\u001b[39;49m\u001b[39mj3\u001b[39;49m\u001b[39m'\u001b[39;49m]\u001b[39m.\u001b[39;49mto_dask()\n\u001b[1;32m 5\u001b[0m ds\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/intake_xarray/base.py:69\u001b[0m, in \u001b[0;36mDataSourceMixin.to_dask\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 67\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mto_dask\u001b[39m(\u001b[39mself\u001b[39m):\n\u001b[1;32m 68\u001b[0m \u001b[39m\"\"\"Return xarray object where variables are dask arrays\"\"\"\u001b[39;00m\n\u001b[0;32m---> 69\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mread_chunked()\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/intake_xarray/base.py:44\u001b[0m, in \u001b[0;36mDataSourceMixin.read_chunked\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 42\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mread_chunked\u001b[39m(\u001b[39mself\u001b[39m):\n\u001b[1;32m 43\u001b[0m \u001b[39m\"\"\"Return xarray object (which will have chunks)\"\"\"\u001b[39;00m\n\u001b[0;32m---> 44\u001b[0m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_load_metadata()\n\u001b[1;32m 45\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_ds\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/intake/source/base.py:236\u001b[0m, in \u001b[0;36mDataSourceBase._load_metadata\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 234\u001b[0m \u001b[39m\"\"\"load metadata only if needed\"\"\"\u001b[39;00m\n\u001b[1;32m 235\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_schema \u001b[39mis\u001b[39;00m \u001b[39mNone\u001b[39;00m:\n\u001b[0;32m--> 236\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_schema \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_get_schema()\n\u001b[1;32m 237\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mdtype \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_schema\u001b[39m.\u001b[39mdtype\n\u001b[1;32m 238\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mshape \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_schema\u001b[39m.\u001b[39mshape\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/intake_xarray/base.py:18\u001b[0m, in \u001b[0;36mDataSourceMixin._get_schema\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 15\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39murlpath \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_get_cache(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39murlpath)[\u001b[39m0\u001b[39m]\n\u001b[1;32m 17\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_ds \u001b[39mis\u001b[39;00m \u001b[39mNone\u001b[39;00m:\n\u001b[0;32m---> 18\u001b[0m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_open_dataset()\n\u001b[1;32m 20\u001b[0m metadata \u001b[39m=\u001b[39m {\n\u001b[1;32m 21\u001b[0m \u001b[39m'\u001b[39m\u001b[39mdims\u001b[39m\u001b[39m'\u001b[39m: \u001b[39mdict\u001b[39m(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_ds\u001b[39m.\u001b[39mdims),\n\u001b[1;32m 22\u001b[0m \u001b[39m'\u001b[39m\u001b[39mdata_vars\u001b[39m\u001b[39m'\u001b[39m: {k: \u001b[39mlist\u001b[39m(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_ds[k]\u001b[39m.\u001b[39mcoords)\n\u001b[1;32m 23\u001b[0m \u001b[39mfor\u001b[39;00m k \u001b[39min\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_ds\u001b[39m.\u001b[39mdata_vars\u001b[39m.\u001b[39mkeys()},\n\u001b[1;32m 24\u001b[0m \u001b[39m'\u001b[39m\u001b[39mcoords\u001b[39m\u001b[39m'\u001b[39m: \u001b[39mtuple\u001b[39m(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_ds\u001b[39m.\u001b[39mcoords\u001b[39m.\u001b[39mkeys()),\n\u001b[1;32m 25\u001b[0m }\n\u001b[1;32m 26\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mgetattr\u001b[39m(\u001b[39mself\u001b[39m, \u001b[39m'\u001b[39m\u001b[39mon_server\u001b[39m\u001b[39m'\u001b[39m, \u001b[39mFalse\u001b[39;00m):\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/intake_xarray/xzarr.py:46\u001b[0m, in \u001b[0;36mZarrSource._open_dataset\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 44\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_ds \u001b[39m=\u001b[39m xr\u001b[39m.\u001b[39mopen_mfdataset(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39murlpath, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkw)\n\u001b[1;32m 45\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[0;32m---> 46\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_ds \u001b[39m=\u001b[39m xr\u001b[39m.\u001b[39;49mopen_dataset(\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49murlpath, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkw)\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/xarray/backends/api.py:495\u001b[0m, in \u001b[0;36mopen_dataset\u001b[0;34m(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, backend_kwargs, *args, **kwargs)\u001b[0m\n\u001b[1;32m 483\u001b[0m decoders \u001b[39m=\u001b[39m _resolve_decoders_kwargs(\n\u001b[1;32m 484\u001b[0m decode_cf,\n\u001b[1;32m 485\u001b[0m open_backend_dataset_parameters\u001b[39m=\u001b[39mbackend\u001b[39m.\u001b[39mopen_dataset_parameters,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 491\u001b[0m decode_coords\u001b[39m=\u001b[39mdecode_coords,\n\u001b[1;32m 492\u001b[0m )\n\u001b[1;32m 494\u001b[0m overwrite_encoded_chunks \u001b[39m=\u001b[39m kwargs\u001b[39m.\u001b[39mpop(\u001b[39m\"\u001b[39m\u001b[39moverwrite_encoded_chunks\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39mNone\u001b[39;00m)\n\u001b[0;32m--> 495\u001b[0m backend_ds \u001b[39m=\u001b[39m backend\u001b[39m.\u001b[39;49mopen_dataset(\n\u001b[1;32m 496\u001b[0m filename_or_obj,\n\u001b[1;32m 497\u001b[0m drop_variables\u001b[39m=\u001b[39;49mdrop_variables,\n\u001b[1;32m 498\u001b[0m \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mdecoders,\n\u001b[1;32m 499\u001b[0m \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs,\n\u001b[1;32m 500\u001b[0m )\n\u001b[1;32m 501\u001b[0m ds \u001b[39m=\u001b[39m _dataset_from_backend_dataset(\n\u001b[1;32m 502\u001b[0m backend_ds,\n\u001b[1;32m 503\u001b[0m filename_or_obj,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 510\u001b[0m \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs,\n\u001b[1;32m 511\u001b[0m )\n\u001b[1;32m 512\u001b[0m \u001b[39mreturn\u001b[39;00m ds\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/xarray/backends/zarr.py:800\u001b[0m, in \u001b[0;36mZarrBackendEntrypoint.open_dataset\u001b[0;34m(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, synchronizer, consolidated, chunk_store, storage_options, stacklevel)\u001b[0m\n\u001b[1;32m 780\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mopen_dataset\u001b[39m(\n\u001b[1;32m 781\u001b[0m \u001b[39mself\u001b[39m,\n\u001b[1;32m 782\u001b[0m filename_or_obj,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 796\u001b[0m stacklevel\u001b[39m=\u001b[39m\u001b[39m3\u001b[39m,\n\u001b[1;32m 797\u001b[0m ):\n\u001b[1;32m 799\u001b[0m filename_or_obj \u001b[39m=\u001b[39m _normalize_path(filename_or_obj)\n\u001b[0;32m--> 800\u001b[0m store \u001b[39m=\u001b[39m ZarrStore\u001b[39m.\u001b[39;49mopen_group(\n\u001b[1;32m 801\u001b[0m filename_or_obj,\n\u001b[1;32m 802\u001b[0m group\u001b[39m=\u001b[39;49mgroup,\n\u001b[1;32m 803\u001b[0m mode\u001b[39m=\u001b[39;49mmode,\n\u001b[1;32m 804\u001b[0m synchronizer\u001b[39m=\u001b[39;49msynchronizer,\n\u001b[1;32m 805\u001b[0m consolidated\u001b[39m=\u001b[39;49mconsolidated,\n\u001b[1;32m 806\u001b[0m consolidate_on_close\u001b[39m=\u001b[39;49m\u001b[39mFalse\u001b[39;49;00m,\n\u001b[1;32m 807\u001b[0m chunk_store\u001b[39m=\u001b[39;49mchunk_store,\n\u001b[1;32m 808\u001b[0m storage_options\u001b[39m=\u001b[39;49mstorage_options,\n\u001b[1;32m 809\u001b[0m stacklevel\u001b[39m=\u001b[39;49mstacklevel \u001b[39m+\u001b[39;49m \u001b[39m1\u001b[39;49m,\n\u001b[1;32m 810\u001b[0m )\n\u001b[1;32m 812\u001b[0m store_entrypoint \u001b[39m=\u001b[39m StoreBackendEntrypoint()\n\u001b[1;32m 813\u001b[0m \u001b[39mwith\u001b[39;00m close_on_error(store):\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/xarray/backends/zarr.py:368\u001b[0m, in \u001b[0;36mZarrStore.open_group\u001b[0;34m(cls, store, mode, synchronizer, group, consolidated, consolidate_on_close, chunk_store, storage_options, append_dim, write_region, safe_chunks, stacklevel)\u001b[0m\n\u001b[1;32m 365\u001b[0m zarr_group \u001b[39m=\u001b[39m zarr\u001b[39m.\u001b[39mopen_group(store, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mopen_kwargs)\n\u001b[1;32m 366\u001b[0m \u001b[39melif\u001b[39;00m consolidated:\n\u001b[1;32m 367\u001b[0m \u001b[39m# TODO: an option to pass the metadata_key keyword\u001b[39;00m\n\u001b[0;32m--> 368\u001b[0m zarr_group \u001b[39m=\u001b[39m zarr\u001b[39m.\u001b[39;49mopen_consolidated(store, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mopen_kwargs)\n\u001b[1;32m 369\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m 370\u001b[0m zarr_group \u001b[39m=\u001b[39m zarr\u001b[39m.\u001b[39mopen_group(store, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mopen_kwargs)\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/zarr/convenience.py:1304\u001b[0m, in \u001b[0;36mopen_consolidated\u001b[0;34m(store, metadata_key, mode, **kwargs)\u001b[0m\n\u001b[1;32m 1299\u001b[0m \u001b[39mraise\u001b[39;00m \u001b[39mValueError\u001b[39;00m(\n\u001b[1;32m 1300\u001b[0m \u001b[39m\"\u001b[39m\u001b[39mpath must be provided to open a Zarr 3.x consolidated store\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[1;32m 1301\u001b[0m )\n\u001b[1;32m 1303\u001b[0m \u001b[39m# setup metadata store\u001b[39;00m\n\u001b[0;32m-> 1304\u001b[0m meta_store \u001b[39m=\u001b[39m ConsolidatedStoreClass(store, metadata_key\u001b[39m=\u001b[39;49mmetadata_key)\n\u001b[1;32m 1306\u001b[0m \u001b[39m# pass through\u001b[39;00m\n\u001b[1;32m 1307\u001b[0m chunk_store \u001b[39m=\u001b[39m kwargs\u001b[39m.\u001b[39mpop(\u001b[39m'\u001b[39m\u001b[39mchunk_store\u001b[39m\u001b[39m'\u001b[39m, \u001b[39mNone\u001b[39;00m) \u001b[39mor\u001b[39;00m store\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/zarr/storage.py:2853\u001b[0m, in \u001b[0;36mConsolidatedMetadataStore.__init__\u001b[0;34m(self, store, metadata_key)\u001b[0m\n\u001b[1;32m 2850\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mstore \u001b[39m=\u001b[39m Store\u001b[39m.\u001b[39m_ensure_store(store)\n\u001b[1;32m 2852\u001b[0m \u001b[39m# retrieve consolidated metadata\u001b[39;00m\n\u001b[0;32m-> 2853\u001b[0m meta \u001b[39m=\u001b[39m json_loads(\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mstore[metadata_key])\n\u001b[1;32m 2855\u001b[0m \u001b[39m# check format of consolidated metadata\u001b[39;00m\n\u001b[1;32m 2856\u001b[0m consolidated_format \u001b[39m=\u001b[39m meta\u001b[39m.\u001b[39mget(\u001b[39m'\u001b[39m\u001b[39mzarr_consolidated_format\u001b[39m\u001b[39m'\u001b[39m, \u001b[39mNone\u001b[39;00m)\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/zarr/storage.py:1369\u001b[0m, in \u001b[0;36mFSStore.__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m 1367\u001b[0m key \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_normalize_key(key)\n\u001b[1;32m 1368\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m-> 1369\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mmap[key]\n\u001b[1;32m 1370\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mexceptions \u001b[39mas\u001b[39;00m e:\n\u001b[1;32m 1371\u001b[0m \u001b[39mraise\u001b[39;00m \u001b[39mKeyError\u001b[39;00m(key) \u001b[39mfrom\u001b[39;00m \u001b[39me\u001b[39;00m\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/fsspec/mapping.py:137\u001b[0m, in \u001b[0;36mFSMap.__getitem__\u001b[0;34m(self, key, default)\u001b[0m\n\u001b[1;32m 135\u001b[0m k \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_key_to_str(key)\n\u001b[1;32m 136\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m--> 137\u001b[0m result \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mfs\u001b[39m.\u001b[39;49mcat(k)\n\u001b[1;32m 138\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mmissing_exceptions:\n\u001b[1;32m 139\u001b[0m \u001b[39mif\u001b[39;00m default \u001b[39mis\u001b[39;00m \u001b[39mnot\u001b[39;00m \u001b[39mNone\u001b[39;00m:\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/fsspec/asyn.py:86\u001b[0m, in \u001b[0;36msync_wrapper..wrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 83\u001b[0m \u001b[39m@functools\u001b[39m\u001b[39m.\u001b[39mwraps(func)\n\u001b[1;32m 84\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mwrapper\u001b[39m(\u001b[39m*\u001b[39margs, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs):\n\u001b[1;32m 85\u001b[0m \u001b[39mself\u001b[39m \u001b[39m=\u001b[39m obj \u001b[39mor\u001b[39;00m args[\u001b[39m0\u001b[39m]\n\u001b[0;32m---> 86\u001b[0m \u001b[39mreturn\u001b[39;00m sync(\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mloop, func, \u001b[39m*\u001b[39;49margs, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/fsspec/asyn.py:66\u001b[0m, in \u001b[0;36msync\u001b[0;34m(loop, func, timeout, *args, **kwargs)\u001b[0m\n\u001b[1;32m 64\u001b[0m \u001b[39mraise\u001b[39;00m FSTimeoutError \u001b[39mfrom\u001b[39;00m \u001b[39mreturn_result\u001b[39;00m\n\u001b[1;32m 65\u001b[0m \u001b[39melif\u001b[39;00m \u001b[39misinstance\u001b[39m(return_result, \u001b[39mBaseException\u001b[39;00m):\n\u001b[0;32m---> 66\u001b[0m \u001b[39mraise\u001b[39;00m return_result\n\u001b[1;32m 67\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m 68\u001b[0m \u001b[39mreturn\u001b[39;00m return_result\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/fsspec/asyn.py:26\u001b[0m, in \u001b[0;36m_runner\u001b[0;34m(event, coro, result, timeout)\u001b[0m\n\u001b[1;32m 24\u001b[0m coro \u001b[39m=\u001b[39m asyncio\u001b[39m.\u001b[39mwait_for(coro, timeout\u001b[39m=\u001b[39mtimeout)\n\u001b[1;32m 25\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m---> 26\u001b[0m result[\u001b[39m0\u001b[39m] \u001b[39m=\u001b[39m \u001b[39mawait\u001b[39;00m coro\n\u001b[1;32m 27\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mException\u001b[39;00m \u001b[39mas\u001b[39;00m ex:\n\u001b[1;32m 28\u001b[0m result[\u001b[39m0\u001b[39m] \u001b[39m=\u001b[39m ex\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/fsspec/asyn.py:398\u001b[0m, in \u001b[0;36mAsyncFileSystem._cat\u001b[0;34m(self, path, recursive, on_error, batch_size, **kwargs)\u001b[0m\n\u001b[1;32m 396\u001b[0m ex \u001b[39m=\u001b[39m \u001b[39mnext\u001b[39m(\u001b[39mfilter\u001b[39m(is_exception, out), \u001b[39mFalse\u001b[39;00m)\n\u001b[1;32m 397\u001b[0m \u001b[39mif\u001b[39;00m ex:\n\u001b[0;32m--> 398\u001b[0m \u001b[39mraise\u001b[39;00m ex\n\u001b[1;32m 399\u001b[0m \u001b[39mif\u001b[39;00m (\n\u001b[1;32m 400\u001b[0m \u001b[39mlen\u001b[39m(paths) \u001b[39m>\u001b[39m \u001b[39m1\u001b[39m\n\u001b[1;32m 401\u001b[0m \u001b[39mor\u001b[39;00m \u001b[39misinstance\u001b[39m(path, \u001b[39mlist\u001b[39m)\n\u001b[1;32m 402\u001b[0m \u001b[39mor\u001b[39;00m paths[\u001b[39m0\u001b[39m] \u001b[39m!=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_strip_protocol(path)\n\u001b[1;32m 403\u001b[0m ):\n\u001b[1;32m 404\u001b[0m \u001b[39mreturn\u001b[39;00m {\n\u001b[1;32m 405\u001b[0m k: v\n\u001b[1;32m 406\u001b[0m \u001b[39mfor\u001b[39;00m k, v \u001b[39min\u001b[39;00m \u001b[39mzip\u001b[39m(paths, out)\n\u001b[1;32m 407\u001b[0m \u001b[39mif\u001b[39;00m on_error \u001b[39m!=\u001b[39m \u001b[39m\"\u001b[39m\u001b[39momit\u001b[39m\u001b[39m\"\u001b[39m \u001b[39mor\u001b[39;00m \u001b[39mnot\u001b[39;00m is_exception(v)\n\u001b[1;32m 408\u001b[0m }\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/asyncio/tasks.py:442\u001b[0m, in \u001b[0;36mwait_for\u001b[0;34m(fut, timeout, loop)\u001b[0m\n\u001b[1;32m 437\u001b[0m warnings\u001b[39m.\u001b[39mwarn(\u001b[39m\"\u001b[39m\u001b[39mThe loop argument is deprecated since Python 3.8, \u001b[39m\u001b[39m\"\u001b[39m\n\u001b[1;32m 438\u001b[0m \u001b[39m\"\u001b[39m\u001b[39mand scheduled for removal in Python 3.10.\u001b[39m\u001b[39m\"\u001b[39m,\n\u001b[1;32m 439\u001b[0m \u001b[39mDeprecationWarning\u001b[39;00m, stacklevel\u001b[39m=\u001b[39m\u001b[39m2\u001b[39m)\n\u001b[1;32m 441\u001b[0m \u001b[39mif\u001b[39;00m timeout \u001b[39mis\u001b[39;00m \u001b[39mNone\u001b[39;00m:\n\u001b[0;32m--> 442\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mawait\u001b[39;00m fut\n\u001b[1;32m 444\u001b[0m \u001b[39mif\u001b[39;00m timeout \u001b[39m<\u001b[39m\u001b[39m=\u001b[39m \u001b[39m0\u001b[39m:\n\u001b[1;32m 445\u001b[0m fut \u001b[39m=\u001b[39m ensure_future(fut, loop\u001b[39m=\u001b[39mloop)\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/gcsfs/core.py:781\u001b[0m, in \u001b[0;36mGCSFileSystem._cat_file\u001b[0;34m(self, path, start, end, **kwargs)\u001b[0m\n\u001b[1;32m 779\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m 780\u001b[0m head \u001b[39m=\u001b[39m {}\n\u001b[0;32m--> 781\u001b[0m headers, out \u001b[39m=\u001b[39m \u001b[39mawait\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_call(\u001b[39m\"\u001b[39m\u001b[39mGET\u001b[39m\u001b[39m\"\u001b[39m, u2, headers\u001b[39m=\u001b[39mhead)\n\u001b[1;32m 782\u001b[0m \u001b[39mreturn\u001b[39;00m out\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/gcsfs/core.py:392\u001b[0m, in \u001b[0;36mGCSFileSystem._call\u001b[0;34m(self, method, path, json_out, info_out, *args, **kwargs)\u001b[0m\n\u001b[1;32m 387\u001b[0m \u001b[39masync\u001b[39;00m \u001b[39mdef\u001b[39;00m \u001b[39m_call\u001b[39m(\n\u001b[1;32m 388\u001b[0m \u001b[39mself\u001b[39m, method, path, \u001b[39m*\u001b[39margs, json_out\u001b[39m=\u001b[39m\u001b[39mFalse\u001b[39;00m, info_out\u001b[39m=\u001b[39m\u001b[39mFalse\u001b[39;00m, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs\n\u001b[1;32m 389\u001b[0m ):\n\u001b[1;32m 390\u001b[0m logger\u001b[39m.\u001b[39mdebug(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m{\u001b[39;00mmethod\u001b[39m.\u001b[39mupper()\u001b[39m}\u001b[39;00m\u001b[39m: \u001b[39m\u001b[39m{\u001b[39;00mpath\u001b[39m}\u001b[39;00m\u001b[39m, \u001b[39m\u001b[39m{\u001b[39;00margs\u001b[39m}\u001b[39;00m\u001b[39m, \u001b[39m\u001b[39m{\u001b[39;00mkwargs\u001b[39m.\u001b[39mget(\u001b[39m'\u001b[39m\u001b[39mheaders\u001b[39m\u001b[39m'\u001b[39m)\u001b[39m}\u001b[39;00m\u001b[39m\"\u001b[39m)\n\u001b[0;32m--> 392\u001b[0m status, headers, info, contents \u001b[39m=\u001b[39m \u001b[39mawait\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_request(\n\u001b[1;32m 393\u001b[0m method, path, \u001b[39m*\u001b[39margs, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs\n\u001b[1;32m 394\u001b[0m )\n\u001b[1;32m 395\u001b[0m \u001b[39mif\u001b[39;00m json_out:\n\u001b[1;32m 396\u001b[0m \u001b[39mreturn\u001b[39;00m json\u001b[39m.\u001b[39mloads(contents)\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/decorator.py:221\u001b[0m, in \u001b[0;36mdecorate..fun\u001b[0;34m(*args, **kw)\u001b[0m\n\u001b[1;32m 219\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m kwsyntax:\n\u001b[1;32m 220\u001b[0m args, kw \u001b[39m=\u001b[39m fix(args, kw, sig)\n\u001b[0;32m--> 221\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mawait\u001b[39;00m caller(func, \u001b[39m*\u001b[39m(extras \u001b[39m+\u001b[39m args), \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkw)\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/gcsfs/retry.py:115\u001b[0m, in \u001b[0;36mretry_request\u001b[0;34m(func, retries, *args, **kwargs)\u001b[0m\n\u001b[1;32m 113\u001b[0m \u001b[39mif\u001b[39;00m retry \u001b[39m>\u001b[39m \u001b[39m0\u001b[39m:\n\u001b[1;32m 114\u001b[0m \u001b[39mawait\u001b[39;00m asyncio\u001b[39m.\u001b[39msleep(\u001b[39mmin\u001b[39m(random\u001b[39m.\u001b[39mrandom() \u001b[39m+\u001b[39m \u001b[39m2\u001b[39m \u001b[39m*\u001b[39m\u001b[39m*\u001b[39m (retry \u001b[39m-\u001b[39m \u001b[39m1\u001b[39m), \u001b[39m32\u001b[39m))\n\u001b[0;32m--> 115\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mawait\u001b[39;00m func(\u001b[39m*\u001b[39margs, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs)\n\u001b[1;32m 116\u001b[0m \u001b[39mexcept\u001b[39;00m (\n\u001b[1;32m 117\u001b[0m HttpError,\n\u001b[1;32m 118\u001b[0m requests\u001b[39m.\u001b[39mexceptions\u001b[39m.\u001b[39mRequestException,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 121\u001b[0m aiohttp\u001b[39m.\u001b[39mclient_exceptions\u001b[39m.\u001b[39mClientError,\n\u001b[1;32m 122\u001b[0m ) \u001b[39mas\u001b[39;00m e:\n\u001b[1;32m 123\u001b[0m \u001b[39mif\u001b[39;00m (\n\u001b[1;32m 124\u001b[0m \u001b[39misinstance\u001b[39m(e, HttpError)\n\u001b[1;32m 125\u001b[0m \u001b[39mand\u001b[39;00m e\u001b[39m.\u001b[39mcode \u001b[39m==\u001b[39m \u001b[39m400\u001b[39m\n\u001b[1;32m 126\u001b[0m \u001b[39mand\u001b[39;00m \u001b[39m\"\u001b[39m\u001b[39mrequester pays\u001b[39m\u001b[39m\"\u001b[39m \u001b[39min\u001b[39;00m e\u001b[39m.\u001b[39mmessage\n\u001b[1;32m 127\u001b[0m ):\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/gcsfs/core.py:369\u001b[0m, in \u001b[0;36mGCSFileSystem._request\u001b[0;34m(self, method, path, headers, json, data, *args, **kwargs)\u001b[0m\n\u001b[1;32m 364\u001b[0m \u001b[39m@retry_request\u001b[39m(retries\u001b[39m=\u001b[39mretries)\n\u001b[1;32m 365\u001b[0m \u001b[39masync\u001b[39;00m \u001b[39mdef\u001b[39;00m \u001b[39m_request\u001b[39m(\n\u001b[1;32m 366\u001b[0m \u001b[39mself\u001b[39m, method, path, \u001b[39m*\u001b[39margs, headers\u001b[39m=\u001b[39m\u001b[39mNone\u001b[39;00m, json\u001b[39m=\u001b[39m\u001b[39mNone\u001b[39;00m, data\u001b[39m=\u001b[39m\u001b[39mNone\u001b[39;00m, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs\n\u001b[1;32m 367\u001b[0m ):\n\u001b[1;32m 368\u001b[0m \u001b[39mawait\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_set_session()\n\u001b[0;32m--> 369\u001b[0m \u001b[39masync\u001b[39;00m \u001b[39mwith\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39msession\u001b[39m.\u001b[39mrequest(\n\u001b[1;32m 370\u001b[0m method\u001b[39m=\u001b[39mmethod,\n\u001b[1;32m 371\u001b[0m url\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_format_path(path, args),\n\u001b[1;32m 372\u001b[0m params\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_get_params(kwargs),\n\u001b[1;32m 373\u001b[0m json\u001b[39m=\u001b[39mjson,\n\u001b[1;32m 374\u001b[0m headers\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_get_headers(headers),\n\u001b[1;32m 375\u001b[0m data\u001b[39m=\u001b[39mdata,\n\u001b[1;32m 376\u001b[0m timeout\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39mrequests_timeout,\n\u001b[1;32m 377\u001b[0m ) \u001b[39mas\u001b[39;00m r:\n\u001b[1;32m 379\u001b[0m status \u001b[39m=\u001b[39m r\u001b[39m.\u001b[39mstatus\n\u001b[1;32m 380\u001b[0m headers \u001b[39m=\u001b[39m r\u001b[39m.\u001b[39mheaders\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/aiohttp/client.py:1138\u001b[0m, in \u001b[0;36m_BaseRequestContextManager.__aenter__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 1137\u001b[0m \u001b[39masync\u001b[39;00m \u001b[39mdef\u001b[39;00m \u001b[39m__aenter__\u001b[39m(\u001b[39mself\u001b[39m) \u001b[39m-\u001b[39m\u001b[39m>\u001b[39m _RetType:\n\u001b[0;32m-> 1138\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_resp \u001b[39m=\u001b[39m \u001b[39mawait\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_coro\n\u001b[1;32m 1139\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_resp\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/aiohttp/client.py:507\u001b[0m, in \u001b[0;36mClientSession._request\u001b[0;34m(self, method, str_or_url, params, data, json, cookies, headers, skip_auto_headers, auth, allow_redirects, max_redirects, compress, chunked, expect100, raise_for_status, read_until_eof, proxy, proxy_auth, timeout, verify_ssl, fingerprint, ssl_context, ssl, proxy_headers, trace_request_ctx, read_bufsize)\u001b[0m\n\u001b[1;32m 504\u001b[0m \u001b[39mwith\u001b[39;00m suppress(\u001b[39mLookupError\u001b[39;00m):\n\u001b[1;32m 505\u001b[0m proxy, proxy_auth \u001b[39m=\u001b[39m get_env_proxy_for_url(url)\n\u001b[0;32m--> 507\u001b[0m req \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_request_class(\n\u001b[1;32m 508\u001b[0m method,\n\u001b[1;32m 509\u001b[0m url,\n\u001b[1;32m 510\u001b[0m params\u001b[39m=\u001b[39;49mparams,\n\u001b[1;32m 511\u001b[0m headers\u001b[39m=\u001b[39;49mheaders,\n\u001b[1;32m 512\u001b[0m skip_auto_headers\u001b[39m=\u001b[39;49mskip_headers,\n\u001b[1;32m 513\u001b[0m data\u001b[39m=\u001b[39;49mdata,\n\u001b[1;32m 514\u001b[0m cookies\u001b[39m=\u001b[39;49mall_cookies,\n\u001b[1;32m 515\u001b[0m auth\u001b[39m=\u001b[39;49mauth,\n\u001b[1;32m 516\u001b[0m version\u001b[39m=\u001b[39;49mversion,\n\u001b[1;32m 517\u001b[0m compress\u001b[39m=\u001b[39;49mcompress,\n\u001b[1;32m 518\u001b[0m chunked\u001b[39m=\u001b[39;49mchunked,\n\u001b[1;32m 519\u001b[0m expect100\u001b[39m=\u001b[39;49mexpect100,\n\u001b[1;32m 520\u001b[0m loop\u001b[39m=\u001b[39;49m\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_loop,\n\u001b[1;32m 521\u001b[0m response_class\u001b[39m=\u001b[39;49m\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_response_class,\n\u001b[1;32m 522\u001b[0m proxy\u001b[39m=\u001b[39;49mproxy,\n\u001b[1;32m 523\u001b[0m proxy_auth\u001b[39m=\u001b[39;49mproxy_auth,\n\u001b[1;32m 524\u001b[0m timer\u001b[39m=\u001b[39;49mtimer,\n\u001b[1;32m 525\u001b[0m session\u001b[39m=\u001b[39;49m\u001b[39mself\u001b[39;49m,\n\u001b[1;32m 526\u001b[0m ssl\u001b[39m=\u001b[39;49mssl,\n\u001b[1;32m 527\u001b[0m proxy_headers\u001b[39m=\u001b[39;49mproxy_headers,\n\u001b[1;32m 528\u001b[0m traces\u001b[39m=\u001b[39;49mtraces,\n\u001b[1;32m 529\u001b[0m )\n\u001b[1;32m 531\u001b[0m \u001b[39m# connection timeout\u001b[39;00m\n\u001b[1;32m 532\u001b[0m \u001b[39mtry\u001b[39;00m:\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/aiohttp/client_reqrep.py:283\u001b[0m, in \u001b[0;36mClientRequest.__init__\u001b[0;34m(self, method, url, params, headers, skip_auto_headers, data, cookies, auth, version, compress, chunked, expect100, loop, response_class, proxy, proxy_auth, timer, session, ssl, proxy_headers, traces)\u001b[0m\n\u001b[1;32m 281\u001b[0m \u001b[39mif\u001b[39;00m params:\n\u001b[1;32m 282\u001b[0m q \u001b[39m=\u001b[39m MultiDict(url\u001b[39m.\u001b[39mquery)\n\u001b[0;32m--> 283\u001b[0m url2 \u001b[39m=\u001b[39m url\u001b[39m.\u001b[39;49mwith_query(params)\n\u001b[1;32m 284\u001b[0m q\u001b[39m.\u001b[39mextend(url2\u001b[39m.\u001b[39mquery)\n\u001b[1;32m 285\u001b[0m url \u001b[39m=\u001b[39m url\u001b[39m.\u001b[39mwith_query(q)\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/yarl/_url.py:984\u001b[0m, in \u001b[0;36mURL.with_query\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 970\u001b[0m \u001b[39m\"\"\"Return a new URL with query part replaced.\u001b[39;00m\n\u001b[1;32m 971\u001b[0m \n\u001b[1;32m 972\u001b[0m \u001b[39mAccepts any Mapping (e.g. dict, multidict.MultiDict instances)\u001b[39;00m\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 980\u001b[0m \n\u001b[1;32m 981\u001b[0m \u001b[39m\"\"\"\u001b[39;00m\n\u001b[1;32m 982\u001b[0m \u001b[39m# N.B. doesn't cleanup query/fragment\u001b[39;00m\n\u001b[0;32m--> 984\u001b[0m new_query \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_get_str_query(\u001b[39m*\u001b[39;49margs, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\u001b[1;32m 985\u001b[0m \u001b[39mreturn\u001b[39;00m URL(\n\u001b[1;32m 986\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_val\u001b[39m.\u001b[39m_replace(path\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_val\u001b[39m.\u001b[39mpath, query\u001b[39m=\u001b[39mnew_query), encoded\u001b[39m=\u001b[39m\u001b[39mTrue\u001b[39;00m\n\u001b[1;32m 987\u001b[0m )\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/yarl/_url.py:945\u001b[0m, in \u001b[0;36mURL._get_str_query\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 943\u001b[0m \u001b[39melif\u001b[39;00m \u001b[39misinstance\u001b[39m(query, Mapping):\n\u001b[1;32m 944\u001b[0m quoter \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_QUERY_PART_QUOTER\n\u001b[0;32m--> 945\u001b[0m query \u001b[39m=\u001b[39m \u001b[39m\"\u001b[39;49m\u001b[39m&\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39m.\u001b[39;49mjoin(\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_query_seq_pairs(quoter, query\u001b[39m.\u001b[39;49mitems()))\n\u001b[1;32m 946\u001b[0m \u001b[39melif\u001b[39;00m \u001b[39misinstance\u001b[39m(query, \u001b[39mstr\u001b[39m):\n\u001b[1;32m 947\u001b[0m query \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_QUERY_QUOTER(query)\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/yarl/_url.py:908\u001b[0m, in \u001b[0;36mURL._query_seq_pairs\u001b[0;34m(cls, quoter, pairs)\u001b[0m\n\u001b[1;32m 906\u001b[0m \u001b[39myield\u001b[39;00m quoter(key) \u001b[39m+\u001b[39m \u001b[39m\"\u001b[39m\u001b[39m=\u001b[39m\u001b[39m\"\u001b[39m \u001b[39m+\u001b[39m quoter(\u001b[39mcls\u001b[39m\u001b[39m.\u001b[39m_query_var(v))\n\u001b[1;32m 907\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[0;32m--> 908\u001b[0m \u001b[39myield\u001b[39;00m quoter(key) \u001b[39m+\u001b[39m \u001b[39m\"\u001b[39m\u001b[39m=\u001b[39m\u001b[39m\"\u001b[39m \u001b[39m+\u001b[39m quoter(\u001b[39mcls\u001b[39;49m\u001b[39m.\u001b[39;49m_query_var(val))\n",
"File \u001b[0;32m~/opt/miniconda3/envs/po-cookbook-dev/lib/python3.9/site-packages/yarl/_url.py:923\u001b[0m, in \u001b[0;36mURL._query_var\u001b[0;34m(v)\u001b[0m\n\u001b[1;32m 921\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39missubclass\u001b[39m(\u001b[39mcls\u001b[39m, \u001b[39mint\u001b[39m) \u001b[39mand\u001b[39;00m \u001b[39mcls\u001b[39m \u001b[39mis\u001b[39;00m \u001b[39mnot\u001b[39;00m \u001b[39mbool\u001b[39m:\n\u001b[1;32m 922\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mstr\u001b[39m(\u001b[39mint\u001b[39m(v))\n\u001b[0;32m--> 923\u001b[0m \u001b[39mraise\u001b[39;00m \u001b[39mTypeError\u001b[39;00m(\n\u001b[1;32m 924\u001b[0m \u001b[39m\"\u001b[39m\u001b[39mInvalid variable type: value \u001b[39m\u001b[39m\"\u001b[39m\n\u001b[1;32m 925\u001b[0m \u001b[39m\"\u001b[39m\u001b[39mshould be str, int or float, got \u001b[39m\u001b[39m{!r}\u001b[39;00m\u001b[39m \u001b[39m\u001b[39m\"\u001b[39m\n\u001b[1;32m 926\u001b[0m \u001b[39m\"\u001b[39m\u001b[39mof type \u001b[39m\u001b[39m{}\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39mformat(v, \u001b[39mcls\u001b[39m)\n\u001b[1;32m 927\u001b[0m )\n",
"\u001b[0;31mTypeError\u001b[0m: Invalid variable type: value should be str, int or float, got None of type "
]
}
],
"source": [
"cat = open_catalog(\"https://raw.githubusercontent.com/pangeo-data/pangeo-datastore/master/intake-catalogs/ocean/altimetry.yaml\")\n",
"print(list(cat))\n",
"ds = cat['j3'].to_dask()\n",
"ds"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Load some data into memory:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Select latitude, longitude, and sea level anomaly\n",
"ds_ll = ds[['latitude', 'longitude', 'sla_filtered']].reset_coords().astype('f4').load()\n",
"ds_ll"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Convert to pandas dataframe:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = ds_ll.to_dataframe()\n",
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Visualize with hvplot"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.hvplot.scatter(x='longitude', y='latitude', datashade=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Bin using xhistogram"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lon_bins = np.arange(0, 361, 2)\n",
"lat_bins = np.arange(-70, 71, 2)\n",
"\n",
"# helps with memory management\n",
"ds_ll_chunked = ds_ll.chunk({'time': '5MB'})\n",
"\n",
"sla_variance = histogram(ds_ll_chunked.longitude, ds_ll_chunked.latitude,\n",
" bins=[lon_bins, lat_bins],\n",
" weights=ds_ll_chunked.sla_filtered.fillna(0.)**2)\n",
"\n",
"norm = histogram(ds_ll_chunked.longitude, ds_ll_chunked.latitude,\n",
" bins=[lon_bins, lat_bins])\n",
"\n",
"\n",
"# let's get at least 200 points in a box for it to be unmasked\n",
"thresh = 200\n",
"sla_variance = sla_variance / norm.where(norm > thresh)\n",
"sla_variance"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sla_variance.load()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# plot the sea level anomaly variance\n",
"sla_variance.plot(x='longitude_bin', figsize=(12, 6), vmax=0.2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this example we visualized sea level anomalies using along-track altimetry data using hvplot. Then, we used xhistogram to calculate and plot the variance of the data.\n",
"\n",
"### What's next?\n",
"\n",
"Other examples will look at other datasets to visualize sea surface temeratures, ocean depth, and currents."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Resources and references\n",
" - This notebook is based on the Pangeo physical oceanography gallery example: https://gallery.pangeo.io/repos/pangeo-gallery/physical-oceanography/02_along_track.html"
]
}
],
"metadata": {
"interpreter": {
"hash": "b69aafa6ae0f1ee3b353273515d208690be9963f9396d970b6e02ccbd7d3e2ec"
},
"kernelspec": {
"display_name": "Python 3.8.13 ('geocat-examples')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
|