Overview¶
In this tutorial we are going to use a large collection of pre-generated Kerchunk reference files and open them with Xarray’s new DataTree functionality. This chapter is heavily inspired by this blog post.
About the Dataset¶
This collection of reference files were generated from the NASA NEX-GDDP-CMIP6 (Global Daily Downscaled Projections) dataset. A version of this dataset is hosted on s3 as a collection of NetCDF files.
Prerequisites¶
| Concepts | Importance | Notes |
|---|---|---|
| Kerchunk Basics | Required | Core |
| Multiple Files and Kerchunk | Required | Core |
| Kerchunk and Dask | Required | Core |
| Multi-File Datasets with Kerchunk | Required | IO/Visualization |
| Xarray-Datatree Overview | Required | IO |
Time to learn: 30 minutes
Motivation¶
In total the dataset is roughly 12TB in compressed blob storage, with a single NetCDF file per yearly timestep, per variable. Downloading this entire dataset for analysis on a local machine would difficult to say the least. The collection of Kerchunk reference files for this entire dataset is only 272 Mb, which is about 42,000 times smaller!
Imports¶
import dask
import hvplot.xarray # noqa
import pandas as pd
import xarray as xr
from xarray import DataTree
from distributed import Client
from fsspec.implementations.reference import ReferenceFileSystemLoading...
Loading...
Loading...
Loading...
Read the reference catalog¶
The NASA NEX-GDDP-CMIP6 dataset is organized by GCM, Scenario and Ensemble Member. Each of these Scenario/GCM combinations is represented as a combined reference file, which was created by merging across variables and concatenating along time-steps. All of these references are organized into a simple .csv catalog in the schema:
| GCM/Scenario | url |
|---|
Organzing with Xarray-Datatree¶
Not all of the GCM/Scenario reference datasets have shared spatial coordinates and many of the have slight differences in their calendar and thus time dimension.
Because of this, these cannot be combined into a single Xarray-Dataset. Fortunately Xarray-Datatree provides a higher level abstraction where related Xarray-Datasets are organized into a tree structure where each dataset corresponds to a leaf.
# Read the reference catalog into a Pandas DataFrame
cat_df = pd.read_csv(
"s3://carbonplan-share/nasa-nex-reference/reference_catalog_nested.csv"
)
# Convert the DataFrame into a dictionary
catalog = cat_df.set_index("ID").T.to_dict("records")[0]Load Reference Datasets into Xarray-DataTree¶
In the following cell we create a function load_ref_ds, which can be parallelized via Dask to load Kerchunk references into a dictionary of Xarray-Datasets.
def load_ref_ds(url: str):
fs = ReferenceFileSystem(
url,
remote_protocol="s3",
target_protocol="s3",
remote_options={"anon": True},
target_options={"anon": True},
lazy=True,
)
return xr.open_dataset(
fs.get_mapper(),
engine="zarr",
backend_kwargs={"consolidated": False},
chunks={"time": 300},
)
tasks = {id: dask.delayed(load_ref_ds)(url) for id, url in catalog.items()}Use Dask Distributed to load the Xarray-Datasets from Kerchunk reference files¶
Using Dask, we are loading 164 reference datasets into memory. Since they are are Xarray datasets the coordinates are loaded eagerly, but the underlying data is still lazy.
client = Client(n_workers=8)
clientLoading...
catalog_computed = dask.compute(tasks)2026-08-26 00:57:24,164 - distributed.worker - ERROR - Compute Failed
Key: load_ref_ds-f0955b8a-64a5-4fd4-8dec-333d3d002029
State: executing
Task: <Task 'load_ref_ds-f0955b8a-64a5-4fd4-8dec-333d3d002029' load_ref_ds(...)>
Exception: 'ValueError("Reference-FS\'s target filesystem must have same value of asynchronous")'
Traceback: ' File "/tmp/ipykernel_5580/3963997076.py", line 10, in load_ref_ds\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/api.py", line 613, in open_dataset\n backend_ds = backend.open_dataset(\n filename_or_obj,\n ...<2 lines>...\n **kwargs,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1741, in open_dataset\n store = ZarrStore.open_group(\n filename_or_obj,\n ...<10 lines>...\n cache_members=cache_members,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 771, in open_group\n ) = _get_open_params(\n ~~~~~~~~~~~~~~~~^\n store=store,\n ^^^^^^^^^^^^\n ...<9 lines>...\n zarr_format=zarr_format,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1983, in _get_open_params\n zarr_group = zarr.open_group(store, **open_kwargs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/synchronous.py", line 552, in open_group\n sync(\n ~~~~^\n async_api.open_group(\n ^^^^^^^^^^^^^^^^^^^^^\n ...<11 lines>...\n )\n ^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 154, in sync\n raise return_result\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 114, in _runner\n return await coro\n ^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/asynchronous.py", line 859, in open_group\n store_path = await make_store_path(store, mode=mode, storage_options=storage_options, path=path)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 471, in make_store_path\n store = await make_store(store_like, mode=mode, storage_options=storage_options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 403, in make_store\n return FsspecStore.from_mapper(store_like, read_only=_read_only)\n ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 206, in from_mapper\n fs = _make_async(fs_map.fs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 57, in _make_async\n return fsspec.AbstractFileSystem.from_json(json.dumps(fs_dict))\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 1500, in from_json\n return json.loads(blob, cls=FilesystemJSONDecoder)\n ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/json/__init__.py", line 365, in loads\n return cls(**kw).decode(s)\n ~~~~~~~~~~~~~~~~^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/json/decoder.py", line 345, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/json/decoder.py", line 361, in raw_decode\n obj, end = self.scan_once(s, idx)\n ~~~~~~~~~~~~~~^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/json.py", line 92, in custom_object_hook\n return AbstractFileSystem.from_dict(dct)\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 1576, in from_dict\n return cls(\n *json_decoder.unmake_serializable(dct.pop("args", ())),\n **json_decoder.unmake_serializable(dct),\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 87, in __call__\n obj = super().__call__(*args, **kwargs, **strip_tokenize_options)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/implementations/reference.py", line 784, in __init__\n raise ValueError(\n ...<2 lines>...\n )\n'
2026-08-26 00:57:24,178 - distributed.worker - ERROR - Compute Failed
Key: load_ref_ds-db406361-7fd9-42b2-9ff3-0e4115765a49
State: executing
Task: <Task 'load_ref_ds-db406361-7fd9-42b2-9ff3-0e4115765a49' load_ref_ds(...)>
Exception: 'ValueError("Reference-FS\'s target filesystem must have same value of asynchronous")'
Traceback: ' File "/tmp/ipykernel_5580/3963997076.py", line 10, in load_ref_ds\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/api.py", line 613, in open_dataset\n backend_ds = backend.open_dataset(\n filename_or_obj,\n ...<2 lines>...\n **kwargs,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1741, in open_dataset\n store = ZarrStore.open_group(\n filename_or_obj,\n ...<10 lines>...\n cache_members=cache_members,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 771, in open_group\n ) = _get_open_params(\n ~~~~~~~~~~~~~~~~^\n store=store,\n ^^^^^^^^^^^^\n ...<9 lines>...\n zarr_format=zarr_format,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1983, in _get_open_params\n zarr_group = zarr.open_group(store, **open_kwargs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/synchronous.py", line 552, in open_group\n sync(\n ~~~~^\n async_api.open_group(\n ^^^^^^^^^^^^^^^^^^^^^\n ...<11 lines>...\n )\n ^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 154, in sync\n raise return_result\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 114, in _runner\n return await coro\n ^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/asynchronous.py", line 859, in open_group\n store_path = await make_store_path(store, mode=mode, storage_options=storage_options, path=path)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 471, in make_store_path\n store = await make_store(store_like, mode=mode, storage_options=storage_options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 403, in make_store\n return FsspecStore.from_mapper(store_like, read_only=_read_only)\n ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 206, in from_mapper\n fs = _make_async(fs_map.fs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 57, in _make_async\n return fsspec.AbstractFileSystem.from_json(json.dumps(fs_dict))\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 1500, in from_json\n return json.loads(blob, cls=FilesystemJSONDecoder)\n ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/json/__init__.py", line 365, in loads\n return cls(**kw).decode(s)\n ~~~~~~~~~~~~~~~~^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/json/decoder.py", line 345, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/json/decoder.py", line 361, in raw_decode\n obj, end = self.scan_once(s, idx)\n ~~~~~~~~~~~~~~^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/json.py", line 92, in custom_object_hook\n return AbstractFileSystem.from_dict(dct)\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 1576, in from_dict\n return cls(\n *json_decoder.unmake_serializable(dct.pop("args", ())),\n **json_decoder.unmake_serializable(dct),\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 87, in __call__\n obj = super().__call__(*args, **kwargs, **strip_tokenize_options)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/implementations/reference.py", line 784, in __init__\n raise ValueError(\n ...<2 lines>...\n )\n'
2026-08-26 00:57:24,191 - distributed.worker - ERROR - Compute Failed
Key: load_ref_ds-d519eff8-7cfa-4be0-a80d-7212d2705fe8
State: executing
Task: <Task 'load_ref_ds-d519eff8-7cfa-4be0-a80d-7212d2705fe8' load_ref_ds(...)>
Exception: 'ValueError("Reference-FS\'s target filesystem must have same value of asynchronous")'
Traceback: ' File "/tmp/ipykernel_5580/3963997076.py", line 10, in load_ref_ds\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/api.py", line 613, in open_dataset\n backend_ds = backend.open_dataset(\n filename_or_obj,\n ...<2 lines>...\n **kwargs,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1741, in open_dataset\n store = ZarrStore.open_group(\n filename_or_obj,\n ...<10 lines>...\n cache_members=cache_members,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 771, in open_group\n ) = _get_open_params(\n ~~~~~~~~~~~~~~~~^\n store=store,\n ^^^^^^^^^^^^\n ...<9 lines>...\n zarr_format=zarr_format,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1983, in _get_open_params\n zarr_group = zarr.open_group(store, **open_kwargs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/synchronous.py", line 552, in open_group\n sync(\n ~~~~^\n async_api.open_group(\n ^^^^^^^^^^^^^^^^^^^^^\n ...<11 lines>...\n )\n ^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 154, in sync\n raise return_result\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 114, in _runner\n return await coro\n ^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/asynchronous.py", line 859, in open_group\n store_path = await make_store_path(store, mode=mode, storage_options=storage_options, path=path)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 471, in make_store_path\n store = await make_store(store_like, mode=mode, storage_options=storage_options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 403, in make_store\n return FsspecStore.from_mapper(store_like, read_only=_read_only)\n ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 206, in from_mapper\n fs = _make_async(fs_map.fs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 57, in _make_async\n return fsspec.AbstractFileSystem.from_json(json.dumps(fs_dict))\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 1500, in from_json\n return json.loads(blob, cls=FilesystemJSONDecoder)\n ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/json/__init__.py", line 365, in loads\n return cls(**kw).decode(s)\n ~~~~~~~~~~~~~~~~^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/json/decoder.py", line 345, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/json/decoder.py", line 361, in raw_decode\n obj, end = self.scan_once(s, idx)\n ~~~~~~~~~~~~~~^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/json.py", line 92, in custom_object_hook\n return AbstractFileSystem.from_dict(dct)\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 1576, in from_dict\n return cls(\n *json_decoder.unmake_serializable(dct.pop("args", ())),\n **json_decoder.unmake_serializable(dct),\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 87, in __call__\n obj = super().__call__(*args, **kwargs, **strip_tokenize_options)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/implementations/reference.py", line 784, in __init__\n raise ValueError(\n ...<2 lines>...\n )\n'
2026-08-26 00:57:24,216 - distributed.worker - ERROR - Compute Failed
Key: load_ref_ds-d5156c1f-c5cc-4ea8-9482-bcd5ca6e2497
State: executing
Task: <Task 'load_ref_ds-d5156c1f-c5cc-4ea8-9482-bcd5ca6e2497' load_ref_ds(...)>
Exception: 'ValueError("Reference-FS\'s target filesystem must have same value of asynchronous")'
Traceback: ' File "/tmp/ipykernel_5580/3963997076.py", line 10, in load_ref_ds\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/api.py", line 613, in open_dataset\n backend_ds = backend.open_dataset(\n filename_or_obj,\n ...<2 lines>...\n **kwargs,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1741, in open_dataset\n store = ZarrStore.open_group(\n filename_or_obj,\n ...<10 lines>...\n cache_members=cache_members,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 771, in open_group\n ) = _get_open_params(\n ~~~~~~~~~~~~~~~~^\n store=store,\n ^^^^^^^^^^^^\n ...<9 lines>...\n zarr_format=zarr_format,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1983, in _get_open_params\n zarr_group = zarr.open_group(store, **open_kwargs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/synchronous.py", line 552, in open_group\n sync(\n ~~~~^\n async_api.open_group(\n ^^^^^^^^^^^^^^^^^^^^^\n ...<11 lines>...\n )\n ^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 154, in sync\n raise return_result\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 114, in _runner\n return await coro\n ^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/asynchronous.py", line 859, in open_group\n store_path = await make_store_path(store, mode=mode, storage_options=storage_options, path=path)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 471, in make_store_path\n store = await make_store(store_like, mode=mode, storage_options=storage_options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 403, in make_store\n return FsspecStore.from_mapper(store_like, read_only=_read_only)\n ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 206, in from_mapper\n fs = _make_async(fs_map.fs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 57, in _make_async\n return fsspec.AbstractFileSystem.from_json(json.dumps(fs_dict))\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 1500, in from_json\n return json.loads(blob, cls=FilesystemJSONDecoder)\n ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/json/__init__.py", line 365, in loads\n return cls(**kw).decode(s)\n ~~~~~~~~~~~~~~~~^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/json/decoder.py", line 345, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/json/decoder.py", line 361, in raw_decode\n obj, end = self.scan_once(s, idx)\n ~~~~~~~~~~~~~~^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/json.py", line 92, in custom_object_hook\n return AbstractFileSystem.from_dict(dct)\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 1576, in from_dict\n return cls(\n *json_decoder.unmake_serializable(dct.pop("args", ())),\n **json_decoder.unmake_serializable(dct),\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 87, in __call__\n obj = super().__call__(*args, **kwargs, **strip_tokenize_options)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/implementations/reference.py", line 784, in __init__\n raise ValueError(\n ...<2 lines>...\n )\n'
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[5], line 1
----> 1 catalog_computed = dask.compute(tasks)
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/dask/base.py:769, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
766 expr = expr.optimize()
767 keys = list(flatten(expr.__dask_keys__()))
--> 769 results = schedule(expr, keys, **kwargs)
771 return repack(results)
Cell In[3], line 10
---> 10 return xr.open_dataset(
11 fs.get_mapper(),
12 engine="zarr",
13 backend_kwargs={"consolidated": False},
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/api.py:613, in open_dataset()
601 decoders = _resolve_decoders_kwargs(
602 decode_cf,
603 open_backend_dataset_parameters=backend.open_dataset_parameters,
(...) 609 decode_coords=decode_coords,
610 )
612 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 613 backend_ds = backend.open_dataset(
614 filename_or_obj,
615 drop_variables=drop_variables,
616 **decoders,
617 **kwargs,
618 )
619 ds = _dataset_from_backend_dataset(
620 backend_ds,
621 filename_or_obj,
(...) 632 **kwargs,
633 )
634 return ds
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py:1741, in open_dataset()
1739 filename_or_obj = _normalize_path(filename_or_obj)
1740 if not store:
-> 1741 store = ZarrStore.open_group(
1742 filename_or_obj,
1743 group=group,
1744 mode=mode,
1745 synchronizer=synchronizer,
1746 consolidated=consolidated,
1747 consolidate_on_close=False,
1748 chunk_store=chunk_store,
1749 storage_options=storage_options,
1750 zarr_version=zarr_version,
1751 use_zarr_fill_value_as_mask=None,
1752 zarr_format=zarr_format,
1753 cache_members=cache_members,
1754 )
1756 store_entrypoint = StoreBackendEntrypoint()
1757 with close_on_error(store):
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py:771, in open_group()
745 @classmethod
746 def open_group(
747 cls,
(...) 764 cache_members: bool = True,
765 ):
766 (
767 zarr_group,
768 consolidate_on_close,
769 close_store_on_close,
770 use_zarr_fill_value_as_mask,
--> 771 ) = _get_open_params(
772 store=store,
773 mode=mode,
774 synchronizer=synchronizer,
775 group=group,
776 consolidated=consolidated,
777 consolidate_on_close=consolidate_on_close,
778 chunk_store=chunk_store,
779 storage_options=storage_options,
780 zarr_version=zarr_version,
781 use_zarr_fill_value_as_mask=use_zarr_fill_value_as_mask,
782 zarr_format=zarr_format,
783 )
785 return cls(
786 zarr_group,
787 mode,
(...) 796 cache_members=cache_members,
797 )
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py:1983, in _get_open_params()
1979 if _zarr_v3():
1980 # we have determined that we don't want to use consolidated metadata
1981 # so we set that to False to avoid trying to read it
1982 open_kwargs["use_consolidated"] = False
-> 1983 zarr_group = zarr.open_group(store, **open_kwargs)
1985 close_store_on_close = zarr_group.store is not store
1987 # we use this to determine how to handle fill_value
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/synchronous.py:552, in open_group()
482 def open_group(
483 store: StoreLike | None = None,
484 *,
(...) 494 use_consolidated: bool | str | None = None,
495 ) -> Group:
496 """Open a group using file-mode-like semantics.
497
498 Parameters
(...) 549 The new group.
550 """
551 return Group(
--> 552 sync(
553 async_api.open_group(
554 store=store,
555 mode=mode,
556 cache_attrs=cache_attrs,
557 synchronizer=synchronizer,
558 path=path,
559 chunk_store=chunk_store,
560 storage_options=storage_options,
561 zarr_format=zarr_format,
562 meta_array=meta_array,
563 attributes=attributes,
564 use_consolidated=use_consolidated,
565 )
566 )
567 )
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py:154, in sync()
151 return_result = next(iter(finished)).result()
153 if isinstance(return_result, BaseException):
--> 154 raise return_result
155 else:
156 return return_result
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py:114, in _runner()
109 """
110 Await a coroutine and return the result of running it. If awaiting the coroutine raises an
111 exception, the exception will be returned.
112 """
113 try:
--> 114 return await coro
115 except Exception as ex:
116 return ex
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/asynchronous.py:859, in open_group()
794 """Open a group using file-mode-like semantics.
795
796 Parameters
(...) 847 The new group.
848 """
850 _warn_unimplemented_kwargs(
851 {
852 "cache_attrs": cache_attrs,
(...) 856 }
857 )
--> 859 store_path = await make_store_path(store, mode=mode, storage_options=storage_options, path=path)
860 if attributes is None:
861 attributes = {}
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py:471, in make_store_path()
466 raise ValueError(
467 "'path' was provided but is not used for FSMap store_like objects. Specify the path when creating the FSMap instance instead."
468 )
470 else:
--> 471 store = await make_store(store_like, mode=mode, storage_options=storage_options)
472 return await StorePath.open(store, path=path_normalized, mode=mode)
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py:403, in make_store()
398 return FsspecStore.from_url(
399 store_like, storage_options=storage_options, read_only=_read_only
400 )
402 elif _has_fsspec and isinstance(store_like, FSMap):
--> 403 return FsspecStore.from_mapper(store_like, read_only=_read_only)
405 else:
406 raise TypeError(f"Unsupported type for store_like: '{type(store_like).__name__}'")
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py:206, in from_mapper()
182 @classmethod
183 def from_mapper(
184 cls,
(...) 187 allowed_exceptions: tuple[type[Exception], ...] = ALLOWED_EXCEPTIONS,
188 ) -> FsspecStore:
189 """
190 Create an FsspecStore from an FSMap object.
191
(...) 204 FsspecStore
205 """
--> 206 fs = _make_async(fs_map.fs)
207 return cls(
208 fs=fs,
209 path=fs_map.root,
210 read_only=read_only,
211 allowed_exceptions=allowed_exceptions,
212 )
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py:57, in _make_async()
55 fs_dict = json.loads(fs.to_json())
56 fs_dict["asynchronous"] = True
---> 57 return fsspec.AbstractFileSystem.from_json(json.dumps(fs_dict))
59 if fsspec_version < parse_version("2024.12.0"):
60 raise ImportError(
61 f"The filesystem '{fs}' is synchronous, and the required "
62 "AsyncFileSystemWrapper is not available. Upgrade fsspec to version "
63 "2024.12.0 or later to enable this functionality."
64 )
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py:1500, in from_json()
1479 """
1480 Recreate a filesystem instance from JSON representation.
1481
(...) 1496 at import time.
1497 """
1498 from .json import FilesystemJSONDecoder
-> 1500 return json.loads(blob, cls=FilesystemJSONDecoder)
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/json/__init__.py:365, in loads()
363 if parse_constant is not None:
364 kw['parse_constant'] = parse_constant
--> 365 return cls(**kw).decode(s)
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/json/decoder.py:345, in decode()
340 def decode(self, s, _w=WHITESPACE.match):
341 """Return the Python representation of ``s`` (a ``str`` instance
342 containing a JSON document).
343
344 """
--> 345 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
346 end = _w(s, end).end()
347 if end != len(s):
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/json/decoder.py:361, in raw_decode()
352 """Decode a JSON document from ``s`` (a ``str`` beginning with
353 a JSON document) and return a 2-tuple of the Python
354 representation and the index in ``s`` where the document ended.
(...) 358
359 """
360 try:
--> 361 obj, end = self.scan_once(s, idx)
362 except StopIteration as err:
363 raise JSONDecodeError("Expecting value", s, err.value) from None
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/json.py:92, in custom_object_hook()
90 if "cls" in dct:
91 if (obj_cls := self.try_resolve_fs_cls(dct)) is not None:
---> 92 return AbstractFileSystem.from_dict(dct)
93 if (obj_cls := self.try_resolve_path_cls(dct)) is not None:
94 return obj_cls(dct["str"])
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py:1576, in from_dict()
1573 dct.pop("cls", None)
1574 dct.pop("protocol", None)
-> 1576 return cls(
1577 *json_decoder.unmake_serializable(dct.pop("args", ())),
1578 **json_decoder.unmake_serializable(dct),
1579 )
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py:87, in __call__()
85 return cls._cache[token]
86 else:
---> 87 obj = super().__call__(*args, **kwargs, **strip_tokenize_options)
88 # Setting _fs_token here causes some static linters to complain.
89 obj._fs_token_ = token
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/implementations/reference.py:784, in __init__()
782 self.fss[k] = AsyncFileSystemWrapper(f, asynchronous=self.asynchronous)
783 elif self.asynchronous ^ f.asynchronous:
--> 784 raise ValueError(
785 "Reference-FS's target filesystem must have same value "
786 "of asynchronous"
787 )
ValueError: Reference-FS's target filesystem must have same value of asynchronousBuild an Xarray-Datatree from the dictionary of datasets¶
dt = DataTree.from_dict(catalog_computed[0])Accessing the Datatree¶
A Datatree is a collection of related Xarray datasets. We can access individual datasets using UNIX syntax. In the cell below, we will access a single dataset from the datatree.
dt["ACCESS-CM2/ssp585"]
# or
dt["ACCESS-CM2"]["ssp585"]Convert a Datatree node to a Dataset¶
dt["ACCESS-CM2"]["ssp585"].to_dataset()Operations across a Datatree¶
A Datatree contains a collection of datasets with related coordinates and variables. Using some in-built methods, we can analyze it as if it were a single dataset. Instead of looping through hundreds of Xarray datasets, we can apply operations across the Datatree. In the example below, we will lazily create a time-series.
ts = dt.mean(dim=["lat", "lon"])Visualize a single dataset with HvPlot¶
display( # noqa
dt["ACCESS-CM2/ssp585"].to_dataset().pr.hvplot("lon", "lat", rasterize=True)
)Shut down the Dask cluster¶
client.shutdown()