Overview¶
In this tutorial we are going to use a large collection of pre-generated Kerchunk reference files and open them with Xarray’s new DataTree functionality. This chapter is heavily inspired by this blog post.
About the Dataset¶
This collection of reference files were generated from the NASA NEX-GDDP-CMIP6 (Global Daily Downscaled Projections) dataset. A version of this dataset is hosted on s3 as a collection of NetCDF files.
Prerequisites¶
| Concepts | Importance | Notes |
|---|---|---|
| Kerchunk Basics | Required | Core |
| Multiple Files and Kerchunk | Required | Core |
| Kerchunk and Dask | Required | Core |
| Multi-File Datasets with Kerchunk | Required | IO/Visualization |
| Xarray-Datatree Overview | Required | IO |
Time to learn: 30 minutes
Motivation¶
In total the dataset is roughly 12TB in compressed blob storage, with a single NetCDF file per yearly timestep, per variable. Downloading this entire dataset for analysis on a local machine would difficult to say the least. The collection of Kerchunk reference files for this entire dataset is only 272 Mb, which is about 42,000 times smaller!
Imports¶
import dask
import hvplot.xarray # noqa
import pandas as pd
import xarray as xr
from xarray import DataTree
from distributed import Client
from fsspec.implementations.reference import ReferenceFileSystemLoading...
Loading...
Loading...
Loading...
Read the reference catalog¶
The NASA NEX-GDDP-CMIP6 dataset is organized by GCM, Scenario and Ensemble Member. Each of these Scenario/GCM combinations is represented as a combined reference file, which was created by merging across variables and concatenating along time-steps. All of these references are organized into a simple .csv catalog in the schema:
| GCM/Scenario | url |
|---|
Organzing with Xarray-Datatree¶
Not all of the GCM/Scenario reference datasets have shared spatial coordinates and many of the have slight differences in their calendar and thus time dimension.
Because of this, these cannot be combined into a single Xarray-Dataset. Fortunately Xarray-Datatree provides a higher level abstraction where related Xarray-Datasets are organized into a tree structure where each dataset corresponds to a leaf.
# Read the reference catalog into a Pandas DataFrame
cat_df = pd.read_csv(
"s3://carbonplan-share/nasa-nex-reference/reference_catalog_nested.csv"
)
# Convert the DataFrame into a dictionary
catalog = cat_df.set_index("ID").T.to_dict("records")[0]Load Reference Datasets into Xarray-DataTree¶
In the following cell we create a function load_ref_ds, which can be parallelized via Dask to load Kerchunk references into a dictionary of Xarray-Datasets.
def load_ref_ds(url: str):
fs = ReferenceFileSystem(
url,
remote_protocol="s3",
target_protocol="s3",
remote_options={"anon": True},
target_options={"anon": True},
lazy=True,
)
return xr.open_dataset(
fs.get_mapper(),
engine="zarr",
backend_kwargs={"consolidated": False},
chunks={"time": 300},
)
tasks = {id: dask.delayed(load_ref_ds)(url) for id, url in catalog.items()}Use Dask Distributed to load the Xarray-Datasets from Kerchunk reference files¶
Using Dask, we are loading 164 reference datasets into memory. Since they are are Xarray datasets the coordinates are loaded eagerly, but the underlying data is still lazy.
client = Client(n_workers=8)
client/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/distributed/node.py:195: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 35773 instead
warnings.warn(
Loading...
catalog_computed = dask.compute(tasks)2026-10-18 02:34:38,710 - distributed.worker - ERROR - Compute Failed
Key: load_ref_ds-fd8b805f-9f75-43ca-b69c-de5ffe4c5609
State: executing
Task: <Task 'load_ref_ds-fd8b805f-9f75-43ca-b69c-de5ffe4c5609' load_ref_ds(...)>
Exception: 'ValueError("Reference-FS\'s target filesystem must have same value of asynchronous")'
Traceback: ' File "/tmp/ipykernel_5742/3963997076.py", line 10, in load_ref_ds\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/api.py", line 619, in open_dataset\n backend_ds = backend.open_dataset(\n filename_or_obj,\n ...<2 lines>...\n **kwargs,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1747, in open_dataset\n store = ZarrStore.open_group(\n filename_or_obj,\n ...<7 lines>...\n cache_members=cache_members,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 797, in open_group\n ) = _get_open_params(\n ~~~~~~~~~~~~~~~~^\n store=store,\n ^^^^^^^^^^^^\n ...<6 lines>...\n zarr_format=zarr_format,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1951, in _get_open_params\n zarr_group = zarr.open_group(store, **open_kwargs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/synchronous.py", line 567, in open_group\n sync(\n ~~~~^\n async_api.open_group(\n ^^^^^^^^^^^^^^^^^^^^^\n ...<11 lines>...\n )\n ^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 155, in sync\n raise return_result\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 116, in _runner\n return await coro\n ^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/asynchronous.py", line 894, in open_group\n store_path = await make_store_path(store, mode=mode, storage_options=storage_options, path=path)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 481, in make_store_path\n store = await make_store(store_like, mode=mode, storage_options=storage_options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 413, in make_store\n return FsspecStore.from_mapper(store_like, read_only=_read_only)\n ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 210, in from_mapper\n fs = _make_async(fs_map.fs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 57, in _make_async\n return type(fs)(*fs.storage_args, **{**fs.storage_options, "asynchronous": True})\n ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 128, in __call__\n obj = super().__call__(*args, **kwargs, **strip_tokenize_options)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/implementations/reference.py", line 785, in __init__\n raise ValueError(\n ...<2 lines>...\n )\n'
2026-10-18 02:34:38,726 - distributed.worker - ERROR - Compute Failed
Key: load_ref_ds-f87487fd-6ecc-47db-acc0-23a72b02a306
State: executing
Task: <Task 'load_ref_ds-f87487fd-6ecc-47db-acc0-23a72b02a306' load_ref_ds(...)>
Exception: 'ValueError("Reference-FS\'s target filesystem must have same value of asynchronous")'
Traceback: ' File "/tmp/ipykernel_5742/3963997076.py", line 10, in load_ref_ds\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/api.py", line 619, in open_dataset\n backend_ds = backend.open_dataset(\n filename_or_obj,\n ...<2 lines>...\n **kwargs,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1747, in open_dataset\n store = ZarrStore.open_group(\n filename_or_obj,\n ...<7 lines>...\n cache_members=cache_members,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 797, in open_group\n ) = _get_open_params(\n ~~~~~~~~~~~~~~~~^\n store=store,\n ^^^^^^^^^^^^\n ...<6 lines>...\n zarr_format=zarr_format,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1951, in _get_open_params\n zarr_group = zarr.open_group(store, **open_kwargs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/synchronous.py", line 567, in open_group\n sync(\n ~~~~^\n async_api.open_group(\n ^^^^^^^^^^^^^^^^^^^^^\n ...<11 lines>...\n )\n ^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 155, in sync\n raise return_result\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 116, in _runner\n return await coro\n ^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/asynchronous.py", line 894, in open_group\n store_path = await make_store_path(store, mode=mode, storage_options=storage_options, path=path)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 481, in make_store_path\n store = await make_store(store_like, mode=mode, storage_options=storage_options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 413, in make_store\n return FsspecStore.from_mapper(store_like, read_only=_read_only)\n ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 210, in from_mapper\n fs = _make_async(fs_map.fs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 57, in _make_async\n return type(fs)(*fs.storage_args, **{**fs.storage_options, "asynchronous": True})\n ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 128, in __call__\n obj = super().__call__(*args, **kwargs, **strip_tokenize_options)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/implementations/reference.py", line 785, in __init__\n raise ValueError(\n ...<2 lines>...\n )\n'
2026-10-18 02:34:38,751 - distributed.worker - ERROR - Compute Failed
Key: load_ref_ds-edecaa35-3478-49fc-aabc-db948b8cecdb
State: executing
Task: <Task 'load_ref_ds-edecaa35-3478-49fc-aabc-db948b8cecdb' load_ref_ds(...)>
Exception: 'ValueError("Reference-FS\'s target filesystem must have same value of asynchronous")'
Traceback: ' File "/tmp/ipykernel_5742/3963997076.py", line 10, in load_ref_ds\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/api.py", line 619, in open_dataset\n backend_ds = backend.open_dataset(\n filename_or_obj,\n ...<2 lines>...\n **kwargs,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1747, in open_dataset\n store = ZarrStore.open_group(\n filename_or_obj,\n ...<7 lines>...\n cache_members=cache_members,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 797, in open_group\n ) = _get_open_params(\n ~~~~~~~~~~~~~~~~^\n store=store,\n ^^^^^^^^^^^^\n ...<6 lines>...\n zarr_format=zarr_format,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1951, in _get_open_params\n zarr_group = zarr.open_group(store, **open_kwargs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/synchronous.py", line 567, in open_group\n sync(\n ~~~~^\n async_api.open_group(\n ^^^^^^^^^^^^^^^^^^^^^\n ...<11 lines>...\n )\n ^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 155, in sync\n raise return_result\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 116, in _runner\n return await coro\n ^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/asynchronous.py", line 894, in open_group\n store_path = await make_store_path(store, mode=mode, storage_options=storage_options, path=path)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 481, in make_store_path\n store = await make_store(store_like, mode=mode, storage_options=storage_options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 413, in make_store\n return FsspecStore.from_mapper(store_like, read_only=_read_only)\n ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 210, in from_mapper\n fs = _make_async(fs_map.fs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 57, in _make_async\n return type(fs)(*fs.storage_args, **{**fs.storage_options, "asynchronous": True})\n ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 128, in __call__\n obj = super().__call__(*args, **kwargs, **strip_tokenize_options)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/implementations/reference.py", line 785, in __init__\n raise ValueError(\n ...<2 lines>...\n )\n'
2026-10-18 02:34:38,772 - distributed.worker - ERROR - Compute Failed
Key: load_ref_ds-eb108c4f-caa4-4d88-ba22-39e1680390ca
State: executing
Task: <Task 'load_ref_ds-eb108c4f-caa4-4d88-ba22-39e1680390ca' load_ref_ds(...)>
Exception: 'ValueError("Reference-FS\'s target filesystem must have same value of asynchronous")'
Traceback: ' File "/tmp/ipykernel_5742/3963997076.py", line 10, in load_ref_ds\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/api.py", line 619, in open_dataset\n backend_ds = backend.open_dataset(\n filename_or_obj,\n ...<2 lines>...\n **kwargs,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1747, in open_dataset\n store = ZarrStore.open_group(\n filename_or_obj,\n ...<7 lines>...\n cache_members=cache_members,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 797, in open_group\n ) = _get_open_params(\n ~~~~~~~~~~~~~~~~^\n store=store,\n ^^^^^^^^^^^^\n ...<6 lines>...\n zarr_format=zarr_format,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1951, in _get_open_params\n zarr_group = zarr.open_group(store, **open_kwargs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/synchronous.py", line 567, in open_group\n sync(\n ~~~~^\n async_api.open_group(\n ^^^^^^^^^^^^^^^^^^^^^\n ...<11 lines>...\n )\n ^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 155, in sync\n raise return_result\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 116, in _runner\n return await coro\n ^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/asynchronous.py", line 894, in open_group\n store_path = await make_store_path(store, mode=mode, storage_options=storage_options, path=path)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 481, in make_store_path\n store = await make_store(store_like, mode=mode, storage_options=storage_options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 413, in make_store\n return FsspecStore.from_mapper(store_like, read_only=_read_only)\n ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 210, in from_mapper\n fs = _make_async(fs_map.fs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 57, in _make_async\n return type(fs)(*fs.storage_args, **{**fs.storage_options, "asynchronous": True})\n ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 128, in __call__\n obj = super().__call__(*args, **kwargs, **strip_tokenize_options)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/implementations/reference.py", line 785, in __init__\n raise ValueError(\n ...<2 lines>...\n )\n'
2026-10-18 02:34:38,856 - distributed.worker - ERROR - Compute Failed
Key: load_ref_ds-eae5f708-c380-4528-a82b-d93737abad9e
State: executing
Task: <Task 'load_ref_ds-eae5f708-c380-4528-a82b-d93737abad9e' load_ref_ds(...)>
Exception: 'ValueError("Reference-FS\'s target filesystem must have same value of asynchronous")'
Traceback: ' File "/tmp/ipykernel_5742/3963997076.py", line 10, in load_ref_ds\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/api.py", line 619, in open_dataset\n backend_ds = backend.open_dataset(\n filename_or_obj,\n ...<2 lines>...\n **kwargs,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1747, in open_dataset\n store = ZarrStore.open_group(\n filename_or_obj,\n ...<7 lines>...\n cache_members=cache_members,\n )\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 797, in open_group\n ) = _get_open_params(\n ~~~~~~~~~~~~~~~~^\n store=store,\n ^^^^^^^^^^^^\n ...<6 lines>...\n zarr_format=zarr_format,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py", line 1951, in _get_open_params\n zarr_group = zarr.open_group(store, **open_kwargs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/synchronous.py", line 567, in open_group\n sync(\n ~~~~^\n async_api.open_group(\n ^^^^^^^^^^^^^^^^^^^^^\n ...<11 lines>...\n )\n ^\n )\n ^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 155, in sync\n raise return_result\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py", line 116, in _runner\n return await coro\n ^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/asynchronous.py", line 894, in open_group\n store_path = await make_store_path(store, mode=mode, storage_options=storage_options, path=path)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 481, in make_store_path\n store = await make_store(store_like, mode=mode, storage_options=storage_options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py", line 413, in make_store\n return FsspecStore.from_mapper(store_like, read_only=_read_only)\n ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 210, in from_mapper\n fs = _make_async(fs_map.fs)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py", line 57, in _make_async\n return type(fs)(*fs.storage_args, **{**fs.storage_options, "asynchronous": True})\n ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py", line 128, in __call__\n obj = super().__call__(*args, **kwargs, **strip_tokenize_options)\n File "/home/runner/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/implementations/reference.py", line 785, in __init__\n raise ValueError(\n ...<2 lines>...\n )\n'
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[5], line 1
----> 1 catalog_computed = dask.compute(tasks)
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/dask/base.py:769, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
766 expr = expr.optimize()
767 keys = list(flatten(expr.__dask_keys__()))
--> 769 results = schedule(expr, keys, **kwargs)
771 return repack(results)
Cell In[3], line 10
---> 10 return xr.open_dataset(
11 fs.get_mapper(),
12 engine="zarr",
13 backend_kwargs={"consolidated": False},
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/api.py:619, in open_dataset()
607 decoders = _resolve_decoders_kwargs(
608 decode_cf,
609 open_backend_dataset_parameters=backend.open_dataset_parameters,
(...) 615 decode_coords=decode_coords,
616 )
618 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 619 backend_ds = backend.open_dataset(
620 filename_or_obj,
621 drop_variables=drop_variables,
622 **decoders,
623 **kwargs,
624 )
625 ds = _dataset_from_backend_dataset(
626 backend_ds,
627 filename_or_obj,
(...) 638 **kwargs,
639 )
640 return ds
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py:1747, in open_dataset()
1745 filename_or_obj = _normalize_path(filename_or_obj)
1746 if not store:
-> 1747 store = ZarrStore.open_group(
1748 filename_or_obj,
1749 group=group,
1750 mode=mode,
1751 consolidated=consolidated,
1752 consolidate_on_close=False,
1753 storage_options=storage_options,
1754 use_zarr_fill_value_as_mask=None,
1755 zarr_format=zarr_format,
1756 cache_members=cache_members,
1757 )
1759 store_entrypoint = StoreBackendEntrypoint()
1760 with close_on_error(store):
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py:797, in open_group()
774 @classmethod
775 def open_group(
776 cls,
(...) 790 cache_members: bool = True,
791 ):
792 (
793 zarr_group,
794 consolidate_on_close,
795 close_store_on_close,
796 use_zarr_fill_value_as_mask,
--> 797 ) = _get_open_params(
798 store=store,
799 mode=mode,
800 group=group,
801 consolidated=consolidated,
802 consolidate_on_close=consolidate_on_close,
803 storage_options=storage_options,
804 use_zarr_fill_value_as_mask=use_zarr_fill_value_as_mask,
805 zarr_format=zarr_format,
806 )
808 return cls(
809 zarr_group,
810 mode,
(...) 819 cache_members=cache_members,
820 )
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/xarray/backends/zarr.py:1951, in _get_open_params()
1947 else:
1948 # we have determined that we don't want to use consolidated metadata
1949 # so we set that to False to avoid trying to read it
1950 open_kwargs["use_consolidated"] = False
-> 1951 zarr_group = zarr.open_group(store, **open_kwargs)
1953 close_store_on_close = zarr_group.store is not store
1955 # we use this to determine how to handle fill_value
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/synchronous.py:567, in open_group()
496 def open_group(
497 store: StoreLike | None = None,
498 *,
(...) 508 use_consolidated: bool | str | None = None,
509 ) -> Group:
510 """Open a group using file-mode-like semantics.
511
512 Parameters
(...) 564 The new group.
565 """
566 return Group(
--> 567 sync(
568 async_api.open_group(
569 store=store,
570 mode=mode,
571 cache_attrs=cache_attrs,
572 synchronizer=synchronizer,
573 path=path,
574 chunk_store=chunk_store,
575 storage_options=storage_options,
576 zarr_format=zarr_format,
577 meta_array=meta_array,
578 attributes=attributes,
579 use_consolidated=use_consolidated,
580 )
581 )
582 )
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py:155, in sync()
152 return_result = next(iter(finished)).result()
154 if isinstance(return_result, BaseException):
--> 155 raise return_result
156 else:
157 return return_result
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/core/sync.py:116, in _runner()
111 """
112 Await a coroutine and return the result of running it. If awaiting the coroutine raises an
113 exception, the exception will be returned.
114 """
115 try:
--> 116 return await coro
117 except Exception as ex: # noqa: BLE001 -- the caller re-raises the returned exception
118 return ex
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/api/asynchronous.py:894, in open_group()
828 """Open a group using file-mode-like semantics.
829
830 Parameters
(...) 882 The new group.
883 """
885 _warn_unimplemented_kwargs(
886 {
887 "cache_attrs": cache_attrs,
(...) 891 }
892 )
--> 894 store_path = await make_store_path(store, mode=mode, storage_options=storage_options, path=path)
895 if attributes is None:
896 attributes = {}
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py:481, in make_store_path()
476 raise ValueError(
477 "'path' was provided but is not used for FSMap store_like objects. Specify the path when creating the FSMap instance instead."
478 )
480 else:
--> 481 store = await make_store(store_like, mode=mode, storage_options=storage_options)
482 return await StorePath.open(store, path=path_normalized, mode=mode)
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_common.py:413, in make_store()
408 return FsspecStore.from_url(
409 store_like, storage_options=storage_options, read_only=_read_only
410 )
412 elif _has_fsspec and isinstance(store_like, FSMap):
--> 413 return FsspecStore.from_mapper(store_like, read_only=_read_only)
415 else:
416 raise TypeError(f"Unsupported type for store_like: '{type(store_like).__name__}'")
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py:210, in from_mapper()
186 @classmethod
187 def from_mapper(
188 cls,
(...) 191 allowed_exceptions: tuple[type[Exception], ...] = ALLOWED_EXCEPTIONS,
192 ) -> FsspecStore:
193 """
194 Create an FsspecStore from an FSMap object.
195
(...) 208 FsspecStore
209 """
--> 210 fs = _make_async(fs_map.fs)
211 return cls(
212 fs=fs,
213 path=fs_map.root,
214 read_only=read_only,
215 allowed_exceptions=allowed_exceptions,
216 )
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/zarr/storage/_fsspec.py:57, in _make_async()
52 return fs
53 if fs.async_impl:
54 # Convert sync instance of an async fs to an async instance. Reuse the original
55 # constructor arguments rather than round-tripping through JSON, since storage
56 # options may hold objects that are not JSON-serializable (e.g. credentials).
---> 57 return type(fs)(*fs.storage_args, **{**fs.storage_options, "asynchronous": True})
59 if fsspec_version < parse_version("2024.12.0"):
60 raise ImportError(
61 f"The filesystem '{fs}' is synchronous, and the required "
62 "AsyncFileSystemWrapper is not available. Upgrade fsspec to version "
63 "2024.12.0 or later to enable this functionality."
64 )
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/spec.py:128, in __call__()
125 if inst is not None:
126 return inst
--> 128 obj = super().__call__(*args, **kwargs, **strip_tokenize_options)
129 # Setting _fs_token here causes some static linters to complain.
130 obj._fs_token_ = token
File ~/micromamba/envs/kerchunk-cookbook/lib/python3.14/site-packages/fsspec/implementations/reference.py:785, in __init__()
783 self.fss[k] = AsyncFileSystemWrapper(f, asynchronous=self.asynchronous)
784 elif self.asynchronous ^ f.asynchronous:
--> 785 raise ValueError(
786 "Reference-FS's target filesystem must have same value "
787 "of asynchronous"
788 )
ValueError: Reference-FS's target filesystem must have same value of asynchronousBuild an Xarray-Datatree from the dictionary of datasets¶
dt = DataTree.from_dict(catalog_computed[0])Accessing the Datatree¶
A Datatree is a collection of related Xarray datasets. We can access individual datasets using UNIX syntax. In the cell below, we will access a single dataset from the datatree.
dt["ACCESS-CM2/ssp585"]
# or
dt["ACCESS-CM2"]["ssp585"]Convert a Datatree node to a Dataset¶
dt["ACCESS-CM2"]["ssp585"].to_dataset()Operations across a Datatree¶
A Datatree contains a collection of datasets with related coordinates and variables. Using some in-built methods, we can analyze it as if it were a single dataset. Instead of looping through hundreds of Xarray datasets, we can apply operations across the Datatree. In the example below, we will lazily create a time-series.
ts = dt.mean(dim=["lat", "lon"])Visualize a single dataset with HvPlot¶
display( # noqa
dt["ACCESS-CM2/ssp585"].to_dataset().pr.hvplot("lon", "lat", rasterize=True)
)Shut down the Dask cluster¶
client.shutdown()