Interactive Visualuzation using `hvPlot`¶

Overview¶

ERA-5 Dataset is available from NCAR RDA in netcdf format. A subset of this dataset is processed into Zarr format and available from NCAR RDA endpoints. To learn how you can create Zarr files from NCAR RDA netcdf files, please see this notebook.

By the end of this notebook, you should be able to:

Understand the importance for interactive plots and the challenges associated with them
Use hvPlot to generate basic interactive plots with Xarray

Prerequisites¶

Concepts	Importance	Notes
Intro to Xarray	Necessary

Time to learn: 30 minutes

Imports¶

import holoviews as hv
import xarray as xr
from holoviews import opts

hv.extension("bokeh")

Data¶

As we mentioned above a subset of NCAR RDA data is available in Zarr format.

rda_url = "https://data.rda.ucar.edu/"
annual_means = rda_url + "pythia_era5_24/annual_means/"
xrds = xr.open_dataset(annual_means + "temp_2m_annual_1940_2023.zarr", engine="zarr")
xrds = xrds.isel(time=slice(0, 5))
xrds.load()

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/fsspec/mapping.py:155, in FSMap.__getitem__(self, key, default)
    154 try:
--> 155     result = self.fs.cat(k)
    156 except self.missing_exceptions as exc:

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/fsspec/asyn.py:118, in sync_wrapper.<locals>.wrapper(*args, **kwargs)
    117 self = obj or args[0]
--> 118 return sync(self.loop, func, *args, **kwargs)

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/fsspec/asyn.py:103, in sync(loop, func, timeout, *args, **kwargs)
    102 elif isinstance(return_result, BaseException):
--> 103     raise return_result
    104 else:

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/fsspec/asyn.py:56, in _runner(event, coro, result, timeout)
     55 try:
---> 56     result[0] = await coro
     57 except Exception as ex:

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/fsspec/asyn.py:464, in AsyncFileSystem._cat(self, path, recursive, on_error, batch_size, **kwargs)
    463     if ex:
--> 464         raise ex
    465 if (
    466     len(paths) > 1
    467     or isinstance(path, list)
    468     or paths[0] != self._strip_protocol(path)
    469 ):

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/fsspec/asyn.py:244, in _run_coros_in_chunks.<locals>._run_coro(coro, i)
    243 try:
--> 244     return await asyncio.wait_for(coro, timeout=timeout), i
    245 except Exception as e:

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/asyncio/tasks.py:507, in wait_for(fut, timeout)
    506 async with timeouts.timeout(timeout):
--> 507     return await fut

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/fsspec/implementations/http.py:246, in HTTPFileSystem._cat_file(self, url, start, end, **kwargs)
    245     out = await r.read()
--> 246     self._raise_not_found_for_status(r, url)
    247 return out

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/fsspec/implementations/http.py:228, in HTTPFileSystem._raise_not_found_for_status(self, response, url)
    227 if response.status == 404:
--> 228     raise FileNotFoundError(url)
    229 response.raise_for_status()

FileNotFoundError: https://data.rda.ucar.edu/pythia_era5_24/annual_means/temp_2m_annual_1940_2023.zarr/.zmetadata

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/zarr/storage.py:1446, in FSStore.__getitem__(self, key)
   1445 try:
-> 1446     return self.map[key]
   1447 except self.exceptions as e:

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/fsspec/mapping.py:159, in FSMap.__getitem__(self, key, default)
    158         return default
--> 159     raise KeyError(key) from exc
    160 return result

KeyError: '.zmetadata'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/xarray/backends/zarr.py:1867, in _get_open_params(store, mode, synchronizer, group, consolidated, consolidate_on_close, chunk_store, storage_options, zarr_version, use_zarr_fill_value_as_mask, zarr_format)
   1866 try:
-> 1867     zarr_root_group = zarr.open_consolidated(store, **open_kwargs)
   1868 except (ValueError, KeyError):
   1869     # ValueError in zarr-python 3.x, KeyError in 2.x.

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/zarr/convenience.py:1362, in open_consolidated(store, metadata_key, mode, **kwargs)
   1361 # setup metadata store
-> 1362 meta_store = ConsolidatedStoreClass(store, metadata_key=metadata_key)
   1364 # pass through

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/zarr/storage.py:3045, in ConsolidatedMetadataStore.__init__(self, store, metadata_key)
   3044 # retrieve consolidated metadata
-> 3045 meta = json_loads(self.store[metadata_key])
   3047 # check format of consolidated metadata

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/zarr/storage.py:1448, in FSStore.__getitem__(self, key)
   1447 except self.exceptions as e:
-> 1448     raise KeyError(key) from e

KeyError: '.zmetadata'

During handling of the above exception, another exception occurred:

GroupNotFoundError                        Traceback (most recent call last)
File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/xarray/backends/zarr.py:1871, in _get_open_params(store, mode, synchronizer, group, consolidated, consolidate_on_close, chunk_store, storage_options, zarr_version, use_zarr_fill_value_as_mask, zarr_format)
   1870 try:
-> 1871     zarr_root_group = zarr.open_group(store, **open_kwargs)
   1872     emit_user_level_warning(
   1873         "Failed to open Zarr store with consolidated metadata, "
   1874         "but successfully read with non-consolidated metadata. "
   (...)   1884         RuntimeWarning,
   1885     )

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/zarr/hierarchy.py:1578, in open_group(store, mode, cache_attrs, synchronizer, path, chunk_store, storage_options, zarr_version, meta_array)
   1577             raise ContainsArrayError(path)
-> 1578         raise GroupNotFoundError(path)
   1580 elif mode == "w":

GroupNotFoundError: group not found at path ''

The above exception was the direct cause of the following exception:

FileNotFoundError                         Traceback (most recent call last)
Cell In[2], line 3
      1 rda_url = "https://data.rda.ucar.edu/"
      2 annual_means = rda_url + "pythia_era5_24/annual_means/"
----> 3 xrds = xr.open_dataset(annual_means + "temp_2m_annual_1940_2023.zarr", engine="zarr")
      4 xrds = xrds.isel(time=slice(0, 5))
      5 xrds.load()

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/xarray/backends/api.py:596, in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, create_default_indexes, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)
    584 decoders = _resolve_decoders_kwargs(
    585     decode_cf,
    586     open_backend_dataset_parameters=backend.open_dataset_parameters,
   (...)    592     decode_coords=decode_coords,
    593 )
    595 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 596 backend_ds = backend.open_dataset(
    597     filename_or_obj,
    598     drop_variables=drop_variables,
    599     **decoders,
    600     **kwargs,
    601 )
    602 ds = _dataset_from_backend_dataset(
    603     backend_ds,
    604     filename_or_obj,
   (...)    615     **kwargs,
    616 )
    617 return ds

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/xarray/backends/zarr.py:1659, in ZarrBackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, synchronizer, consolidated, chunk_store, storage_options, zarr_version, zarr_format, store, engine, use_zarr_fill_value_as_mask, cache_members)
   1657 filename_or_obj = _normalize_path(filename_or_obj)
   1658 if not store:
-> 1659     store = ZarrStore.open_group(
   1660         filename_or_obj,
   1661         group=group,
   1662         mode=mode,
   1663         synchronizer=synchronizer,
   1664         consolidated=consolidated,
   1665         consolidate_on_close=False,
   1666         chunk_store=chunk_store,
   1667         storage_options=storage_options,
   1668         zarr_version=zarr_version,
   1669         use_zarr_fill_value_as_mask=None,
   1670         zarr_format=zarr_format,
   1671         cache_members=cache_members,
   1672     )
   1674 store_entrypoint = StoreBackendEntrypoint()
   1675 with close_on_error(store):

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/xarray/backends/zarr.py:714, in ZarrStore.open_group(cls, store, mode, synchronizer, group, consolidated, consolidate_on_close, chunk_store, storage_options, append_dim, write_region, safe_chunks, align_chunks, zarr_version, zarr_format, use_zarr_fill_value_as_mask, write_empty, cache_members)
    688 @classmethod
    689 def open_group(
    690     cls,
   (...)    707     cache_members: bool = True,
    708 ):
    709     (
    710         zarr_group,
    711         consolidate_on_close,
    712         close_store_on_close,
    713         use_zarr_fill_value_as_mask,
--> 714     ) = _get_open_params(
    715         store=store,
    716         mode=mode,
    717         synchronizer=synchronizer,
    718         group=group,
    719         consolidated=consolidated,
    720         consolidate_on_close=consolidate_on_close,
    721         chunk_store=chunk_store,
    722         storage_options=storage_options,
    723         zarr_version=zarr_version,
    724         use_zarr_fill_value_as_mask=use_zarr_fill_value_as_mask,
    725         zarr_format=zarr_format,
    726     )
    728     return cls(
    729         zarr_group,
    730         mode,
   (...)    739         cache_members=cache_members,
    740     )

File ~/micromamba/envs/ERA5_interactive/lib/python3.13/site-packages/xarray/backends/zarr.py:1887, in _get_open_params(store, mode, synchronizer, group, consolidated, consolidate_on_close, chunk_store, storage_options, zarr_version, use_zarr_fill_value_as_mask, zarr_format)
   1872             emit_user_level_warning(
   1873                 "Failed to open Zarr store with consolidated metadata, "
   1874                 "but successfully read with non-consolidated metadata. "
   (...)   1884                 RuntimeWarning,
   1885             )
   1886         except missing_exc as err:
-> 1887             raise FileNotFoundError(
   1888                 f"No such file or directory: '{store}'"
   1889             ) from err
   1891 # but the user should still receive a DataTree whose root is the group they asked for
   1892 if group and group != "/":

FileNotFoundError: No such file or directory: 'https://data.rda.ucar.edu/pythia_era5_24/annual_means/temp_2m_annual_1940_2023.zarr'

Considerations for Interactive Plots¶

Add some markdown text on some of the following ideas:

What are some reasons we want to make data visualuzation interactive?

Baisc Interactivity using `hvPlot`¶

The hvPlot package is a familiar and high level API for data exploration and visualuzation.

One of the most powerfull features of hvPlot is that it provides an alternative plotting API that directly attaches to existing Python objects through the .hvplot() attribute. For the case of Xarray, importing hvplot.xarray adds a brand new set of plotting routines accessible either through xr.DataArray.hvplot() or xr.Dataset.hvplot()

import hvplot.xarray

Before using hvPlot, let’s take a look at the default Xarray plotting methods.

xrds["VAR_2T"].plot()

We can replace the .plot() function call with .hvplot(). By default, hvPlot uses the Bokeh backend, which has naitive interactive tools, such as :

Panning
Box Select
Scroll Zoom
Saving
Resetting

xrds["VAR_2T"].hvplot()

If we wanted to plot ...

xrds["VAR_2T"].isel(time=0).plot()

Switching

xrds["VAR_2T"].isel(time=0).hvplot()

Climate data typically comes with multiple timesteps. We can create a basic widget that allows us to seek through time by setting the groupby='time' parameter in our .hvplot() call.

xrds["VAR_2T"].hvplot(groupby="time", widget_location="bottom")

You may notice that our colorbar is dynamically changing as we change our time steps. We can fix the colorbar by setting a clim value, which is a tuple of the minimum and maximum desired colorbar range.

One suggestion is to use the minimum and maximum of the data variable you are visualuzing across time.

clim = (xrds["VAR_2T"].values.min(), xrds["VAR_2T"].values.max())

xrds["VAR_2T"].hvplot(clim=clim, groupby="time", widget_location="bottom")

You may have noticed that there is a slight lag when switching time steps. This is due to hvPlot plotting the full resolution of our dataset. We can instead rasterize the output by setting rasterize=True, which will significantly improve the perfromance of our interactive plot.

xrds["VAR_2T"].hvplot(
    rasterize=True, clim=clim, groupby="time", widget_location="bottom"
)

Another usefull interactive feature is animations. Instead of manually scrolling through time, we can set up a widget that lets us animate our data across time. This can be achieved by adding a Scrubber widget to our plot by setting widget_type="scrubber"

xrds["VAR_2T"].hvplot(
    rasterize=True,
    groupby="time",
    widget_type="scrubber",
    widget_location="bottom",
)

Visualization Notebooks

Interactive Visualization using GeoViews

Visualization Notebooks

Creating an Interactive Dashboard using hvPlot and Panel

Interactive Visualuzation using `hvPlot`¶

Overview¶

Prerequisites¶

Imports¶

Data¶

Considerations for Interactive Plots¶

Baisc Interactivity using `hvPlot`¶

Time Widget¶

Animation Widget¶

Interactive Visualuzation using hvPlot

Interactive Visualuzation using hvPlot¶

Overview¶

Prerequisites¶

Imports¶

Data¶

Considerations for Interactive Plots¶

Baisc Interactivity using hvPlot¶

Time Widget¶

Animation Widget¶

Interactive Visualuzation using `hvPlot`¶

Baisc Interactivity using `hvPlot`¶