Using PelicanFS via FSSpec to Access Data on the OSDF

Overview¶

Now that you’ve learned about the OSDF and the Pelican command line client, you may be wondering how you can easily access that data from within a notebook using python.

You can do this using PelicanFS, which is an FSSPec implementation of the Pelican client.

This notebook will contain:¶

A brief explanation of FSSPec and PelicanFS
A real-world example using FSSPec, Pelican, Xarray, and Zarr
Other common access patterns
FAQs

Prerequisites¶

To better understand this notebook, please familiarize yourself with the following concepts:

Concepts	Importance	Notes
Intro to OSDF	Necessary
Understanding of Xarray	Helpful	To better understand the example workflow
Overview of FSSpec	Helpful	To better understand the FSSpec library

Time to learn: 20-30 minutes

Imports¶

import xarray as xr
import numpy as np
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
import cartopy.feature as cfeature
import metpy.calc as mpcalc
from metpy.units import units
import fsspec
import intake

What are PelicanFS and FSSPec?¶

First, let’s understand PelicanFS and how it integrates with FSSpec

FSSPec¶

FileSystem Spec (fsspec) is a python library which endeavors to provide a unified interface to many different storage backends. This includes, but is not limited to, POSIX, https, and S3. It’s used by various data processing libraries such as xarray, pandas, and intake, just to name a few.

To learn more about FSSPec, visit its information page.

Schemes¶

FSSpec figures out how to interact with data from different storage backends through the scheme in the data path. For example, FSSpec knows to use the “Hyper Text Transfer Protocol” interface whenever it sees URLs with the https: scheme. This lets users interact with data from a variety of storage technologies without forcing them to know how those technologies work under the hood.

PelicanFS¶

PelicanFS is an implementation of FSSpec that introduces two new schemes to FSSpec: pelican and osdf. PelicanFS enables you to use the pelican:// scheme to access data via Pelican Federations like the OSDF in any software that already understands FSSpec. To use it, you must specify the federation host name. A Pelican path looks like:

pelican://<federation-host-name>/<namespace-path>

The osdf scheme is a specific instance of the pelican scheme that knows how to access the OSDF. A path using the osdf scheme should not provide the federation root. An OSDF path looks like:

osdf:///<namespace-path>

PelicanFS teaches FSSpec how to interact with the Pelican protocol using the above pelican:-schemed or osdf:-schemed URLs.

If you’d like to understand more about how pelican works, check out the documentation here.

Putting it all together¶

What does this mean in practice?

If you want to access data from the OSDF using FSSpec or any library that uses FSSpec, build the osdf-schemed URL for the data and use that URL as your data path and then FSSpec and PelicanFS will do all the work to resolve it behind the scenes.

A PelicanFS Example using Real Data¶

The following is an example that shows how PelicanFS works on real world data using FSSPec and Xarray to access Zarr data from AWS.

This portion of the notebook is based off of the Project Pythia HRRR AWS Cookbook

Setting the Proper Path¶

The data for this tutorial is part of AWS Open Data, hosted in the us-west-1 region. The OSDF provides access to that region using the /aws-opendata/us-west-1 namespace.

Let’s first create a path which uses the osdf scheme.

# Set the date, hour, variable, and level for the HRRR data
date = '20211016'
hour = '21'
var = 'TMP'
level = '2m_above_ground'

# Construct object paths for the Zarr datasets using the osdf scheme
namespace_object1 = f'osdf:///aws-opendata/us-west-1/hrrrzarr/sfc/{date}/{date}_{hour}z_anl.zarr/{level}/{var}/{level}/'
namespace_object2 = f'osdf:///aws-opendata/us-west-1/hrrrzarr/sfc/{date}/{date}_{hour}z_anl.zarr/{level}/{var}/'

Using FSSpec to access the data¶

Now we can access the data using XArray as usual. The two objects will be accessed using fsspec’s get_mapper function, which knows to use PelicanFS because we created the path using the osdf scheme.

# Get mappers for the Zarr datasets

object1 = fsspec.get_mapper(namespace_object1)
object2 = fsspec.get_mapper(namespace_object2)

# Open the datasets
ds = xr.open_mfdataset([object1, object2], engine='zarr', decode_timedelta=True)

# Display the dataset
ds

Continue the workflow¶

As you can see, Xarray streamed the data correctly into the datasets. To prove the workflow works, the next cell continues the computation and generates two plots. This tutorial will not go in depth as to what this code is accomplishing.

If you’d like to know more about the following workflow, please refer to the Project Pythia HRRR AWS Cookbook

# Define coordinates for projection
lon1 = -97.5
lat1 = 38.5
slat = 38.5

# Define the Lambert Conformal projection
projData = ccrs.LambertConformal(
    central_longitude=lon1,
    central_latitude=lat1,
    standard_parallels=[slat, slat],
    globe=ccrs.Globe(
        semimajor_axis=6371229,
        semiminor_axis=6371229
    )
)

# Display dataset coordinates
ds.coords

# Extract temperature data
airTemp = ds.TMP

# Display the temperature data
airTemp

# Convert temperature units to Celsius
airTemp = airTemp.metpy.convert_units('degC')

# Display the converted temperature data
airTemp

# Extract projection coordinates
x = airTemp.projection_x_coordinate
y = airTemp.projection_y_coordinate

# Plot temperature data
airTemp.plot(figsize=(11, 8.5))

# Compute minimum and maximum temperatures
minTemp = airTemp.min().compute()
maxTemp = airTemp.max().compute()

# Display minimum and maximum temperature values
minTemp.values, maxTemp.values

# Define contour levels
fint = np.arange(np.floor(minTemp.values), np.ceil(maxTemp.values) + 2, 2)

# Define plot bounds and resolution
latN = 50.4
latS = 24.25
lonW = -123.8
lonE = -71.2
res = '50m'

# Create a figure and axis with projection
fig = plt.figure(figsize=(18, 12))
ax = plt.subplot(1, 1, 1, projection=projData)
ax.set_extent([lonW, lonE, latS, latN], crs=ccrs.PlateCarree())
ax.add_feature(cfeature.COASTLINE.with_scale(res))
ax.add_feature(cfeature.STATES.with_scale(res))

# Add the title
tl1 = 'HRRR 2m temperature ($^\\circ$C)'
tl2 = f'Analysis valid at: {hour}00 UTC {date}'
plt.title(f'{tl1}\n{tl2}', fontsize=16)

# Contour fill
CF = ax.contourf(x, y, airTemp, levels=fint, cmap=plt.get_cmap('coolwarm'))

# Make a colorbar for the ContourSet returned by the contourf call
cbar = fig.colorbar(CF, shrink=0.5)
cbar.set_label(r'2m Temperature ($^\circ$C)', size='large')

# Show the plot
plt.show()

/home/runner/micromamba/envs/osdf-cookbook/lib/python3.12/site-packages/cartopy/io/__init__.py:242: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/50m_physical/ne_50m_coastline.zip
  warnings.warn(f'Downloading: {url}', DownloadWarning)

/home/runner/micromamba/envs/osdf-cookbook/lib/python3.12/site-packages/cartopy/io/__init__.py:242: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/50m_cultural/ne_50m_admin_1_states_provinces_lakes.zip
  warnings.warn(f'Downloading: {url}', DownloadWarning)

Other Ways to Access¶

There are other common ways to access data and use data with FSSpec and PelicanFS. This section will will cover the following topics

Using an Intake Catalog
Directly Accessing Data

Intake Catalog¶

In order to use PelicanFS with an Intake catalog, the paths in the catalog itself need to use the osdf or pelican schemes.

Here’s an example using the catalog located at https://data.rda.ucar.edu/d850001/catalogs/osdf/cmip6-aws/cmip6-osdf-zarr.json

An entry in the catalog’s csv file looks like:

HighResMIP,CMCC,CMCC-CM2-HR4,highresSST-present,r1i1p1f1,Amon,ta,gn,osdf:///aws-opendata/us-west-2/cmip6-pds/CMIP6/HighResMIP/CMCC/CMCC-CM2-HR4/highresSST-present/r1i1p1f1/Amon/ta/gn/v20170706/,,20170706

Notice how the path is using the ‘osdf’ scheme and the ‘/aws-opendata/us-west-2’ namespace. If all the paths in the csv file are formatted like this, then you can use the Intake catalog exactly as usual.

Here is a workflow and plot which uses an Intake catalog and the osdf scheme. If you want to understand more about the underlying workflow, please look at the Global Mean Surface Temperature Anomalies (GMSTA) from CMIP6 data notebook.

rda_url     =  'https://data.rda.ucar.edu/'
cat_url     = rda_url +  'd850001/catalogs/osdf/cmip6-aws/cmip6-osdf-zarr.json'

col = intake.open_esm_datastore(cat_url)

expts = ['historical']

query = dict(
    experiment_id=expts,
    table_id='Amon',
    variable_id=['tas'],
    member_id = 'r1i1p1f1',
    #activity_id = 'CMIP',
)

col_subset = col.search(require_all_on=["source_id"], **query)

ds = xr.open_zarr(col_subset.df['zstore'][0])

ds.tas.isel(time=0).plot()

/home/runner/micromamba/envs/osdf-cookbook/lib/python3.12/site-packages/intake_esm/__init__.py:6: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  from pkg_resources import DistributionNotFound, get_distribution

Direct Access of Data¶

You can also access the data directly using normal file system calls.

For example, let’s say you want to read in a csv object from the OSDF. Just use the same pattern we’ve shown before of

osdf:///<namespace-path>

for your path.

with fsspec.open('osdf:///ndp/burnpro3d/YosemiteBurnExample/burnpro3d-yosemite-example.csv') as ex_csv:
    content = ex_csv.read()
    print(content.decode())

Reminder: This tool is in beta testing through and this report has been provided to users for the purpose of getting feedback.,,,,,,,,,,,,,,,,
id,name,fdfm,live_fm,wind_speed_mph,wind_direction,surface,midstory,canopy,outside_surface_growth_rate,outside_canopy_consumption_rate,time_to_traverse_buffer,run_max_mem_rss_MB,elapsed_model_s,fuels_dens_surface_final_plot,run_zarr,fuel_zarr
bcbb40c0-2875-4753-b77a-a7d382b8045f,Yosemite valentine's day burn,0.08,0.9,3,30,0.9069966673851013,0.7914127111434937,0.229936882853508,-1,-1,-1,7116,53844,https://wifire-data.sdsc.edu/data/burnpro3d/d/bc/bb/run_bcbb40c0-2875-4753-b77a-a7d382b8045f/png/run_bcbb40c0-2875-4753-b77a-a7d382b8045f_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/bc/bb/run_bcbb40c0-2875-4753-b77a-a7d382b8045f/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
d27d6ab2-b9ad-42d5-95b8-d191d88dc98a,Yosemite valentine's day burn,0.08,0.9,5,45,0.9437029361724854,0.8316463232040405,0.2760632634162903,-1,-1,-1,7116,26488,https://wifire-data.sdsc.edu/data/burnpro3d/d/d2/7d/run_d27d6ab2-b9ad-42d5-95b8-d191d88dc98a/png/run_d27d6ab2-b9ad-42d5-95b8-d191d88dc98a_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/d2/7d/run_d27d6ab2-b9ad-42d5-95b8-d191d88dc98a/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
cbfc1b23-239f-4c29-8dc8-77f8430a116c,Yosemite valentine's day burn,0.1,0.9,3,30,0.882082998752594,0.7743878364562988,0.22590337693691254,-1,-1,-1,7116,33863,https://wifire-data.sdsc.edu/data/burnpro3d/d/cb/fc/run_cbfc1b23-239f-4c29-8dc8-77f8430a116c/png/run_cbfc1b23-239f-4c29-8dc8-77f8430a116c_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/cb/fc/run_cbfc1b23-239f-4c29-8dc8-77f8430a116c/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
b864daaf-321b-4fdf-bd20-b09eebfc57a1,Yosemite valentine's day burn,0.12,0.9,5,60,0.9061393737792969,0.795659065246582,0.2641012668609619,-1,-1,-1,7116,33903,https://wifire-data.sdsc.edu/data/burnpro3d/d/b8/64/run_b864daaf-321b-4fdf-bd20-b09eebfc57a1/png/run_b864daaf-321b-4fdf-bd20-b09eebfc57a1_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/b8/64/run_b864daaf-321b-4fdf-bd20-b09eebfc57a1/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
9674ad06-cf9b-41e9-b338-6be7ccd1bc3b,Yosemite valentine's day burn,0.12,0.9,3,30,0.8347817659378052,0.7263342142105103,0.22154957056045532,-1,-1,-1,7116,34119,https://wifire-data.sdsc.edu/data/burnpro3d/d/96/74/run_9674ad06-cf9b-41e9-b338-6be7ccd1bc3b/png/run_9674ad06-cf9b-41e9-b338-6be7ccd1bc3b_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/96/74/run_9674ad06-cf9b-41e9-b338-6be7ccd1bc3b/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
f0dceead-7f65-45c9-bef6-6a619de114c3,Yosemite valentine's day burn,0.1,0.9,5,30,0.9168633818626404,0.8168119788169861,0.27730774879455566,-1,-1,-1,7116,34171,https://wifire-data.sdsc.edu/data/burnpro3d/d/f0/dc/run_f0dceead-7f65-45c9-bef6-6a619de114c3/png/run_f0dceead-7f65-45c9-bef6-6a619de114c3_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/f0/dc/run_f0dceead-7f65-45c9-bef6-6a619de114c3/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
a555b89a-ce2a-455a-b42b-92df90b5d11f,Yosemite valentine's day burn,0.12,0.9,7,30,0.9233599305152893,0.8224429488182068,0.30096957087516785,-1,-1,-1,7106,27057,https://wifire-data.sdsc.edu/data/burnpro3d/d/a5/55/run_a555b89a-ce2a-455a-b42b-92df90b5d11f/png/run_a555b89a-ce2a-455a-b42b-92df90b5d11f_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/a5/55/run_a555b89a-ce2a-455a-b42b-92df90b5d11f/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
4d22a945-2b69-4516-9b85-35127faab17e,Yosemite valentine's day burn,0.1,0.9,7,45,0.9538415670394897,0.8491777181625366,0.2990766167640686,-1,-1,-1,7116,16885,https://wifire-data.sdsc.edu/data/burnpro3d/d/4d/22/run_4d22a945-2b69-4516-9b85-35127faab17e/png/run_4d22a945-2b69-4516-9b85-35127faab17e_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/4d/22/run_4d22a945-2b69-4516-9b85-35127faab17e/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
1cbe99a7-91b0-4844-b689-54aeb31465ae,Yosemite valentine's day burn,0.08,0.9,5,30,0.9422920942306519,0.835054337978363,0.2832014560699463,-1,-1,-1,7116,34215,https://wifire-data.sdsc.edu/data/burnpro3d/d/1c/be/run_1cbe99a7-91b0-4844-b689-54aeb31465ae/png/run_1cbe99a7-91b0-4844-b689-54aeb31465ae_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/1c/be/run_1cbe99a7-91b0-4844-b689-54aeb31465ae/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
947820fe-0248-4c75-b919-d4f638bee427,Yosemite valentine's day burn,0.1,0.9,3,45,0.8786376118659973,0.7682275772094727,0.22101251780986786,-1,-1,-1,7116,34331,https://wifire-data.sdsc.edu/data/burnpro3d/d/94/78/run_947820fe-0248-4c75-b919-d4f638bee427/png/run_947820fe-0248-4c75-b919-d4f638bee427_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/94/78/run_947820fe-0248-4c75-b919-d4f638bee427/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
ae71abf7-f373-4b0c-8783-6470e79993e5,Yosemite valentine's day burn,0.12,0.9,3,45,0.8379149436950684,0.7212843894958496,0.21654847264289856,-1,-1,-1,7116,34550,https://wifire-data.sdsc.edu/data/burnpro3d/d/ae/71/run_ae71abf7-f373-4b0c-8783-6470e79993e5/png/run_ae71abf7-f373-4b0c-8783-6470e79993e5_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/ae/71/run_ae71abf7-f373-4b0c-8783-6470e79993e5/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
142a0e10-366d-4260-af90-b4d004d14f20,Yosemite valentine's day burn,0.08,0.9,7,45,0.9550843238830566,0.8489208817481995,0.299629271030426,-1,-1,-1,7116,26769,https://wifire-data.sdsc.edu/data/burnpro3d/d/14/2a/run_142a0e10-366d-4260-af90-b4d004d14f20/png/run_142a0e10-366d-4260-af90-b4d004d14f20_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/14/2a/run_142a0e10-366d-4260-af90-b4d004d14f20/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
33b3b976-a01e-4d4e-8ab8-679e6b5a3269,Yosemite valentine's day burn,0.08,0.9,7,30,0.9565696716308594,0.8489348888397217,0.3054349422454834,-1,-1,-1,7116,26870,https://wifire-data.sdsc.edu/data/burnpro3d/d/33/b3/run_33b3b976-a01e-4d4e-8ab8-679e6b5a3269/png/run_33b3b976-a01e-4d4e-8ab8-679e6b5a3269_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/33/b3/run_33b3b976-a01e-4d4e-8ab8-679e6b5a3269/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
3c198ece-d5e5-49ab-83c4-5ed9df384e82,Yosemite valentine's day burn,0.08,0.9,5,60,0.9494196772575378,0.837781548500061,0.27108460664749146,-1,-1,-1,7116,28218,https://wifire-data.sdsc.edu/data/burnpro3d/d/3c/19/run_3c198ece-d5e5-49ab-83c4-5ed9df384e82/png/run_3c198ece-d5e5-49ab-83c4-5ed9df384e82_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/3c/19/run_3c198ece-d5e5-49ab-83c4-5ed9df384e82/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
59f90dba-4839-40f1-9086-c8afc900fc14,Yosemite valentine's day burn,0.08,0.9,3,45,0.9095155000686646,0.795427143573761,0.22799168527126312,-1,-1,-1,7116,36761,https://wifire-data.sdsc.edu/data/burnpro3d/d/59/f9/run_59f90dba-4839-40f1-9086-c8afc900fc14/png/run_59f90dba-4839-40f1-9086-c8afc900fc14_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/59/f9/run_59f90dba-4839-40f1-9086-c8afc900fc14/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
82fb4e4a-d5e5-4dc7-a768-b1ebfcaf6ae1,Yosemite valentine's day burn,0.12,0.9,5,45,0.9058856964111328,0.7998331785202026,0.2666092813014984,-1,-1,-1,7116,27035,https://wifire-data.sdsc.edu/data/burnpro3d/d/82/fb/run_82fb4e4a-d5e5-4dc7-a768-b1ebfcaf6ae1/png/run_82fb4e4a-d5e5-4dc7-a768-b1ebfcaf6ae1_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/82/fb/run_82fb4e4a-d5e5-4dc7-a768-b1ebfcaf6ae1/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
5b5af879-ab09-4067-a50a-72ed590470e4,Yosemite valentine's day burn,0.1,0.9,5,45,0.9250667095184326,0.8165096640586853,0.2700793147087097,-1,-1,-1,7116,27161,https://wifire-data.sdsc.edu/data/burnpro3d/d/5b/5a/run_5b5af879-ab09-4067-a50a-72ed590470e4/png/run_5b5af879-ab09-4067-a50a-72ed590470e4_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/5b/5a/run_5b5af879-ab09-4067-a50a-72ed590470e4/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
a1071cbc-d668-45ee-8b84-eb55a5ddfe4c,Yosemite valentine's day burn,0.1,0.9,5,60,0.9303156137466431,0.8205060362815857,0.2682588994503021,-1,-1,-1,7116,27839,https://wifire-data.sdsc.edu/data/burnpro3d/d/a1/07/run_a1071cbc-d668-45ee-8b84-eb55a5ddfe4c/png/run_a1071cbc-d668-45ee-8b84-eb55a5ddfe4c_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/a1/07/run_a1071cbc-d668-45ee-8b84-eb55a5ddfe4c/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
bc4c3bc3-68a7-4440-9b07-6f79f4577420,Yosemite valentine's day burn,0.08,0.9,7,60,0.9706637263298035,0.8596436381340027,0.2899647653102875,-1,-1,-1,7116,29163,https://wifire-data.sdsc.edu/data/burnpro3d/d/bc/4c/run_bc4c3bc3-68a7-4440-9b07-6f79f4577420/png/run_bc4c3bc3-68a7-4440-9b07-6f79f4577420_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/bc/4c/run_bc4c3bc3-68a7-4440-9b07-6f79f4577420/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
fd946d56-a80d-4b14-82e5-526471666d90,Yosemite valentine's day burn,0.1,0.9,7,30,0.9405875205993652,0.8391811847686768,0.30287861824035645,-1,-1,-1,7116,20800,https://wifire-data.sdsc.edu/data/burnpro3d/d/fd/94/run_fd946d56-a80d-4b14-82e5-526471666d90/png/run_fd946d56-a80d-4b14-82e5-526471666d90_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/fd/94/run_fd946d56-a80d-4b14-82e5-526471666d90/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
b3f88e96-d5ff-4b0f-a4ff-3207e3a8125f,Yosemite valentine's day burn,0.12,0.9,5,30,0.8966860175132751,0.796311616897583,0.2770472764968872,-1,-1,-1,7116,43694,https://wifire-data.sdsc.edu/data/burnpro3d/d/b3/f8/run_b3f88e96-d5ff-4b0f-a4ff-3207e3a8125f/png/run_b3f88e96-d5ff-4b0f-a4ff-3207e3a8125f_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/b3/f8/run_b3f88e96-d5ff-4b0f-a4ff-3207e3a8125f/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
86957bbf-69fd-4ab6-89bc-dc8ac7ae9dec,Yosemite valentine's day burn,0.12,0.9,7,60,0.9344711899757385,0.8276606798171997,0.28594765067100525,-1,-1,-1,7116,21984,https://wifire-data.sdsc.edu/data/burnpro3d/d/86/95/run_86957bbf-69fd-4ab6-89bc-dc8ac7ae9dec/png/run_86957bbf-69fd-4ab6-89bc-dc8ac7ae9dec_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/86/95/run_86957bbf-69fd-4ab6-89bc-dc8ac7ae9dec/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
08ad49f3-9ae4-4226-8ec5-ee15ed563809,Yosemite valentine's day burn,0.12,0.9,7,45,0.9358673095703125,0.8323848247528076,0.29675960540771484,-1,-1,-1,7116,21993,https://wifire-data.sdsc.edu/data/burnpro3d/d/08/ad/run_08ad49f3-9ae4-4226-8ec5-ee15ed563809/png/run_08ad49f3-9ae4-4226-8ec5-ee15ed563809_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/08/ad/run_08ad49f3-9ae4-4226-8ec5-ee15ed563809/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
b666f7e3-01b3-4e9a-b546-cf7e9fecb0f9,Yosemite valentine's day burn,0.12,0.9,3,60,0.8292223215103149,0.716907799243927,0.21364882588386536,-1,-1,-1,7115,43930,https://wifire-data.sdsc.edu/data/burnpro3d/d/b6/66/run_b666f7e3-01b3-4e9a-b546-cf7e9fecb0f9/png/run_b666f7e3-01b3-4e9a-b546-cf7e9fecb0f9_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/b6/66/run_b666f7e3-01b3-4e9a-b546-cf7e9fecb0f9/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
3508e39e-f40c-4145-904a-b4c9e23284ec,Yosemite valentine's day burn,0.1,0.9,7,60,0.9536229372024536,0.8402692079544067,0.2832769453525543,-1,-1,-1,7116,22065,https://wifire-data.sdsc.edu/data/burnpro3d/d/35/08/run_3508e39e-f40c-4145-904a-b4c9e23284ec/png/run_3508e39e-f40c-4145-904a-b4c9e23284ec_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/35/08/run_3508e39e-f40c-4145-904a-b4c9e23284ec/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
762cf136-60da-4c6e-8c2b-a3a320ba9992,Yosemite valentine's day burn,0.1,0.9,3,60,0.8621399402618408,0.7489991784095764,0.22054122388362885,-1,-1,-1,7116,45320,https://wifire-data.sdsc.edu/data/burnpro3d/d/76/2c/run_762cf136-60da-4c6e-8c2b-a3a320ba9992/png/run_762cf136-60da-4c6e-8c2b-a3a320ba9992_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/76/2c/run_762cf136-60da-4c6e-8c2b-a3a320ba9992/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr
70d0a4f1-95d3-40df-89e6-4b5f50d07c10,Yosemite valentine's day burn,0.08,0.9,3,60,0.9052510857582092,0.7845945954322815,0.2249608039855957,-1,-1,-1,7116,46190,https://wifire-data.sdsc.edu/data/burnpro3d/d/70/d0/run_70d0a4f1-95d3-40df-89e6-4b5f50d07c10/png/run_70d0a4f1-95d3-40df-89e6-4b5f50d07c10_fuels-dens_7080_000.png,https://wifire-data.sdsc.edu/data/burnpro3d/d/70/d0/run_70d0a4f1-95d3-40df-89e6-4b5f50d07c10/quicfire.zarr,http://wifire-data.sdsc.edu/data/burnpro3d/d/49/7f/fuel_497fc5d4-9a4a-48c5-90f1-6fc50778f0b9/fastfuels.zarr

Summary¶

In this notebook we demonstrated how to use PelicanFS and gave an overview of a few different common usages. The main example showed how to use PelicanFS and Xarray to open a Zarr store. We also showed how to use PelicanFS and an intake catalog.

What’s next?¶

The following notebooks all demonstrate various workflows which will use PelicanFS to access data from the OSDF.

Introduction to OSDF

Streaming Data with the Open Science Data Federation (OSDF)

NCAR examples

Introduction to the Research Data Archive