OSDF Cookbook

This Project Pythia Cookbook covers examples that demonstrate data access via the Open Science Data Federation.

Motivation¶

The Open Science Data Federation (OSDF) is a service designed to support the sharing of files staged in autonomous data ``origins”. The OSDF provides a way to efficiently access those files from anywhere in the world via a global namespace and network of caches. In particular, the Pelican platform helps users access climate data from a vast variety of datasets and transfer it to computational platforms like the OSPool. In this cookbook, we describe how to use PelicanFS, which is a file system interface (fsspec) for the Pelican Platform package to access data.

Authors¶

Harsha Hampapura, Second Author

Contributors¶

Structure¶

This cookbook is broken up into two main sections - “Foundations” and “Example Workflows.”

Section 1 ( Foundations)¶

The foundational content includes an introduction to the Open Science Data Federation, its data origins and PelicanFS, which is a file system interface (fsspec) for the Pelican Platform.

Section 2 ( Example workflows )¶

Example workflows include

• i) Accessing CESM2 LENS data from the AWS open data origin and the
• ii) NCAR data origin
• iii) Accessing NOAA SONAR data from the AWS data origin

Running the Notebooks¶

You can either run the notebook using Binder or on your local machine.

Running on Binder¶

The simplest way to interact with a Jupyter Notebook is through Binder, which enables the execution of a Jupyter Book in the cloud. The details of how this works are not important for now. All you need to know is how to launch a Pythia Cookbooks chapter via Binder. Simply navigate your mouse to the top right corner of the book chapter you are viewing and click on the rocket ship icon, (see figure below), and be sure to select “launch Binder”. After a moment you should be presented with a notebook that you can interact with. I.e. you’ll be able to execute and even change the example programs. You’ll see that the code cells have no output at first, until you execute them by pressing Shift+Enter. Complete details on how to interact with a live Jupyter notebook are described in Getting Started with Jupyter.

Running on Your Own Machine¶

If you are interested in running this material locally on your computer, you will need to follow this workflow:

1. Clone the https://github.com/ProjectPythia/osdf-cookbook repository:

    git clone https://github.com/ProjectPythia/osdf-cookbook.git

2. Move into the osdf-cookbook directory

   cd osdf-cookbook

3. Create and activate your conda environment from the environment.yml file

   conda env create -f environment.yml
   conda activate osdf-cookbook

4. Move into the notebooks directory and start up Jupyterlab

   cd notebooks/
   jupyter lab