Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Contextual Integration of NCAR/NOAA Environmental Data with the NOAA Water Column Sonar Archive

Contextual Integration of NCAR/NOAA Environmental Data with the NOAA Water Column Sonar Archive

This notebook walks through an end-to-end workflow to relate shipboard sonar backscatter (Sv) to local environmental conditions. We (1) open EK60 data from a public NOAA S3 Zarr, (2) gather co-located environmental variables from OISST and IOOS ERDDAP, (3) compute hourly mean Sv, (4) assemble a depth×time error map for reference, and (5) synchronize timestamps to produce an interactive line-plus-heatmap visualization. All selections (time/depth/frequency) and conversions are kept explicit for reproducibility.

  1. Imports Load core libraries for data access (xarray, s3fs), analysis (numpy, pandas), plotting (plotly), and I/O.

  2. Initializing the datasets Access HB1906 EK60 Zarr data from public S3; subset by time/depth, select 38 kHz, and mask bins below bottom.

  3. Access buoy data Define Georges Bank buoy coordinates, sample daily OISST SST at the nearest grid cell (±1 day), and download the model error map (.npy).

  4. Calculate the temperature anomaly, sun elevation in degree and azimuth

  5. Downloading external error map for the specific location Downloading the error map comes from a fixed file

  6. Helper Function: Mean Sv (dB) Convert Sv from dB→linear, compute mean, convert back to dB.

  7. Group Cruise Data into Hourly Chunks Add an hourly label and split the EK60 dataset into per-hour xarray.Dataset chunks.

  8. Compute Hourly Mean Sv & Attach to env_df Aggregate Sv per hour and append results as a new column in the environmental dataframe.

  9. Build Depth×Time Error-Map DataFrame & Align Timestamps Construct a depth-by-time matrix from the error map, guard for size mismatches, and align env_df endpoints to the heatmap timestamps.

  10. Data Visualization: Synchronized Lines + Heatmap Plot environmental time series above a depth×time heatmap with shared x-axis; save interactive HTML output.


Prerequisites

This section was inspired by this template of the wonderful The Turing Way Jupyter Book.

This notebook opens public NOAA EK60 Zarr data from S3, subsets by time/depth, reads daily OISST SST near a buoy, fetches ERDDAP environmental variables, computes hourly mean Sv (dB), aligns with a depth×time error map, and renders synchronized line/heatmap plots.

Label the importance of each concept explicitly as helpful/necessary.

ConceptsImportanceNotes
Xarray + Zarr basicsNecessaryOpening Zarr stores, selecting by coords/dims, .compute() semantics
s3fs & public S3 accessNecessaryAnonymous reads from AWS S3 (anon=True)
Pandas time seriesNecessaryDatetimeIndex, sorting, filtering, timezone-naive vs. aware
NumPy fundamentalsNecessaryArray slicing, stats, type conversion
Acoustic backscatter (Sv) & dB averagingNecessaryConvert dB→linear, mean, then linear→dB, Understanding results
ERDDAP tabledap & info endpointsHelpfulReading CSV responses; unit metadata lookup
Plotly fundamentalsHelpfulSubplots, heatmaps, interactive HTML export
Understanding of NetCDF/CFHelpfulVariable metadata and geospatial conventions
Dask awarenessHelpfulLazy arrays; when/why to call .compute()
Geographic coordinatesHelpful0–360 vs. −180–180 longitude handling
HTTP/IO with requestsHelpfulDownloading .npy assets for local use
  • Time to learn: ~75 minutes

  • System requirements:

    • Python 3.9+ with Jupyter Notebook/Lab

    • Required packages: xarray, s3fs, numpy, pandas, plotly, requests, netCDF4 (optional but helpful: dask)

Note: Run the cell below only in a local environment to install the required packages. If you’re using Binder, skip this step.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[1], line 34
     30 
     31 # Install Conda packages
     32 if conda_pkgs:
     33     if not conda_exe:
---> 34         raise RuntimeError("Conda/mamba not found in PATH. Run this inside a Conda environment.")
     35     # Install into the current environment prefix
     36     env_prefix = os.environ.get("CONDA_PREFIX", sys.prefix)
     37     cmd = [conda_exe, "install", "-y", "-p", env_prefix]

RuntimeError: Conda/mamba not found in PATH. Run this inside a Conda environment.

1) Imports

Core libraries used throughout the notebook.

Key roles:

xarray/s3fs for reading NOAA Zarr data from S3
numpy/pandas for arrays & tables
plotly for interactive plotting
requests/io/os for file I/O and downloads
datetime for time calculations

2) Initializing the datasets

Builds the S3 path to the HB1906 EK60 Zarr dataset and opens it anonymously. Subsets by time window and depth range, selects the 38 kHz channel, and masks samples below the estimated bottom. .compute() materializes the selection; hm_timestamps will be reused for time alignment later. All datasets are accessed using the OSDF infrastructure

3) Accessing buoy data

Defines a buoy location on Georges Bank (longitude converted to 0–360).

  1. Optional: Loads three daily OISST files and samples SST at the nearest grid point (day before, day of, day after).

  2. ERDDAP buoy environmental data.

Sets ERDDAP dataset parameters and enforces a max_days cap by adjusting end_date_time if needed. Reads station metadata to extract lon/lat and wind-speed units; prepares a conversion to knots. Pulls a table of time, wind_speed, SST, significant wave height, converts wind speed to knots, indexes by time. Filters to the requested window and keeps the first nine rows (intentional truncation for later alignment).

4) Calculate the temperature anomaly, sun elevation in degree and azimuth

Extracts World Ocean Atlas 2023 temperature data for a specific location and month and calculates temperature anomaly (optional), sun elevation in degree and azimuth (optional).

5) Downloading external error map for the specific location.

Currently the error map comes from a fixed file; our plan is to switch to a dynamic AWS download that accepts location parameters.


6) Helper: mean Sv in dB

Computes the mean of Sv correctly by converting dB → linear, averaging, then linear → dB. Accepts array-like input (NumPy/xarray/dask); returns a scalar in dB.

7) Group cruise data into hourly chunks

Adds an hourly label and groups the cruise data by hour. Produces a list of per-hour xarray.Dataset chunks for downstream aggregation.

8) Compute hourly mean Sv and attach to env_df

Iterates over hourly chunks, computes mean Sv per hour using calculate_sv_mean. Converts dask→NumPy→Python float and appends to a list. Assigns the resulting hourly series to env_df[“sv_hourly”]. Assumes the number/order of hours matches rows retained in env_df.

9) Build (depth × time) error-map DataFrame and align timestamps

Extracts one channel from sonar_clusters and pairs it with cruise depths and timestamps to form a DataFrame. Uses min(...) to guard against size mismatches in depth/time dimensions. Aligns only the first and last timestamps in env_df to the heatmap’s time range (keeps interior indices unchanged, sets UTC).

10) Data Visualization: Synchronized Lines + Heatmap

Plots synchronized data: top = time series from line_df; bottom = depth×time heatmap from heatmap_df. Expects line_df to have a DatetimeIndex (timezone-naive or converted). Depth axis is reversed (surface at top). Saves an interactive HTML file to the parent directory (correlations.html) and shows the figure if show=True.

Because rendering the plot is computationally intensive and involves downloading approximately 1 GB of data, we present a static image of the result instead.