Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Investigating interhemispheric precipitation changes over the past millennium

Project Pythia Logo LinkedEarth Logo

Investigating interhemispheric precipitation changes over the past millennium


Overview

This CookBook demonstrates how to compare paleoclimate model output and proxy observations using EOF to identify large-scale spatio-temporal patterns in the data. It is inspired from a study by Steinman et al. (2022) although it reuses different datasets.

Prerequisites

ConceptsImportanceNotes
Intro to CartopyNecessary
Intro to MatplotlibNecessary
Intro to PandasNecessary
Understanding of NetCDFNecessary
Using xarrayNecessaryFamiliarity with understanding opening multiple files and merging
EOF (PCA) Analysis - See Chapter 12 of this bookHelpfulFamiliarity with the concepts is helpful for interpretation of the results
eofs packageHelpfulA good introduction on the package can be found in this notebook
Using Pyleoclim for Paleoclimate DataHelpful
SPARQLFamiliarityQuery language for graph database
  • Time to learn: 40 min.


Imports

PCA analysis on proxy observations

After looking at the model data, let’s have a look at the proxy datasets.

Query to remote database

The first step is the query the LinkedEarth Graph Database for relevant datasets for comparison with the model.

The database uses the SPARQL language for queries. We are filtering the database for the following criteria:

  • Datasets bounded by 27°S-27°N and 70°W-150°W

  • Datasets from the Pages2k, Iso2k, CoralHydro2k and SISAL working groups. These working groups identified archived datasets that were sensitive to temperature and the isotopic composition of precipication (precipitation δ18\delta^{18}O) and curated them for use in a standardized database.

  • Timeseries within these datasets representing precipitation.

We asked for the following information back:

  • The name of the dataset

  • Geographical Location of the record expressed in latitude and longitude

  • The type of archive (e.g., speleothem, Lake sediment) the measurements were made on

  • The name of the variable

  • The values and units of the measurements

  • The time information (values and units) associated with the variable of interest.

The following cell points to the query API and creates the query itself.

The following cell sends the query to the database and returns the results in a Pandas Dataframe.

---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/urllib3/connection.py:239, in HTTPConnection._new_conn(self)
    238 try:
--> 239     sock = connection.create_connection(
    240         (self._dns_host, self.port),
    241         self.timeout,
    242         source_address=self.source_address,
    243         socket_options=self.socket_options,
    244     )
    245 except socket.gaierror as e:

File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/urllib3/util/connection.py:85, in create_connection(address, timeout, source_address, socket_options)
     84 try:
---> 85     raise err
     86 finally:
     87     # Break explicitly a reference cycle

File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/urllib3/util/connection.py:73, in create_connection(address, timeout, source_address, socket_options)
     72     sock.bind(source_address)
---> 73 sock.connect(sa)
     74 # Break explicitly a reference cycle

TimeoutError: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

ConnectTimeoutError                       Traceback (most recent call last)
File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/urllib3/connectionpool.py:793, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
    792 # Make the request on the HTTPConnection object
--> 793 response = self._make_request(
    794     conn,
    795     method,
    796     url,
    797     timeout=timeout_obj,
    798     body=body,
    799     headers=headers,
    800     chunked=chunked,
    801     retries=retries,
    802     response_conn=response_conn,
    803     preload_content=preload_content,
    804     decode_content=decode_content,
    805     **response_kw,
    806 )
    808 # Everything went great!

File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/urllib3/connectionpool.py:494, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
    493         new_e = _wrap_proxy_error(new_e, conn.proxy.scheme)
--> 494     raise new_e
    496 # conn.request() calls http.client.*.request, not the method in
    497 # urllib3.request. It also calls makefile (recv) on the socket.

File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/urllib3/connectionpool.py:470, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
    469 try:
--> 470     self._validate_conn(conn)
    471 except (SocketTimeout, BaseSSLError) as e:

File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/urllib3/connectionpool.py:1125, in HTTPSConnectionPool._validate_conn(self, conn)
   1124 if conn.is_closed:
-> 1125     conn.connect()
   1127 # TODO revise this, see https://github.com/urllib3/urllib3/issues/2791

File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/urllib3/connection.py:827, in HTTPSConnection.connect(self)
    826 sock: socket.socket | ssl.SSLSocket
--> 827 self.sock = sock = self._new_conn()
    828 server_hostname: str = self.host

File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/urllib3/connection.py:248, in HTTPConnection._new_conn(self)
    247 except SocketTimeout as e:
--> 248     raise ConnectTimeoutError(
    249         self,
    250         f"Connection to {self.host} timed out. (connect timeout={self.timeout})",
    251     ) from e
    253 except OSError as e:

ConnectTimeoutError: (<HTTPSConnection(host='linkedearth.graphdb.mint.isi.edu', port=443) at 0x79108bb27680>, 'Connection to linkedearth.graphdb.mint.isi.edu timed out. (connect timeout=None)')

The above exception was the direct cause of the following exception:

MaxRetryError                             Traceback (most recent call last)
File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/requests/adapters.py:696, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    695 try:
--> 696     resp = conn.urlopen(
    697         method=request.method,
    698         url=url,
    699         body=request.body,  # type: ignore[arg-type]  # urllib3 stubs don't accept Iterable[bytes | str]
    700         headers=request.headers,  # type: ignore[arg-type]  # urllib3#3072
    701         redirect=False,
    702         assert_same_host=False,
    703         preload_content=False,
    704         decode_content=False,
    705         retries=self.max_retries,
    706         timeout=resolved_timeout,
    707         chunked=chunked,
    708     )
    710 except (ProtocolError, OSError) as err:

File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/urllib3/connectionpool.py:847, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
    845     new_e = ProtocolError("Connection aborted.", new_e)
--> 847 retries = retries.increment(
    848     method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
    849 )
    850 retries.sleep()

File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/urllib3/util/retry.py:555, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
    554     reason = error or ResponseError(cause)
--> 555     raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    557 log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)

MaxRetryError: HTTPSConnectionPool(host='linkedearth.graphdb.mint.isi.edu', port=443): Max retries exceeded with url: /repositories/LiPDVerse-dynamic (Caused by ConnectTimeoutError(<HTTPSConnection(host='linkedearth.graphdb.mint.isi.edu', port=443) at 0x79108bb27680>, 'Connection to linkedearth.graphdb.mint.isi.edu timed out. (connect timeout=None)'))

During handling of the above exception, another exception occurred:

ConnectTimeout                            Traceback (most recent call last)
Cell In[3], line 1
----> 1 response = requests.post(url, data = {'query': query})
      2 
      3 data = io.StringIO(response.text)
      4 df_res = pd.read_csv(data, sep=",")

File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/requests/api.py:134, in post(url, data, json, **kwargs)
    117 def post(
    118     url: _t.UriType,
    119     data: _t.DataType = None,
    120     json: _t.JsonType = None,
    121     **kwargs: Unpack[_t.PostKwargs],
    122 ) -> Response:
    123     r"""Sends a POST request.
    124 
    125     :param url: URL for the new :class:`Request` object.
   (...)    131     :rtype: requests.Response
    132     """
--> 134     return request("post", url, data=data, json=json, **kwargs)

File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/requests/api.py:71, in request(method, url, **kwargs)
     67 # By using the 'with' statement we are sure the session is closed, thus we
     68 # avoid leaving sockets open which can trigger a ResourceWarning in some
     69 # cases, and look like a memory leak in others.
     70 with sessions.Session() as session:
---> 71     return session.request(method=method, url=url, **kwargs)

File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/requests/sessions.py:651, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    646 send_kwargs = {
    647     "timeout": timeout,
    648     "allow_redirects": allow_redirects,
    649 }
    650 send_kwargs.update(settings)
--> 651 resp = self.send(prep, **send_kwargs)
    653 return resp

File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/requests/sessions.py:784, in Session.send(self, request, **kwargs)
    781 start = preferred_clock()
    783 # Send the request
--> 784 r = adapter.send(request, **kwargs)
    786 # Total elapsed time of the request (approximately)
    787 elapsed = preferred_clock() - start

File ~/micromamba/envs/paleoPCA-Cookbook/lib/python3.12/site-packages/requests/adapters.py:717, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    714 if isinstance(e.reason, ConnectTimeoutError):
    715     # TODO: Remove this in 3.0.0: see #2811
    716     if not isinstance(e.reason, NewConnectionError):
--> 717         raise ConnectTimeout(e, request=request)
    719 if isinstance(e.reason, ResponseError):
    720     raise RetryError(e, request=request)

ConnectTimeout: HTTPSConnectionPool(host='linkedearth.graphdb.mint.isi.edu', port=443): Max retries exceeded with url: /repositories/LiPDVerse-dynamic (Caused by ConnectTimeoutError(<HTTPSConnection(host='linkedearth.graphdb.mint.isi.edu', port=443) at 0x79108bb27680>, 'Connection to linkedearth.graphdb.mint.isi.edu timed out. (connect timeout=None)'))

We have retrieved the following number of proxy records:

Make sure we have unique timeseries (some may be found across compilations):

The first step is to make sure that everything is on the same representation of the time axis. Year is considered prograde while age is considered retrograde:

Since we have records expressed in both year and age, let’s convert everything to year. First let’s have a look at the units:

The units for age are expressed in BP (before present), if we assume the present to be 1950 by convention, then we can transform:

It is obvious that some of the timeseries do not have correct time information (e.g., row 2). Let’s filter the dataframe to make sure that the time values are within 0-2000 and that there is at least 1500 years of record:

We are now left with:

Let’s also make sure that the records are long enough (i.e., more than 1500 years long):

This leaves us with the following number of datasets:

Let’s filter for records with a resolution finer or equal to 60 years:

This leaves us with the following number of datasets:

Next, let’s use the Pyleoclim software package and create individual GeoSeries objects:

Now let’s use a MultipleGeoSeries object for visualization and analysis:

Let’s first map the location of the records by the type of archive:

Let’s have a look at the records, sliced for the 0-2000 period

Run PCA Analysis

Let’s place them on a common time axis for analysis and standardize:

Let’s have a look at the screeplot:

As is nearly always the case with geophysical timeseries, the first few of eigenvalues trully overwhelm the rest. In this case, let’s have a look at the first three.

Let’s have a look at the second mode:

Finally, let’s have a look at the third mode:

As you can see, the first three modes explain 60% of the variance.

PCA analysis on the CESM Last Millennium Ensemble

For comparison with the proxy record, we will be using output from the CESM Last Millennium Ensemble.

The first step is the calculate precipitation δ18O\delta^{18}O for the all forcings simulation. The following section demonstrates how to get the data from JetStream2 and pre-process each file to save the needed variable and place them into a new xarray.Dataset. This process can be time-consuming.

Get the CESM Last Millennium Ensemble data from JetStream2 and select the tropical Central and South America region

Let’s open the needed files for this analysis, which consists of various precipitation isotopes. The data is also sliced for the relevant region bounded by 27°S-27°N and 70°W-150°W. All data have been made available on NSF JetStream2. In case JetSteam2 is not available we have made the relevant, pre-calculated subset needed for this work available on Figshare.

Warning: Note that this cell may take some time to run! On a 2024 MacBook pro with an M3 chip, it took about 7minutes.

For model-data comparison, it might be useful to resample the model data on the proxy scales:

And resample to the proxy resolution (~20yr as calculated above):

Calculate Precipitation δ18O\delta^{18}O

Run PCA analysis

Let’s first standardize the data:

Create an EOF solver to do the EOF analysis.

Retrieve the leading EOF, expressed as the covariance between the leading PC time series and the input d18O anomalies at each grid point.

Plot the leading EOF expressed covariance

Let’s have a look at the first three PCs:

Model-Data Comparison

Let’s map the first EOF for the model and data and the first PC.


Summary

In this Cookbook, you learned to run PCA analysis on gridded and point datasets to allow for model-data comparison. Although this was focued on the paleoclimate domain, the technique is broadly applicable to the instrumental era.

Resources and references

Model Output

  • CESM LME: Otto-Bliesner, B.L., E.C. Brady, J. Fasullo, A. Jahn, L. Landrum, S. Stevenson, N. Rosenbloom, A. Mai, G. Strand. Climate Variability and Change since 850 C.E. : An Ensemble Approach with the Community Earth System Model (CESM), Bulletin of the American Meteorological Society, 735-754 (May 2016 issue)

Proxy Compilations

  • Iso2k: Konecky, B. L., McKay, N. P., Churakova (Sidorova), O. V., Comas-Bru, L., Dassié, E. P., DeLong, K. L., Falster, G. M., Fischer, M. J., Jones, M. D., Jonkers, L., Kaufman, D. S., Leduc, G., Managave, S. R., Martrat, B., Opel, T., Orsi, A. J., Partin, J. W., Sayani, H. R., Thomas, E. K., Thompson, D. M., Tyler, J. J., Abram, N. J., Atwood, A. R., Cartapanis, O., Conroy, J. L., Curran, M. A., Dee, S. G., Deininger, M., Divine, D. V., Kern, Z., Porter, T. J., Stevenson, S. L., von Gunten, L., and Iso2k Project Members: The Iso2k database: a global compilation of paleo-δ18O and δ2H records to aid understanding of Common Era climate, Earth Syst. Sci. Data, 12, 2261–2288, Konecky et al. (2020), 2020.

  • PAGES2kTemperature: PAGES2k Consortium. A global multiproxy database for temperature reconstructions of the Common Era. Sci. Data 4:170088 doi: 10.1038/sdata.2017.88 (2017).

  • CoralHydro2k: Walter, R. M., Sayani, H. R., Felis, T., Cobb, K. M., Abram, N. J., Arzey, A. K., Atwood, A. R., Brenner, L. D., Dassié, É. P., DeLong, K. L., Ellis, B., Emile-Geay, J., Fischer, M. J., Goodkin, N. F., Hargreaves, J. A., Kilbourne, K. H., Krawczyk, H., McKay, N. P., Moore, A. L., Murty, S. A., Ong, M. R., Ramos, R. D., Reed, E. V., Samanta, D., Sanchez, S. C., Zinke, J., and the PAGES CoralHydro2k Project Members: The CoralHydro2k database: a global, actively curated compilation of coral δ18O and Sr ∕ Ca proxy records of tropical ocean hydrology and temperature for the Common Era, Earth Syst. Sci. Data, 15, 2081–2116, Walter et al. (2023), 2023.

  • SISAL: Comas-Bru, L., Rehfeld, K., Roesch, C., Amirnezhad-Mozhdehi, S., Harrison, S. P., Atsawawaranunt, K., Ahmad, S. M., Brahim, Y. A., Baker, A., Bosomworth, M., Breitenbach, S. F. M., Burstyn, Y., Columbu, A., Deininger, M., Demény, A., Dixon, B., Fohlmeister, J., Hatvani, I. G., Hu, J., Kaushal, N., Kern, Z., Labuhn, I., Lechleitner, F. A., Lorrey, A., Martrat, B., Novello, V. F., Oster, J., Pérez-Mejías, C., Scholz, D., Scroxton, N., Sinha, N., Ward, B. M., Warken, S., Zhang, H., and SISAL Working Group members: SISALv2: a comprehensive speleothem isotope database with multiple age–depth models, Earth Syst. Sci. Data, 12, 2579–2606, Comas-Bru et al. (2020), 2020.

Software

Khider, D., Emile-Geay, J., Zhu, F., James, A., Landers, J., Ratnakar, V., & Gil, Y. (2022). Pyleoclim: Paleoclimate timeseries analysis and visualization with Python. Paleoceanography and Paleoclimatology, 37, e2022PA004509. Khider et al. (2022)

Khider, D., Emile-Geay, J., Zhu, F., James, A., Landers, J., Kwan, M., Athreya, P., McGibbon, R., & Voirol, L. (2024). Pyleoclim: A Python package for the analysis and visualization of paleoclimate data (Version v1.0.0) [Computer software]. Khider et al. (2026)

  • eofs: Dawson, A. (2016) ‘eofs: A Library for EOF Analysis of Meteorological, Oceanographic, and Climate Data’, Journal of Open Research Software, 4(1), p. e14. Available at: Dawson (2016).

References
  1. Steinman, B. A., Stansell, N. D., Mann, M. E., Cooke, C. A., Abbott, M. B., Vuille, M., Bird, B. W., Lachniet, M. S., & Fernandez, A. (2022). Interhemispheric antiphasing of neotropical precipitation during the past millennium. Proceedings of the National Academy of Sciences, 119(17). 10.1073/pnas.2120015119
  2. Konecky, B. L., McKay, N. P., Churakova (Sidorova), O. V., Comas-Bru, L., Dassié, E. P., DeLong, K. L., Falster, G. M., Fischer, M. J., Jones, M. D., Jonkers, L., Kaufman, D. S., Leduc, G., Managave, S. R., Martrat, B., Opel, T., Orsi, A. J., Partin, J. W., Sayani, H. R., Thomas, E. K., … von Gunten, L. (2020). The Iso2k database: a global compilation of paleo- δ 18 O and δ 2 H records to aid understanding of Common Era climate. Earth System Science Data, 12(3), 2261–2288. 10.5194/essd-12-2261-2020
  3. Emile-Geay, J., McKay, N. P., Kaufman, D. S., von Gunten, L., Wang, J., Anchukaitis, K. J., Abram, N. J., Addison, J. A., Curran, M. A. J., Evans, M. N., Henley, B. J., Hao, Z., Martrat, B., McGregor, H. V., Neukom, R., Pederson, G. T., Stenni, B., Thirumalai, K., Werner, J. P., … Zinke, J. (2017). A global multiproxy database for temperature reconstructions of the Common Era. Scientific Data, 4(1). 10.1038/sdata.2017.88
  4. Walter, R. M., Sayani, H. R., Felis, T., Cobb, K. M., Abram, N. J., Arzey, A. K., Atwood, A. R., Brenner, L. D., Dassié, É. P., DeLong, K. L., Ellis, B., Emile-Geay, J., Fischer, M. J., Goodkin, N. F., Hargreaves, J. A., Kilbourne, K. H., Krawczyk, H., McKay, N. P., Moore, A. L., … Zinke, J. (2023). The CoralHydro2k database: a global, actively curated compilation of coral δ 18 O and Sr ∕ Ca proxy records of tropical ocean hydrology and temperature for the Common Era. Earth System Science Data, 15(5), 2081–2116. 10.5194/essd-15-2081-2023
  5. Comas-Bru, L., Rehfeld, K., Roesch, C., Amirnezhad-Mozhdehi, S., Harrison, S. P., Atsawawaranunt, K., Ahmad, S. M., Brahim, Y. A., Baker, A., Bosomworth, M., Breitenbach, S. F. M., Burstyn, Y., Columbu, A., Deininger, M., Demény, A., Dixon, B., Fohlmeister, J., Hatvani, I. G., Hu, J., … Zhang, H. (2020). SISALv2: a comprehensive speleothem isotope database with multiple age–depth models. Earth System Science Data, 12(4), 2579–2606. 10.5194/essd-12-2579-2020
  6. Hoyer, S., & Hamman, J. (2017). xarray: N-D labeled Arrays and Datasets in Python. Journal of Open Research Software, 5(1), 10. 10.5334/jors.148
  7. Khider, D., Emile‐Geay, J., Zhu, F., James, A., Landers, J., Ratnakar, V., & Gil, Y. (2022). Pyleoclim: Paleoclimate Timeseries Analysis and Visualization With Python. Paleoceanography and Paleoclimatology, 37(10). 10.1029/2022pa004509
  8. Khider, D., Emile-Geay, J., Zhu, F., James, A., Landers, J., Kwan, M., Athreya, P., McGibbon, R., Voirol, L., & Stiller, D. (2026). Pyleoclim: A Python package for the analysis and visualization of paleoclimate data. Zenodo. 10.5281/ZENODO.1205661
  9. Dawson, A. (2016). eofs: A Library for EOF Analysis of Meteorological, Oceanographic, and Climate Data. Journal of Open Research Software, 4(1), 14. 10.5334/jors.122