Querying the NOAA NCEI Database¶

Authors¶

Deborah Khider , Dhiren Oswal

Preamble¶

In a previous chapter, you learned how to create a PangaeaDataset object and its many functionalities. We won’t be revisiting those here; just know that regardless of how many datasets are returned, the functions will apply.

This tutorial demonstrates how to query the PANGAEA database. A list of search parameters available in PyleoTUPS can be found here. Note that many PANGAEA search terms have been mapped to their NOAA NCEI equivalents for consistency.

• study_ids (int, str, or list, optional) – One or more PANGAEA dataset identifiers (numeric ID or DOI string). If provided, performs direct lookup and ignores other filters.

• topic (str, optional) – Filter datasets by PANGAEA topic classification. Must be one of the predefined topics: - “all” (default) - “Agriculture” - “Atmosphere” - “Biological Classification” - “Biosphere” - “Chemistry” - “Cryosphere” - “Ecology” - “Fisheries” - “Geophysics” - “Human Dimensions” - “Lakes & Rivers” - “Land Surface” - “Lithosphere” - “Oceans” - “Paleontology” If set to “all” or omitted, no topic filtering is applied.

• search_text (str, optional) – Free-text search query applied across dataset metadata. Maps to PANGAEA full-text search parameter ‘q’. Example: ‘stable carbon and oxygen isotopes’.

• investigators (str or list[str], optional) – Author names. Mapped internally to PANGAEA query syntax: author:

• variable_name (str or list[str], optional) – Name of parameters/variables (columns) present in dataset tables. Internally mapped to PANGAEA query term: parameter:<variable_name>

• min_lat (float, optional) – Latitude bounds (–90..90).

• max_lat (float, optional) – Latitude bounds (–90..90).

• min_lon (float, optional) – Longitude bounds (–180..180)

• max_lon (float, optional) – Longitude bounds (–180..180)

• limit (int, default 100, maximum 500) – Maximum number of results returned.

• skip (int, default 0) – Number of results to skip (pagination). Maps to PANGAEA ‘offset’

A list of available search term is available here.

Goals¶

• Perform query on PANGAEA database using search parameters.

• Extend query using the search_text parameter and PANGAEA’s fields

• Dealing with Collections

Prerequisite¶

• Understanding of PANGAEA datasets and associated search API

• PyleoTUPS PANGAEADataset object

Reading time¶

15 min

Let’s import our packages!

Study Query¶

In the notebook introducing the PangaeaDataset object, we introduced the concept of searching by study_ids. If you are familiar with the pangaeapy package, this is one of the easiest way to get data from the database.

These IDs correspond to the last six digits of the DOI PANGAEA mints for each of this dataset. Let’s use the study by van der Bilt et al. (2016), whose data are available here. The DOI is 10.1594/PANGAEA.868935, resuting in the following search:

You may have noticed that the parameter name is in a plural form. Yes, this means that you can query data from multiple datasets at the same time.

Let’s get the second dataset in the collection corresponding to the van der Bilt et al. (2016) study.

Investigator query¶

Let’s search for datasets with a specific investigator.

The query returned 31 datasets. Note that PyleoTUPS warns us that some of these are collections. As discussed in the PangaeaObject tutorial, collections group datasets that belong to the same study. Collections are necessary in PANGAEA because each data table is its own dataset with a unique DOI. Consider a study in which two sediment cores are analyzed for Mg/Ca and $\delta^{18}O$. he data for both cores might appear in the same table under different Event labels, but if the data differ substantially, they may be reported in separate tables, and therefore separate datasets.

This is one of the key differences between the two repositories. NOAA NCEI groups data by study, allowing a one-to-many relationship between a study and its data tables; a query returns all tables in the study. PANGAEA, by contrast, creates an individual dataset for each table (a one-to-one relationship), with collections serving as the grouping mechanism.

Also notice the string returned by PyleoTUPS under the q key. As mentioned in the preamble, PANGAEA searches are string-based, with certain keys (such as author) that can be passed to the search. The q value is the actual query string that PyleoTUPS constructed from your search parameters.

Let’s have a look at multiple authors search:

We have hit the limit! Like we have seen with NOAA queries, PyleoTUPS is conducting an OR search by defaults. So it is looking for all the studies authored by either one of the investigators.

To do an AND query:

Geographical query¶

Let’s perform the same query as we did previously on NOAA NCEI for all the datasets within 5°S-5°N and 109-125°E, roughly corresponding to the Indo-Pacific Warm Pool. For speed, we will limit the search to the first 50 datasets.

Variable queries¶

Let’s search using variable names. In PANGAEA, this concept is closely related to Parameter and this is the reason why we mapped this concept to our key variable_name. But Parameter also include units. For a complete list of available parameters in PANGAEA, see this wiki page.

As you can see on the sample above, many studies on PANGAEA will not be paleoclimate related. When doing a variable search, it is strongly advised to narrow the topics if possible:

Text search¶

It is possible to further refine queries by using the open search_text parameter. For instance, let’s try to look for Holocene records:

As long as the word “Holocene” appears anywhere in the metadata (dataset name, study name, or description), the search will return those datasets. However, this does not mean that other datasets covering the Holocene are absent from PANGAEA — a study described as spanning “the past 15,000 years” would not be matched. Because age bounds are not part of PANGAEA’s standard metadata, time-based searches are severely limited.

Dealing with Collections¶

Collections link Datasets together. They can be very powerful, but also difficult to deal with when interested in the data only.

Let’s do a search query for the data is supplement to van der Bilt et al. (2016). To do so we will be using:

• investigators parameter in search_studies

• search_text to narrow down the publication year using the PANGAEA field citation:year: as seen here:

As you can see, PyleoTUPS created the string query q from both the exposed parameter investigators and the PANGAEA field citation:year:.

Let’s have a look at the returned datasets:

As noted earlier, the first dataset is actually a collection. If you scroll to the last column, you’ll see a list of its collection members. One way to filter out collections is to remove all entries where collectionMembers is not NaN, then recreate a PangaeaDataset object:

Summary¶

In this tutorial, you learned how to use the search_studies function with the PANGAEA database. A few things to keep in mind:

1. PANGAEA is more general-purpose than NOAA NCEI for paleo. While this means instrumental datasets may also appear in results, search parameters common in paleoclimate queries (e.g., age) are unavailable on PANGAEA. Additionally, PANGAEA lacks the rich controlled vocabulary of NOAA NCEI, which can make broad searches more difficult.

2. Some datasets require login credentials to access.

3. Collections are returned alongside individual datasets, so care is needed when linking them back together.