Retrieving textual information from LiPD files

Retrieving textual information from LiPD files#

Authors#

Deborah Khider

Preamble#

PyLiPD is a Python package that allows you to read, manipulate, and write LiPD formatted datasets. In this tutorial, we will demonstrate how you can use pre-defined APIs that allows getting specific information from a LiPD file.

Goals#

  • Use existing APIs to get information about the datasets loaded in the workspace, their location, the variables available, the types of geologic archives.

  • Obtain a BibTeX file of references to properly credit scholarly contributions

Reading Time: 5 minutes

Keywords#

LiPD

Pre-requisites#

None. This tutorial assumes basic knowledge of Python and Pandas. If you are not familiar with this coding language and this particular library, check out this tutorial: http://linked.earth/ec_workshops_py/.

Relevant Packages#

pylipd

Data Description#

This notebook uses the following datasets, in LiPD format:

  • McCabe-Glynn, S., Johnson, K., Strong, C. et al. Variable North Pacific influence on drought in southwestern North America since AD 854. Nature Geosci 6, 617–621 (2013). https://doi.org/10.1038/ngeo1862

  • Lawrence, K. T., Liu, Z. H., & Herbert, T. D. (2006). Evolution of the eastern tropical Pacific through Plio-Pleistocne glaciation. Science, 312(5770), 79-83.

  • PAGES2k Consortium., Emile-Geay, J., McKay, N. et al. A global multiproxy database for temperature reconstructions of the Common Era. Sci Data 4, 170088 (2017). doi:10.1038/sdata.2017.88

Demonstration#

Extracting infomation about the content of a LiPD object#

Let’s start by importing our favorite package and load our datasets.

from pylipd.lipd import LiPD

Let’s load some diverse datasets to highlight to capabilities:

path = '../data/Pages2k/'

D = LiPD()
D.load_from_dir(path)

Loading 16 LiPD files

  0%|          | 0/16 [00:00<?, ?it/s]

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:41:56 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:41:56 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:41:56 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:46:16 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:41:56 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:41:56 U'

 31%|███▏      | 5/16 [00:00<00:00, 41.67it/s]

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:41:56 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:41:56 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:41:56 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:41:56 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:46:49 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:41:56 U'

 62%|██████▎   | 10/16 [00:00<00:00, 44.48it/s]

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:41:56 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:41:56 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:47:20 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:41:56 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:41:56 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:46:12 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:41:56 U'

 94%|█████████▍| 15/16 [00:00<00:00, 34.63it/s]

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-23 23:41:56 U'

100%|██████████| 16/16 [00:00<00:00, 37.05it/s]

Loaded..



data_path = ['../data/Crystal.McCabe-Glynn.2013.lpd', '../data/ODP846.Lawrence.2006.lpd', 'https://lipdverse.org/data/iso2k100_CO06MOPE/1_0_2//CO06MOPE.lpd']

D.load(data_path)

Loading 3 LiPD files

  0%|          | 0/3 [00:00<?, ?it/s]

 33%|███▎      | 1/3 [00:00<00:00,  2.71it/s]

 67%|██████▋   | 2/3 [00:01<00:00,  1.18it/s]

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-16 16:21:45 U'

Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_xsd_date at 0x7ffaa84996c0>
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/term.py", line 2163, in _castLexicalToPython
    return conv_func(lexical)  # type: ignore[arg-type]
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/rdflib/xsd_datetime.py", line 593, in parse_xsd_date
    return parse_date(date_string if not minus else ("-" + date_string))
  File "/usr/share/miniconda/envs/pylipd/lib/python3.10/site-packages/isodate/isodates.py", line 193, in parse_date
    raise ISO8601Error("Unrecognised ISO 8601 date format: %r" % datestring)
isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: '2022-08-12 20:17:06 U'

100%|██████████| 3/3 [00:01<00:00,  1.75it/s]

100%|██████████| 3/3 [00:01<00:00,  1.67it/s]

Loaded..

Getting information about Datasets#

From the introductory notebooks on loading LiPD datasets and working with LiPD objects, you should be already familiar with the functions to get all the names of the datasets.

D.get_all_dataset_names()

['Ocn-RedSea.Felis.2000',
 'Eur-CoastofPortugal.Abrantes.2011',
 'Eur-LakeSilvaplana.Trachsel.2010',
 'Arc-Kongressvatnet.D_Andrea.2012',
 'Ocn-AlboranSea436B.Nieto-Moreno.2013',
 'Ocn-SinaiPeninsula_RedSea.Moustafa.2000',
 'Eur-Stockholm.Leijonhufvud.2009',
 'Eur-NorthernScandinavia.Esper.2012',
 'Eur-SpannagelCave.Mangini.2005',
 'Eur-NorthernSpain.Martin-Chivelet.2011',
 'Asi-SourthAndMiddleUrals.Demezhko.2007',
 'Ocn-PedradeLume-CapeVerdeIslands.Moses.2006',
 'Ocn-FeniDrift.Richter.2009',
 'Eur-SpanishPyrenees.Dorado-Linan.2012',
 'Ant-WAIS-Divide.Severinghaus.2012',
 'Eur-FinnishLakelands.Helama.2014',
 'Crystal.McCabe-Glynn.2013',
 'ODP846.Lawrence.2006',
 'CO06MOPE']

len(D.get_all_dataset_names())

19

In fact, this function has been used throughout these notebooks to be able to extract other types of information. Another equivalent function returns all the datasetIDs. datasetIDs are unique identifiers for each LiPD dataset. This notion was introduced as the name may not be unique enough for unique identification. All datasets from the LiPDGraph will have an ID but it is not mandatory.

D.get_all_dataset_ids()

['4fZQAHmeuJn8ipLfurWv',
 '33wLrOlZRR8hw53DVKSr',
 '23GDZxTEJsBQAH05hU4g',
 'pwY7bQRstXsZc6iOpgRI',
 'fYUegig785BJMl3NrZcz',
 'wH1adV7y36OC0h3kwDRF',
 'uOhCAmcuPO5Xo9rSniHn',
 'fyUORoSbcL0GP0J3wyoj',
 '19nwWA48PSW3uSoDRiA4',
 'WX0GIjmoc46FH1Oj4c5r',
 'mE7P31hoHDXy1Q9yfQlq',
 'HH7jd52QFWaBgs9OvMqP',
 'IVVTVphliHduuTjQhlTM',
 'PPWjMBBkRAcCv6bkL58K',
 '5oHqINxYpL0XCaLcIjhR',
 'ZDMEZiVVO4eFNwBA4D3o',
 'iso2k100_CO06MOPE']

len(D.get_all_dataset_ids())

17

Notice that the function returned only 17 items (2 less than the dataset names). The reason is these files were created before datasetIDs were prevalent on the Lipdverse.

Note: `datasetIDs` will be used in future versions of `PyLiPD` to directly query the LiPDverse, without the need to pass the direct URL.

Another function that allows to look up information stored at the dataset level is get_all_archiveTypes. This one works a little bit differently than the previous functions in that it will only return the unique names present in these datasets:

D.get_all_archiveTypes()

['Coral',
 'Marine sediment',
 'Lake sediment',
 'Documents',
 'Wood',
 'Speleothem',
 'Borehole']

This function is particularly useful to know what terms can be used to filter with specific queries. You can see that coral appears with two different capitalizations. For filtering, this won’t matter as we will see in the next tutorial.

You can get information about the location of each dataset as follows:

df_loc = D.get_all_locations()

df_loc
dataSetName geo_meanLat geo_meanLon geo_meanElev
0 Ocn-RedSea.Felis.2000 27.8500 34.3200 -6.0
1 Eur-CoastofPortugal.Abrantes.2011 41.1000 -8.9000 -80.0
2 Eur-LakeSilvaplana.Trachsel.2010 46.5000 9.8000 1791.0
3 Arc-Kongressvatnet.D'Andrea.2012 78.0217 13.9311 94.0
4 Ocn-AlboranSea436B.Nieto-Moreno.2013 36.2053 -4.3133 -1108.0
5 Ocn-SinaiPeninsula,RedSea.Moustafa.2000 27.8483 34.3100 -3.0
6 Eur-Stockholm.Leijonhufvud.2009 59.3200 18.0600 10.0
7 Eur-NorthernScandinavia.Esper.2012 68.0000 25.0000 300.0
8 Eur-SpannagelCave.Mangini.2005 47.1000 11.6000 2347.0
9 Eur-NorthernSpain.Martin-Chivelet.2011 42.9000 -3.5000 1250.0
10 Asi-SourthAndMiddleUrals.Demezhko.2007 55.0000 59.5000 1900.0
11 Ocn-PedradeLume-CapeVerdeIslands.Moses.2006 16.7600 -22.8883 -5.0
12 Ocn-FeniDrift.Richter.2009 55.5000 -13.9000 -2543.0
13 Eur-SpanishPyrenees.Dorado-Linan.2012 42.5000 1.0000 1200.0
14 Ant-WAIS-Divide.Severinghaus.2012 -79.4630 -112.1250 1766.0
15 Eur-FinnishLakelands.Helama.2014 62.0000 28.3250 130.0
16 Crystal.McCabe-Glynn.2013 36.5900 -118.8200 1386.0
17 ODP846.Lawrence.2006 -3.1000 -90.8000 -3296.0
18 CO06MOPE 16.7500 -22.8883 -5.0

Getting information about variables#

To get information about available variable names, you can do the following:

D.get_all_variable_names()

['d18O',
 'year',
 'temperature',
 'Uk37',
 'MXD',
 'Mg_Ca',
 'depth_top',
 'notes',
 'depth_bottom',
 'trsgi',
 'uncertainty_temperature',
 '230th age',
 'corr_age_uncert',
 'age',
 'sample',
 '230th/238u_uncertainty',
 '230th age_uncertaity',
 '230th/232th',
 'd234u_undertainty',
 'depth',
 'd18o',
 'd234uinitial',
 '230th age_uncertainty',
 '230th/232th_uncertainty',
 'd234uinitial_uncertainty',
 '232th',
 'corr_age',
 '232th_uncertainty',
 'd234u',
 '230th/238u',
 '238u_uncertainty',
 'depth_dating',
 'Year',
 '238u',
 'section',
 'depth comp',
 'lower95',
 'c37 total',
 'u. peregrina d18o',
 'u. peregrina d13c',
 'd180',
 'depth cr',
 'upper95',
 'sample label',
 'c. wuellerstorfi d18o',
 'site/hole',
 'event',
 'ukprime37',
 'interval',
 'c. wuellerstorfi d13c',
 'median',
 'temp muller',
 'sst',
 'temp prahl']

Note that like the functions retrieving the various archiveTypes, this function also only returns the unique names. As we have explored previously, the Euro2k database contains more than one record correspoonding to temperature. Again, this function can be used to figure out what to filter by.

If you want to have more granularity about which variable is available in which datasets and their associated unique IDs, you can use the following function:

D.get_all_variables()
uri TSID variableName
0 http://linked.earth/lipd/Ocn-RedSea.Felis.2000... Ocean2kHR_019 d18O
1 http://linked.earth/lipd/Ocn-SinaiPeninsula_Re... Ocean2kHR_018 d18O
2 http://linked.earth/lipd/Eur-SpannagelCave.Man... Eur_001 d18O
3 http://linked.earth/lipd/Eur-NorthernSpain.Mar... Eur_008 d18O
4 http://linked.earth/lipd/Ocn-PedradeLume-CapeV... Ocean2kHR_107 d18O
... ... ... ...
93 http://linked.earth/lipd/paleo0measurement1.PY... PYT19MC8WE2 c. wuellerstorfi d13c
94 http://linked.earth/lipd/chron0model0summary0.... PYT7DLYN7X4 median
95 http://linked.earth/lipd/paleo0measurement0.PY... PYTGO6NV72Y temp muller
96 http://linked.earth/lipd/paleo0model0ensemble0... PYTDW6AIJPW sst
97 http://linked.earth/lipd/paleo0measurement0.PY... PYT95DVDUU3 temp prahl

98 rows × 3 columns

Get a bibliography#

pylipd makes is easy to retrieve the publication information from the subset of records you used and export this to a .bib file:

bibs, df = D.get_bibtex(remote = True, save = True, path = '../data/mybiblio.bib', verbose = False)

Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (http://nbn-resolving.de/urn:nbn:de:gbv:46-ep000102745), creating the entry manually
Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (None), creating the entry manually
Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (None), creating the entry manually

Cannot find a matching record for the provided DOI (None), creating the entry manually

Let’s decompose the parameters for this function:

  • remote: If set to True, PyLipd will use the crossref function in the doi2bib package to retrieve the bilbiography. You can only use this option online. If the retrieval fails, the entry will be created from the information in the LiPD file. If set to False, only the information in the file will be used.

  • save, path: If save is set to True, PyliPD will save the entries in a .bib file. In this example, we saved the file to the data folder contained in this repository.

  • verbose if set to True, the bibliography will print on the screen.

In addition to saving the file, the function returns bibs, a list of text bliography and df, which presents the information in a Pandas DataFrame.

df.head()
dsname title authors doi pubyear year journal volume issue pages type publisher report citeKey edition institution url url2
0 Ocn-RedSea.Felis.2000 Tropical sea surface temperatures for the past... Casey P. Saenger and Nerilie J. Abram and Cyri... 10.1002/2014PA002717 None 2015.0 Paleoceanography 30 3.0 226-252 journal-article Wiley-Blackwell None tierney2015tropicalseasurfacetempera None None None None
1 Ocn-RedSea.Felis.2000 World Data Center for Paleoclimatology T. Felis None None NaN None None None None dataCitation None None felis2000httpswwwncdcnoaagovpaleostudy1861Data... None World Data Center for Paleoclimatology None https://www.ncdc.noaa.gov/paleo/study/1861
2 Ocn-RedSea.Felis.2000 A coral oxygen isotope record from the norther... Yossi Loya and Thomas Felis and Maoz Fine and ... 10.1029/1999PA000477 None 2000.0 Paleoceanography 15 6.0 679-694 article Wiley-Blackwell None felis2000acoraloxygenisotoperecord None None None None
3 Eur-CoastofPortugal.Abrantes.2011 PANGAEA F. Abrantes None None NaN None None None None dataCitation None None abrantes2011httpsdoipangaeade101594pangaea7849... None PANGAEA None https://doi.pangaea.de/10.1594/PANGAEA.784992
4 Eur-CoastofPortugal.Abrantes.2011 Climate of the last millennium at the southern... C Lopes and C Santos and F Abrantes and B Mont... 10.3354/cr01010 None 2011.0 Climate Research 48 2.0 261-280 article Inter-Research Science Center None abrantes2011climateofthelastmillenniu None None None None