Structured paleoclimate tasks
Overview
This part of the PaleoPAL evaluation focuses on a structured paleoclimate analysis. These analysis workflows are similar to homework or tutorial-style scientific exercises: they have a defined goal and an expected solution, but they require both scientific understanding and technical implementation.
The evaluation compares three conditions: completing the task without AI assistance, completing a comparable task with a general-purpose LLM, and completing a comparable task with PaleoPAL. The goal is to assess whether PaleoPAL can help users translate scientific prompts into appropriate analyses, generate useful code, choose suitable tools and methods, avoid hallucinated functions or APIs, reach a working result efficiently, and respond to corrections or clarifications.
| Condition | Description |
|---|---|
| No AI assistance | Participants complete the task manually using documentation, tutorials, and non-AI programming resources. |
| General-purpose LLM assistance | Participants complete a comparable task using a general-purpose AI assistant, such as ChatGPT, Claude, or Gemini. |
| PaleoPAL assistance | Participants complete a comparable task using PaleoPAL, with an LLM of their choice. |
The goal is not to evaluate whether PaleoPAL can solve an entire research problem worthy of a standalone paper. Instead, we are evaluating whether PaleoPAL can support smaller pieces of scientific analyses that paleogeoscientists commonly perform. Each of the evaluation analyses include retrieving a dataset, loading and inspecting paleoclimate data, running a standard analysis, and creating a visualization. Each of these tasks involve one or more of the documents and libraries that PaleoPAL has indexed, and each step requires both scientific understanding and technical implementation.
This activity primarily evaluates:
| Evaluation criterion | Role in this activity |
|---|---|
| User Experience and Ease of Use | Does AI assistance make the task easier, faster, or less frustrating? |
| Tool and Method Appropriateness | Does the system select suitable libraries, functions, methods, and workflows? |
| Factuality and Hallucination Control | Does the system avoid inventing functions, misusing APIs, or making unsupported claims? |
| Steering and Adaptability | Does the system respond appropriately to corrections, clarifications, and constraints? |
| Time to First Correct Result | How quickly does the participant reach a working result, even if it still needs refinement? |
Participants
Participants are researchers in paleoclimatology with varying levels of experience in the field. Participants will be asked to document their process, including where they encountered difficulty, what resources they used, when they reached a working result, and how much manual correction was required. The comparison across participants and conditions is part of the evaluation. We are interested not only in whether a task is completed, but also in how the experience differs depending on the participant’s background and the type of assistance available.
Evaluation Design
For each condition, participants will complete a structured paleoclimate workflow. Each workflow is based on a small but meaningful part of a typical paleoclimate study and includes the following components:
- Dataset retrieval: Participants will retrieve paleoclimate data from multiple source types, including the [LiPDGraph](https://linkedearth.graphdb.mint.isi.edu, a dataset from NOAA or PANGAEA accessed through PyleoTUPS, and a local LiPD file.
- Data inspection: Participants will load the datasets, inspect metadata, and identify relevant variables.
- Analysis: Participants will apply a standard time series analysis, such as spectral analysis, coherence analysis, correlation, or change point detection.
- Visualization: Participants will create plots or maps to visualize the data, analysis results, or metadata.
Each task is time-constrained. If a participant does not complete a task within the allotted time, that is still a valid evaluation outcome. Incomplete tasks should be documented clearly, including what was completed, where the participant got stuck, and what additional information or assistance would have been needed.
How the Task Notebooks Were Prepared
Before the evaluation, each participant contributed a structured paleoclimate analysis based on their current work. Each analysis notebook included three components: data gathering, analysis, and visualization. The data-gathering step was designed to involve multiple source types, including records from the LiPDGraph, datasets from NOAA or PANGAEA accessible through PyleoTUPS, and local LiPD files.
The notebooks were then standardized into high-level task descriptions, similar to homework-style assignments. These standardized versions preserved the scientific goals of the original analyses while removing tool-specific instructions, so that participants could be evaluated on how they approached the task under different assistance conditions.
Conditions
PaleoPAL will be evluated through three conditions:
For the No AI assistance condition, participants may use package documentation, tutorials, example notebooks, and programming resources such as Stack Overflow. Participants should not use AI assistants. Participants should also avoid using search workflows that surface AI-generated answers. When possible, go directly to the relevant documentation or known resource.
For the General-purpose LLM assistance condition, participants may use an AI assistant such as ChatGPT, Claude, Gemini, or a combination of these systems. Participants should record which system they used, and, when possible, the model or version. Participants should also preserve prompts and responses, either in the notebook or in a separate record.
For the PaleoPAL assistance condition, participants will use PaleoPAL with an LLM of their choice. Installing PaleoPAL does not count toward the task time. However, understanding how to use it properly does.
Required tools
Before beginning this evaluation activity, participants should make sure they have access to the required tools, documentation, and computing environment.
Participants should have:
| Requirement | Notes |
|---|---|
| Python environment | A working Python environment (Python=3.12) with Pyleoclim, PyLiPD, PyleoTUPS, and Ammonyte installed. |
| Jupyter notebook interface | VS Code with all appropriate extension (Python, Jupyter Notebook, PaleoPAL) |
| Evaluation notebooks | The task notebooks provided by the PaleoPAL evaluation team. |
| Time tracker | Participants should track how much time they spend on each task and subtask. |
For all evaluation conditions, participants may use the following tools and resources:
Resources
These resources were used to build the PaleoPAL libraries.
LiPD resources
The following websites contain useful information about LiPD and the controlled vocabulary:
| Tool or resource | Purpose |
|---|---|
| LiPD Overview | An overview on how LiPD files are organized |
| Controlled Vocabulary | Controlled Vocabulary used for certain fields in LiPD datasets |
LiPDGraph Resources
The following contain information about where to find the graph, the ontology underpinning the knowledge base, and useful resources for SPARQL
| Tool or resource | Purpose |
|---|---|
| LiPDGraph endpoint | URL to access the SPARQL Endpoint |
| LinkedEarth Ontology | Represents the schema underlying the knowledge base. Can be used to infer relationship between classes and properties |
| Paleoclimate Dataset Representation | An overview of the ontology and SPARQL |
| Querying the LiPDGraph | Tutorial on how to query the LiPDGraph |
| PyLiPD queries | Since PyLiPD operates on a local graph similar to LiPDGraph, most of its functionalities are wrappers around SPARQL queries. These queries can be reused in the concept of LiPDGraph |
| PaleoBooks | Example workflows and tutorials that may serve as references. For graph queries, the Books on Updating LMRv2.1, Fun in the Frequency Domain, and PaleoPCA are relevant |
| A gentle introduction to SPARQL | This website contains some basic SPARQL commands |
| regex buddy | This website has quick shortcuts on regular expressions, which you can use in your query |
Software
| Tool or resource | Purpose |
|---|---|
| PyLiPD documentation | Overview of PyLiPD and API documentation |
| PyLiPD tutorials | Tutorials on how to use PyLiPD in scientific workflows |
| PyleoTUPS documentation | Overview of PyleoTUPS and API documentation |
| PyleoTUPS tutorials | Tutorials on how to use PyleoTUPS, including an overview of how NOAA and PANGAEA organize their databases |
| Pyleoclim documentation | Overview of Pyleoclim and API documentation |
| Pyleoclim tutorials | Tutorials on how to use Pyleoclim in scientific worklfows |
| Ammonyte documentation and tutorials | Overview of Ammonyte, API documentation and tutorials |
Science Examples
| Tool or resource | Purpose |
|---|---|
| PaleoBooks | A gallery of scientific notebooks that showcase the various tools supported by PaleoPAL |
Evaluation Activities
Scientific Analysis
For each task, participants should follow the same general procedure.
- Open the assigned task notebook.
- Read the task description and expected deliverables.
- Start tracking time.
- Work on the task under the assigned condition: no AI, general-purpose LLM, or PaleoPAL.
- Complete the task to the best of your ability within the allotted time. When using an AI assistant, make sure that you preserve prompts in the notebook.
- Stop after the allotted task time, even if the task is incomplete.
- Clean up the notebook enough that another person can understand what was attempted.
- Complete the evaluation form.
The evaluation notebooks are divided into three tasks: data gathering (which itself is divided into three subtasks), analysis, and visualization. Each task has been assigned a specific time allocation. At these checkpoints, participants may be given part of the expected solution or “ground truth” so that they can continue with the next stage rather than getting stuck indefinitely. When this happens, participants should clearly mark where they used the provided checkpoint information. At the end of a task (whether completed or not), participants should stop the clock and look at the provided solution to ensure that they start on the next task with a similar output. Participants should restart the clock at the beginning of each new task, after looking at the provided solution.
Evaluation Form
Each condition will be accompanied by an evaluation form. Participants will complete the form after the period to describe their experience and assess the system used for that condition.
The form will ask participants to reflect on their experience, including what worked well, what was challenging, and what could be improved. Participants will also rate the system across several evaluation criteria, including ease of use, tool and method appropriateness, factuality, adaptability, and time to first correct result.
In addition, participants will report their familiarity with the scientific concepts and software tools used in the task. They will also indicate whether they were able to complete the task and, if so, how much time it took.
Time Commitment
Each structured paleoclimate task is expected to require approximately three hours total.
| Activity | Time |
|---|---|
| Science task | 2 hours |
| Evaluation form | 1 hour |
Participants should stop the science task after two hours, even if the task is incomplete. Incomplete tasks are informative and should be documented. The evaluation is designed to capture where progress was made, where participants got stuck, and how different forms of assistance affected the process.