Evaluation

This section provides information about how the performance of PaleoPAL has been evaluated.

Purpose of the Evaluation

PaleoPAL is an AI assistant designed to support paleoclimate research by helping users find relevant data, identify appropriate methods, and develop reproducible analysis workflows. Because PaleoPAL is embedded in the scientific research process, evaluating it requires more than checking whether it returns a single “correct” answer.

Many AI systems are evaluated using automated metrics, such as accuracy, precision, recall, or F1 score. These metrics are useful when there is a clearly defined ground truth: for example, whether a document belongs in a category, whether a known answer was retrieved, or whether a prediction matches an observed label. PaleoPAL is different. A successful interaction may involve interpreting an open-ended research question, identifying relevant datasets, explaining assumptions, suggesting possible methods, generating code, or helping a user refine an analysis. In these cases, there may be several valid paths forward rather than one correct answer.

The goal of this evaluation is therefore to assess how well PaleoPAL supports real paleoclimate research workflows. We are interested not only in whether PaleoPAL provides accurate information, but also whether its responses are useful, transparent, scientifically appropriate, reproducible, and aligned with community practices. Feedback from evaluators will help us identify where PaleoPAL performs well, where it needs improvement, and how it can better support researchers, educators, and students working with paleoclimate data.

What Participants Will Evaluate

Participants will evaluate how well PaleoPAL supports common paleoclimate research tasks, from interpreting a scientific question to helping build a reproducible workflow. The evaluation focuses on both the final output and the interaction process: how PaleoPAL responds, explains its choices, adapts to feedback, and helps the user make progress.

Participants may be asked to test whether PaleoPAL can understand paleoclimate research questions, including relevant proxy systems, archives, time periods, regions, variables, and scientific context. They will also evaluate whether PaleoPAL can retrieve relevant datasets, metadata, documentation, or methodological resources, and whether those materials are appropriate for the scientific task.

The evaluation will also examine PaleoPAL’s ability to recommend suitable methods and tools, build coherent analysis workflows, and generate or refine code in notebooks. Participants will consider whether PaleoPAL can connect steps such as data discovery, preprocessing, analysis, visualization, and interpretation into a workflow that is technically usable and scientifically meaningful.

Finally, participants will assess how well PaleoPAL explains results, communicates assumptions and limitations, and adapts when the user provides clarifications, corrections, or additional constraints.

Evaluation Activities

The PaleoPAL evaluation includes three activities that increase in complexity. The goal is to assess not only whether PaleoPAL can produce correct outputs, but also whether it can support the process of scientific work: interpreting questions, selecting tools, generating code, building workflows, and adapting to feedback.

Evaluation activity	What participants will do
Structured paleoclimate tasks	Participants will complete short, well-scoped paleoclimate analysis tasks under three conditions: without AI assistance, with a general-purpose LLM, and with PaleoPAL. These tasks are similar to homework or tutorial exercises: they have a defined goal and an expected solution, but require scientific understanding as well as technical implementation. They are designed to evaluate whether PaleoPAL can help users translate scientific prompts into appropriate analyses, generate useful code, choose suitable tools and methods, avoid hallucinated functions or APIs, reach a working result efficiently, and respond to corrections or clarifications.
Reproducing a scientific study	Participants will use PaleoPAL to help reproduce part of an existing paleoclimate study. This activity tests whether PaleoPAL can support a connected research workflow, including understanding the scientific goal, identifying relevant data and methods, generating or refining code, explaining choices, and helping interpret results.
Exploring an unknown science problem	Participants will use PaleoPAL to investigate a less structured scientific problem. This activity tests PaleoPAL in an exploratory setting where the question, data, methods, and interpretation may need to be refined during the interaction. It is intended to evaluate the full set of criteria, including usability, factuality, reasoning transparency, workflow coherence, steering, and domain alignment.

Evaluation Criteria

PaleoPAL will be evaluated across several dimensions that reflect how researchers actually use AI assistants in scientific workflows. Rather than relying on a single score, the evaluation considers whether PaleoPAL helps users move from a research question to a scientifically appropriate, reproducible result.

Evaluation criterion	What we are assessing
User Experience and Ease of Use	Does PaleoPAL make the task easier, faster, or less frustrating, and how does this vary by user experience level?
Reasoning Transparency	Does PaleoPAL explain why it chose a dataset, method, tool, or workflow in a way the user can follow?
Tool and Method Appropriateness	Does PaleoPAL select suitable libraries, functions, methods, and data structures, both with and without user guidance?
Workflow Coherence	Does PaleoPAL produce a connected workflow rather than fragmented steps?
Factuality and Hallucination Control	Does PaleoPAL avoid inventing functions, misusing APIs, citing nonexistent features, or making unsupported claims?
Steering and Adaptability	Does PaleoPAL adapt to clarifications, corrections, and constraints provided by the user?
Domain Alignment	Does PaleoPAL respect paleoclimate terminology, concepts, and community practices?
Time to First Correct Result	How quickly does the user obtain a result that works, even if it still needs refinement?