# Evaluation Protocol & Statistical Confidence Intervals

This page describes the evaluation protocol and judge settings used for the published report.

## 1. GeoKTools

- Prompt the agent with `case["query"]`. Queries instruct the system to answer with a bracketed JSON list filling the `__` blanks.
- **Report protocol (LLM-judged):** a response that exactly matches `golden_answer_raw` (string-equal after stripping, or JSON-equal) scores **1.0 without a model call**. Otherwise a *geospatial-expert* LLM judge compares response and golden element by element and returns a continuous score in [0, 1] (arithmetic mean of element scores; wrong primary entity zeroes dependent elements; empty response → 0). Judge: **Gemini 3.8 Flash**, model-default temperature, single call with structured JSON output, system prompt in [`geoktools/scoring/llm_judge_system_prompt.txt`](../geoktools/scoring/llm_judge_system_prompt.txt) (also printed in the report's appendix).
  Reference implementation: `python -m geoktools.scoring.run_llm_judge --predictions <jsonl>`.
- **Deterministic alternative (offline):** `python -m geoktools.scoring.run_scoring --predictions <jsonl>` — `score_structured_answer` with 10 % relative tolerance, `1 − MAPE` partial credit and ROUGE‑L for strings. Same inputs, no API key. It agrees with the LLM protocol on exact matches and clear misses and differs only on near-misses; use it for quick iteration, and the LLM protocol when comparing with the report.
- Report macro-average accuracy overall and stratified by K ∈ {1, 2, 3, 4}. Both runners print this.

## 2. GeoExpertQueries

- Condense the agent's execution trajectory (tool calls, arguments, code outputs, final response) into text and evaluate with `tool_agnostic_autorater_prompt.txt` (primary; used for the report) or `workflow_autorater_prompt.txt` (tool-aware variant).
- Judge: **Gemini 3.8 Flash**, model-default temperature, **3 independent runs** per response, each returning 0 / 1 / 2 → mapped to 0.0 / 0.5 / 1.0 and averaged. A "Today is <date>" header precedes the prompt, as for the report.
- Reference implementation: `python -m geoexpertqueries.autorater.run_autorater --traces <jsonl>` (public Gemini API or any LLM via `--judge command`).

## 3. Confidence intervals

- 95 % percentile bootstrap intervals over the per-case scores (`scoring_functions.compute_confidence_intervals` with `bayesian_smoothing=False`, `n_boot = 10000`, seeded with `--seed`, default 0); both `GeoKTools` runners compute them overall and per K. The report additionally repeats the *full pipeline* three times and reports mean ± s.d. across repeats — that is a property of the agent runs, not of the scorer.

## 4. Reproducibility pins

| Component | Setting used for the report |
|---|---|
| `GeoKTools` goldens | `geoktools/dev_queries.json` v1.0.0 (per-case `snapshot_date`) |
| `GeoKTools` scoring | exact match → 1.0, else LLM judge (`run_llm_judge.py`); judge Gemini 3.8 Flash, default temperature, structured JSON output |
| `GeoKTools` deterministic scorer | `scoring_functions.py`, `tolerance_ratio = 0.10`, ROUGE‑L F1 — *secondary, offline* |
| `GeoExpertQueries` goldens | `geoexpertqueries/dev_queries.json` v1.0.0 |
| `GeoExpertQueries` judge prompt | `tool_agnostic_autorater_prompt.txt` (+ date header); `Available Tools` listed the tools of the agent evaluated in the report (not released) |
| `GeoExpertQueries` judge | Gemini 3.8 Flash, default temperature, `num_runs = 3`, mean of 0/0.5/1 |
| Rating map | `0 → 0.0`, `1 → 0.5`, `2 → 1.0` (`likert_scales.json`) |
| CI | percentile bootstrap, `n_boot = 10000`, 95 % |

Judge models evolve; results produced with a different judge (or a later version of the same one) rank agents consistently under that judge but are not directly comparable with the report's figures. Always report the judge configuration (both runners write it to `judge_config` in their output).