# decide-lab

Independent evaluation, calibration and LoRA fine-tuning experiments for
[fastino/GLiNER2.5-Decide](https://huggingface.co/fastino/GLiNER2.5-Decide), a 340M local zero-shot
classifier, with a harness to compare it against TypeSafe's Jev on the same data.

Everything runs locally: CPU works, a CUDA GPU is ~10x faster.

## What Decide is

- A fine-tune of `fastino/gliner2-large-v1`: DeBERTa-v3-large encoder, **span** architecture
  (`config.json` says `"architecture": "span"`), despite the 2.5 name. Apache 2.0.
- The task and candidate labels are encoded next to the text (`[P] task [L] label1 [L] label2 ... [SEP] text`);
  a small head scores each label. Labels are free-form at call time.
- The base GLiNER2 recipe ([paper](https://arxiv.org/abs/2507.18546)): 254k GPT-4o-annotated examples
  (136k real, 119k synthetic), 5 epochs, lr 1e-5 encoder / 2e-5 heads. Decide's own fine-tuning data is not published.
- The open-source trainer uses per-label BCE; single-label inference applies softmax over those logits.
  That mismatch is a plausible cause of the overconfidence measured below (not proven for Decide).

## Results (fast-decisions dev split, 1,700 rows / 2,900 heads)

These are on the **public dev split**, not Fastino's held-out test split, so they are an independent
sanity check, not the benchmark. Fastino reports 60.2% (Decide) vs 57.6% (JevK5) on the test split.

**Accuracy** (exact match per head, vendor scoring)

| | Accuracy |
|---|---|
| All heads (vendor protocol: plain label list) | 61.4% |
| Single-label heads | 66.5% |
| Multi-label heads | 17–23% |

Latency: ~300 ms/decision on CPU (Ryzen 7 9800X3D), ~25 ms on an RTX 5080.

**Calibration** (`fd_logits.py` + `fd_calibrate.py`; fit on a seeded half of rows, reported on the other half)

| Single-label | Mean confidence | Accuracy | ECE | NLL |
|---|---|---|---|---|
| As shipped (T=1) | 82.4% | 66.5% | 0.160 | 1.107 |
| Temperature T=2.2 | 66.2% | 66.5% | 0.028 | 0.892 |

Risk–coverage at T=2.2: the 50% most confident answers are 84.8% accurate; the top 30% are 91.0%.

| Multi-label (sigmoid threshold) | Exact-set | Micro-F1 |
|---|---|---|
| 0.5 (default) | 15.3% | 67.2% |
| 0.24 (fitted, global) | 24.7% | 74.6% |

Per-head thresholds did worse than one global threshold (too few examples per head).
Refit T and thresholds on your own labels, and after any fine-tuning.

**LoRA pilot** (`fd_lora_pilot.py`, r=16, 3 epochs, 140 training rows from two weak domains)

Trained in 36 s on an RTX 5080 (13.9 GB peak VRAM). The 28 MB adapter reloads from disk on GPU and CPU
(`fd_lora_verify.py`) with identical results.

| Held-out 30% | Base | Tuned |
|---|---|---|
| support_topic | 12/30 | 19/30 |
| ticket_route (3 heads) | 44/90 | 44/90 (31 predictions changed, net zero) |
| review_sentiment (not trained on) | 28/30 | 28/30 |

Tiny test sets: this shows the pipeline works, not a measured gain. The dev rows are synthetic, so
train/test rows may share templates.

## Setup

```bash
python -m venv .venv && . .venv/Scripts/activate        # Windows; use bin/activate elsewhere
pip install torch --index-url https://download.pytorch.org/whl/cu128   # or /whl/cpu
pip install -r requirements.txt
python fetch_data.py
```

Windows gotchas:
- set `PYTHONUTF8=1` (gliner2 prints emoji; cp1252 consoles crash)
- training: `num_workers=0`, or DataLoader workers re-execute the script
- training: pass `fp16=False` when using `bf16=True` (fp16 defaults on)
- the Hugging Face symlink warning is harmless

## Scripts

| Script | What it does |
|---|---|
| `fetch_data.py` | Download the dev split into `fast_decisions/` |
| `fd_decide.py` | Decide predictions for every head → `results/fd_decide_preds.jsonl` |
| `fd_logits.py` | Raw per-label logits for every head → `results/fd_decide_logits.jsonl` |
| `fd_calibrate.py` | Temperature / threshold fitting and reliability report from the logits |
| `fd_lora_pilot.py` | LoRA fine-tune on two domains, score before/after |
| `fd_lora_verify.py` | Reload the saved adapter and re-score (`cuda` or `cpu`) |
| `fd_jev.py` | Jev on a seeded N-per-domain sample (Choice per single-label head, Noul per label for multi-label) |
| `fd_score.py` | Score one or more prediction files, overall and on shared rows |

All long runs print flushed progress and append results line by line, so they can be resumed.

## Applying calibration

`classify_text` does not expose a temperature, so capture the logits with a hook:

```python
import torch
from gliner2 import AutoExtractor

m = AutoExtractor.from_pretrained("fastino/GLiNER2.5-Decide")
cap = []
m.classifier.register_forward_hook(lambda mod, i, o: cap.append(o.detach().squeeze(-1)))

labels = ["refund_request", "cancel_subscription", "other"]
m.classify_text("Can I get that charge refunded?", {"intent": labels})
probs = torch.softmax(cap[-1] / 2.2, -1)          # calibrated single-label distribution

# multi-label: the threshold is a public option
m.classify_text(text, {"tags": {"labels": tag_labels, "multi_label": True, "cls_threshold": 0.24}})
```

## Training on your own data

Use the same JSONL shape as fast-decisions, one row per document:

```json
{"input": "text", "output": {"classifications": [
  {"task": "intent", "labels": ["a", "b", "c"], "true_label": ["b"]},
  {"task": "tags", "labels": ["x", "y", "z"], "true_label": ["x", "z"], "multi_label": true}]}}
```

Optional per-head fields: `label_descriptions`, `prompt`, `examples`. A few hundred rows split
train/dev/test by document is a reasonable start; use dev to pick the checkpoint and refit calibration.