# Training

mlx-decision can fit a decision model to your own labelled examples, on
the Mac it runs on, and serve the result like any other model. Five kinds
of fit cover the nine model families, from a calibration in seconds to
fine-tuning a whole encoder or adapting a network with LoRA. None of
them needs more than MLX.

Training is newer than the rest of mlx-decision: the flags of `data` and
`training`, the Python training API and `dataset.json` are provisional
and may change in a minor release ([Stability](STABILITY.md)). Fitted
models and training files in JSON lines are stable.

The workflow has four steps:

1. **Data**: turn a dataset into labelled examples, one JSON request per
   line plus its labels (`mlx-decision data`, or your own script).
2. **Fit**: `mlx-decision training fit -m BASE --data FILE` trains and
   writes a *fitted model*. This is a small folder that points at its
   base and holds only what was trained.
3. **Compare**: `mlx-decision training eval -m FITTED --data TEST --compare
   BASE` scores both on data neither has seen.
4. **Use**: pass the folder to `-m` in `run`, `chat`, `server` or `mcp`,
   or to `mlx_decision.load`.

The Studio does the same without the command line: its Datasets page
downloads or adds a dataset, and Fit on It… and the Training page fit
and evaluate ([Studio](STUDIO.md#training)).

```bash
pip install "mlx-decision[training]"     # `data` reading from the Hugging Face Hub

mlx-decision data --preset trec --split train --split test -o trec
mlx-decision training fit -m SupersonicLabs/Julia-1 --data trec -o julia-trec
mlx-decision training eval -m julia-trec --data trec --compare SupersonicLabs/Julia-1
mlx-decision run -m julia-trec -s "Who wrote Hamlet?" -q trec/dataset.json
```

(`trec` is a [dataset folder](#dataset-folders) with one file per
split: `training fit` trains on its train split and reports on its test
split, `training eval` scores the test split.)

(`-q` takes the folder's `dataset.json`, which holds the questions the
examples are asked; for a plain training file, the `questions` object of
any of its lines.)

Run in a terminal without flags, `data` and `training fit` ask for what
they need in menus. When they finish, they print the command line that
does the same thing without menus.

## Choosing a fit

Start with `--method auto`: it picks the kind your model's family takes
and ends with a calibration. The numbers below were measured on an
Apple M5 Pro (20-core GPU, 64 GB); the runs behind them are in
[What a fit gains](#what-a-fit-gains).

| Your model | `auto` trains | Takes | Try instead |
|---|---|---|---|
| pplx, d3, Base Labs, H2O-Lightning, d1 LFM3, d1 | `readout`: each option's readout row | one scoring pass, then seconds | `lora` on d3 and d1 LFM3 when the network has to see something new |
| Laya, Julia | `finetune`: the whole encoder | minutes | `--head-only` when memory is short |
| Clef | `head`: the decision head | minutes, and disk for stored states | `calibrate`, often the useful one |

**Only the probabilities are off?** `--method calibrate` works on every
family. It fits one temperature per question (and a bias per option
when there are enough rows) in under a second after one scoring pass,
and leaves accuracy as it is. Clef on ANLI-R3 went from an ECE of 20.9%
to 2.8% this way. Fit it on rows like the ones it will answer:
[Calibration needs data like the target](#calibration-needs-data-like-the-target).

**How many rows.** Most of the gain comes by 1,000 rows. A d3-edge
readout fit, five samples each:

| Rows | 100 | 300 | 1,000 | 3,000 | all |
|---|---|---|---|---|---|
| TREC (6 answer types) | 82.6 | 89.9 | 93.6 | 94.5 | 94.8 |
| SST-5 (5 sentiment levels) | 45.5 | 46.5 | 49.3 | 50.1 | 51.1 |

These runs ended with a bias per option in the calibration
(`--calibration full`), which below 1,000 rows moved accuracy by up to
2 points either way; `auto` leaves those biases out after a trained
kind. A fine-tuned encoder can need far fewer rows: Laya reached 94.8%
on TREC from 100. Under 50 rows, or under 10 for an option, the fit
warns.

**When LoRA pays.** `lora` (d3 and d1 LFM3) adapts the network itself
where a readout fit only retrains the answer step. On images that
matters: d3-edge on 37 pet breeds went from 86.2% (readout) to 90.2%
(LoRA). On text, where the network already reads what the answer needs,
it lands within a point or two of the readout fit (SST-5: 52.5 against
about 51), at minutes instead of seconds. Its defaults (rank 8, learning
rate 1e-4, two epochs) came out of a sweep in which only the learning
rate mattered: above about 3e-4 the fit breaks down. It needs an
unquantized base. Details: [Kinds](#kinds).

**More rows than you can label.** A larger model's answers can label
data for a smaller one, with its probabilities as soft labels. That
helps the students that change weights (Laya, LoRA) by about 2 points
over its top answers, but not a readout fit, and the student inherits
the teacher's calibration: fit the teacher first.
[A teacher's labels](#a-teachers-labels-training-label).

**Several tasks in one model.** A
[mix of datasets](#mixes-of-datasets) fits all their questions at once.
Expect a trade, not a lift on every task: the adapters or encoder that
all tasks share have a fixed size, unlike tasks pull them different
ways, one learning rate and one kept epoch serve them all, and equal
caps ignore that some tasks are harder. In the measured mix of five,
four datasets stayed within one standard error of their own fit and
OpenBookQA lost 2.6 points; the report per dataset shows such a loss. A question left out of the mix
gains nothing from it, and after a fine-tune its answers can get worse.

**Thresholds in your code.** A fit ends with a calibration. If a program
compares the model's probabilities with thresholds tuned on the base,
`--calibration none` keeps the fitted kind's answers as they are:
[The closing calibration](#the-closing-calibration).

**Check the result** with `training eval -m FITTED --data TEST --compare
BASE` on rows the fit has not seen. The fit's own report scores rows it
held back, but they come from the same split as the training rows, and
a train split can be easier than the test split (ANLI-R3's is).

## What a fit gains

Each model was fitted on its dataset's train split with the default
settings (`--method auto`) and scored on the test split. Before → after:
accuracy / macro-F1 / ECE in percent (ECE: expected calibration error,
lower is better). Measured on an Apple M5 Pro (20-core GPU, 64 GB):

| Model | Kind | Training data | Base | Fitted | Time | Peak memory |
|---|---|---|---|---|---|---|
| clef-flash (8-bit) | head | ANLI-R3, 2,000 rows | 50.3 / 49.9 / 20.9 | 52.7 / 52.5 / 21.4 | 10.4 min | 15.1 GB |
| clef-flash (8-bit) | calibrate | 2,500 rows, five datasets | ANLI-R3: 50.3 / 49.9 / 20.9 | 52.2 / 52.3 / 2.8 | 10.6 min | 11.9 GB |
| H2O-Lightning-4B | readout | TREC, 5,452 rows | 85.2 / 86.8 / 4.7 | 97.4 / 97.0 / 1.4 | 12.8 min | 10.6 GB |
| d1-3B | readout | TREC, 5,452 rows | 86.6 / 86.8 / 5.5 | 96.4 / 94.4 / 1.4 | 4.0 min | 6.1 GB |
| laya | finetune | TweetEval Offensive, 11,916 rows | 77.8 / 61.9 / 7.7 | 84.7 / 78.4 / 5.0 | 17.2 min | 8.9 GB |
| Julia-1 | finetune | TREC, 5,452 rows | 16.0 / 14.0 / 63.1 | 97.4 / 96.0 / 1.9 | 2.3 min | 4.8 GB |

Every kind also fits a calibration at the end (see below). These runs
were made with version 1.2, whose calibration always had a bias per
option, as `--calibration full` does now; the default since 1.3 leaves
the biases out after a trained kind
([The closing calibration](#the-closing-calibration)). A fit holds
back a fifth of the training data to report on, so these runs trained on
four fifths of each train split. The clef-flash calibration was a single
fit over 500 rows from each of the five datasets, one question each. On
the other four datasets it lowered ECE as well (TREC 4.4 → 2.1, Offensive
9.2 → 3.5, OpenBookQA 4.5 → 1.9, SST-5 10.8 → 6.5) and left accuracy
within half a point.

The laya run trained three epochs and kept the first. With early
stopping, the default since then, the same run stops after the second
epoch and takes about 11.5 minutes.

Soft labels make the difference on data with uncertain answers. On
[typed-decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions)
(`--preset typed-decisions`: 1,200 training cases of five questions,
its gold distributions as soft labels; scored on the 400 test cases with
the benchmark's own scorer, accuracy and KL divergence from the gold,
lower is better), laya fine-tuned for 31 minutes went from 0.363 / 0.572
to 0.784 / 0.094, ahead of the published laya-typed-decisions (0.766 /
0.117), and a 5-minute d1-3B readout fit from 0.648 / 0.286 to 0.776 /
0.097. The same d1-3B fit on hard labels (each gold's most likely
answer) reached 0.765 but KL 1.240: trained on hard labels, a model
learns to be sure where the data is not.

Images train like text. d3-edge (0.6B), fitted with `readout` on the
3,680 photos of [Oxford-IIIT Pet](https://huggingface.co/datasets/timm/oxford-iiit-pet)'s
train split (37 breeds, one photo per line, an empty state) in 3.3
minutes, went from 31.6 / 27.6 / 4.6 to 86.2 / 86.2 / 3.4 on 1,850 test
photos (50 per breed). With a one-line state saying what the photos show
("A photo of a pet cat or dog.") it reached 87.1.

LoRA adapts the network where the readout fit stops. On the same photos
and state, `--method lora` on d3-edge went from 30.3 / 26.8 / 5.9 to
90.2 / 90.1 / 1.8, in 16.7 minutes (13.9 of them training, two epochs)
and about 4 GB. On text it added less: on 3,000 rows of SST-5 (five
sentiment levels, 2,210 test rows), d3-edge went from 42.5 to 52.5
accuracy in 2.9 minutes, where a readout fit of the same rows reaches
about 51.

What to expect in general:

- **Julia** gains the most: on TREC its release guesses (16%), and after
  2 minutes of fine-tuning it answers 97% right. Fine-tuning helps it on
  any task with fixed options.
- **H2O-Lightning and d1** gain 5 to 12 points on a task with fixed
  options and enough examples. A readout fit leaves the network alone and
  retrains only the answer step, so a fitted model shares its base in
  memory.
- **laya** gains on tasks its pre-training covered less well, such as
  offensive language.
- **d3 and d1 LFM3** gain most from `lora` where the network has to
  see something new, such as pet breeds in photos; on text tasks it has
  landed within a point or two of the readout fit, at many times its
  time.
- **Clef** gains little from its head alone (2 to 3 points on entailment).
  Its calibration fit is the useful one: it makes the probabilities
  trustworthy when Clef is overconfident, as on ANLI-R3 (ECE 20.9 → 2.8).

## Training data

### Format

A training file has JSON lines. Each line is a request body as the
server takes it (a `state` and its `questions`) plus `labels`, the
correct answer for each question, by question id:

```json
{"id": "t1", "state": "My card was charged twice.", "questions": {"team": {"type": "choice", "instructions": "Which team handles this ticket?", "criteria": {"billing": "Payments, refunds, invoices", "tech": "Crashes, bugs, errors", "sales": "Plans, pricing, upgrades"}}, "urgent": {"type": "noul", "instructions": "Does the customer need an answer today?"}}, "labels": {"team": "billing", "urgent": true}}
```

- `choice`: the label is an option id (a key of `criteria`).
- `noul`: `true` or `false` (also as strings).
- `score`: the level, counted from 0 in the order of the `criteria`
  list (`2` or `"2"` for the third).
- A label may also be soft: how likely each answer is, for data whose
  answers are uncertain (several annotators, a teacher model's
  probabilities). For `choice` and `score` it is an object from option
  id (a level as a string) to probability, the probabilities summing to
  1 and options left out counting as 0; for `noul` it is the probability
  of true, a number from 0 to 1:
  `"labels": {"team": {"billing": 0.7, "tech": 0.3}, "urgent": 0.8}`.
  A soft label that puts everything on one answer is that answer.
- `id` is optional: a non-empty string, unique in the file. It defaults
  to the line number.
- A question without a label is asked but not trained on.
- A line may carry images, as a request does: `images` with file paths
  or data URLs (`"images": ["photos/4711.jpg"]`), or data URLs inside
  the state. A path is relative to the file's folder (in a
  [dataset folder](#dataset-folders), to the folder), so a training file
  works from any working folder. Videos are not accepted.

The whole file is checked before any model loads: every request must be
valid (the same checks as `run` and the server), every label must name
an option of its question, and every image must open (a missing,
unreadable or too large image is an error). An error names its line.
Image data needs a model that reads images: every family but d1-3B, Laya
and Julia; the others refuse it before loading. A fit records a hash of
its data, and for image data the hash covers the image files too, so a
changed photo shows as changed data.

Soft labels train against their distribution: the trained kinds' loss
and the calibration's log loss are measured against it, without label
smoothing, so a fitted model learns to answer as uncertain as the data
is. Accuracy, macro-F1 and the confusion matrix count a soft label's
most likely answer (on a tie the question's first option; a `noul` at
0.5 counts as true); the log loss that `training fit` and
`training eval` report is against the distribution.

Two rules decide what a fit can learn:

- **Ask a question the same way on every line.** A fit applies to a
  question by its id and its options. Calibration biases and readout rows
  are fitted per option, so they need the same option ids in the same
  order everywhere. Option biases also need the same instructions and
  criteria. A question asked in several wordings is still fitted (a
  temperature, readout rows) but gets no option biases.
- **The data must look like what the model will be asked.** See
  [Calibration](#calibration-needs-data-like-the-target).

### Dataset folders

A dataset folder keeps one dataset together: `dataset.json` and one
JSON-lines file per split, named after the split (`train.jsonl`,
`test.jsonl`). `data` writes one when `-o` has no `.jsonl`. Its lines
hold the state and the raw values of the dataset's label column; the
questions and what each value means live once, in `dataset.json`:

```json
{
  "format": 1,
  "questions": {"topic": {"type": "choice", "instructions": "What is the tweet about?",
                          "criteria": {"earnings": "Earnings", "fed": "Fed, central banks"}}},
  "answers": {"topic": {"column": "label", "map": {"5": "earnings", "1": "fed"},
                        "left_out": ["0", "2"]}},
  "columns": {"label": {"values": {"0": {"rows": 1789}, "1": {"rows": 826}}}},
  "splits": {"train": {"role": "train", "split": "train", "rows": 16990, "examples": 16990},
             "validation": {"role": "report", "split": "validation", "rows": 4117, "examples": 4117}},
  "source": {"source": "hf:zeroshot/twitter-financial-news-topic", "shape": "text"}
}
```

```json
{"id": "twitter-financial-news-topic:train/0", "state": "Here are Thursday's biggest analyst calls ...", "values": {"label": "0"}}
```

- `answers` maps each value of a column to an answer of a question: an
  option id, `true` / `false` for a yes/no question, a level for a
  score. Several values may map to one option. Values in `left_out`
  are not trained on for that question. A value that is neither mapped
  nor left out stops `training fit` and `training eval`, which name it.
- Changing the wording, merging classes, leaving some out or asking a
  yes/no question about some of them ("Is this about earnings?") only
  changes `dataset.json`; the split files stay as they are.
- A dataset may have no questions yet: `data` without `--question`
  writes the rows and their values, and `training fit` says what is
  missing until you add `questions` and `answers`.
- A line may still carry its own `questions` and `labels`, where they
  differ per row (OpenBookQA's options, a yes/no dataset's question);
  they replace the dataset's for that line.
- Each split has a `role`: `train` (what a fit learns from), `report`
  (what a fit's report scores) or `evaluate` (what `training eval`
  scores). `training fit --data FOLDER` (and Python's `fit`) trains on
  the train split and reports on the report split (unless
  `--validation` or `--holdout` is given); `training eval --data FOLDER` scores the evaluate split, else
  the report split. Pass a split's file (`--data trec/test.jsonl`) to
  pick another.
- A plain JSON-lines file works everywhere a folder does, as a dataset
  of one split.
- Images live in the folder's `images/`; the lines name them relative to
  the folder (`"images": ["images/train-00000.jpg"]`).
- `--value-names "0=Analyst Update,1=Fed | Central Banks"` gives the
  label's values names, kept in `columns` beside their row counts (the
  Studio reads them from the dataset card when it lists them).
- `format` is the version of `dataset.json` (a file without it is
  format 1). A dataset written by a newer release with a higher format
  is refused with a message naming it, by `training` and the Studio
  alike. Keys this release does not know are kept when it writes the
  file again (the Studio saving questions, for one).

### Mixes of datasets

A *mix* is a dataset folder whose `dataset.json` lists other datasets
instead of holding rows. A fit on a mix learns every source's questions
at once, and its report shows each source before and after; `training
eval` on a mix shows each source too.

```json
{"format": 2,
 "mix": {"sources": [{"path": "../trec"},
                     {"path": "../sst5", "cap": 3000},
                     {"path": "../openbookqa", "held_out": true}],
         "cap": 2000, "seed": 0}}
```

```bash
mlx-decision data --preset trec --split train --split test -o trec
mlx-decision data --preset sst5 --split train --split test -o sst5
mkdir mix
echo '{"format": 2, "mix": {"sources": [{"path": "../trec"}, {"path": "../sst5"}], "cap": 2000}}' > mix/dataset.json
mlx-decision training fit -m vllm-sr/d3-edge --method lora --data mix
mlx-decision training eval -m d3-edge-mix --data mix --compare vllm-sr/d3-edge
```

- `path` is a dataset folder, one of its split files or a plain
  JSON-lines file, relative to the mix's folder (or absolute). Sources
  are read as they are when the fit runs, and the fit's data hash
  covers all of them, the caps and the seed.
- `cap` is the most train rows taken from a source: its own `cap`, else
  the mix's, every row without either. They are sampled with `seed`.
  Equal caps were enough to keep every source about as good as fitted
  alone (below).
- The mix has three splits, `train`, `report` and `evaluate`, each the
  sources' splits of that role together. A source without a report
  split reports on a fifth of its capped train rows, which are then not
  trained on. `evaluate` takes each source's evaluate split, else its
  report split.
- A `held_out` source is scored in the report and by `training eval`
  but not trained on: it shows what the fit does to a question outside
  its data. The report marks it with `*`.
- Example ids start with their source's name (`trec/17`). Questions of
  different types mix. Two sources may share a question id only when
  they ask it the same way; otherwise the fit names both.
- A mix cannot hold another mix. Mixes are `format` 2, so releases
  before mixes refuse them with the format message.

What a mix keeps, measured with 2,000 train rows from each of the five
coverage datasets and scored on their test splits (accuracy, %): each
dataset fitted on its own ("alone", scored on its own dataset) against
one fit on all five.

| | TREC | Offensive | ANLI-R3 | OpenBookQA | SST-5 |
|---|---|---|---|---|---|
| d3-edge base | 73.8 | 77.2 | 37.9 | 57.8 | 42.5 |
| d3-edge `lora`, alone | 95.2 | 83.7 | 44.2 | 68.4 | 49.2 |
| d3-edge `lora`, the mix | 95.4 | 82.9 | 43.2 | 67.2 | 52.9 |
| Laya base | 79.0 | 77.8 | 39.5 | 39.2 | 35.1 |
| Laya `finetune`, alone | 96.2 | 83.6 | 43.5 | 54.2 | 53.8 |
| Laya `finetune`, the mix | 96.2 | 85.7 | 43.0 | 54.8 | 54.6 |

In this run every dataset in the mix is within 0.6 standard errors of
its own fit, and SST-5 is better in the mix. Three more runs of the
d3-edge mix, as a mix folder through `training fit` with seeds 0 to 2,
kept TREC, Offensive, ANLI-R3 and SST-5 within one standard error on
average, but OpenBookQA came out at 65.8% against its own fit's 68.4%,
on every seed. So a mix can cost a dataset a couple of points; the
report per source shows it. The mix took about as long as the five
fits one after another (10 and 11 minutes on an M5 Pro) and as much
memory as one.

A fit also changes the questions it was not fitted on. The Laya model
fitted on TREC alone lost 9.4 points on OpenBookQA and gained 5.9 on
SST-5, and those questions' ECE rose as high as 28%, since only the
fitted questions get a new calibration. d3-edge's LoRA fits moved the
other datasets by about a point (3.1 at most). Readout and calibration
fits are per question and leave other questions as they were. Nothing
carries over to a question outside the data: held out of a mix of the
other four, OpenBookQA stayed at the base's score on d3-edge and lost
5.0 points on Laya. So put every question a fitted model should keep
into its mix. A `finetune`, `head` or `lora` fit says so in a warning.

### `data`: datasets to training files

```bash
mlx-decision data SOURCE -o FOLDER [options]     # a dataset folder
mlx-decision data SOURCE -o FILE.jsonl [options] # one file
mlx-decision data SOURCE --show 3                # preview: three lines and the label counts
```

SOURCE is `hf:ORG/NAME` (a Hugging Face dataset, with `--subset`,
`--split` and `--revision`), a `.csv`, `.jsonl` or `.parquet` file, or a
folder written by the `datasets` library's `save_to_disk`. Everything
except CSV and JSONL files needs the `training` extra. A dataset with
several subsets (configs, in the `datasets` library) or splits must be
told which one to use; the error lists them. `--limit N` takes the
first rows, `--sample N --seed S` draws rows at random (in the
dataset's order). The same flags always write the same
file.

Several splits in one run: repeat `--split`; `-o` then names a dataset
folder, with one file per split. `--sample` is given once (for every
split) or once per `--split`, in the same order; `0` takes a whole
split. The folder is written whole: when a split fails, or the run is
stopped, nothing is left.

```bash
mlx-decision data --preset sst5 --split train --split validation \
  --sample 2000 --sample 0 -o sst5
```

`--roles` says what each split is for, one entry per `--split`: `train`,
`report` or `evaluate`. By default the names decide: `train` trains, a
`validation` (or `dev`) split reports, `test` evaluates when another
split reports and reports otherwise. A split can also be cut into
parts, each a role with its share, joined by `+`; the parts are drawn
with `--seed`, keep the labels' proportions and are written as split
files of their own (`train`, `validation`, `test`):

```bash
# One file, cut three ways
mlx-decision data tickets.csv --roles "train:0.8+report:0.1+evaluate:0.1" -o tickets \
  --question "Which team handles this ticket?"
# train and validation on the Hub, no test: half of validation evaluates
mlx-decision data hf:zeroshot/twitter-financial-news-topic --split train \
  --split validation --roles "train,report:0.5+evaluate:0.5" -o tweets
```

Each dataset row becomes one example with one question. The *shape*
says how columns map onto it. It is guessed from the column names and
types (a class-label column gives the label names); `--shape` and the
column flags override the guess. `object` is never guessed: naming its
`--columns` picks it. When a guess is ambiguous, the command
stops and lists the columns.

| Shape | Columns | The question | Example |
|---|---|---|---|
| `text` | `--text` → the state, `--label` → the answer | a `choice` with one option per class, or a `noul` with `--noul` (two classes, the second is yes) | emotion, TREC |
| `pair` | `--premise` and `--hypothesis` → the state as `{"premise": ..., "hypothesis": ...}`, `--label` | a `choice` (e.g. entailment / neutral / contradiction) | SNLI, ANLI |
| `choice` | `--choices` → the options, `--answer`, `--context` (if any) → the state | a `choice` per row with the row's options | ARC, OpenBookQA |
| `yesno` | the passage → the state, `--question-column` → the question, a bool label | each row's own `noul` | BoolQ |
| `rating` | `--text` → the state, a numeric `--label` | a `score` with one level per value (`--levels "bad,ok,good"` describes them) | yelp_review_full |
| `object` | `--columns A,B` → the state as an object of those columns, `--label` | as `text` | MMLU (question and choices) |
| `image` | `--image` → the line's image, `--label`, `--text` (if any) → the state | as `text` | Oxford-IIIT Pet, cats_vs_dogs |

**Images.** The `image` shape takes a Hugging Face `Image` column, or a
column of image file paths in a CSV or JSON-lines file (relative to the
file). It writes a dataset folder only: each row's image goes into
`images/` (`images/train-00000.jpg`). PNG, JPEG and WebP files are kept
as they are, and other formats are converted to PNG. A text column
becomes the state only when it is named as one (`caption`,
`description`, `text` and the like, or `--text`); otherwise the state is
empty and the model reads the image and the question alone. Rows whose
image is missing, unreadable or too large are left out and counted.
`--show` lists each line's image with its size and type, and the
summary the images' count and total size.

```bash
mlx-decision data hf:timm/oxford-iiit-pet --split train --split test -o pets \
  --question "Which breed is the animal in the photo?"
```

The question's wording has no default. Without `--question`, `-q` or a
preset, the command stops and names these flags. In a terminal, it asks
instead.

- `--question TEXT`: the instructions.
- `--id ID`: the question id (default: the dataset's name).
- `--options "id=description,..."`: the option ids and their
  descriptions (default: the label names). Use `\,` for a comma inside a
  description.
- `--label-map "0=negative,1=positive"`: raw labels to option ids.
- `-q FILE`: the whole question as JSON, as in `run`.

Rows without a usable label are left out and counted in the summary.

```bash
mlx-decision data hf:dair-ai/emotion --subset split --split train -o emotion.jsonl \
  --question "Which emotion does the text express most?"

mlx-decision data hf:Yelp/yelp_review_full --split train --sample 5000 \
  --shape rating --question "How positive is this review?" \
  --levels "very negative,negative,neutral,positive,very positive" -o yelp.jsonl
```

**Presets.** `--preset trec|offensive|sst5|openbookqa|anli-r3` (with
`--split`) gives the five datasets of the accuracy benchmark's coverage
suite, with its wording and label maps, pinned to the commits it was
measured on. These are the datasets of the table above. Their splits:
the dataset's own (`train`, `test`; ANLI's `train_r3`, `dev_r3` and
`test_r3`). `--preset typed-decisions` gives
[typed-decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions)
(splits `train`, 1,200 cases, and `test`, 400), its requests as they
are, five questions over four workflows, with soft labels from its gold
distributions.

**Menus.** Without a source in a terminal, `data` asks step by step,
with its guesses preselected: the source (presets, a Hub dataset, a
file), subset and split, shape, columns (with each column's type and a
sample value), label names, the question (the same builder as `chat`)
and the output file. It then shows the preview, asks for confirmation
and finally prints the equivalent command. `--no-input` turns the menus
off.

### A teacher's labels: `training label`

A larger model's answers can label data for a smaller one
(distillation): the teacher answers every row, and its probabilities
become the rows' labels. Rows need questions, not labels, so a pool of
unlabelled rows works too.

```bash
mlx-decision data --preset sst5 --split train --sample 3000 -o sst5
mlx-decision training label -m perplexity-ai/pplx-decider-flash-v1.2 --data sst5
mlx-decision training fit -m convaiinnovations/laya --data sst5-pplx-decider-flash-v1.2
```

`training label` reads any dataset `training fit` reads (its train
split, `--split` another; mixes too) and writes a new dataset folder
(`-o`, by default `<data>-<teacher>`) with one train split. The labels
are each answer's probabilities, as [soft labels](#format); `--hard`
writes the top answer instead. `--threshold P` leaves out answers less
sure than a [confidence](STUDIO.md#running) of P, and a row with none
left is left out; `--sample N` (with `--seed`) labels N rows. When the
rows have labels of their own, they are not kept, but the teacher is
scored on them: the report shows its accuracy and ECE there. The
folder's `dataset.json` records the teacher (and for a fitted one its
base, kinds and calibration) in `source`. A fitted teacher answers with
its fit. Progress, `--events` and Ctrl-C work as in
[`training fit`](#progress-events-cancel). In the Studio, a batch's
Save Answers as a Dataset does the same ([Studio](STUDIO.md#batches)).

What it buys, on SST-5 with pplx flash (60.5% on the test split) as the
teacher, students fitted on 3,000 rows and scored on the 2,210 test
rows (accuracy, %):

| Student | the dataset's labels | teacher, top answers | teacher, probabilities |
|---|---|---|---|
| Laya `finetune` | 55.3 | 55.7 | 57.4 |
| d3-edge `lora` | 52.6 | 53.0 | 54.5 |
| d3-edge `readout` | 51.1 | 50.5 to 52.0 | 50.5 to 52.0 |

The probabilities beat the top answers by about 2 points for the
students that change weights (`finetune`, `lora`); a readout fit stays
at its own plateau whatever labels it gets. On all 8,544 rows the
teacher's probabilities were worth about what the dataset's labels are
(Laya 59.5 against 58.2): a teacher labels an unlabelled pool about as
well as people would, but does not lift a student past its own limit.

**The student inherits the teacher's calibration.** Its closing
calibration is fitted to the teacher's labels, so a teacher whose
probabilities are off passes that on: pplx flash as released (ECE
13.5%) gave its students ECEs of 11 to 16%, the same teacher after a
readout fit (ECE 2.7%) about 2%. So fit the teacher first, a
calibration alone takes seconds, and label with the fitted model.
`training label` and the Studio say so in a note when a teacher's
calibration is poor (an ECE above 5% on the data's labels or its fit's
report) or unknown (a release, or a fit without a closing calibration
for the question).

## Fitting

```bash
mlx-decision training fit -m BASE --data FILE.jsonl [-o FOLDER] [--method KIND]
```

BASE is any model `-m` takes: a folder or a repo id, in any precision. It
can also be a fitted model; its fit is then replaced and its base kept.
The output folder defaults to `<base>-<data name>` in the current folder;
an existing one is replaced only with `--overwrite`.
Progress goes to stderr and the report to stdout. The report shows, per
question, accuracy, macro-F1, ECE and log loss (`score` questions also
mean absolute error) before and after, the kept epoch and the time
taken; for a [mix](#mixes-of-datasets), a table per source follows. It
is also saved in the folder as `training.json`.

### Splits

A fifth of the data (`--holdout 0.2`, stratified by label, seeded by
`--seed`) is held out. It only scores the report: nothing is tuned on it.
`--validation FILE` replaces the holdout with a file of its own. The rest
is the training part. A tenth of it (at least 50 examples, at most half)
is the *dev* part, which picks the readout fit's settings and the best epoch and fits
the closing calibration. The rest is what the weights are trained on.

Fewer than 50 training examples, or fewer than 10 for an option, is a
warning. The fit still runs, but its numbers are noisy.

### Kinds

`--method auto`, the default, picks the family's own kind and follows it
with a calibration ([The closing calibration](#the-closing-calibration)):

| Family | `auto` | What is trained | Written to the fitted folder | Also available |
|---|---|---|---|---|
| Clef | `head` | the decision head on top of the frozen backbone | the head (clef-flash: 244 MB) | `calibrate` |
| pplx, d3, Base Labs, H2O-Lightning, d1 LFM3, d1 | `readout` | each option's readout row and a bias, on the frozen network | rows and biases (kilobytes) | `calibrate`; `lora` for d3 and d1 LFM3 |
| Laya, Julia | `finetune` | the whole encoder and the marker head (`--head-only`: the head alone) | the encoder and head (Julia 0.3 GB, laya 0.8 GB) | `calibrate` |

- **`calibrate`** (every family): one temperature per question, and one
  bias per option when the question is asked the same way on every line
  and has enough examples ([The closing calibration](#the-closing-calibration)).
  It is fitted to the model's probabilities by minimum log loss, in under
  a second after one scoring pass. Accuracy changes only through the
  biases. The probabilities then match how often the model is right.
- **`readout`** (pplx, d3, Base Labs, H2O-Lightning, d1 LFM3, d1): these models read an
  answer from the last hidden state through one row per option (of the
  language model's output layer, or pplx's and d3's own readout). The fit
  trains those rows plus a bias per
  option, starting from the release's rows. The hidden states are
  computed once per example and kept. The learning rate (1e-5, 1e-4,
  1e-3) and the pull towards the release's rows (L2 0 or 0.01) are picked
  on the dev part, over up to 400 full-batch steps. A question whose
  release rows are best on dev is only calibrated.
- **`head`** (Clef): the backbone runs once per example and its hidden
  states are stored on disk, beside the output folder. Their size (2.6
  MB per ANLI-R3 request, about 9 MB per photo of 500x375 pixels on
  clef-flash) is shown before the run and checked against the
  free space. The folder is removed when the run ends, also on errors and
  Ctrl-C. The joint head trains on those states in float32: AdamW 5e-5,
  weight decay 0.01, 5% warm-up and a linear decay, 16 requests a step.
- **`finetune`** (Laya, Julia): AdamW 2e-5 (head alone 1e-4) with the
  same schedule. Batches hold up to 4,096 tokens and 32 rows. Compute is
  in bf16 with float32 master weights and moments, and activations are
  recomputed per layer (gradient checkpointing). The fitted weights are
  stored in bf16.
- **`lora`** (d3, d1 LFM3; never `auto`'s pick): adapts the network
  itself. Small low-rank adapters (LoRA) go on the MLP projections of
  every decoder layer, and train together with each question's readout
  rows and option biases (starting from the release's rows, or from a
  fitted base's readout fit). The loss runs through the model's own
  prompt and readout, as it answers; the vision tower runs once per
  image and stays frozen. AdamW 1e-4 (`--learning-rate`; above about
  3e-4 the fit breaks down), a short warm-up and a linear decay; rank 8
  (`--rank`; the adapters' scale is 16 / rank, so the learning rate
  stays right when the rank changes). Text examples go in batches of up
  to 4,096 tokens and 16 rows, examples with images one at a time with 8
  summed per step; activations are recomputed per layer. Where a
  readout fit takes seconds after its scoring pass, LoRA takes minutes:
  about 0.12 s per photo on d3-edge and 0.35 s per Doom frame on d1-2B
  per epoch on the M5 Pro above. It needs an unquantized base (a
  quantized copy is refused; fit the model it was converted from).

The trained kinds use cross-entropy with label smoothing 0.1 plus a Brier
term (against a soft label's distribution: no smoothing); the readout fit
uses the cross-entropy alone. `head` and
`finetune` train for 3 epochs, `lora` for 2 (`--epochs`). After each epoch they score
the dev part, keep the epoch with the lowest dev loss, and stop when an
epoch is worse than the best one before it. `--no-early-stop` trains
every epoch (the best is still kept).

A family's other kinds are refused with a message naming the ones it
takes. With `--seed` fixed, `calibrate` and `readout` give the same
fitted model again. Fine-tuning and the head can differ by a point or so
between runs.

### The closing calibration

A fit ends with a calibration, fitted on the dev part: a temperature per
question, which makes the probabilities match how often the model is
right without changing which answer wins, and, where the question is
asked the same way on every line, a bias per option, which can change
answers. `--calibration` picks it:

- `auto` (the default): the temperature alone after `readout`, `head` or
  `finetune`. For `calibrate` alone, the biases too when the training
  part holds at least 10 examples of every option; otherwise the
  temperature alone.
- `full`: a temperature and a bias per option, as every fit had before
  1.3.
- `temperature`: no biases.
- `none`: no closing calibration; the fitted model answers as the
  trained kind leaves it. For a program whose thresholds were tuned on
  the base's probabilities, which a calibration would move. Not with
  `--method calibrate`, which is the calibration.

After a trained kind the biases add little and can cost: they are
fitted on the dev part, a tenth of the training part, so on a question
with many options each bias rests on a few examples, while the readout
or head has already learned the options' balance from many more. On
Oxford-IIIT Pet (37 breeds, about 10 dev photos per breed) d3-edge's
readout fit lost 2.9 points of accuracy to its biases; with the
temperature alone it lost none, and its ECE was 3.5 against 6.4. A calibration
alone is where the biases pay: they correct a base that leans towards
some options (on the same photos, +14.5 points from 10 examples per
breed). The report says what was used (`calibration (auto): breed
temperature`), and `training.json` records it.

### Calibration needs data like the target

A fit's calibration is fitted on the dev part, a slice of
your training data. It can only correct the model's confidence as it is
on that data. If the model finds the training data much easier (or
harder) than what it will be asked later, the calibration corrects in
the wrong direction.

ANLI-R3 shows this. Clef answers 87% of its train split right but only
50% of its test split. A calibration fitted on the train split made Clef
more confident (temperature 0.89), and on the test split ECE rose from
20.9 to 23.0. Fitted on 500 rows of the `dev_r3` split, which is like
the test split, it fell to 2.8. A head fit on train rows shows the same
effect: its calibration, fitted on train-like dev rows, raises ECE on
the test split (table above).

So use data drawn from the same source as the requests the model will
answer. When a public dataset's train split differs from your use, fit
the calibration (`--method calibrate`) on a sample that does not.

### Memory and time

Before loading, `training fit` estimates the peak memory: the model's
weights plus what the kind keeps beside them (optimizer state,
activations, kept hidden states). It shows the estimate and checks it
against this Mac's GPU working set, the same check the other commands
make, which is about two thirds of the memory on a 16 GB Mac. A run over
it is refused, and the message names smaller runs that would fit
(`--head-only`, `--method readout` for a LoRA fit, `--method
calibrate`) or a quantized copy to convert first. A head fit also shows
the disk its stored states need. A LoRA fit counts the adapters with
their optimizer state, one recomputed layer of its longest example and
the vision tower's rows of every training image, which it keeps for the
run (about 0.3 MB per pet photo on d3-edge, 1.2 MB per Doom frame on
d1-2B).

Peak memory of the `auto` runs, estimated by `training fit` for these
datasets (measured, where the run above was made), and the smallest Mac
whose working set holds it:

| Model | Peak | Mac |
|---|---|---|
| clef-flash, 4-bit copy | about 10 GB (`calibrate` 7 GB) | 16 GB (at the edge) |
| clef-flash, 8-bit copy | 15.1 GB (`calibrate` 11.9 GB) | 24 GB |
| clef-flash, bf16 | about 24 GB (`calibrate` 21 GB) | 48 GB (`calibrate` 32 GB) |
| pplx v1.1 or v1.2 (27B), 4-bit copy | about 17 GB | 32 GB |
| pplx v1.1 or v1.2 (27B), 8-bit copy | about 31 GB | 48 GB |
| d3-flash, 8-bit copy | about 10.6 GB | 16 GB (at the edge) |
| d3-flash, bf16 | about 18.8 GB | 32 GB |
| d3-mini (bf16) | about 10.3 GB | 16 GB (at the edge) |
| H2O-Lightning-4B, 8-bit / 4-bit copy | about 6 / 3.5 GB | 16 GB |
| H2O-Lightning-4B, bf16 | 10.6 GB | 16 GB (at the edge) |
| d1-3B (bf16) | 6.1 GB | 16 GB |
| laya, laya-typed-decisions | 8.9 GB (`--head-only` 1.4 GB) | 16 GB |
| laya-multilingual | about 10.2 GB (`--head-only` 1 GB) | 16 GB (at the edge) |
| Julia-1 | 4.8 GB (`--head-only` 0.4 GB) | 16 GB |

The clef-flash 4-bit, the pplx and the d3 fits have not been measured. Clef 27B
takes the same kinds as clef-flash; on its 8-bit copy the head needs
about 36 GB (a 64 GB Mac), also unmeasured.

Time depends mostly on the Mac's chip, so `training fit` does not
estimate it before the run. During the run, the progress bar shows the
time left, measured from the steps so far. As a guide, on the M5 Pro
above (see the table at the top):

- Scoring: about 0.04 s per example for d1, 0.14 s for H2O-Lightning,
  0.26 s for clef-flash (8-bit) on ANLI-R3's long premises. A readout fit
  scores every example once and then trains in a few seconds.
- Head: about 40 s per epoch over 1,440 ANLI-R3 requests, after the
  scoring pass that stores their states.
- Fine-tuning: Julia about 40 s per epoch over 3,900 TREC questions,
  laya about 5 minutes per epoch over 8,600 tweets. `--head-only` is
  several times faster but gains less.
- Calibration alone: one scoring pass.

A Mac with fewer GPU cores, or an older chip, takes longer: on a base M1,
expect several times these figures.

### Progress, events, cancel

The progress bar runs on stderr. `--events` prints the progress as JSON
lines on stdout instead, one per event, for other programs to follow:

```json
{"stage": "check", "metrics": {"examples": 600, "holdout": 120, "fit": 430, "dev": 50, "labels": {...}, "kinds": ["calibrate"], "estimate": {"peak_gb": 0.9}}, "message": "loading laya"}
{"stage": "before", "step": 3, "total": 120, "eta_s": 9.5}
```

The stages are `check`, `before`, `features`, `train`, `calibrate`,
`after`, `write` and `done`; a run goes through those its kinds need. A
run that fails ends with `{"stage": "error", "message": ...}`, a
cancelled one with `{"stage": "cancelled"}`. The lines' keys and the
three final stages (`done`, `error`, `cancelled`) are stable; the stages
before them and what `metrics` holds are provisional
([Stability](STABILITY.md#provisional)).

Ctrl-C stops the run at its next step: the next example scored, the next
training batch, or the next 20 steps of a search. A second Ctrl-C stops
it at once. A stopped run writes nothing: there is no half-written
folder.

A program that runs `training fit --events` can have the run stop when
the program ends: it sets `MLX_DECISION_WATCH_STDIN=1` in the run's
environment, gives it a pipe as stdin and never writes to it. When the
program ends, even when it is killed, the pipe closes and the run stops
as after Ctrl-C; a run still going 4 seconds later (in the middle of a
long step) removes what it had written so far and exits at once (code
130). The Studio does this for its jobs. Without the variable, or with
a terminal, `/dev/null` or a file on stdin, nothing is watched, so a
wrapper that closes stdin at once (`conda run`) does not stop the run.

### Menus

Run in a terminal without `--method` (or without the data, base or
output), `training fit` asks for what is missing. It offers the kinds
the family takes, each with its estimated peak memory (marked when it
does not fit), then the holdout share or a validation file, the epochs
and the output folder. It then shows a summary with the equivalent
command line and asks before starting. `--yes` takes the defaults, and
`--events` never asks.

## Fitted models

A fitted model is a folder:

- `mlx_decision.json`: the family, the base (a Hugging Face repo id at
  its commit, or an absolute path), a fingerprint of the base and the
  algorithm that made it, the kinds, the questions with their options,
  the metrics, the seed and the data file's hash.
- `fit.safetensors`: the trained tensors only. The base's weights are
  not copied. A LoRA fit's adapters are fused into the base's weights
  when it loads, so it answers as fast as the base.
- `training.json`: the full report and the options of the run.

It loads anywhere a model does, and its name is the folder's name:

```bash
mlx-decision run -m julia-trec -s "Who wrote Hamlet?" -q trec/dataset.json
mlx-decision chat -m julia-trec
mlx-decision server -m ~/Models          # lists fitted models beside their bases
mlx-decision mcp -m julia-trec
```

```python
import mlx_decision

model = mlx_decision.load("julia-trec")
model.info()["fit"]      # kinds, questions, metrics, seed
```

Through the server, the Jev SDK picks a fitted model with
`model="julia-trec"`.

**The base must not change.** Loading resolves the base (a repo id from
the Hugging Face cache, downloaded if needed, at the recorded commit)
and compares its fingerprint: its config files and the names, shapes and
types of its weights. A missing, replaced or re-quantized base is
refused with a message naming it. A fit made on a quantized copy needs
that copy; fit again to use another.

**Fits from other versions.** A fitted model written by mlx-decision 0.9
or later loads in every 1.x release. One written by a newer release that
uses something this one does not know (a new kind of fit, tensors it
does not read, another way of fingerprinting the base) is refused with a
message naming it, instead of being answered without its fit; upgrade
mlx-decision to load it.

**Fits on unchanged weights share their base.** A calibration or readout
fit changes only probabilities and readout rows, so the server loads the
base once for all its fitted models of these kinds. Each such fit adds a
few kilobytes. A head or fine-tune fit loads its own copy of the base,
with its trained weights.

**Which questions a fit applies to.** Calibrations and readout rows
apply to a question with the same id, type and options as in the
training data. Other questions are answered without them: by the base
for a calibration or readout fit, and by the trained weights without
their calibration for a head fit or fine-tune. `run` and `chat` warn
once per such question, the server and MCP log it, and the MCP tool
description names the fitted questions. A fine-tuned encoder or a
trained Clef head applies to every question. A fine-tune on one task
can make the model worse at others, so check the questions you rely on
with `training eval`.

H2O-Lightning's yes/no floor still applies after a fit, and every
family keeps its input limits and checks. An H2O-Lightning fit answers
in the [reading](MODELS.md#h2o-lightning) its base was read in when it
was made: two-way for v1.4 unless the fit was made one-way (in Python,
`fit(..., reading="one-way")`), one-way for v1.2.3. Loading it takes that
reading by itself; asking for the other one (`--reading`) is refused.

## Evaluating

```bash
mlx-decision training eval -m julia-trec --data trec --compare SupersonicLabs/Julia-1
```

```
answer_type (choice, 500 examples)
             julia-trec     Julia-1
  accuracy        96.6%       16.0%
  macro-F1        96.2%       14.0%
  ECE              1.3%       63.1%
  log loss        0.134       7.649
  confusion, julia-trec (rows: gold, columns: predicted)
  ...
```

For each labelled question, `training eval` shows accuracy, macro-F1,
ECE and log loss (`score` questions also mean absolute error), refused and cut
examples when there are any, and a confusion matrix. `--compare` scores
a second model after the first; only one is loaded at a time. `--json`
prints the results as JSON. `--events` prints the progress as JSON
lines instead (stages `load`, `score` and `done`, whose `metrics` hold
the results as `--json` prints them), and, with
`MLX_DECISION_WATCH_STDIN=1`, stops when a pipe on stdin closes, as
`training fit` does. Evaluation needs no extra.

Scores from `training eval` are measured on data the fit has not seen
only if you give it such data: use a test split or your own held-back
examples, not the training file.

## Python

```python
import threading

from mlx_decision.training import evaluate, fit, label

stop = threading.Event()            # stop.set() from another thread cancels

result = fit(
    "SupersonicLabs/Julia-1",
    "trec",                        # a dataset folder (trains on its train split,
                                   # reports on its test split), a file, or a
                                   # list of data.format.Example
    output="julia-trec",
    method="auto",                 # calibrate, readout, head, finetune, lora
    seed=0,
    epochs=None,                   # the kind's default: 3, lora 2
    on_event=lambda event: print(event.as_json()),
    cancel=stop.is_set,
)
print(result.metrics["answer_type"]["after"])

report = evaluate("julia-trec", "trec", compare="SupersonicLabs/Julia-1")  # its test split
print(report.as_json())

taught = label(
    "julia-trec",                  # the teacher
    "trec",                        # rows with questions; labels optional
    output="trec-taught",
    labels="soft",                 # or "hard": the top answers
    threshold=None, sample=None, seed=0,
)
print(taught.examples, taught.measured, taught.note)
```

`fit` raises `Cancelled` when cancelled (nothing written),
`mlx_decision.DataError` for a bad training file (with its `line`),
and `mlx_decision.ModelTooLargeError` when the run does not fit (both
are `DecisionError`s). Further
options: `validation=` and `holdout=` (as `--validation` and
`--holdout`; with neither, data without a report split holds back a
fifth),
`calibration=` (as `--calibration`), `learning_rate=` and `rank=` (lora),
`head_only=`, `early_stop=`, `overwrite=` and
the loader's options (`max_input_tokens=`, H2O-Lightning's `reading=`).
On a [mix](#mixes-of-datasets), `result.sources` holds the report per
source (`held_out`, and per question `before` and `after`), and each of
`evaluate`'s models has `sources` (per source and question, its
metrics). `label` takes `split=`, `overwrite=`, `on_event=`, `cancel=`
and the loader's options too, and raises `FileExistsError` for an
output that exists (without `overwrite=True`).

## Not in this version

- LoRA for Clef, pplx, Base Labs, H2O-Lightning and d1 (their networks
  stay frozen, and only the head or the readout rows train), and full
  fine-tuning of any of the LLM-based families.
- LoRA on a quantized base, or on the attention and DeltaNet projections.
- Videos in training data.
- Multi-label tasks (several correct options for one question).
- Generating training data: a teacher labels rows that exist
  ([`training label`](#a-teachers-labels-training-label)), it does not
  write new ones.
- Publishing fitted models to the Hugging Face Hub. A fitted folder
  refers to its base, so share it together with the base's repo id.