# Release classification Before Questarr grabs a release unattended, it can check that the release is the game and not a patch, a DLC, a crack, a demo or a soundtrack. This page describes the layers, how to pick a mode, how to install the local model and what changes for existing installs. ## Layers ```text release name → rules deterministic, always on, no model → local model tf-idf + logistic regression, in-process, works offline (hybrid/local) → cloud model TypeSafe Jev or a compatible model, only if still unresolved (hybrid/cloud) → classification (full_game | patch | dlc | unknown, plus flags such as demo or crack_only) ``` Classification is separate from download authorization. The pipeline only answers "what is this?". The auto-download hold in `server/cron.ts` decides what to do with the answer. A release that no layer can classify is `unknown`. It is never turned into `full_game`. ## Settings page Everything below can be changed in Settings > Discovery & Downloads > Release classification, without restarting: the classification type, the local model confidence, holding unclassifiable releases, and the active-learning options (collection, cloud suggestions, the low-confidence threshold and the candidate limit). The card also shows whether a local model is installed (and its version) and whether a cloud provider is configured. **An environment variable wins.** When one of the variables in this page is set, the matching setting is shown read-only with the variable's name, and the API refuses to change it. Unset the variable to manage the setting from the page. The low-confidence threshold and the candidate limit have no variable: only the page changes them. Changing the type or the local confidence rebuilds the pipeline and clears its cache at once. ## Modes (`CLASSIFICATION_MODE`) | Mode | Behaviour | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `hybrid` (default) | Rules, then the local model. The cloud provider is asked only when the local model abstains (below its threshold), fails, times out or is not installed. The cloud model decides alone: it does not see the local result, and it can still answer `unknown`. | | `local` | Rules, then the local model. No request ever leaves the server. | | `cloud` (legacy) | The historical Jev-only check, unchanged. Without a TypeSafe key nothing is checked. Needs a TypeSafe key, so it is not the default. | The cloud provider is the TypeSafe client already configured in Settings. Its model id (`typesafe.model`, for example `jev-latest` or an OpenRouter id) is the only thing that changes to use another compatible model. ## One pipeline for search and auto-download The manual search results go through the same pipeline as the auto-download check. Each result keeps the existing categories (Main, Update, DLC, Extra), but its type now comes from the rules, the local model and, in `hybrid` mode, the cloud layer. So the type you see in a search is the one auto-download would act on. In `hybrid` and `cloud` mode only the top 15 results of a search are sent through a layer that may call a provider, to bound the calls; `local` mode types every result. In `cloud` mode the search keeps the Jev-only call it always made. ## What the auto-download hold does In `local` and `hybrid` mode an unattended download is held for review when the pipeline says: - `patch`, `dlc` or `expansion`, or - a flag `crack_only`, `demo`, `soundtrack` or `other_artifact`, or - (cloud layer only) the file size is judged implausible. A result of `unknown` goes through, as it does today. Set `CLASSIFICATION_HOLD_UNKNOWN=true` to hold those too. That is a behaviour change, so it is off by default. Any error in the pipeline lets the download proceed, like the existing Jev check. ## The local model A model trained on 964 labeled releases ships with the application, in `shared/models/release-classifier.json` (about 1.4 MB of plain JSON, also inside the Docker image). Nothing needs installing: `hybrid` and `local` mode use it out of the box. ### Performance In short: out of 10 releases the local model puts about 8 in the right category (full game, patch, DLC or unknown), against about 6 for the keyword rules alone. Its accuracy is close to the cloud model's (the cloud figure comes from an earlier split), runs in under a millisecond and needs no account. How it was measured: the 964 labeled releases (123 games, including console releases, software, movies and series) were split by game, so the model is tested on games it has never seen. 187 test releases were scored from the release name alone. The shipped file is retrained on all 964 releases, so it has no test score of its own; these numbers come from the same method on the split. The full method and the other candidates are in [`scripts/release-classifier-training/DECISION.md`](../scripts/release-classifier-training/DECISION.md). | Layer | Accuracy | Macro F1 | Time per release | Size | | ------------------------------------ | -------: | -------: | ----------------: | ------- | | Rules only | 55.6% | 49.7% | 0.06 ms | none | | Local model (tf-idf + logistic reg.) | 79.1% | 75.1% | 0.06 ms | ~1.4 MB | | Cloud model (Jev, for comparison) | 79.5% | 74.8% | ~840 ms (network) | none | The cloud row comes from the earlier 88-release test and was not re-measured on the new split. - **Accuracy** is the share of releases given the right category. - **Macro F1** also scores each category separately and then averages them, so a model cannot look good just by getting the most common category right. It is also a percentage, and higher is better. For example, always answering "patch" can score a decent accuracy but a low macro F1. - **Cross-validation** repeats the test five times on different groups of games. On the earlier 459-release dataset it gave 78.9% for the local model, give or take 10.9 points depending on which games are tested. With this little data, differences under about 5 points are not significant. Because the cloud figure comes from a different split, these numbers do not show that the two models are equivalent. - **The 70% confidence threshold** is how sure the model must be before its answer is used. Below it, the release is passed on to the cloud model (`hybrid`) or left `unknown` (`local`) instead of being guessed. - **The main mistake** is a full game read as a DLC; patches and unknown releases are the most reliable categories. More labeled games is what improves it most; the earlier 459-release evaluation showed accuracy still rising with its 49-game dataset. To use your own, for example after retraining on more data (the dataset is not published), generate the file and place it where the server looks first: ```powershell npx tsx scripts/release-benchmark/export-model.ts --dataset releases-labeled.csv --out release-classifier.json ``` Copy the exported file to `/models/release-classifier.json`, or keep it anywhere and set `CLASSIFIER_MODEL_PATH` to its location. Lookup order: `CLASSIFIER_MODEL_PATH`, then `/models/release-classifier.json` (in Docker: `/app/data/models/`), then the bundled model. Restart after changing the file. The file is validated on load; a missing or damaged one is logged and the next in the list is tried. To replace the bundled model for everyone, overwrite `shared/models/release-classifier.json` in a pull request. ### Share your dataset The bundled model is only as good as the 964 releases it learned from (123 games), and more labeled games is what improves it most. If you have a dataset of your own, for example the verified labels exported from **Settings → Release classification**, you are invited to share it so the public classifier can be retrained on more data. Two ways, both your choice: - **From the app (opt-in).** In the review card, tick **Share my verified labels automatically** (off by default). While it is ticked, a task ("Label sharing", every 6 hours, visible in System Tasks) sends the labels you verified since the last run. **See what will be sent** shows the exact rows, and **Send now** sends them without waiting. Only verified rows go, and each carries four fields and nothing else: the release name, the game title, the label you chose and the version of the local model. No account, path, address or library data. Rows already sent are not sent again, and a request carries at most 200 rows. The address is fixed in the code and the request needs no key; it follows no redirect. If the collection point cannot be reached, the rows stay unsent and the next run tries again. Unticking the box stops all sending; rows already received cannot be recalled from the app. - **By hand.** Open an issue on [Doezer/Questarr](https://github.com/Doezer/Questarr/issues) and attach the CSV from **Export verified**. Nothing is shared unless you tick the box. A shared label is a proposal: the maintainer only trains on a class that several independent senders agree on, or after checking it, and never retrains automatically. Only release names and their labels are needed, so with the manual route remove anything you do not want to publish before attaching it. Run `scripts/release-benchmark/run.ts` on a held-out split before trusting a new model (see `scripts/release-benchmark/README.md`). Do not expect a small model to reach high automatic precision: that is why the local layer abstains below `CLASSIFIER_LOCAL_THRESHOLD` (0.7) and why `unknown` is never turned into `full_game`. Whether an `unknown` release is downloaded is the hold's decision, not the classifier's: it goes through unless `CLASSIFICATION_HOLD_UNKNOWN=true`. ## Active learning: label the releases the classifier is unsure about Off by default. When `ACTIVE_LEARNING_COLLECT=true`, the unattended auto-download check keeps the releases it could not classify with confidence, so you can label the ones that teach the model the most. Behaviour is unchanged when it is off: nothing is collected, no extra request is made and the review card does not appear. What is collected (only in `local` and `hybrid` mode, and never while a layer failed): | Reason | Meaning | | -------------------- | ----------------------------------------------------------------------------- | | `unresolved` | No layer could say what the release is. | | `low_confidence` | A layer decided, but below 85 % confidence. | | `disagreement` | The rules and the local model's best guess name different classes. | | `model_disagreement` | The cloud model answered a different class than the local model's best guess. | Each release is stored once per game (case and spacing ignored) with its name, game, platform, size, what each layer answered, and a count of how often it came back. A release you already reviewed is never reset to pending. At most 5,000 are kept. **Four kinds of label stay apart.** Rule output and the local model's guess are recorded as signals for you to read. A cloud suggestion is stored with its model, confidence and time and is shown as "not verified". Only a label you give in the review card (status `verified`) is a training example. ### Review Settings > Discovery & Downloads > Release classification review lists pending candidates, most useful first: unresolved and disagreeing releases come before low-confidence ones, and a release whose likely class has few verified examples gets a bonus so the thin classes fill up. For each one you can verify a label (the local model's guess or the cloud suggestion is prefilled, not applied), mark it ambiguous, or mark it as not an example. Ambiguous and rejected releases are never exported. Every time the waiting list reaches a multiple of 25 releases, each user gets one notification, "Releases need your review". Clicking it opens this card. `ACTIVE_LEARNING_CLOUD_SUGGEST=true` adds an "Ask the cloud model" button. It sends one release name (with its size and platform) to the configured provider, costs what one Jev call costs, and only when you press it. If its answer differs from the local model's best guess, the release is flagged `model_disagreement` and moves up the list. In `hybrid` mode the same flag is set without an extra request when the cloud was already asked because the local model was unsure. It needs a TypeSafe key; without one the button is hidden. If the provider fails, the candidate is left as it was. The API behind the card is under `/api/classification-candidates` and uses the same authentication as the rest of `/api`. ### Export, retrain, re-evaluate Nothing retrains or deploys a model by itself. When you have enough verified examples: ```powershell # 1. Download the verified examples (the "Export verified" button, same file as the API): # release_name,query,human_label,notes # 2. Merge them into your labeled dataset. Duplicates are skipped, a different label for a # release already in the dataset is reported as a conflict and never overwrites it, and # games that appear in the held-out file are not added. npx tsx scripts/release-benchmark/merge-candidates.ts --dataset releases-labeled.csv --candidates verified-candidates.csv --heldout releases-heldout.csv --out releases-labeled.csv # 3. Re-evaluate on the held-out set, then compare with the previous model npx tsx scripts/release-benchmark/run.ts --dataset releases-heldout.csv --classifiers "deterministic,tfidf" # 4. Only if it is better, generate the model and install it (see "Install the local model") npx tsx scripts/release-benchmark/export-model.ts --dataset releases-labeled.csv --out release-classifier.json ``` #### Labels shared by other installs (collector) If you run the label collector (Doezer/Questarr-mgmt#60, repository `questarr-label-collector`), step 1 above is replaced by an export from its database. `consolidate.sql` keeps only labels confirmed by at least 3 different senders with no contradicting majority. In PowerShell, from the collector's folder (this snippet is untested on real data, check the CSV before merging): ```powershell $rows = (npx wrangler d1 execute questarr-labels --remote --file consolidate.sql --json | ConvertFrom-Json)[0].results # Neutralize spreadsheet formulas in any cell foreach ($r in $rows) { foreach ($p in 'release_name','query','human_label') { if ($r.$p -match '^[=+\-@]') { $r.$p = "'" + $r.$p } } } $rows | Select-Object release_name, query, human_label | Export-Csv verified-candidates.csv -NoTypeInformation -Encoding utf8 ``` Then follow steps 2 to 4 above with `verified-candidates.csv`. Nothing from the collector reaches a model by itself: the merge reports conflicts, and you decide what to retrain and ship. Keep the **held-out evaluation set in its own file** and never merge candidates into it: the benchmark splits by game, and a game that is both in the held-out file and in the training file would leak. The merge script enforces that for games already in the held-out file. Growing the held-out file is a deliberate step you do by hand. ### Delete "Delete all candidates" in the card (or `DELETE /api/classification-candidates`, optionally `?status=rejected`) removes the collected releases. Turn collection off in the Settings page (or by unsetting `ACTIVE_LEARNING_COLLECT`); existing candidates stay until you delete them. ## Operations - Timeouts: each model layer is bounded to 10 s, a failing layer counts as "unresolved". - Cache: results are kept in memory for 24 h (2,000 entries). The key includes the pipeline version, the mode, the local model version, the cloud model id and whether a provider is configured, so installing a new model or changing the provider never serves stale answers. Results from a failing layer (error, timeout, unusable answer) are not cached. - Cloud mode: the historical Jev call keeps its own in-memory cache (24 h, 2,000 entries, keyed by release name, size, platform, model and provider settings), so a name seen on every search is asked once. A failed call is never cached. Neither cache survives a restart. - Logs: one `debug` line per release in the `classification` module with the layers consulted, class, confidence, source and model version. - Docker: no new container and no native dependency. The model file ships in the image; an optional one in the data volume overrides it. ## Compatibility - Default `CLASSIFICATION_MODE=hybrid`. It needs no key: without a TypeSafe key and without a model file only the rules run. Releases the rules identify as a patch, DLC, crack, demo or soundtrack are held for review before an unattended download, which the Jev-only check never did without a key. Set `CLASSIFICATION_MODE=cloud` to get the previous behaviour back. - API and UI are unchanged unless `ACTIVE_LEARNING_COLLECT` is set (see above). - Search-result AI badges (`enrichWithAiAnalysis`) still use Jev only, in every mode. - One new table, `classification_candidates` (SQLite and Postgres migrations), used only by active learning. Settings are saved in the existing system configuration (`classification.settings`), so no migration.