# Changelog All notable changes to Ojuri will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] ## [1.8.1] - 2026-09-29 Four instructions that could not be followed and three defects behind them, found by running every documented script against a live stack rather than reading it. ### Added - **`ojuri reset-admin`.** Issues a new admin password and prints it once, with `--password`, `--username` and `--tenant`. It runs inside the RDA container, which carries bcrypt, knex and the connection details. ### Fixed - **The synthetic load generator declined everything it sent.** `npm run seed:load --count N` omitted the trust context the curated demo rows carry (`is_authenticated`, `device_is_trusted`, `account_age_days`, `session_to_txn_seconds`, channel and countries). Every absent field defaults to its least trusted value, so a run reported `accept=0 decline=20` no matter what `--fraud-ratio` said, and the decision mix it printed meant nothing. The generated transactions now carry that context, weaker on the fraud-flavoured ones: at `--fraud-ratio 0.02` a run now accepts, at 1.0 it sends everything to review. - **Both load paths sent the timestamp in seconds** where the predict contract takes milliseconds, so every generated transaction claimed to be from 1970 and every calendar feature derived from it was wrong. `feature-builder.ts` already carried a comment about this exact hazard from a previous occurrence. - **The two install paths could not migrate each other's database.** knex records whichever migration filename it loaded, and the two paths load different ones: the container runs compiled JavaScript from `dist`, a checkout runs TypeScript from `src` through ts-node. So a database created by `ojuri up` refused `npm run db:migrate` from a checkout, and the reverse, with `Error: The migration directory is corrupt, the following files are missing:` followed by all 38 of them. Nothing was corrupt, and nothing in the message suggested what to do. Both paths now retarget the recorded names to their own extension before migrating, which is idempotent and a no-op on a database that has never been migrated. - **The decision replay CLI could not replay anything.** `npm run replay` re-posted each audited transaction under its original `transaction_id`, which the target rejects as a duplicate, so every row came back as a decision called `ERROR` and the comparison it exists to print was always empty. Rows now replay under a `replay--` key, which keeps the original recoverable in SQL, and a rejection is reported with its code rather than flattened into `ERROR`. It also sent the timestamp in seconds where the API takes milliseconds, putting every replayed transaction in 1970 and skewing every feature derived from its age. - **The dashboard's Backtest action printed a command that could not work.** `npm run replay --target --since 24h` names a script that needs a checkout, omits the `--` that npm needs to pass flags through at all, passes a model version where `--target` takes the URL of the deployment to replay against, and passes `24h` where `--since` goes into a SQL timestamp comparison. There is no backtest endpoint behind the menu item; it only ever printed text. It now describes what actually has to happen. - **`ojuri up` told you to run a command that cannot exist.** Three lines of its summary, and the dashboard's own login page, pointed at `npm run reset:admin` — a script in the repository's `package.json`. An adopter who installed with `npx @ojuri/cli up` has no repository, so the one instruction printed for a lost password was impossible to follow. They now name `ojuri reset-admin`, and say that `ADMIN_SEED_PASSWORD` in `.env` is the password on a database this stack created. ## [1.8.0] - 2026-09-29 What an install gives you when you touch nothing. `npx @ojuri/cli up` used to bring up the scoring path and stop there: no dashboard, no drift monitoring, and nothing in the output saying either was missing. It now starts everything except FIA. Nothing on the `docker compose` path changed, and an `ojuri.yaml` you already have is read exactly as written. ### Changed - **Everything but FIA is on by default.** The Sentinel dashboard and MLA now start with the stack. Sentinel was the bigger omission: decisions were going to `decisionAuditLog` and the only way to read them was SQL or the API, on a product whose dashboard is half of what it is. MLA off meant no drift monitoring and no retraining, so the model an evaluation judged was the one it started with. FIA stays off, and is now the only service that is: its language model is a 7.6 GB download and it wants 16 GB of RAM, which is more than an install should spend uninvited. `services.fia.enabled: true` and another `ojuri up` turns it on. - **RDA defaults to one replica under the CLI**, down from three. Three copies of the same Node process on one laptop buys nothing an evaluation can measure and costs three times the memory; raise it when a single process stops keeping up. A bare `docker compose up` still defaults to three, and `RDA_REPLICAS` still overrides either way. - **`ojuri up` prints where to go.** Every URL it started, in one table, with what each is for: Sentinel, the predict endpoint, Grafana, PAA's stats, MLA's. The admin block now names the dashboard it signs you into, and says plainly that Grafana has its own credentials, which the shipped pair still being `admin / admin` made easy to miss. When FIA is off the output says so, and gives the two steps to turn it on. - **Switching a service off in the manifest now removes its container.** Withholding a Compose profile is not enough: `up -d` simply does not mention the container and it keeps running, and `--remove-orphans` leaves it too, because a service in an inactive profile is still a defined service. So turning MLA, FIA or Sentinel off and re-running `ojuri up` left it running, with the manifest and `ojuri status` both saying otherwise. Switching observability off removes Prometheus and Grafana the same way. Volumes are kept, so turning FIA off for an afternoon does not cost the 7.6 GB download. Two things worth knowing. A bundled datastore you have pointed at your own is **named, not removed** — the container holds your data, and that is not this command's to delete. And Compose projects are named after the directory, so `ojuri up` in `~/work/ojuri` addresses the same containers as a bare `docker compose` in a checkout of the same basename: a FIA started there by hand is one the default manifest considers switched off. The output says what it removed. - **Dropped the `sentinel-without-fia` warning.** It described the shipped default once Sentinel came on, and a warning every install earns is a warning everybody learns to scroll past. The manifest comment and the `up` output both say what FIA-off costs. ### Fixed - **Sentinel's drift and retrain controls could not reach MLA.** With the dashboard served from the stack, NGINX routed `/mla/` to `host.docker.internal`, which resolves from inside the container but does not reach a published port back out, so every call the Settings page makes (`drift-config`, `retrain`, `retrain-runs`, `stats`) timed out as a 502. The config that fronts Sentinel now names the `mla` service, as the AWS deploy config already did. The config used when Sentinel is off is unchanged, since that is the layout where MLA typically runs on the host. - **System health showed a red failure for a service that was deliberately off.** Every probe failure was reported as `DOWN` with the message `fetch failed`, which is what Node calls every network error. With FIA off by default, the first health page an operator opened had a red card and a meaningless error on it. A hostname with no DNS record now reads as `UNKNOWN` — "not running in this deployment" — and a real failure shows the underlying cause (`connect ECONNREFUSED …`) rather than the wrapper. - **The compose stack ran MLA's drift check at a threshold nothing else agreed with.** `docker-compose.yml` defaulted `DRIFT_F1_THRESHOLD` to 0.92 while MLA's own default, `docs/ARCHITECTURE.md` and `CLAUDE.md` all say 0.4, and the repo's measured F1 on IEEE-CIS is 0.554. Every windowed check would therefore read as drift (tracked as OJR-05 in `docs/REVIEW_FINDINGS.md`). It was latent while MLA was off by default; it is not any more, so the compose default now matches the documented one. - **`ojuri --version` printed `1.6.0`.** The number came from a constant kept by hand next to the one in `package.json`, and the two had been apart since 1.7.0, so anyone reporting a bug reported it against the wrong version. It is read from the package now, which is the only copy that can be wrong. ### Security - **`ojuri init` generates `MLA_SERVICE_TOKEN`.** RDA accepts it as a bearer credential for `models:register` and `models:set_status`, and `.env.example` ships a published development string long enough to clear RDA's 32-character floor. It was inert while MLA was off by default. With MLA on it would have been a live credential, identical on every install, for registering and activating a model. Existing installs keep whatever their `.env` holds: rotate it by hand if it still starts with `dev-only`. `ojuri validate`, which `up` runs, now says so on every run while it holds: a warning outside production and an error in it, matching how the other published defaults are treated. - **`ojuri init` generates `GRAFANA_PASSWORD`.** Grafana publishes 3001 on every interface and its admin can add a datasource, so `admin / admin` was not a pair to ship either. As with the MLA token, an existing `.env` is never rewritten; the `up` summary prints the credentials while they are still the shipped ones and only the variable names once they are not. ## [1.7.2] - 2026-09-28 Friction found by auditing the paths an adopter actually takes, rather than by waiting for each one to be reported. No migrations, no schema changes. One behaviour change worth reading before upgrading: a decision can now only be reviewed once. ### Fixed - **A retry could be rejected as a changed request.** The idempotency hash was taken over the request body as your client happened to serialise it, so the same transaction sent with its fields in a different order looked like a different request and the retry came back 422 "reused with a different request body" — sending you looking for a field that changed when none had. Any client that rebuilds the body from a map or a database row could hit this. Field order no longer matters; the order of items in a list still does, because a reordered list really is a different request. Cached responses from before the upgrade will not match, so a retry in flight across it is scored again rather than replayed, for one cache lifetime. - **Errors said what went wrong only in prose.** Every failure the platform raises now carries a `code` alongside the message. Three conditions return 409 and four return 503, so the status alone never said what to do next, and the only way to tell them apart was matching the wording of a message that was written for a human reading a log. `duplicate_transaction` and `idempotency_in_flight` are both 409 and mean opposite things. The message wording is not part of the contract; the code is. - **A decision could be reviewed twice, and the second review won silently.** Two reviewers acting on the same transaction both succeeded: the later decision replaced the earlier one with nothing recording that it had happened, in the table an auditor reads, and each sent its own `decision.overridden` webhook and its own label to the learning agent, which then trained on whichever arrived last. A second override is now refused, naming the reviewer whose decision stands. Reopening a review is not supported. - **A repeated webhook could not be recognised as a repeat.** `X-Webhook-Delivery` changed on every attempt, and deliveries are retried, so a subscriber had nothing stable to deduplicate on and could act twice on one event. It is now the same for every attempt at the same delivery. - **The same investigation report had two shapes.** Whether it looked reviewed depended on which endpoint returned it. - **`docs/TROUBLESHOOTING.md` listed a port nothing uses** and omitted two that are published, the same error the README carried. ## [1.7.1] - 2026-09-28 A fix to the one-command install, which failed on exactly the run it exists for. Nothing else changes; `@ojuri/sdk` is unaffected and stays at 0.0.1. ### Fixed - **`npx @ojuri/cli up` failed on a first install, on any connection slow enough to take ten minutes over the images.** Everything the CLI shelled out to Docker carried a ten-minute ceiling, and starting the stack pulls roughly three gigabytes on a fresh host, so the install died partway through with `spawnSync docker ETIMEDOUT`, no containers created, and nothing explaining why. Found by installing the published package on a machine with no images cached; CI had never caught it because its runners fetch well inside the ceiling. The ceiling was only half of it. Output was buffered rather than shown, so the longest part of the install printed nothing at all and read as a hang even when it was working. The pull is now its own step that renders Docker's own progress as it happens, and the limit on a step like that is an hour, there to catch a stuck daemon rather than to bound a download. Reading container state and logs stays as it was, because that output is parsed. Building from source skips the pull, since there is nothing published to fetch, and a failure now points at what Docker printed instead of leaving a blank where the reason belongs. ## [1.7.0] - 2026-09-28 Additive only. Two new publishable packages, a new optional manifest, no migrations and no schema changes. Every existing install path is unchanged. ### Added - **A one-command install, as an additional option.** `npx @ojuri/cli@1.7.0 up` brings the stack up on a machine with Docker and Node 20 and **no clone**: it writes `ojuri.yaml` and a `.env` with fresh secrets, unpacks the compose file and the assets it bind-mounts, pulls the images, waits for the migration and for RDA to answer `/ready`, then prints a runnable `curl` and the admin password. Neither existing flow changes: `cp .env.example .env` plus the two-file Compose invocation, and the build-from-source path, work exactly as documented. The new `ojuri.yaml` manifest is a layer over `.env`, not a replacement, and rendering the committed default produces a byte-identical Compose project to the one you get today, which CI asserts. It adds `ojuri init`, `up`, `down`, `status`, `doctor` and `render`, a JSON Schema, and validation that refuses what the architecture cannot do: `paa.replicas` above 1 is an error, because a second copy sees half the transaction graph and silently stops catching rings that span both. `mla`, `fia` and `sentinel` ship disabled, so a default install starts what it starts today. FIA stays off because it downloads roughly 7.6 GB of model weights on first start. - **`@ojuri/sdk`, a typed client for the predict path.** Adopters calling `POST /v1/predict` no longer hand-roll `fetch`: `packages/sdk` ships a zero-dependency, dual ESM/CJS client (Node 18+) covering the predict call and inbound webhook signature verification, published to npm on release. It deliberately does not wrap the audit reads or the FIA report endpoints — those are Sentinel's surface, and the dashboard calls them directly. `test/contracts/sdk-types.contract.test.ts` compares every copied type and enum against the server declarations and fails this repo's CI on drift in either direction. ### Fixed - **`ReasonCodeDto` omitted `basis`.** The wire DTO had drifted from `ReasonCode`, which carries the field and is what the response factory actually passes through, so the documented response shape understated what `/v1/predict` returns. - **Deploys served a stale dashboard until the CDN cache lapsed.** Sentinel is a hashed-bundle SPA: a new image writes a new `index.html`, but CloudFront kept handing visitors the cached one, pointing at the previous bundle. The stack was current and every artefact check passed — only the browser was behind. `ojuri-up` now invalidates the distribution, keyed on the image the sentinel container is actually running rather than on the floating `:v1` tag, so a boot that changed nothing costs no invalidation. `VERSIONING.md` records why `curl -H 'Cache-Control: no-cache'` is the wrong way to check for it. ## [1.6.0] - 2026-08-15 Additive only. One new unauthenticated endpoint, two new optional `.env` variables that default to off, no migrations, no schema changes. ### Added - **The sign-in page adapts to a public sandbox.** New unauthenticated `GET /v1/auth/sign-in-options` reports whether this deployment seeded the shared `demo` account. Where it did, the form prefills `demo` and explains that the account is shared and its password published; everywhere else the page is unchanged, keeping the `admin` prefill and the first-run `npm run db:migrate` instructions. The answer comes from the `users` table rather than from `SEED_DEMO_USER`, so a migration that ran before the variable was set cannot point visitors at an account that does not exist. The password is never returned. Optional `DEMO_CREDENTIALS_URL` adds a link to wherever it is published. ## [1.5.1] - 2026-08-11 Bug fixes only. No API surface changes an integrator relies on, no migrations, no new `.env` variables to set. ### Fixed - **API keys were never persisted.** Sentinel posted the expiry select's label (`"30 days"`) where a timestamp was expected. `new Date()` produced an Invalid Date, Postgres rejected the insert, and because the token is minted *before* the insert the operator was shown a plausible `fdk_…` key that had never been stored — every request with it then returned 401. The dashboard now sends an ISO timestamp, and `expiresAt` is validated: a malformed value returns 400 naming the field instead of a 500 with the failing SQL in the logs. - **nginx stopped reaching RDA after any restart or slow boot.** The upstream resolved `rda:3000` once at startup, so nginx booting before RDA cached an address that no longer existed. `/v1/*` then 502'd while `/` kept serving the dashboard — the API looked broken while the UI looked fine. Now resolved per request through the Docker resolver, as `/fia/` and `/mla/` already were. - **The login form blamed `AUTH_JWT_SECRET` for unrelated outages.** Any 503 was reported as a missing secret. RDA does return 503 when the secret is absent, but so does anything in front of it that is down, so a routing failure sent operators to fix a correctly-set variable. It now distinguishes the two. - **The sandbox's auto-stop timers were never enabled.** They were armed after the stack under `set -e`, so the first boot where the stack failed aborted the bootstrap before reaching them, leaving an instance with no idle stop and no nightly stop. Timers are now armed first, and a stack that will not start is reported rather than fatal. - **A stack half-started when Kafka lost its health race.** Kafka not being healthy when compose evaluated `depends_on` aborted RDA, PAA and nginx; Kafka became healthy moments later but the oneshot unit never retried. Now retries on failure, with a health-check grace period so the first attempt usually wins. ## [1.5.0] - 2026-08-11 Ships a one-instance AWS deployment and fixes three defects found while standing it up — two of which made services unrunnable rather than merely awkward. The MLA and FIA fixes change the published images, so adopters on the floating `:v1` tag receive them automatically. ### Added - **AWS test environment (`deploy/aws/`)** — Terraform for a single EC2 instance running the whole stack behind CloudFront with AWS-managed TLS, plus a runbook aimed at operators who are not DevOps engineers. Ingress is limited to CloudFront's origin-facing prefix list, shell access is SSM Session Manager (no port 22, no key pair), data stores bind to loopback, IMDSv2 is required with a hop limit of 1 so no container can reach the instance role, and every secret is generated into SSM Parameter Store rather than living in the repo. The instance stops itself when idle and can be woken from a public button, which keeps a continuously-available environment at roughly a third of the cost of leaving it running. - **`sentinel` service in `docker-compose.yml`**, behind a `sentinel` profile. The image was already published but had no compose service and no ingress route, so the dashboard could not be served from the shipped stack. A plain `docker compose up` is unchanged. - **`ADMIN_SEED_PASSWORD` passed through to `db-migrate`**, so the seed migration's existing support for a supplied password can actually be used. Unset still generates a random one and prints it once. - **Optional public-demo account (`SEED_DEMO_USER`, default off)** — seeds a shared `demo` user on a read-mostly `DEMO` role so a throwaway sandbox can be signed into by anyone, without handing out the super-admin credential. The role carries reads plus `api_keys:issue` / `api_keys:revoke`, and deliberately excludes everything that changes decisioning (`rules:update`, `models:set_status`, `models:set_threshold`, `settings:write`, `review_queue:override`) or identity (`users:*`, `roles:*`). Requires `DEMO_USER_PASSWORD` to be set explicitly and refuses to run without it. `mustChangePassword` is false by design: the credential is shared, and forcing a rotation would let the first visitor lock out everyone after them. **Never enable this on a real deployment** — a known-credential account on a self-hosted fraud platform is a vulnerability, which is why it is opt-in rather than seeded-then-disabled. - **`extra_cors_origins` for the AWS deployment** — additional browser origins permitted to call the API, for a site hosted somewhere other than the front door. The front door is always included; anything beyond it is a listed, reviewable decision rather than a wildcard. ### Fixed - **RDA could not boot with `NODE_ENV=production` from the shipped compose file** ([#121](https://github.com/ojuri-io/ojuri/issues/121)). `SENTINEL_CORS_ORIGINS` was never passed into the container, and the production guard treats its absence as fatal — so the default configuration crash-looped on *"Refusing to boot with unsafe defaults"*. Setting it in `.env` did not help: `.env` populates Compose's interpolation scope, and a variable only reaches a container if the service's `environment:` block names it. - **MLA crash-looped on modern kernels** ([#122](https://github.com/ojuri-io/ojuri/issues/122)). `onnxruntime==1.16.3` ships a native extension marked as requiring an executable stack, which current loaders refuse — the import failed before the service started. Now `1.19.2`. The runtime is independent of the `onnx` / `onnxmltools` / `onnxconverter-common` pins that exist for XGBoost conversion; those are unchanged and `tests/test_onnx_conversion.py` passes against the new runtime. - **FIA chat answers bled into invented follow-up turns.** The follow-up prompt was a plain `USER:/ASSISTANT:` transcript rather than Phi-3's chat template, so the model never emitted `<|end|>` — and `<|end|>` was not in the stop set anyway, since the tokenizer's configured `eos_token` is `<|endoftext|>`. The model finished its answer, was not stopped, and spent the remaining `max_new_tokens` writing `QUESTION:`/`ANSWER:` pairs the user never asked, all returned and persisted as the assistant turn. Measured on a live deployment: same question, same report, 248,237 ms with four invented turns before, 66,960 ms and a clean answer after. Report generation shares the stop-token fix; it never showed the artefact because the parser keeps the first JSON object and discards what trails it. - **The wake page is no longer served to callers that wanted JSON.** CloudFront's error configuration is distribution-wide, so a sleeping sandbox answered an API call with an HTML page. It now content-negotiates on `Accept`, and carries CORS headers so a cross-origin caller can read the error at all rather than seeing an opaque network failure. - **A stopped instance no longer takes 30 seconds to report itself.** CloudFront's default three attempts at ten seconds were spent in full, because a stopped instance swallows connections rather than refusing them. One attempt at three seconds. - **`/fia/` and `/mla/` were never stripped before proxying** ([#123](https://github.com/ojuri-io/ojuri/issues/123)), so Sentinel's investigation reports and retrain controls 404'd against the shipped stack. nginx substitutes the location prefix only when `proxy_pass` names a literal upstream; both routes use a variable so the resolver is consulted per request, and in that form the trailing slash is inert. - **FIA silently degraded to rule-based reports on any CUDA host.** `phi3_generator` requested `attn_implementation="flash_attention_2"` whenever CUDA was present, but `flash-attn` is not a dependency. transformers raised, the fallback caught it, and the only outward sign was `llmModelVersion` ending in `-fallback` — on a GPU rented specifically to avoid that. It now falls back to `eager` when `flash_attn` is unavailable. The bug was invisible because every measured run had been on Apple Silicon MPS, which takes the `eager` branch. ## [1.4.0] - 2026-08-09 This release remediates a full line-by-line architecture review — 45 findings across three rounds (OJR-01–45, per-finding evidence in `docs/REVIEW_FINDINGS.md`) — and ships a flag-gated log-first audit pipeline. It also makes permission changes apply to live sessions and the RDA replica count configurable. No API surface changes, but several decision-path behaviours differ from 1.3.0 (see Changed) and every fix was re-verified against a live compose stack: 2,000-request load test all HTTP 200, p99 84.5 ms at 16-way concurrency vs the 295 ms pre-fix baseline. ### Added - **Log-first audit pipeline (`AUDIT_PIPELINE=stream`, default `queue`)** — the decision event carries the full audit payload and is published with an awaited `acks=all` send before the response; `AuditStreamConsumer` materialises `decisionAuditLog` from the topic, and late values (shadow scores, early-PRE feature snapshots) follow as `audit.enrichments` events applied as idempotent UPDATEs. Audit durability moves from "row survives unless the process dies before flush" to "row durable in Kafka before the caller sees the decision". Measured: p50/p95/RPS parity with the queue, +10–19 ms p99 end-to-end; server-side decision path p50 1 ms / p99 5 ms; enrichment delivery 100% (queue mode's patch race dropped ~15% of early-PRE snapshots). Closes OJR-07 when enabled; the checklist for making it the default (a major-release change — Kafka becomes a hard dependency of `/v1/predict`) is in `docs/REVIEW_FINDINGS.md`. Full measurements: `docs/LOG_FIRST_AUDIT_PROTOTYPE.md`. The prod compose stack threads `AUDIT_PIPELINE` through to the RDA replicas, so the `.env` opt-in works on `docker compose up` (previously dev-overlay only). - **Score calibration reaches serving** — RDA loads the isotonic breakpoints from `meta.json` and applies them after ONNX output. `ONNX_CALIBRATION_MODE` defaults to `observe`: the calibrated score is recorded as `decisionAuditLog.calibratedScore` while decisions still use the raw score, because every shipped threshold was tuned against the raw distribution. Flip to `enforce` only after re-deriving thresholds from calibrated audit data. - **PAA leader lease** — PAA takes a Redis lease (`ojuri:paa:leader`) before consuming; a second instance waits for handover then exits instead of joining the consumer group and splitting the graph. Renewal fails closed on elapsed TTL, and a fenced-out instance discards its buffers rather than flushing a partial-graph snapshot over the new leader's writes. `PAA_REQUIRE_LEADER_LEASE=false` disables the fence. ### Changed - **The ONNX breaker fallback is REVIEW, not DECLINE.** opossum fires the fallback on per-call timeouts too, so a timeout below the measured p95 under concurrency turned ordinary contention into customer-facing declines. `CB_ONNX_TIMEOUT` now defaults to 750 ms, the fallback returns `{ degraded: true }`, and `PredictService` maps it to `CB_ONNX_FALLBACK_DECISION` (default REVIEW) with `decisionSource = BREAKER_FALLBACK`. Degraded declines are no longer published to `transactions.blocked` — there is no model signal for FIA to investigate. - **Drift F1 threshold anchors to the deployed champion** — the window alarm fires at (champion validation F1 − `DRIFT_F1_MARGIN`) instead of the static `DRIFT_F1_THRESHOLD`, which stays as the fallback when no champion metrics exist. - **Model hot-reload serves the version artefact directly** — the copy into the canonical `MODEL_PATH` is best-effort (the compose files mount `models/` read-only; EROFS is expected), and a champion that was ACTIVE before boot is applied at startup, so cold restarts no longer silently fall back to whatever the canonical file holds. - **Dev containers run the mounted source directly** — `module-alias` registers only in compiled builds (`src/register-aliases.ts`); under ts-node, `tsconfig-paths` resolves aliases to `src/*.ts`, keeping one module universe (two resolvers duplicated every tsyringe singleton). nodemon uses `legacyWatch` so host-side edits through the Docker mount trigger restarts on macOS. No in-container `npx tsc` after edits any more. - **Configurable RDA replica count (`RDA_REPLICAS`, default 3)** — the three hand-copied `rda-1/2/3` compose services collapse into a single `rda` service scaled via `deploy.replicas`. NGINX balances across the replica IPs Docker DNS returns (restart nginx after changing the count so it re-resolves), and Prometheus discovers replicas via DNS service discovery instead of static targets. The GHCR overlay and CI reference the `rda` service name; per-replica names (`rda-1:3000` upstreams, static scrape targets) are gone — update any scripts that addressed replicas by name. ### Fixed 45 findings; the tracker has per-finding detail. Highlights by service: - **MLA** — the isotonic calibrator previously affected only the reported Brier score and never a served score; the calibration split is carved before SMOTE (fitting on oversampled rows targeted a ~50% synthetic base rate); drift windows are fed from Postgres ground truth on the label poll (they previously never received a sample, so F1/PSI drift could not fire); the label-volume watermark anchors to the last succeeded `retrainRuns` row, so labels arriving while MLA is down still count after a restart. - **PAA** — graph cap is a true LRU; velocity windows are retained by time, not count; replayed events dedupe instead of double-counting; graph metadata snapshots for both parties on every event. - **RDA** — audit ids are tenant-scoped; the audit requeue path bounds poison entries; the LevelDB replay buffer preserves topic and partition key per entry (`{ v: 2, ... }` envelope). - **FIA** — poison messages are bounded by a retry counter and committed past instead of wedging a partition; offsets commit per-partition only. - **Dev/compose** — `db-migrate` and dev services carry the env they need (`MLA_SERVICE_TOKEN`, `AUDIT_PIPELINE` passthrough). - **Permission changes now apply mid-session** — the JWT authenticates identity only; `requireAuth` resolves permissions, `isActive`, and `mustChangePassword` from Postgres per request through a 30 s in-process grants cache (`AuthService.verifyTokenLive`). Role edits, role (un)assignment, user deactivation/deletion, and forced password rotation take effect on live sessions within the cache TTL — instantly on the replica that handled the admin call — instead of on the user's next login (previously up to the 8 h token TTL). The token's permission claim is still minted for compatibility but is never consulted for authorization. - **`/readyz` through NGINX no longer 404s** — `location /ready` prefix-matched `/readyz` and the `proxy_pass` URI rewrite sent `/readyzz` upstream. The `/health` and `/ready` aliases are exact matches now, so the app's own `/livez` and `/readyz` pass through the catch-all unchanged. - **`scripts/demo-traffic.mjs` defaulted to an unreachable URL** — the default `http://localhost:3000` is not published by the production compose stack (NGINX answers on `:80`), so the documented host-side invocation exited with "RDA is unreachable". It now probes `:80` then `:3000`, so both the Docker stack and `npm run start:dev` work with no `RDA_URL` set. An explicit `RDA_URL` is still used verbatim. ### Documentation - **README restructured around the adopter journey** — quick start first, then what the platform does, then a path split for evaluating / integrating / operating / contributing. Install prerequisites and the build-from-source path moved into collapsed sections, and symptoms moved to the new [`docs/TROUBLESHOOTING.md`](docs/TROUBLESHOOTING.md). Prose rewritten in plain language throughout. - **Corrected README instructions that failed when executed** — the quick-start `curl` returns 409 on a second run (fixed `transaction_id`, now called out); the sample response showed `threshold` 0.65 on a `TRANSFER` payload where the seeded segment threshold is 0.30, and omitted the `basis` field on reason codes; `"model_version": "default"` was described as no model being registered, when `00_initial_model_version` seeds exactly that version as ACTIVE on purpose; `frontend/.env.example` already targets the Docker stack, so the instruction to uncomment that block was stale; and `python --version` fails where no bare `python` exists, so the MLA setup now creates the venv with `python3.11`. - **`docs/ARCHITECTURE.md` performance table** now carries the post-review contended measurement (p99 84.5 ms, ~516 RPS) alongside the superseded 295 ms baseline, which the table had been presenting as current. ### Notes - Round 2 of the review (OJR-27–40) consisted of defects introduced by round 1's fixes, found by three independent re-reviews — the tracker keeps the full chain visible. - Reference performance in CLAUDE.md was re-measured post-fix; the honest contended baseline is now p99 84.5 ms (was 295 ms). ## [1.3.0] - 2026-07-07 This release acts on an independent efficacy validation of the shipped stack (`efficacy-validation/`). It fixes four correctness gaps between what the platform did and what its own contracts/README claimed, and adds two detection improvements. Existing deployments are unaffected — both seed changes skip rows that already exist — but **fresh installs behave differently on day 1** (see Changed). ### Added - **Behavioral rule pack (`04_behavioral_rule_pack.ts`)** — two POST-stage REVIEW rules on PAA-derived features (`velocity_1h`, `unique_receivers_24h`), guarded to exclude agents and corporates. Turns PAA's velocity/graph signals into verdicts the ML model misses on trusted, authenticated traffic. Measured against the efficacy harness: velocity-anomaly recall 0.00 → 0.80 with zero added false positives on agent-network, payroll, airtime, and remittance traffic. Every feature it reads defaults to 0 on a Redis miss, so the pack is cold-start-safe. - **Context-field dropout augmentation (MLA)** — training appends context-zeroed copies of a fraction (`CONTEXT_DROPOUT_FRACTION`, default 0.4) of the training rows so the model scores on behaviour, not payload richness, fixing the shipped model's trust-context degeneracy (a bare payload scored ~1.0, full context ~0.0). `CONTEXT_DROPOUT_ENABLED` (default on). See `mla-service/docs/CONTEXT_DROPOUT_RETRAIN.md` for the runbook and the measured result: the mechanism closes the context gap (0.9998 → 0.0009) but does not by itself add behavioural typology recall — that requires training data containing those patterns as fraud. - **Context-sensitivity probe (RDA)** — alongside the calibration probe, every model load and hot-swap scores the same transaction with and without context fields and warns (exposing the gap via `getModelInfo()`) when the model keys on integration context rather than behaviour. ### Changed - **The demo rule pack now seeds inactive by default.** Its amount thresholds are demo-dataset props that flag large slices of real traffic (declining every txn ≥ ₦100k, reviewing every ₦500–10k PAYMENT) and shadow the FATF pack. Set `SEED_DEMO_RULES_ACTIVE=true` before the first seed run to restore the old behaviour for the demo dataset. Existing rules keep their operator-set `isActive` state. - **Fresh installs now register the shipped model and per-type thresholds.** The `db-migrate` container previously lacked the `models/` mount, so the initial-model seed silently skipped — every decision fell back to a flat 0.65 threshold with an empty registry. Fresh installs now score with the registered `default` model and its segment thresholds (CASH_OUT 0.70, TRANSFER 0.30, …), so day-1 verdicts differ from 1.2.0's. `v1.x` stays reserved for adopter-trained models. - **Calendar features are computed correctly for the first time.** Both RDA serving and MLA training mis-handled the millisecond event timestamp (RDA multiplied by 1000; MLA parsed as nanoseconds and trained on ingestion time), so `hour_of_day` / `day_of_week` / `is_weekend` / `is_payday_window` / `is_off_hours` were noise. Verified zero decision change on the deployed model (it barely weighted these), but retrain before relying on time-of-day signals. ### Notes - Deployments upgrading from 1.2.0 will see a one-time context-sensitivity warning in the RDA logs against the shipped model — that is the new probe working, not a regression. ## [1.2.0] - 2026-07-02 This release closes the label feedback loop — the core mechanism that lets a self-hosted deployment improve on its own outcomes the way closed fraud vendors do — and fixes four correctness gaps where the platform silently did less than its own contracts claimed. Models trained before 1.2.0 keep working, but see the "Changed" notes on score distributions before relying on pre-1.2.0 thresholds. ### Added - **Ground-truth label ingestion API** (#81). `POST /v1/admin/labels` (new `labels:write` permission) accepts up to 1,000 verified outcomes per batch — chargebacks, disputes, customer reports — and upserts `transactions.groundTruthFraud` with provenance. Duplicates in a batch collapse last-wins; the response splits `applied` vs `unmatched` so callers can retry rows PAA hasn't flushed yet. `GroundTruthSource` is now a shared enum across the label paths. `docs/ADOPTER_TRAINING.md` §4 documents the flow. - **MLA label-volume retrain trigger** (#81). MLA counts labels recorded since its watermark every `LABEL_CHECK_INTERVAL_SECONDS` (default 900) and retrains once `LABEL_RETRAIN_THRESHOLD` (default 500; `0` disables) accumulate — fresh verified labels reach the model on their own schedule instead of waiting for F1 drift, which is 3–7 days label-delayed by nature. Runs with or without Kafka, respects `autoRetrainEnabled`, and reports `label_volume_checks` / `labels_pending_retrain` in `/stats`. The registry treats the `label_volume` trigger like `drift` (auto-activate on gate pass). - **PAA fraud-proximity graph features** (#82). PAA polls confirmed-fraud transactions (every `FRAUD_LABEL_POLL_INTERVAL_MS`, default 5 min) and marks both parties as fraud nodes, making two previously always-default catalogue features real: `graph_shortest_path_to_fraud` (undirected BFS, depth-capped at 4, visit-capped at 1000) and `graph_neighborhood_fraud_rate` (fraction of flagged 1-hop out-neighbors). `recipient_dispute_rate` is now the only contracted feature without a writer, enforced by test. - **Real shadow-model scoring** (#83). The SHADOW registry status now does what it says: OnnxService keeps a small second session pool for the shadow version (same feature-schema-version refusal as the champion), scores it on every ML-scored predict overlapped with the reason-code/POST-rule stages, and writes `shadowScore` onto the same `decisionAuditLog` row. Strictly observational — failures record null and never touch the decision or `/readyz`. New `shadow_inference` stage metric. - **PAA delivers the full `paa:redis` feature contract** (#78). Previously only 11 of the ~20 contracted features were written; the model scored on catalogue defaults for the rest, permanently. Now implemented: `velocity_1m/5m/15m`, `amount_mean_24h`, `amount_max_24h`, `unique_receivers_24h/7d` (fan-out signal), `hour_dev_from_sender_norm`, `graph_is_hub` (top 0.1% out-degree with an absolute floor), `recipient_lifetime_tx_count` (kept fresh on pure receivers via partial updates), and a full pair block on a new `features:pair:{sender}:{receiver}` hash (30-day TTL): `pair_is_first_send`, `pair_prior_send_count`, `pair_time_since_last_send`, `pair_round_trip_count_30d`, `pair_amount_mean_30d`. RDA fetches sender + pair + receiver hashes in one pipelined round trip. A contract test fails the build if a contracted feature loses its writer. - **REVIEW band on the ML score** (#87). A new `review_margin` runtime setting (seeded 0 = off) routes scores in `[threshold − margin, threshold)` to a REVIEW decision and into the analyst queue instead of a silent ACCEPT — turning model uncertainty into ground-truth labels. Tracks the per-segment threshold resolution; REVIEW is observational on the wire (`fraud: false`, never published to the blocked topic). Settings gains a Review band card with sizing guidance. - **MLA temporal train/test splits** (#88). Training rows are time-ordered and split positionally — train on the past, evaluate on the future. The previous stratified random split leaked future rows into training and inflated every metric the deployment gate reads. Falls back to the stratified split for degenerate datasets. - **MLA absolute deploy floor** (#88). `MIN_DEPLOY_F1` (default 0.3): the validator refuses candidates below the floor even when they beat the incumbent, and cold-start models below it register as CANDIDATE for operator review instead of auto-activating. - **Sentinel Labels page + label-feedback card** (#86). Manual entry for verified outcomes (paste ids, per-line verdicts, source picker) posting to the labels API, and a Settings card showing "labels until next retrain" from MLA `/stats`. - **Fraud simulation benchmark** (#90). `scripts/fraud-sim.mjs` + `scripts/fraud-sim-score.py` + `docs/FRAUD_SIMULATION.md`: a deterministic persona population with six embedded fraud typologies (half deliberately rule-evading) driven through `/v1/predict` over a simulated multi-week window, scored per typology against ground truth. Reference run: 128k transactions, 34.2% of fraud caught cold → 98.8% at 1.1% FPR after one label-driven retrain, on fraud identities the model never saw. - **Live ground-truth metrics in champion-vs-shadow** (#91). The comparison endpoint computes per-model precision/recall/F1 and McNemar's p from labelled decisions in the window instead of hardcoding null; the Models page prefers live values over training metrics, √-scales the score histogram, and fixes the off-plot tooltip. - **Demo traffic seeder** (#84). `docker compose --profile demo run --rm demo-seed` (or `node scripts/demo-traffic.mjs`) posts ~500 realistic NGN transactions — 50 recurring senders plus an embedded money ring, a 30-receiver mule fan-out, VPN sessions, and a structuring sequence — so a fresh install has a populated dashboard and firing rules within a minute. Zero npm dependencies. ### Fixed - **Reason codes were misattributed** (#77). The explainer used pre-catalogue vector indices, so every code read a different feature than its label claimed (`AMOUNT_HIGH` was computed from `unique_receivers_24h`, `PAGERANK` from `velocity_7d`, …). Specs now resolve positions from the feature catalogue by name; a regression test pins the attribution. `PAGERANK` baselines moved to realistic per-node magnitudes so it no longer adds a large constant to every explanation. - **`pair_time_since_last_send` semantics** (#78). Was the sender's time since their last transaction to *anyone*, mislabeled to the model as pair recency. Now genuinely pair-scoped. - **MLA deployment gate had an escape hatch** (#79). A candidate deployed on F1 improvement alone, bypassing both the McNemar significance requirement and the >5% precision/recall regression guards. All three gates are now binding. - **MLA PSI data-drift detection was inert** (#80). The consumer fed the detector fields the Kafka event doesn't carry (fabricated as constant 0) under names that didn't match the baseline's catalogue keys. PSI now monitors only fields actually present on the event (`amount`, `account_age_days`, `session_to_txn_seconds`) under exact catalogue names, omitting absent fields instead of defaulting them. - **PAA first Redis flush no longer silently dropped** (#89). The worker now waits for Redis readiness before consuming Kafka — the lazy client raced the first batch flush and every write in it was counted as an error and discarded. Also quiets the fraud-sync log re-reporting the same users every poll. - **Isotonic calibrator was dropped on automated retrains** (#88). Only the cold-start script persisted it, so the first drift/label-volume retrain silently shipped uncalibrated scores. `upload_model` now writes `calibrator.npz` into every version directory. ### Changed - **Score distributions will shift after deploying 1.2.0.** Models trained before this release learned on default values for the ~9 newly-delivered PAA features (#78) and the two fraud-proximity features (#82); once real values flow, scores move. Plan a retrain shortly after deploying, and re-validate any hand-tuned per-segment thresholds against post-upgrade audit data. - Shadow scoring adds one extra ONNX inference per ML-scored request *while a SHADOW model is staged* (#83). No cost when no shadow is registered. If throughput matters on small hardware, retire the shadow when the comparison window is done. - `POST /v1/admin/labels` requires the new `labels:write` permission — grant it to the role your ops integration uses before pointing a chargeback feed at it (#81). - Post-1.2.0 training metrics will read **lower** than pre-1.2.0 numbers for the same data (#88) — the temporal split removes the future-leak inflation. That drop is honesty, not regression; compare models within the same split methodology only. - The review queue now includes REVIEW-decision rows alongside DECLINEs (#87). Queue volume is unchanged until an operator sets a non-zero `review_margin`. ## [1.1.0] - 2026-06-22 First tagged release. v1.0.0 (2026-06-07) was the soft launch; v1.1.0 is the first version with a formal git tag, signed container images on GHCR (`ghcr.io/ojuri-io/{rda,paa,mla,fia,sentinel}`), and a GitHub Release. Adopters should pin to `:v1` (floating, receives future minors and patches) or `:v1.1.0` (strict). See [`VERSIONING.md`](./VERSIONING.md). This release adds the post-v1.0.0 graph-coherence work, adopter training-data ingest, FATF rule pack, MLA score calibration, per-segment threshold defaults, and several PAA scale/correctness fixes. ### Added - **PAA durable edge + node state in Postgres.** New `transactionEdges` table (Knex migration `20260608000001`) holds the directed-edge list; `graphMetadata` finally gets the `firstSeen` / `lastSeen` / `transactionCount` / `totalAmount` / `inDegree` / `outDegree` columns populated alongside the existing pagerank / community / clustering outputs. On boot, PAA hydrates the in-memory `graphology` instance from these tables first and only tails transactions newer than the latest persisted edge timestamp. Rings whose seed edges fall outside the 30 d / 100 k row replay window stop being forgotten across restarts. - **PAA singleton runtime guard.** New `paa_group_members` Prometheus gauge that must stay at 1; PAA logs ERROR if a second consumer joins the `pattern-analysis` group. Backstops the compose change for ops scenarios (rolling restart overlap, accidental scale-up, helm-chart drift) where the in-memory graph would otherwise be fragmented across replicas. - **Triangle-close recompute trigger.** When a new edge closes a directed 3-cycle (`A → B`, `B → C` already exists, `C → A` already exists), PAA fires `computeNetworkMetrics()` on the next event instead of waiting up to 5 minutes for the scheduled tick. Burst rings whose edges land in a tight window stop being invisible during the gap. Throttled by `TRIANGLE_RECOMPUTE_MIN_INTERVAL_MS` (default `10000`) so a flurry of ring-forming events doesn't thrash the recompute path. - **PAA sizing guidance** in `paa-service/README.md`. Covers what a node represents, what actually happens at the `MAX_GRAPH_NODES` cap (silent crash loop when no nodes are stale enough to evict), and the two binding ceilings (1–2k TPS sustained typical, dropping toward 200–500 with high-degree receiver hubs or graphs past ~500k nodes; ~1M nodes memory) with a profile-by-profile sizing table from demo through Tier-1. - **Adopter training-data ingest.** Operators can now upload labelled transaction CSVs through the Sentinel "Training imports" page or via the file-based API. Upload protocol is chunked (5 MB chunks, SHA-256 verify on assemble) so multi-hundred-thousand-row files survive network blips: `POST /v1/admin/training/upload/init` → `PUT /v1/admin/training/upload/:id/chunk` × N → `POST /v1/admin/training/upload/:id/complete` → `POST /v1/admin/training/import/:jobId/promote` → `POST /mla/v1/admin/retrain`. Column renames and default values are applied in the UI (no need to re-export the CSV); a downloadable template surfaces the canonical headers. New tables: `trainingJobs`, `trainingUploads`, `transactionsStaging`. See `docs/ADOPTER_TRAINING.md`. - **Per-segment fraud-threshold defaults** seeded for every PaySim transaction type in `src/database/seeds/02_segment_thresholds.ts`: CASH_OUT=0.70, TRANSFER=0.30, PAYMENT=0.50, DEBIT=0.50, CASH_IN=0.50. Resolution uses `request.segment ?? request.transaction_type` so adopters get segment-aware thresholds with no UI work. - **FATF default rule pack** under `src/database/seeds/03_fatf_rule_pack.ts`: five rules covering structuring (CASH_OUT under the cash-reporting threshold), VPN+significant-amount, outbound TRANSFER to FATF high-risk corridors, account-takeover signature (rushed session + cross-country IP + high amount), and untrusted device + significant amount. Defaults are NGN-tuned; edit per-market. - **MLA isotonic calibration** wraps the post-XGBoost score with `sklearn.IsotonicRegression`, fitted on a held-out 10% split. The Brier score (calibrated and uncalibrated) is recorded in `meta.json` and persisted to the `modelVersions.brierScore` column so adopters can spot saturation regressions across versions. - **User-controlled training mode.** New `mlaSettings.trainingMode` (`FRESH` | `CONTINUED`) and `mlaSettings.continuedTreesPerRound` let operators choose between full retraining (current behaviour) and incremental boosting on top of the current production model (XGBoost `xgb_model=`). Surfaced as a dropdown on the Settings page; `PUT /mla/v1/admin/drift-config` accepts the two new fields. - **Rule visibility in audit and predict response.** Every `/v1/predict` response now carries a `rule` object when a PRE or POST rule fired (`{ id, name, stage, action, expression }`), and the decision-audit row stores the rule expression so the Sentinel detail page can render exactly which rule caused a DECLINE. - **Curated demo dataset and one-shot loader.** A 20-row `data/demo/sample-transactions.json` hand-built to exercise all three decision buckets, plus `npm run demo:load` (extends `scripts/seed-load.ts` with a `--file` flag that re-mints `transaction_id` / `timestamp` per send so the dataset can be replayed without idempotency collisions). - **Demo rules seed** under `src/database/seeds/01_demo_rules.ts`, installed by `npm run db:seed`. Four PRE-stage rules keyed on `amount` / `transaction_type` / `segment` so the demo dataset produces a visible ~9 ACCEPT / ~4 REVIEW / ~7 DECLINE split out of the box even on a fresh deploy with no PAA cache. - **Regression guard** for the demo dataset shape: seven Jest assertions (`test/demo/demo-dataset.spec.ts`) that fail CI if the JSON drifts from the predict-API contract. - README quickstart now mentions `npm run reset:admin` for the "I lost the seeded password" recovery path. - `.env.example` now documents 11 previously-undocumented env vars (BRAND_NAME, FEATURE_CATALOG_BASE_PATH / ADOPTER_PATH / FEATURE_LOOKUP_ROOT, the four `*_HEALTH_URL` knobs, HEALTH_PROBE_TIMEOUT_MS, MODEL_VERSION_LABEL, MODEL_INPUT_DIMENSION). ### Changed - **PAA is now a singleton by design.** `paa-2` removed from `docker-compose.yml`; `deploy.replicas: 1` pins the remaining `paa-1` service; Prometheus scrape config and CI log-dump list trimmed accordingly. Reason: PAA holds the transaction graph and velocity windows in process memory, and Kafka partition assignment splits the event stream across replicas — running two means each computes PageRank / Louvain on a partial graph and any ring whose members hash to different partitions becomes invisible. Adopters running past the singleton's sustained-throughput ceiling (typically >2k TPS, sooner for graphs >500k nodes or when high- degree receiver hubs dominate) will eventually want the externalised-graph deployment profile (separate work) instead of scaling PAA. - **PAA `graphMetadata` populated on every event for both sender and receiver.** Previously the trigger was `processedCount % 100 === 0` and only the sender was snapshotted, which dropped 99 % of writes and left pure-receiver nodes (mules, drain points) missing from the table entirely. The flush path also switched from a per-row upsert loop to a single bulk `INSERT … ON CONFLICT … MERGE` per batch. - **PAA `pruneOldNodes` switched to event-time clock and an hourly schedule.** Was wall-clock (mis-classifying historical replays as recent) and only cap-driven (so the prune effectively never fired in practice). Now tracks the max observed event timestamp and runs on a 1 h interval in addition to the existing cap path; the cap path still drops the oldest 10 % as a relief valve, but the scheduled path drops everything past the 30 d cutoff. - **NGINX `/mla/` proxy hardened for Linux hosts.** The new `location /mla/` block defers `host.docker.internal` resolution to request time (variable in `proxy_pass` + embedded resolver), and the `nginx` service in `docker-compose.yml` now declares `extra_hosts: host.docker.internal:host-gateway`. Without this, nginx hard-fails at startup on Linux CI runners and the Sentinel "Retrain now" / drift-config calls return 502. - **Predict service refactored** into named stages (`resolveModel`, `loadFeatures`, `evaluatePreRules`, `runInference`, `evaluatePostRules`, `finalize`) backed by factories (`DecisionAuditFactory`, `PredictDecisionContextFactory`, `TransactionEventFactory`, `PredictResponseFactory`). No behavioural change; the request hot path is now legible from a single 40-line method. - Tighter TypeScript strict-family coverage. RDA gained `noImplicitAny`, `strictPropertyInitialization`, `noUncheckedIndexedAccess`, and `noImplicitOverride`. PAA (already on `strict: true`) gained `noUncheckedIndexedAccess` and `noImplicitOverride`. Net: one real bug fixed in `velocity.service.ts` (un-guarded `sorted[0]` access), and the compiler now refuses several whole classes of regression. ### Removed - Unused `appConfig.onnx.modelRegistryUrl` field. The HTTP-polling model registry it once fed was retired in favour of the in-process `ModelRegistryService` subscription flow. ## [1.0.0] - 2026-06-07 Initial public release. The platform is a multi-agent fraud-detection stack with four backend services (RDA, PAA, MLA, FIA) and an operator dashboard (Sentinel), all self-hosted and decoupled by Kafka. ### Added - **RDA (Real-Time Detection Agent).** Fastify HTTP API with `POST /v1/predict` (sub-5 ms p99 on a developer workstation, ONNX inference, per-segment thresholds, idempotency keys, inline reason codes), `/livez` / `/readyz` / `/v1/metrics`, and a full `/v1/admin/*` surface. Three-replica deploy behind NGINX in `docker-compose.yml`. - **PAA (Pattern Analysis Agent).** TypeScript Kafka consumer that maintains a transaction graph (`graphology`) and velocity windows, batching writes to Redis (features) and Postgres (graph metadata every 100 events). - **MLA (Model Learning Agent).** Python 3.11 drift monitor (F1 + PSI), SMOTE-balanced XGBoost retraining, McNemar's test before promotion, ONNX export, filesystem-based model registry under `mla-service/models/versions/`. - **FIA (Fraud Investigation Agent).** Python 3.11 LLM service that consumes `transactions.blocked`, runs a fine-tuned Phi-3-mini-4k-instruct model (fallback to a deterministic rule-based report on LLM failure), writes structured investigation reports to Postgres, and exposes `POST /v1/reports` and conversational follow-ups via `POST /v1/reports/:id/messages`. - **Sentinel dashboard.** Vite + React 18 SPA covering live decisions, transactions, audit log, rules editor, model registry, FIA investigations, webhooks, API keys, and user/role administration. Reads use a `safe(live, empty-fallback)` pattern; the dashboard never displays synthetic data. - **Authentication and authorization.** API-key auth (`fdk__`, SHA-256-hashed at rest), JWT user auth with bcrypt-hashed users and role-based permissions, `requireAuth(...perms)` middleware on every `/v1/admin/*` route. SUPER_ADMIN seeded with a one-time random password printed at migration time. - **Rules engine.** JSON-Logic predicate evaluator with PRE / POST stages, hot-reloaded from Postgres every 30 s. - **Model registry.** CANDIDATE → SHADOW → ACTIVE → RETIRED lifecycle, per-segment thresholds, hot-swap on activation. - **Decision audit log.** Every `/v1/predict` writes a row with champion + shadow scores, threshold, rule hit, reason codes, feature snapshot. Reviewer overrides write back to `groundTruthFraud` so the model does not learn from its own past decisions. - **HMAC-signed webhooks.** SHA-256 over `t.body`, exponential-backoff delivery worker, delivery ledger in `webhookDeliveries`. SSRF guards reject private IP ranges and non-HTTPS schemes. - **Idempotency.** `Idempotency-Key` header on `/v1/predict`, keyed against `(tenantId, apiKeyId, key, requestHash)`. Replay returns the cached response with `Idempotency-Replay: true`. Body divergence returns 422. - **Resilience.** `opossum` circuit breakers around Redis feature lookup and ONNX inference. On breaker open, predictions still succeed against default features. - **Observability.** Prometheus metrics under `/v1/metrics`, Grafana dashboard, structured pino logs with sensitive-field redaction. - **Reason codes.** Lightweight feature-deviation explainer on every prediction (e.g. `AMOUNT_HIGH`, `VELOCITY_24H`, `PAGERANK`). For deep reasoning use the FIA report endpoints. - **Feature catalogue.** Declarative `models/feature-catalog.v1.json` (64 base features across 9 categories) loaded by both RDA and MLA. Adopters add their own features via `models/feature-catalog.adopter.json`. - **Replay CLI.** `scripts/replay.ts` runs candidate models against the live audit log for offline evaluation before promotion. ### Security - Pre-launch hardening pass: CORS allowlist via `SENTINEL_CORS_ORIGINS`, per-IP rate limit on `/v1/auth/login`, JWT algorithm / issuer / audience pinning, dummy bcrypt for unknown users (constant-time login), webhook SSRF guards, idempotency race protection, FIA HTTP API authentication, prompt-injection guards on FIA inputs, MLA model loader switched from `pickle` to XGBoost JSON format, removal of `useragent` package (replaced with `ua-parser-js`).