--- name: fruxon-debug-trace displayName: Debug a Fruxon Trace description: > Work out why a run did what it did, from its execution trace: find the run, read the step tree, drill into a tool call's parameters and result, and reconstruct the state the agent decided from. Load when a run failed, hung, or produced a surprising result — including the common case of a run that reports COMPLETED but did the wrong thing. --- You are now operating as a Fruxon execution-trace forensics specialist. The question is almost never "which step threw". It's "why did the agent decide *that*" — why it processed two items when twenty were available, why it skipped a customer, why it stopped for nine minutes. The trace holds the answer, but only if you read the inputs each step saw, not just its status. > Authoring failures — a revision that won't create, deploy, or run at > all — are a different job. Go to `fruxon-build-agent` ("Common > pitfalls") for those, and `fruxon-agent-mode` for the exit-code and > error-envelope contract. ## Workflow 1. Find the execution record id. 2. `executions get` for the headline, `trace` for the step tree. 3. Find the step where the behaviour diverged from what you expected. 4. Read that step's **inputs** — parameters, and the result of whatever fed it. Most surprises are an input you assumed, not a bug. 5. Confirm across a second run before concluding it's systemic. ## 1. Find the run ``` fruxon agents executions list # newest-first fruxon agents executions list -s FAILED -n 50 fruxon agents executions list --origin TEST # your draft runs fruxon agents executions list --since 2026-08-05 --until 2026-08-06 ``` **When the report came from a person, not an alert**, filter by conversation instead of by time — you rarely know the timestamp, and scanning a busy agent's log for "the run that served this customer" is hopeless: ``` fruxon participants list --query +972545614701 -o id # person → id fruxon agents executions list --participant # → their runs fruxon agents executions list --session ts_9dbaf… # one conversation fruxon agents executions list --tool system.escalate_to_human --since 2026-08-01 fruxon agents executions list --with-tool-errors -n 20 ``` `--query` matches channel addresses and linked account ids, so a phone number, email or handle resolves to a participant id directly. `--tool` takes `integrationId.toolId` or the name a trace shows ("Search Clients"), and is backed by the tool-usage projection, which lags the newest runs by about a minute. **When the question is "what did that tool return, across runs?"**, don't pull traces one by one. `--calls` adds each call's arguments and a result preview under its run (`toolCalls[]` in JSON), and `--result-nonempty` keeps only runs where the tool succeeded with something to show: ``` fruxon agents executions list --tool "Search Clients" --result-nonempty --since 2026-10-17 -n 200 -o json ``` It reads one trace per listed run and filters only those, so size `-n` and the window to cover the runs you mean. **When the question is "is this one customer, or everyone?"**, search what was said. One complaint, checked against every run in the window: ``` fruxon agents executions list --since 2026-10-14T09:30:00Z --until 2026-10-14T12:30:00Z -q "תקלה בשרת" fruxon agents executions list --since 2026-10-14 -q "beta.restigo.co.il" # who was TOLD this ``` `executions list --query` matches the run's `subject`, its full input (the customer's whole message, which the 200-char `subject` mostly cuts off behind the sender line) and its reply, case-insensitively and in any language. Don't pull `-n 200 -o json` and grep `subject` instead: it misses everything past the cut and every run past the page. It needs `--since` and searches at most 31 days per call, so slide the window for longer. A run that failed, was cancelled or is still running has no stored input or reply, so it matches on its `subject` alone. Search for the customer's words in their own language and spelling, and try a shorter stem (`התחבר` catches `להתחבר` and `מתחבר`). Each row carries `subject` (a one-line "who and what"), `deliveryStatus` and `runType`. A run that is part of an automation also carries `automationId`: the trigger it belongs to, whatever started it. For a person's reply to an automation's question, `trigger` is their channel, and `automationId` is the only link back to the automation. **`COMPLETED` is not `DELIVERED`** — a run can finish cleanly while its reply is withheld or never lands, and only `deliveryStatus` says so. Statuses, mirroring the backend's `AgentExecutionRecordStatus`: `IN_PROGRESS`, `WAITING_FOR_ASYNC`, `COMPLETED`, `FAILED`, `CANCELLED`, `REJECTED`. - **`WAITING_FOR_ASYNC` is the only suspended state.** Consults, human approvals and sub-task fan-outs all report it. *What* the run is parked on lives in the trace's `waitingFor.kind`, never in the status. - **`REJECTED` never ran.** Blocked at the pre-flight gate (cost budget, exhausted credits, invalid licence), so there is no trace to read — the record's rejection reason is the whole story. The budget can be the agent's (`fruxon agents budget list `) or, for a run with an `automationId`, the automation's own (`fruxon triggers budget list `). - `--origin` defaults to `PRODUCTION`. Your own `agents draft run` invocations are `TEST`-origin and are invisible until you ask. **From a dashboard URL.** The `id=` query parameter on an executions-tab URL is the execution record id (under `group=session`, the root run's record id) — paste it straight into `executions get`. If it 404s, fall back to `executions list` narrowed to that timestamp. **When the report is about a voice (Talk) call**, start from the call: `fruxon agents voice-calls list ` finds it, and `fruxon agents voice-calls get

` prints it turn by turn with each reply's `executionRecordId`, which `executions trace` takes. Read the turn's `delivery` before the trace: the trace holds the whole reply, but on an `INTERRUPTED` turn the caller heard only `spokenText`. **Listing does not mark sub-tasks.** A fan-out puts the orchestrator and every spawned child in the same list, and the row shape doesn't say which is which. `executions tree

` answers this directly — it returns the whole tree with parentage and depth, from any node in it. Without it: the orchestrator is the one whose trace contains the `spawn_subtasks` step, and each child's record id is inside that step's result (see §5). ## 2. The three facets | Command | Answers | |---|---| | `executions get ` | status, timing, cost, tokens | | `executions trace ` | what ran, in what order, what broke | | `executions result ` | the run's inputs and final output | | `executions tree ` | every run in the spawn tree, and what the whole request cost | Each maps to one endpoint. `get` is deliberately thin — it will not tell you a run is blocked on a named person; only `trace` carries that. `tree` is the one to reach for on a fan-out: it returns the orchestrator, every sub-task and every consult in one request, each with its depth, status and cost. Your run is marked `▸` and is often not the root — asking about a consult returns the tree that consult sits in. ## 3. Read the step tree Text output renders the whole tree, one line per step. Under agent mode (Claude Code, CI) the default is JSON: pass `-o text` for the tree, or `--show-io` / `--show-thinking`, which pick the text tree on their own. ``` ✓ Trace · 20 step(s) · COMPLETED · 1 error(s) agent emailpoliciesparser (revision 59) duration 11m 23s 1 Agent (SUCCESS · 19.4s) 1.1 LLM_STEP · LLM Query (SUCCESS · google/gemini-3.5-flash) 1.1.2 TOOL · Search Emails (SUCCESS · 4.8s · google_gmail/search) 1.1.3 TOOL · Spawn Subtasks (batch) (SUCCESS · 36.7s · system/spawn_subtasks) 1.1.3.1 TOOL · Spawn Subtasks (batch) (SUCCESS · 36.7s · system/spawn_subtasks) 1.1.10 TOOL · Consult for Role (ERROR · system/consult_for_role) ✗ addressing_failed: role 'advisor' no_match — No available participant… ``` The dotted path is a stable handle — use it when talking about a step. **A slow tool step says where its time went**, when the tool records it (`assets/search_assets` does). The indented line beneath the step splits the call: ``` 1.1.1 TOOL · Search Assets (SUCCESS · 12.2s · assets/search_assets) upstream 12.1s · service 11.6s (embed 11.5s, retrieval 55ms, credentials 33ms, window 26ms) · unaccounted 420ms ``` `upstream` is the call as timed from our side; `service` is what the service reports it spent, stages largest first; `unaccounted` is the rest — connection, queueing, and above all a container booting from zero. One large stage means the service's own work was slow (above: RAG's embed call stalled, not our code); a large `unaccounted` is a cold start. A line with only `upstream` is a service that reports nothing back (an older RAG deployment) — the split is unknown, not zero. No line at all means the tool records no timing, and `-o json` has nothing more. A `result 48,113 → 212 chars · offloaded to file` (or `· projected `) line means the model was not handed the result as the tool returned it — see §5. **A quality check prints its verdict.** An agent whose replies pass a blocking check has a `JUDGE` step under each `LLM_STEP` try, and the line beneath it says what the check decided: ``` 1.1 LLM_STEP · LLM Query (SUCCESS · 26.8s · vertex_ai_gemini/gemini-3.8-flash) 1.1.1 TOOL · Search Assets (SUCCESS · 445ms · assets/search_assets) 1.1.2 JUDGE · LLM Query (SUCCESS · 2.9s · vertex_ai_gemini/gemini-3.7-flash) verdict 1.0 · customInstructions 1.0 — all claims supported ``` `verdict` is the weighted aggregate, followed by each criterion's score and the judge's rationale. With several criteria, each one gets its own line. `--show-io` adds the judge's `summary`. The trace does not record the threshold, so this line never says "passed". A refusal is read from the steps around the check, and a line beneath the verdict names it: | Line | What the trace shows | |---|---| | `✗ rejected — rewritten as 1.2` | Another `LLM_STEP` try followed. Only a refusal brings another try, so every try but a step's last was refused. | | `✗ rejected — no retries left, so the step failed` | The last try was refused too. The `LLM_STEP`'s own error line carries the score and the threshold. | | `✗ rejected — handed to a person (1.1.3)` | In a live conversation the step escalates instead of failing. The `escalate_to_human` step after the check is that hand-off. | | `▲ no verdict — the reply went out unchecked` | The check itself failed (its error is on the line above). A first draft is let through rather than held. | No line beneath the verdict means the reply went out with that verdict. **The JSON shape.** `--output json` is where the real detail lives, and the shape is not what the top-level key names suggest: ``` { "status": "COMPLETED" | "WAITING_FOR_ASYNC" | …, "runType": "ROOT" | …, "rootExecutionId": "…", "waitingFor": { … }, # only while suspended "trace": { "agentId", "agentRevision", "startTime", "endTime", "traces": [ # ← the run's top-level steps; an agent's run is one frame { "id", "displayName", "type": "FLOW_STEP", "status", "duration", "steps": [ # ← nested, recursively { "type": "LLM_STEP", "llmStepTrace": { … }, "steps": [ … ] }, { "type": "TOOL", "toolTrace": { … }, "result": { … }, "status": "ERROR", "error": "…" } ] } ] } } ``` Note `trace.traces[]`, **not** `trace.steps[]`; a step's kind is `type` and its label is `displayName`. Useful one-liners: ``` # every tool that ran, in order … trace -o json | jq -r '.. | objects | select(.toolTrace) | .toolTrace.tool | "\(.integrationId)/\(.id)"' # only the steps that errored, anywhere in the tree … trace -o json | jq -r '.. | objects | select(.status=="ERROR") | "\(.displayName): \(.error)"' # what the run is blocked on … trace -o json | jq '.waitingFor' # every quality-check verdict — result.strValue is a JSON string … trace -o json | jq '.. | objects | select(.type=="JUDGE") | .result.strValue? // empty | fromjson | {aggregate, metrics, summary}' ``` `jq`'s recursive descent (`..`) is the right tool here — writing out `.trace.traces[0].steps[0].steps[3]` by hand breaks the moment the run makes one more tool call. ## 4. If the run is suspended ```json "waitingFor": { "kind": "CONSULT", "displayName": "Dor Gelless", "participantKind": "PERSON", "topicId": "b8a2e04a-…", "waitingSince": 1785999774196, "expiresAt": 1786433573679 } ``` - `kind` is `CONSULT` (a participant's answer), `APPROVAL` (a human go/no-go), or `SUBTASK` (delegated children — `childCount` says how many). This is the only signal separating a machine wait from a human one. - `expiresAt` is the TTL after which the sweeper times the wait out. A run that looks stuck may simply be waiting on someone who went home. - Follow a consult with `fruxon agents topics get

` and `topics messages` — that's the conversation, including the reply. > ⚠️ **A suspended trace is live.** Fetch it twice and you may get two > different documents: the consult resolves, the run continues, steps > appear. If you are reasoning about a `WAITING_FOR_ASYNC` run, save the > payload to a file and analyse the file, or re-fetch and re-read before > you conclude anything. `endTime: 0` (not `null`) means still open. ## 4b. Ending a run that will not end itself Two verbs, and picking the wrong one wastes a cycle: ``` fruxon agents executions cancel-wait --yes # skip the consult, run continues fruxon agents executions cancel --yes # end the run, terminally ``` - **`cancel-wait` is the scalpel.** It resolves the consult as cancelled and the run picks up where it left off. Right when a specific consult is the blocker and the run should still finish — someone answered out of band, or the participant is gone. - **`cancel` is the hammer.** Use it when skipping one consult would not help: an agent that re-issues the same consult just parks again. - **`cancel-wait` only skips consults.** An `APPROVAL` or `SUBTASK` wait reports the same `WAITING_FOR_ASYNC` and gets a 400. Read `waitingFor.kind` (§4) *before* choosing. `cancel` answers differently depending on what it caught: - Parked run → killed on the spot; the command says so. - Live run → the worker executing it lives in another process, so this only *requests* cancellation and the output says "requested". It stops at its next checkpoint. **A run that finishes on its own first stays `COMPLETED`** — the cancel is not retroactive. `--wait` polls and then tells you which of the two happened. > Both verbs confirm first, and under agent mode they refuse without > `--yes`. They stop live work on a deployed agent. ## 5. Drill into a step Every `TOOL` step carries what went in and what came back: ``` toolTrace.tool # {integrationId, id} — the identity to grep for toolTrace.parameters # exactly what the model passed toolTrace.resultDelivery # SIZES ONLY — how much the model was shown toolTrace.upstream # {callMs, serverMs, unaccountedMs, serverStages} # — where the time went; some tools only result.jsonValue # structured result … result.strValue # … or a string one error # present iff status == "ERROR" ``` **`result` is a sibling of `toolTrace`, not a field inside it.** Read `toolTrace` alone and every payload looks stripped of its results — `resultDelivery` carries character counts and nothing else, so it reads like a redaction. It isn't: the value is right there one level up. Anyone who concludes "the trace doesn't expose tool results" has looked in `toolTrace` and stopped. For a quick read without jq, `--show-io` prints each tool step's parameters and result beneath its line in the tree, truncated: ``` fruxon agents executions trace --show-io ``` Whole results, unabridged, in JSON: ``` … trace -o json | jq '.. | objects | select(.toolTrace) | {tool: .toolTrace.tool.id, params: .toolTrace.parameters, result: .result}' ``` **Check `resultDelivery` before theorising.** It reports `originalChars` / `deliveredChars` and `offloaded` / `projected`, and the text tree prints a `result` line whenever either flag is set. A tool that returned 22 records but delivered a file link or a projection explains a model that "ignored" most of them — and no amount of prompt-reading will show you that. Equally, if neither flag is set, the model genuinely saw the whole result and the fault is in its reasoning or its instructions. **Trust the flags, not the counts.** `originalChars` is measured before the JSON is un-escaped for the model and `deliveredChars` after, so a non-ASCII result reports a big drop with nothing cut: a Hebrew `get_users_by_phone` result reads `565 → 220`, which is just its `\uXXXX`-escaped length against its readable one. Anything that genuinely cuts a result sets `offloaded`. `LLM_STEP` steps carry: ``` llmStepTrace.systemInstructions # the system prompt the run carried — an # ARRAY of message records, MOST of them # platform blocks, not the author's prompt llmStepTrace.systemBlocks # index-aligned labels for those records: # {descriptor, variant, tag, region, chars}; # region is LEAD | STABLE | VOLATILE llmStepTrace.systemCacheBreakpointIndex # last cached record; -1 = all llmStepTrace.reasoningBlocks # the model's own account of why llmStepTrace.llmMessages # the conversation turns llmStepTrace.tools # the tools the step had available llmStepTrace.{input,output,cached,thinking}Tokens and matching costs ``` `reasoningBlocks` is the highest-value field in the whole payload when the question is "why did it decide that" — it often names the rule it was following. `systemInstructions` lets you read the prompt the run *actually* executed rather than the file you think is deployed. **Most of `systemInstructions` is not the author's prompt.** Identity, ``, ``, `` and the rest are blocks the platform adds; the author's prompt is one record among them (`descriptor: "author_prompt"` in `systemBlocks`). Use `systemBlocks` to tell them apart — it is recorded even when the text isn't captured, and it is the only reliable count, because a step whose system prompt repeats an earlier step's has its `systemInstructions` emptied. `--show-system` prints them labelled, with sizes and the cache breakpoint; add `--show-io` for the text: ``` fruxon agents executions trace --show-system fruxon agents executions trace --show-system --show-io ``` Never copy a platform block's text out of a trace into a prompt — the platform adds it anyway, so the agent would carry it twice. To see the blocks for a draft before it runs, `fruxon agents system-blocks `. A trace recorded before the labels existed names a block by the tag on its first line, else `(unlabelled)`. `--show-thinking` prints the reasoning inline, the way `--show-io` does for tool payloads, so you do not have to drop to `jq` for the one field you came for: ``` fruxon agents executions trace --show-thinking fruxon agents executions trace --show-io --show-thinking ``` The two flags answer different questions — `--show-io` is "what was it handed", `--show-thinking` is "what did it make of that" — and the second is the only one that catches a model reasoning from how a system like this *usually* works instead of from anything it retrieved. That failure leaves no error, no missing call and no odd result. Blocks are numbered (`think 2/4`) because one LLM step spans every model turn: the model re-reasons after each tool result, and which pass a claim first appears in is most of the diagnosis. **Sub-task correlation.** A `spawn_subtasks` step's result embeds each child's `execution_record_id`. Pull them out and trace each child as a first-class run: ``` … trace -o json | jq -r ' [ .. | objects | select(.toolTrace.tool.id=="spawn_subtasks") | .result.strValue? // empty | fromjson | .. | objects | .execution_record_id? // empty ] | unique[]' ``` The shape-agnostic form is deliberate: a batch spawn nests a child step per sub-task, so recursive descent hits **two** result shapes — the parent's `{results: [...]}` and each child's bare `{execution_record_id, status, response, session_id}`. Reaching straight for `.results[]` works on the parent and throws "Cannot iterate over null" on every child. `fromjson` inside `jq` also saves a second process and keeps the whole thing one pipeline. > ⚠️ **Payloads are double-encoded, and non-ASCII is escaped.** A > spawn's `prompts` parameter is a JSON *string* inside the parameters > object; its result is a `strValue` string containing JSON containing > more JSON. Expect two or three `jq`/`json.loads` passes. Non-Latin > text arrives as `\uXXXX` — use `jq -r` and, in Python, > `json.dumps(…, ensure_ascii=False)`, or you will be reading escape > soup instead of the customer's name. > > **Redirect traces to files; don't hold them in shell variables.** Once > `jq -r` has decoded the escapes, the payload contains real UTF-8, and > `payload=$(fruxon … -o json)` in zsh fails on it with `character not > in range` — every downstream `jq` then sees truncated input and dies > on an unfinished string. Looping over many runs? Write each trace to > `.json` and read the files. ## 6. Reconstruct the state the run decided from A step's output is rarely the mystery. The mystery is the state it read. - **Memory is the state machine** for scheduled agents — watermarks, processed-id lists, pending batches. `memory_search` results *inside a trace are truncated*, so the trace alone cannot tell you what the run read. Two ways out: `fruxon agents memory list ` / `memory get` for the value **now**, or the *previous* run's `memory_write` step for the value **as it was**. Prefer the second — memory has usually been overwritten since. - **Triggers set the batch size.** `fruxon triggers get ` gives the cadence, timezone, and the literal parameter mappings the agent is invoked with. A run that processed "too few" items is very often a run that fired sooner than the schedule implies — an off-schedule fire consumed the backlog and advanced the watermark. `triggers list` alone won't tell you: it carries `lastFiredAt` but not the schedule. It also says where each binding **delivers** (`delivers to` / `bindings[].outputDestination`) — the first place to look when a run COMPLETED but its reply never arrived. - **A ledger automation's state is the item, not memory.** When the run's trigger type is `LEDGER_ITEM`, the durable state is the work item the sweep claimed — so the question "what did this run know?" is answered by `fruxon triggers ledger items --item-key ` (its accumulated `data`; with only a customer name or a policy number in hand, `--search ` finds the item by any value in its data instead) and `ledger transitions

` (every attempt, each with the `executionRecordId` behind it). Both are per-item history the trace cannot carry: a run shows one attempt, the transition log shows why there was a third. `fruxon triggers preview-shape ` then renders the exact `user_query` that stage sends, including the item payload appended to the instruction — which is usually most of the prompt and appears nowhere in the shape as authored. - **"Would the stage get it right now?" is a stage test, not a re-drive.** A re-drive runs the item for real, writes and all. `fruxon triggers ledger test-stage --stage --item --wait` runs that one stage, by its agent's *deployed* revision, on the item, in the sandbox — nothing on the ledger is written, writes go to the simulator, and the run is charged to the test budget. The verdict is the transition a real pass would write (`ADVANCE` / `IGNORE` / `DEAD` / `FAIL` / `REFUSE`, with its `cause` and `detail`) plus what a pass only learns by writing: `duplicateOf` (a pass would IGNORE the item as that one's duplicate) and `nextStageMissing` (the next stage would park it DEAD). Its `executionRecordId` opens with `agents executions trace` like any run. An instruction edit is tested before saving with `--shape @draft.json`; an agent edit is tested before deploying it: `fruxon agents revisions create --file rev.json` (no `--deploy`), then `test-stage … --agent-revision --wait` (the step shows `rev N (not deployed)`), then `fruxon agents revisions deploy ` once the verdict is right. Never hand-compose a stage's `user_query` into a plain `agents draft run -p` — that run carries none of the stage's output contract, so its answer is not what a pass would get. To try an edit still in the agent's **draft** on a real run, `fruxon agents draft run --replay ` (optionally `--file .draft.json` and `--stages stages.json` for edited instructions) replays that run as a stage test on the draft: the data the run was handed, the stage's contract, nothing written. `test-stage --execution [--agent-revision ] --wait` is the control: the same input on the deployed (or another) revision — run both before blaming a prompt change. A batch stage runs the item as a batch of one. An ASK stage runs no agent: test it with the reply the person would send — `--answer ` (applied as is, no model) or `--reply ` (read by a model, as a real reply is) — and a reply that leaves the item out is a `WAIT` verdict. `fruxon triggers ledger walk --item --yes` replays the rest of the path, each stage on the data the last one advanced with (`--answer =` for each ASK stage it reaches, `--agent-revision =` for an undeployed revision), and `stopped.code` names where a real pass would stop — test there, then re-drive. - **A WAITING item is not stuck — it is parked, and the question may have reached nobody.** `waitingFor` on the row says what it is parked on. `TIME` is a WAIT stage's due time: `wakeAt` says when, the sweeper ends it within a minute or two of that, and only one 15 minutes past due is a fault (the sweeper is behind — `ledger funnel` counts them as `overdue`; `ledger wake

` ends one by hand). `PERSON` means the item is on an ASK stage's batch, and `ledger items` carries the `batchId` that joins it to one. The pairing is total for those (an item waits on a question only while its batch is open), so one `fruxon triggers ledger batches --open-only` resolves every such row. Read `delivery` there before anything else: `UNDELIVERABLE` and `FAILED` mean **nobody was asked and nobody will answer**, so the items sit until the batch expires while the batch, the count and the stage all look healthy — this is the failure that looks like nothing at all. `QUEUE_ONLY` is not a fault; the contact is queue-bound and the pending row is itself the operator-queue item. The way out is `fruxon triggers questions answer` out of band, or `ledger cancel-batch`, which returns the items to READY. - **An answered item went somewhere wrong: compare what was given with what was handed back.** `fruxon triggers ledger batches --answers ` lists, per item, `given` (what the person wrote — the sheet cell or their words, as the conducting agent copied them), `answer` (what the batch was handed back and wrote into the item) and `current` (what the item holds at that path now); `mismatch` marks where `given` and `answer` differ, which is where the agent's reading went wrong (PassportCard: "מעודכנת" handed back as `SKIP`). A batch answered before the server recorded answers, or by an agent that sent no `given`, gets `given` from the resolving run's sheet read (`givenFrom: "trace"`). The conversation itself is `--transcript `: a batch's conversation belongs to the stage's **asking** agent, not the desk agent that conducts it, so `agents topics messages ` 404s — a current server names the owner in that 404. To find every item a value landed on, `ledger items --where destination=SKIP`. - **A READY item at a batch stage may be gathering, not stuck.** A stage with a `batch` block runs its items in groups, and an item waits READY until its group closes — full, its oldest item has waited the window, or nothing upstream can still add to it. `fruxon triggers ledger gathering ` says which each group is waiting on; `WAITING_FOR_RUNNING_BATCH` means the group's previous batch is still running (one per group at a time). A CLAIMED item there carries `batchRunId`, and `ledger batch-runs` shows that run, its trace and its cost. To run the groups now: `triggers fire --max-admit 0 --stage --close-gathering --yes`. - **An IGNORED item was a decision, and the reason is the diagnosis.** `ledger funnel` stops at a count, which nobody can act on. `fruxon triggers ledger exclusions ` lists every reason a stage recorded with how many items each holds — usually the fastest way to see that a prompt is excluding a whole class of work it should not. Each count is exactly what `redrive-many --ignore-reason` resolves, so the diagnosis and the fix use the same string, matched whole and case-sensitively. - **"What is this automation costing?" is a per-pass question.** `executions get` prices one run; a schedule's bill is the runs its sweeps caused, which is what `fruxon triggers ledger passes ` reports — one row per fire, with `admitted` / `dispatched` / `completed` / `runs` and the all-in USD of that pass. Two readings to get right. `runs` sits below `dispatched` while runs are in flight or when a dispatch was parked on a human, so the gap is not a lost item. And a pass's `cost` **rises after** its `firedAt` as those runs finish — read it once `completed` has caught up with `dispatched`, or you are reading a floor. `ledger funnel` carries the same two fields per stage/status cell, which is how you find *where* the money goes rather than only how much; a `Cost` of `—` there means nothing in the cell has ever been claimed, as against `$0.0000`, which means it ran and was genuinely free. The automation's whole bill — item runs plus the conversations its questions are held in and its report — is `fruxon triggers ledger spend `: the current UTC calendar week (or `--period daily|monthly`) by agent and part, the figure its spend chip shows; `ledger spend-series` is the same per day. It is not the funnel's `cost`, which files item runs under the cell each item sits in *now*: the two overlap, so never add them. What the automation is *allowed* to spend is `fruxon triggers budget list `: its own daily / weekly / monthly limits, each with the window's spend. An enforced one that is spent holds every stage until the window resets. While any of its limits is enforced, those are the only limits its runs meet: its agents' own limits neither stop nor count them (a run record's `automationGoverned` says so). With none enforced, each stage stops at its agent's own limit. - **"Too few items" needs the size of the backlog, not a sample of it.** `fruxon triggers test-source ` lists a page of what the door returns right now; a full page comes back saying *more available*, which is not an answer to how far behind an automation is. `fruxon triggers test-source --count` is: it pages the same door to exhaustion and reports the total, how many a `door.skipWhen` filter drops, how many the ledger already holds, and how many clear the watermark. `total - filtered - already held` is the work a pass would actually admit, and on a healthy automation that difference is small — every pass re-lists its own overlap window, so a large `already held` is the source working, not repeating itself. Add `--apply-watermark` for what the *next* pass would see rather than what the query matches at all. Ceilinged server-side at 2,000 records / 20 pages; when a ceiling stops it the CLI says "at least", and that floor is not a number to quote as a backlog. - **"The model never saw that field" is the projection, not the prompt.** An item carries only what `workShape.door.projection` names, and the stage's `user_query` carries the item — so a field an instruction asks about is absent from the prompt whether the source omitted it or the shape dropped it. `fruxon triggers test-source ` prints each record as the item will carry it, with an `item keeps` line naming the surviving paths, which separates the two. Items admitted BEFORE a projection changed keep what they were admitted with, so an old item and a new one under the same shape legitimately carry different fields — compare `ledger items` against the current `test-source` output rather than assuming the ledger is uniform. An absent projection keeps whole records, and then the field really was never sent. Note that `ledger items` returns one page of 20 and says how many matched in total — do not read a page as the ledger. Narrow it with `--stage` / `--status`, or ask for the walk with `--all`. - **"The automation ignored almost everything" may be a filter, not a decision.** A shape's `workShape.door.skipWhen` drops matching records *before* admission — no item, no transition, no model call — so those records appear in no ledger query at all, and the gap shows up only as a source that lists far more than the ledger holds. `fruxon triggers test-source ` marks each such record `FILTERED` and counts them in the footer. A large IGNORED pile is the opposite finding: that is a stage spending a model call per record to reach a verdict, which is what a `skipWhen` rule exists to replace when the question is one a dotted path can answer. The filter governs admission only — items admitted before it was added stay where they are. - **A knowledge-base hit is state you can re-run.** When the step that decided things was `search_assets`, the trace gives you the query it sent and what came back — but not whether anything *better* was there. Replay it: ``` fruxon assets search -q "" fruxon assets search -q "" ``` Two different answers means the retrieval was the fault, not the reasoning. The `` is in the run's `` system block (`llmStepTrace.systemInstructions`). Read `score` carefully: in the default `HYBRID` mode it is normalised against the best hit, so **the top result always scores 1.0** and tells you nothing about absolute relevance. `--mode VECTOR` gives a real 0–1 similarity when you need to judge that. **When the chunk you expected is indexed and still did not come back, `--explain` says which leg lost it:** ``` fruxon assets search -q "" --explain ``` It runs all three legs, prints where each placed every fused hit, and names what went wrong as a `code` a driver can branch on: | code | what it means | |---|---| | `FTS_LEG_EMPTY` | full-text matched nothing, so HYBRID ran on the vector leg alone | | `VECTOR_LEG_EMPTY` | the reverse — usually embedding-side, not a query problem | | `WEAK_VECTOR_MATCH` | best true similarity is low; nothing here is a confident match | | `FUSED_DROPPED_LEG_TOP` | a leg's best hit fell below the fused cut — try the weights | | `ALL_LEGS_EMPTY` | check ingestion before blaming the query | `FTS_LEG_EMPTY` is the one to expect on an agent's own query. The full-text leg wants every term of the query in a single chunk, so a long query matches *less* than a short one — and a model writes longer queries the harder it tries. A chunk that ranks first for four words can fall out of the top twenty-five for the same four plus four more, with nothing in the ordinary result to show it. Re-run the search shortened before concluding the answer is not indexed. If a search finds nothing you expected to be there, the index — not the query — may be the problem: `fruxon assets documents ` shows what was ingested, and a `chunkCount` of 0 is a file that produced no indexable text at all. - **An answer that follows a notice, not an article.** A knowledge notice (an outage, a known bug) is given to every agent attaching the base on every turn while it is active, and the agent is told it overrides what articles and search results say about the same thing. The step trace keeps the block the agent saw: ``` … trace -o json | jq '.. | objects | select(.knowledgeNotices) | .knowledgeNotices' fruxon knowledge-bases notice

``` `text` is the block as given, `noticeIds` the notices in it, active and ended alike: a conversation that was given one is told on its next turn that it ended. A wrong answer during an incident is usually the notice's wording (its *Applies when* too broad); `edit-notice` fixes it from the next turn, and `withdraw-notice` ends it. - **Compare against the previous run.** `executions list`, take the run before, trace it, and diff the inputs. A watermark, a roster size, a memory value: one of them moved. ## 7. "The run says COMPLETED but it did the wrong thing" The most common real defect. Two shapes: **A step errored and the agent recovered.** Status stays `COMPLETED` because the model routed around the failure. It will be in the tree as an `ERROR` step — the text renderer counts them in the headline (`· 1 error(s)`). These repeat silently forever; check whether the same step errors in the previous run too before dismissing it as a blip. **Every step succeeded and the *decision* was wrong.** Nothing is red. Work backwards: find the step whose output first disagreed with your expectation, then read the inputs it had — the tool result that fed it, the memory it searched, and `reasoningBlocks` for its own account. In practice the answer is usually one of: a watermark or filter excluded more than you thought; a dedupe rule collapsed rows; the result was truncated before the model saw it (`resultDelivery`); or the deployed `systemInstructions` are not the prompt you have open in your editor. **A quality check passed the wrong reply.** When the step has a check, its verdict is printed under the `JUDGE` step (§3). A high score beside a wrong reply means the check is part of the defect too. Compare its rationale with the tool results it judged from (`--show-io`). A grounding check confirms that each claim has a source. It does not confirm that the source was the right one for this customer, so a reply that applied the wrong knowledge-base article still scores "all claims supported". ## 8. "The agent went quiet and I cannot see why" Three separate causes, none of which appear in a trace, because in each one the runtime worked correctly and the *conversation* is where the story is. **The customer's message never reached an agent.** A withheld inbound is recorded and then deliberately kept from every agent — a human owns the chat in the source console, or admission refused the sender. It never becomes a conversation turn, so `topics messages` shows the customer writing and nothing after it, and there is no run to trace because no run happened. ``` fruxon messages list --agent --delivery WITHHELD --since 2026-08-01 fruxon messages list --delivery WITHHELD_PENDING # never acted on at all ``` `withheldReason` is `HUMAN_OWNED_CONVERSATION` (a person had the chat) or `ADMISSION_REFUSED` (the sender is unknown or unenrolled). A row with a reason and no `withheldConsumedAt` was never replayed into a later run and never discarded — nothing will drain it. **A person took the conversation over.** Then the agent is muted on purpose. ``` fruxon escalations history --agent --address +9725... ``` `escalations list` will not show it — that is a queue, and a takeover leaves it the moment it stops needing someone. **`humanObservedAt` is the field that matters**: null means nobody was ever seen actually holding the chat, even on a `RESOLVED` row, because ownership returning to the assistant is equally what a chat nobody touched reports. `openDurationMs` is how long they had it, and is null while the handoff is still open. **Nobody was reachable to take it.** Then the customer was told someone would follow up and the runtime moved on — `escalations unhandled`. To see whether that was a misconfiguration or the design: ``` fruxon escalations policy ``` An empty ladder under `mode: NATIVE_HANDOFF` is correct — the channel's own console supplies the human. An empty `offerableOperators` under `BRIDGE` means an escalation right now would degrade. Read `configuredOperators`, not `offerableOperators`, when reconstructing a *past* escalation: the second is filtered by who is available this minute, and presence today says nothing about presence then. ## 9. "The agent answered as if it had never seen the conversation" The opposite failure: the agent did reply, promptly and fluently, to a customer whose question only makes sense against an exchange it appears not to know about. The trace is clean, because the run genuinely had no history to work from — it was a *different session*. A topic outlives its sessions. The platform ends an episode and mints a fresh one rather than keeping one open, and the fresh one starts with an empty transcript. `topics messages` stamps every turn with its `sessionId`, so the boundary is visible there — but not the reason for it. ``` fruxon agents topics sessions ``` **`endCause` on the *previous* session is the answer.** `CONSOLE_ALREADY_RESOLVED` means a human closed the conversation in the provider's own console; that is what ended the session that knew about the escalation. A follow-up sent seconds later lands in a fresh session, and the agent answers it blind — which is precisely when a customer asks about something a colleague just told them. `seedContextIn` says whether the runtime pointed the new session at that resolved case. A `recentSupportHandoff` block carries the ids — `supportCaseId`, `sourceTopicId`, the handoff, and when it resolved — and its absence on a session that opened moments after a handoff resolved is the agent going in with nothing. The seed's prose is deliberately not on the wire: read `sourceTopicId` with `topics get` for the redacted outcome the agent was actually shown. Two shapes to check while you are here. **A session still `RUNNING` on a `RESOLVED` topic is usually the healthy shape, not a strand.** Resolving a ticket ends the episode it interrupted — the participant's *home* session — and deliberately keeps the ticket's own session: a customer who comes back within the reopen grace window (48 hours from the topic's `resolvedAt`) reopens that ticket, and reopening mints no session, so ending it early would leave the reopened ticket nothing to route to. The stranded-session sweep reaps it once the window has passed. Two checks before calling it a defect: ``` fruxon agents topics get -o json fruxon agents inbox --participant -o json fruxon agents topics sessions ``` - **Age.** A `resolvedAt` less than 48 hours ago is a window still open. Well past it, the sweep should have ended the session; one that is still `RUNNING` is an episode nothing ever closed. - **The home session's `endCause`.** The close is recorded on home, not on the ticket. On a console-routed handoff, the home session that held the escalation ending with `CONSOLE_ALREADY_RESOLVED` is the console closing the chat and the conversation being released (`CASE_RESOLVED` when the handoff ran out its TTL instead). Any other shape — home still open after the ticket resolved, or ended for some other cause — means the ticket's close is not what ended home; and a home still open is why nothing handed the conversation back to its console. And an `endCause` with no `endedAt` is an end that was *requested* and never applied; a new inbound message voids it, so it is not an ended episode and must not be counted as one. ## 9b. "Why does this topic exist? Its transcript starts in the middle" A topic opened from inside a conversation — handed to the team, opened with `open_topic`, or a customer coming back about a closed ticket — is created by a run in the participant's *main* conversation. That run's rows (the customer's message, the agent's reply) are stamped to the home session, so `topics messages` on the topic starts after them, often on an operator's "Done". ``` fruxon agents topics origin -o json ``` `openingTurn.messages` is that turn, and `openingTurn.executionRecordId` is the run to `executions trace`. `reason` / `tried` / `handoff` are the agent's own words for why. For a follow-up, `continues.topicId` is the earlier topic: run `origin` on it too to walk the case back. **`derived: true`** means the topic predates recorded origins and the server matched the run in flight when it was created; `openingTurn: null` with `customerWords` set means the opening run failed before saving rows. ## 10. Reproduce the bad reply, then prove the fix Reading the trace tells you why the agent answered badly; it cannot tell you whether your edit answers better. Replay the same turn instead of waiting for the next customer: ``` fruxon agents topics messages -o json # the id of the reply that went wrong fruxon agents draft pull # edit, then: fruxon agents draft push fruxon agents sandbox fork

--send --draft -o id fruxon agents sandbox stream ``` `fork` copies every turn the agent had loaded when it produced that reply — from the last summary checkpoint before it, so the window matches — into a fresh sandbox session, and `--send` re-sends the customer message that preceded it. `--draft` runs your draft rather than the deployed revision; check `sandbox draft ` if it is refused, or if `onCurrentRevision` is false (your draft forks from an older revision than production runs). What this does **not** reproduce: the copied history is frozen, but tools run live against today's data, so a lookup can answer differently than it did then. Only one topic's history is copied. And the sandbox blocks channel sends and memory writes but **not integration writes** — `--send` holds every create / update / delete call as a `pending_approval` for exactly that reason; answer it with `agents approvals respond`. The forked topic then appears in `topics list` under the real participant, flagged `isSandbox` — it is not their conversation. ## Gotchas - **`role: assistant` does not mean the agent said it.** A human colleague's message is stored as an `ASSISTANT` turn on purpose, so the agent reads it on re-engage instead of re-asking what the person already handled. `author` is the field that separates them: `HUMAN_OPERATOR` is a person, `AGENT` is the agent. `UNSPECIFIED` means the row predates authorship stamping and could be either — it is **not** a synonym for `AGENT`, and treating it as one is how an operator's words get quoted back as the agent's. - **A fresh `sessionId` is not a new conversation.** One topic spans many sessions, and each new one starts with an empty transcript — so an agent can answer a follow-up knowing nothing about the exchange directly above it in `topics messages`. `topics sessions` says why the previous episode ended; `endCause: CONSOLE_ALREADY_RESOLVED` is the common one. - **A gap in a transcript may be a message that was never delivered.** A withheld inbound is absent from `topics messages` entirely rather than shown as unanswered, so the transcript cannot distinguish "the agent chose not to reply" from "the agent never saw it". `fruxon messages list` can. - **`get` won't tell you a run is blocked.** Only `trace` carries `waitingFor`. A suspended run and a hung one look identical from `get`. - **Ids in agent-written memory may be corrupt.** When a prompt has the model re-emit a long list of opaque ids each run, it garbles them. Verify a stored id against a real one before trusting a dedupe list — a near-miss (one hex character off) is the tell. - **A trace's own `duration` on the outermost step is not the run.** A run that suspended and resumed reports only the resumed segment there. The run span is `trace.endTime - trace.startTime`. - **`executions list` has no cursor.** It's a time-ordered log; to go further back, slide `--since` / `--until` rather than paging. - **"Cancellation requested" is not "cancelled".** For a live run the command returns before the worker has stopped. Read the record back (or pass `--wait`) before reporting a run as cancelled — it may have completed instead. - **A tree can be pruned by retention and still look whole.** `executions tree` warns when the retention window dropped runs out of it; the totals then cover only what is left. Without that line the tree just draws short.