---
name: monitor-job-tables
description: >-
  Format HPC job-status reports as box-drawing tables, bucketed by job type (RL · SFT · Datagen · Eval ·
  Catch-all), with the right metric columns, signal thresholds, and red-flags per bucket. Use whenever
  reporting active/recently-terminated job status — during a cron sweep, an ad-hoc "how are my jobs
  doing", or a single-job progress update. Covers which metrics are mandatory (entropy + collapse
  signals for RL, not just step/reward/grad), where to pull live status (SFT .out vs trainer_log.jsonl),
  the RL collapse-warning rule, and which log lines are benign noise vs real faults. Cluster-agnostic —
  resolve paths, active clusters, and credentials from .agents/ops at execution time.
---

# monitor-job-tables

> **Read `.agents/ops/<cluster>/ops.md` first, every sweep.** It is the source of truth for which
> clusters are active, how to locate logs safely, login-node caveats, and current known log noise. This
> skill deliberately names no cluster as active or down — that changes, and a stale list here produces
> confidently wrong reports.
>
> **Locate logs by cluster type, never by guessing a path:**
> - **SLURM clusters** — resolve the log path from the scheduler (`scontrol show job <id> -o`, fields
>   `StdOut=` / `%Z` workdir). **Never `find`/`du` on a parallel filesystem.**
> - **Kubernetes/iris clusters** — there is no scheduler `.out` and no path to `stat`. **Liveness is a
>   state poll of the job lifecycle, never a log-string grep.** Use the iris job-summary/state helpers
>   documented in the ops file; pull metrics from the job logs. **"running-but-0-pods" or a record that
>   has disappeared is TERMINAL** — that is the silent-wedge signature. Keep iris/kubectl calls
>   synchronous.
>
> **Verify a log path EXISTS before concluding "dead."** A scheduler's `StdOut=` may name a file that
> was never created while the real live log sits in the same workdir under a different name. If the
> scheduler path is absent, `ls` the workdir for any `*_<jobid>.out` and read that. Absence at the
> scheduler path is a path mismatch, not a death.
>
> **A failed log fetch is indistinguishable from an idle job.** An API or kubelet error can land in the
> same stream as the logs and parse as "no metrics". Check the line count before concluding anything
> about a job's state, and retry once against a floor.

Report **every** active and recently-terminated job, **bucketed by type**, in the formats below.
**Unify cross-cluster runs of the same type into ONE table.** Give a separate table for jobs still
filling their generation buffer (no metrics yet). Five buckets: **RL · SFT · Datagen · Eval ·
Catch-all**.

Cross-cutting (every bucket):
- **Chain-restart TIMEOUTs are normal, NOT failures** — when a walltime-limited job TIMEOUTs and its
  `afterany` successor is RUNNING/PENDING, report it as a normal restart and name the successor.
- **Completion → matching cleanup skill**: RL by flavor — agentic → `rl-agentic-job-cleanup`, standard
  non-agentic → `rl-standard-job-cleanup`; SFT → `sft-job-cleanup`; datagen → `datagen-job-cleanup`;
  eval → `eval-agentic-cleanup`. Object-store-backed RL routes the same way but leaves no on-disk trial
  tree to reap. On shared-filesystem clusters, cleanup is not done until the artifact's on-disk trial
  tree is removed and inode reclaim is verified — leaving it is the top inode-leak source.
- **Genuine FAILED (exit≠0, not a wall TIMEOUT) → diagnose + dated `agent_logs/` entry.** Recurring
  identical failures are not transient.

---

## RL

```
┌─────────────────────────┬───────┬────────┬─────────────┬───────────┬─────────────────────────────────────────┐
│           Job           │ Step  │ Reward │ Policy Loss │ Grad Norm │                  Trend                  │
├─────────────────────────┼───────┼────────┼─────────────┼───────────┼─────────────────────────────────────────┤
│ <run> (shaped)          │ 15/80 │ 0.619  │ -0.0040     │ 0.006     │ Checkpoint saved. Slight dip from 0.652 │
│ <run> (base)            │ 26/80 │ 0.451  │ -0.0930     │ 0.021     │ Stable, gradients strong                │
└─────────────────────────┴───────┴────────┴─────────────┴───────────┴─────────────────────────────────────────┘
```
Box-drawing tables (┌─┬─┐), **not** markdown — hard user preference for RL. Columns: Job, Step
(`cur/max`), Reward, Policy Loss, Grad Norm, Trend. **Entropy + collapse signals are mandatory**:
include `policy_entropy`, TIS `log_ratio`, and `grad_norm` (in Trend or as extra columns) — without
entropy you cannot apply the collapse rule. A metric not emitted yet → mark `—`. A fresh launch still
in bring-up (gang/queue admission, mesh load, shared-memory broadcast waits, transient image-pull
self-heal — all BENIGN) goes in the buffer-filling table with `—` until its first step lands.

**Rewards from different shaping regimes are not comparable.** Confirm the shaper state from
integrality of `reward × rollouts_per_step` (fractional ⇒ shaping active) before putting two arms in
the same column and drawing a conclusion.

**New/untested RL run? → deep-probe it, don't trust the row.** A row can read "healthy" on a silently
dead run (weight-sync garbage, engine starvation, zero trials completing). For any RL job in a new
setting — new config/geometry/model/image, a smoke test, or the first launch after a code or config
change — dispatch a subagent with **`rl-job-health-deep-dive`**; it reads the literal rollouts and
returns a KILL/NO-KILL recommendation.

**Standard (non-agentic) RL has no Harbor trial artifacts.** Its gates cannot be scored from trial
evidence and must not be marked ERROR for lacking it. Substitute `reward/avg_raw_reward`, banked-step
cadence plus durable checkpoints, `timing/*`, and `generate/avg_num_tokens`.

**Banked steps come from durable evidence, not a progress line.** Take the max `global_step_N` under
the run's checkpoint prefix, and **search every location the launcher may have written to** — a run
that resumed and a run that started fresh can bank to different paths. Corroborate with a
purity-checked log parse. An `exports/global_step_N` signals completion only on a finished run; a
running job also writes periodic saves there.

### Metrics to track per RL run (priority order)
**Core 5 (always):** `reward/avg_raw_reward` (primary), `reward/avg_pass_at_N` (less noisy),
`policy/policy_loss`, `policy/policy_entropy` (direction and magnitude both matter — pre-collapse),
`policy/raw_grad_norm` (most predictive; healthy < 1.0; > 1.0 for ≥2 steps has predicted collapse 2–5
steps early). Under **seqnorm global-denom**, grad/policy_loss/log_ratio are genuinely ~1e-5 or
smaller — that is the regime, NOT vanishing gradient.
**Clip ratio (if tracked):** `policy/ppo_clip_ratio` ≈0 normally; >1 % indicates an LR↔eps_clip
mismatch. Also `policy/z_clip/triggered` for clip-variant ablations.
**TIS:** `tis/imp_ratio_mean` (~0.84–1.56 healthy), `tis/imp_ratio_capped_fraction` (~0 healthy).
**Per-token log-ratio diagnostics, where the trainer emits them:** `log_ratio_abs_{mean,p99,max}`,
`n_tokens_dp_gt_{1,10,50}pct`, positional buckets. Healthy: `mean` ~0.005–0.02, `max` < 0.5,
`gt_50pct` ≈ 0, position buckets even.

### NOT a collapse signal — `rollout_train_prob_diff_mean`
`policy/rollout_train_prob_diff_mean` = `exp(rollout_lp − train_recompute_lp).abs().mean()` — the mean
per-token importance ratio, **dominated by outlier tokens** (a single ~20-nat disagreement gives
`exp(20)≈5e8`). **Millions or billions are NORMAL** on healthy dense arms. Reward is verifier-computed
and independent of logprobs, so this can never "hit the reward". For a per-token divergence read use
the **capped** `tis/imp_ratio_mean` / `imp_ratio_capped_fraction`, the median, or `log_ratio_abs_*` —
not this mean.

### NOT a failure/hang cause — context-overflow + passthrough-exception lines
Engine `... N input tokens > M max`, `ContextLengthExceededError`, and `AgentTimeoutError` are **benign
and expected** in agentic rollouts — they are harbor `passthrough_exceptions`, the verifier still
scores, the rollout completes, and they appear in successful runs. **Never the reason a job hangs or
fails.** Find the real terminal signal: a `Traceback`, OOM / raylet death / SIGKILL, an RPC or sampling
timeout, a `RuntimeError`, or a hung actor/trial that never returns.

### Collapse rule (≥2 fire same step → cancel + salvage)
`raw_grad_norm` > 1.0 (or > 2× its window); `policy_entropy` off its 10-step trend by > 30 %;
`log_ratio_abs_mean` > 2× its window while `max` stays bounded; trial pass-rate < 10 % over the last
100. **Exception:** spike-mitigation ablations are NEVER auto-cancelled on this rule — observing the
recovery IS the experiment.

**Where a no-kill instruction is in force, this rule gates a RECOMMENDATION, not an action.** Capture
the evidence that disappears at termination, record it, report it, and leave the job running.

---

## SFT

```
┌──────────────────────────────┬─────────┬────────┬───────────┬───────────────────────────────────┐
│             Job              │  Step   │  Loss  │ Grad Norm │               Trend               │
├──────────────────────────────┼─────────┼────────┼───────────┼───────────────────────────────────┤
│ <run> cold-start 2ep         │ 320/916 │ 1.21   │ 0.84      │ Loss descending; healthy          │
└──────────────────────────────┴─────────┴────────┴───────────┴───────────────────────────────────┘
```
Columns: Job, Step (`cur/total`), Loss, Grad Norm, Trend. **No reward.**

**For multi-cell SFT grids, also give a grid-completion rollup each sweep:**
- Per RUNNING cell: progress % (step/total) and a rough ETA, plus a one-line running / pending-unique /
  done tally.
- **Dedupe the PENDING count — restart-chain resume copies inflate it several-fold.** Count *distinct*
  cells: list pending job names, strip to the cell name, `sort -u`, then subtract running cells' own
  resume backups.
- **Name the long pole.** In a mixed-scale grid the largest cells gate completion; small cells clear
  fast and their progress is not the campaign's progress.
- **Grep the TRAINING progress bar, not the packing bar.** Verify the denominator matches the cell's
  total optimization steps, not an example count.
- **A single tailed `s/it` is NOT the rate.** Checkpoint-save spikes inflate one line at the save
  cadence. Use a trailing-window rate (average several step lines, or Δwall/Δstep).

**Pull live status from the training `.out`, not `trainer_log.jsonl`.** The `.out` carries the
per-step dicts and is richer (live grad_norm, per-rank loss spread, token coverage, epoch). The JSONL
is unreliable mid-run — sparse, empty, or frozen — and produces false "stale/dead" readings. Use it
only for the completion check before consolidate/upload. Total steps come from the rendered config or
the trainer banner.

**Red flags:** `ChildFailedError` / non-zero exit (read the FIRST real traceback above the elastic
summary — it is usually masked), CUDA OOM at the first forward/backward, `SIGTERM` (node fault or a
masked rank crash — a recurring death at a fixed interval is NOT transient), loss → NaN, grad
explosion.

---

## Datagen

```
┌─────────────────────────────┬──────────────┬─────────┬───────────┬──────┬──────┬──────────────────────────┐
│         Datagen run         │    Chunks    │ Trials  │ avg_turns │ Mean │ exc% │           Trend          │
├─────────────────────────────┼──────────────┼─────────┼───────────┼──────┼──────┼──────────────────────────┤
│ <run> (tracker row #N)      │ 18/20 done   │ ~8.6k   │ 5.1       │ 0.53 │ 19%  │ 2 chunks running         │
└─────────────────────────────┴──────────────┴─────────┴───────────┴──────┴──────┴──────────────────────────┘
```
Columns: run (+ tracker row), Chunks (`done/total`), Trials (`result.json` count), avg_turns, **Mean**
(mean reward, from harbor's `<done>/<total> Mean: <X>` line; mark `—` if there is no verifier), exc%,
Trend. **avg_turns is the realness gate** — `>1` is real multi-step; **`≈1.0` is a dead-engine run, do
NOT consolidate.** An exc% of ~20–25 % AgentTimeout is normal for hard sets.
**Red flags:** a `TIMEOUT` **strands the traces** (the terminal upload is killed — traces are on disk
but not uploaded, so consolidate manually); a hung chunk (log silent for hours with a stalled trial
count while still RUNNING); avg_turns ≈ 1.0.

---

## Eval

```
┌──────────────────────────────┬───────────┬───────────┬───────────┬────────────────────────────────┐
│   Eval (model × benchmark)   │  Trials   │ pass-rate │  top exc  │         Infra / Trend          │
├──────────────────────────────┼───────────┼───────────┼───────────┼────────────────────────────────┤
│ <model> × <benchmark>        │ 142/300   │ 0.21      │ AgentTO   │ tunnel✓ engine✓ ; healthy      │
└──────────────────────────────┴───────────┴───────────┴───────────┴────────────────────────────────┘
```
Columns: model×benchmark, Trials (`result.json`/total), pass-rate (fraction with reward > 0), top
exception type, Infra/Trend. The Infra column is the launch-check set from `eval-agentic-launch`:
tunnel auth and traffic, sandbox `api_base` pointing at the public URL rather than an internal IP,
engine POSTs growing and returning 200, trial progression.
**Red flags:** no `result.json` for 60+ min while RUNNING → stall; engine showing zero running requests
for 10+ min → agents not generating; **all trials done but job RUNNING → zombie, cancel**; instant-fail
(null output tokens, `finished_at` ≈ `started_at`) → tunnel not carrying traffic; repeated auth
failures → sandbox-provider degradation.
**Before calling an eval dead, confirm the RIGHT log and a CURRENT window.** Count `result.json` over
the whole run, not just the tail. A burst of timeouts in the last window is usually the hard-trial tail
of a nearly-done run. Verify the engine is actually down (no recent 200s) before blaming it.

### NOT a reliability problem — a high `AgentTimeoutError` fraction
A large timeout share — **even a majority of trials** — is EXPECTED on hard, long-horizon benchmarks
and does NOT make the eval unreliable. The timeout is a passthrough exception: the trial is still
scored, an unfinished task scores as not-solved, and that reflects genuine capability. If the baseline
ran the same harness, the score and delta stand. The only timeout red flag is the infra case:
essentially every trial failing with zero completions and no `result.json` is a stall, not a score.

---

## Catch-all / other (ad-hoc)

Anything that is not one of the four majors — consolidate, pretokenize, uploads, image builds, feature
smoke tests, GPU-CI runs, measurement and grid probes. **Don't force a metric table** — one line each:

| Job | Type | State | Elapsed | Note |
|---|---|---|---|---|
| `<id>` | datagen-consolidate | running | 12m | pushing N rows → `<dataset>` |
| `<id>` | gpu-ci | COMPLETED | 6m | 2 passed |
| `<id>` | RL upload | running | 3m | `<model>` |

State, elapsed, and a human note: what it is, the one signal that matters, and any follow-up. Flag
terminal COMPLETED/FAILED and whether it needs action.

---

## Benign log-noise (do NOT chase as faults)

- **Shared-memory broadcast "no available block found in N seconds"** is an informational heartbeat,
  not a kill signal. It re-fires on a fixed cadence while the engine waits with nothing scheduled. It
  is fault-indicative **only when co-firing with** a real NCCL hang (a `WorkNCCL(...)` timeout line, a
  "preparing to dump debug info", or a SIGABRT). Alone, look upstream for the engine-idle cause; do not
  relaunch or patch the ring buffer.
- **`rollout_train_prob_diff_mean` in the millions or billions** — outlier-dominated, normal. See RL §.
- **Debug-token and `opCount` chatter** — check the cluster's ops file for the current benign set rather
  than assuming any given line is a fault.