--- name: math-auditor description: > Run an adversarial mathematical-accuracy review of a Julia package's src/ and test/ directories, producing a dated markdown report plus concise, self-contained fix-prompt markdowns suitable for handing to a smaller model (e.g. Sonnet) in a later session. Use this skill whenever the user asks for an adversarial review, a math audit, a full code review focused on mathematical/numerical correctness, a check of algorithmic consistency with the literature, or says things like "review the math in src/", "is the algebra right?", "audit the update equations", "check the statistics/linear algebra for bugs", or "construct a code review as markdown". Trigger even when the user does not say "audit" — any request for a correctness-focused sweep of a scientific Julia codebase qualifies. --- # Math Audit Adversarial review of a scientific Julia package for **mathematical accuracy and consistency** — not software architecture (flag architecture only when it causes mathematical wrongness, e.g. mutation aliasing, accidental type demotion, or inconsistent conventions between modules). The output is written for the package's own developers: findings must cite exact `file:line`, state the correct mathematics, and give a concrete failure scenario. A finding that can't survive an attempt at refutation doesn't ship. ## What "adversarial" means here Each reviewer's job is to *break* the code, not describe it. Concretely, hunt for: - **Wrong equations**: update formulas, gradients, covariances, likelihoods that differ from the cited papers or from the docstring's own LaTeX. Derive the correct expression independently and diff it against the code. - **Convention drift**: rows-vs-columns for ensemble members, `N-1` vs `N` normalization, factor-of-2 / sign errors, Cholesky `L` vs `U`, covariance vs precision, whether noise is added in obs-space or transformed space — especially *inconsistencies between modules that must agree*. - **Statistical validity**: is added noise sampled with the right covariance and scaling (e.g. `Δt` scaling in stochastic dynamics)? Are means/covariances computed over the right dimension? Deterministic vs stochastic variants actually equivalent in expectation? - **Numerical soundness**: unguarded `inv`/`\` on possibly-singular matrices, loss of symmetry/PSD-ness, subtraction-based variance formulas, missing regularization, `sqrt` of negative-by-roundoff eigenvalues. - **Edge cases the math must survive**: ensemble size 1–2, dimension 1 (scalar vs matrix degeneracy), zero variance, NaN/failed ensemble members, empty minibatches. - **Test-math consistency**: do the tests actually pin the mathematics (analytic solutions, invariants, convergence rates), or just check shapes and "it runs"? A wrong equation whose test only checks `size()` is a *double* finding: the bug and the missing test. ## Workflow ### 1. Partition List `src/*.jl` and `test/**/*.jl` with line counts. Group into 4–8 review units of roughly comparable size, pairing each source module with the tests that exercise it. Group modules that *share mathematical conventions* together (e.g. all Kalman-update variants in one or two units) so the reviewer can catch cross-module inconsistencies. ### 2. Fan out reviewers (parallel agents) Spawn one agent per unit, in a single message so they run concurrently. Each agent prompt must include: - the exact file list for its unit, - the "What adversarial means here" hunting list above (copy it in — agents don't see this skill), - instructions to check code against docstrings/comments *and* against the standard form of the algorithm from the literature, - a required output format: a JSON-like list of findings, each with `file`, `line`, `severity` (critical / major / minor / hygiene), `claim` (one sentence), `evidence` (the code vs the correct math), `failure_scenario` (concrete inputs → wrong output), `verified` (`numerical` / `inspection`), and `suggested_fix` (optional, a few lines). Tell agents explicitly: - "Prefer few, well-evidenced findings over many speculative ones — but do report genuine minor inconsistencies. If the module's math is correct, say so and note the strongest invariants the tests pin." - "When a finding concerns a fixed point, a statistical scaling, or a crash, verify it numerically in a scratch script if cheap — a small linear-Gaussian fixed-point run, a quick Monte Carlo of the statistic, or reproducing the error — and tag it `verified: numerical`. Numerically verified findings are worth far more than inspection-only ones." (In one audit, the agents that ran code delivered the critical finding pre-verified to 1e-16; the only refuted claim of the run came from an inspection-only unit.) - "Before claiming anything is 'silent' or 'has no warning/guard', grep for `@warn`, `@error`, and `throw` at the *constructors and call sites* of the code path, not just the function you are reading — guards often live at construction time." As each agent's report arrives, save its raw findings verbatim to a scratchpad file (one per unit). A full audit plus verification is long enough that context summarization mid-run can silently lose findings; the scratchpad files are the durable record the report is assembled from. ### 3. Verify For each critical/major finding, attempt refutation before it enters the report: re-read the cited lines yourself, re-derive the math, and check whether a test or an upstream transformation already accounts for it (common false positives: a transpose hidden in a helper, normalization done at construction time, a convention documented elsewhere, a `@warn` at the constructor that the reviewer never read — re-run the warn/throw grep yourself for any "silent" claim). Prioritise findings tagged `verified: inspection`; numerically verified ones usually need only a sanity re-read. Spawn skeptic agents for findings you can't settle from the main context. Demote or drop findings that don't survive; mark surviving ones **CONFIRMED** vs **PLAUSIBLE** (couldn't fully verify). Then build a small **conventions matrix** from the unit reports before writing anything: rows = modules, columns = the conventions the units commented on (covariance normalization N vs N−1, Δt/noise-scale placement, order of regularization vs localization, RNG threading, rows-vs-columns). Any mismatched cell between modules that must agree is a finding candidate in itself — in practice the worst bugs are a scale factor applied to one block or module but not its sibling, and they only become visible side by side. ### 4. Write the report Create `full-code-review//` (date from `date +%F`, never from memory). Write `review.md`: ```markdown # Adversarial Mathematical Review — () ## Scope and method ## Summary table ## Critical findings ## Major findings ## Minor findings & hygiene ## Cross-module consistency notes ## Test-coverage gaps ## What was checked and found sound ``` Findings get stable IDs used everywhere, including fix prompts: `C1…` critical, `M1…` major, `m1…` minor, `h1…` hygiene; grouped-minor fix prompts get `G1…`. ### 5. Write fix prompts For each finding with an actionable fix (usually critical + major, plus grouped minors), write `full-code-review//fix-prompts/-.md`. These are consumed by a *smaller model in a fresh session with no context*, so each must be self-contained: ```markdown # Fix : **File**: `src/Foo.jl`, function `bar!`, around line NNN. **Problem**: <2–4 sentences: what the code does vs what the math requires. Include the incorrect snippet verbatim.> **Required change**: **Do not**: **Verify**: ``` Keep each under ~40 lines. One finding per file; group only truly mechanical repeats (e.g. the same typo pattern in five docstrings) into one prompt. Quote enough of the offending snippet that the fixer can locate it by function name + snippet — line numbers drift between the audit and the fix session, so present them as hints, not anchors. Also write `fix-prompts/README.md` listing prompts in recommended application order (independent fixes first, same-file prompts sequenced, conflicting ones flagged) and naming which fixes *intentionally change numerical results* — so the fixer checks a failing loose regression test against the analytic reference in the prompt before "fixing" the test. ### 6. Report back Final message: lead with the headline (how many confirmed critical/major findings and the single worst one), then the report path, then a compact summary table. Do not paste the whole report into the chat. ## Calibration - Severity: **critical** = produces mathematically wrong results in mainstream use; **major** = wrong in common configurations or silently degrades statistical properties; **minor** = wrong in edge cases, misleading docs math, dead/misnamed math; **hygiene** = style-level (only if math-adjacent). - A docstring–code mismatch is a real finding even when the code is right — users implement against docstrings. - A statistically unjustified combination that the package explicitly warns about (e.g. a constructor `@warn "... experimental ..."`) caps at **minor** unless the warning itself is wrong — the trap isn't silent. - Don't pad. If a unit is sound, the report says so in one paragraph; an audit that cries wolf gets ignored next time. ## Improving this skill After delivering the report, offer: "Would you like to improve the **math-auditor** skill itself using skill-creator? You can share suggestions, or I can analyse this run — finding quality, false-positive rate, fix-prompt usability — to refine the skill for next time."