--- name: arms-paper-modeler description: "Execute ARMS post-validation production evaluations from frozen validated code, preserving method and parameters. Produce traceable CSVs and figures, reconcile formula-versus-code discrepancies into the canonical audit, and catalog anomalies for manuscript consumers." --- ## Selection and handoffs Scale already validated code to final publication evaluation and own canonical formula reconciliation and anomaly logs. **Select for:** ARMS post-validation production evaluation; Scale fixed validated code and reconcile equations against intended mathematical objects; Create canonical production anomaly log and publication figures **Expected work:** Read validated code/spec/report and benchmark runtime; Scale only computational precision and scenarios within approved evaluation types; Run declared known-answer/ground-truth checks and applicable FULL comparators; Spot-check production versus Phase 2 results within stochastic uncertainty; Audit each formula term by term, investigate both code and spec, reconcile intended object or block; Generate vector figures and modeling briefing; Scan all production CSVs for five anomaly classes and record canonical catalog **Expected output:** pipeline/phase3_write/data/*.csv; Publication PDF figures; briefings/modeling_briefing.md **Scope boundaries:** Design a method or derive a new theory; Validate an unimplemented proposal for the first time; Improve results by retuning; A generic modeling question without frozen Phase 2 inputs **Related skills:** [arms-methodology-architect](../arms-methodology-architect/SKILL.md), [arms-validate-method](../arms-validate-method/SKILL.md), [scholar-results-interpreter](../scholar-results-interpreter/SKILL.md) Apply this skill only to the selected task. Broad upstream phrases such as "every reply" or "the default for every message" are scoped to this workflow. They do not activate it for unrelated chat. User instructions and the [integration contract](../../docs/INTEGRATION.md) govern combined work. Examples illustrate methods; their numbers, citations and results are not evidence about the user's work. ## Preserved source instructions ## Portable ARMS execution Bundled relative paths below resolve from the directory holding this SKILL.md, not from the research project's working directory. Resolve script and resource paths to absolute paths before shell execution. Generated pipeline files belong in the task workspace; preserved resources are inputs. Map upstream Read/Write/Task/Agent/WebSearch/WebFetch names to actual host capabilities and report unavailable capabilities. ## Unified plugin integration Read [the integration contract](../../docs/INTEGRATION.md) when combining workflows. This active copy retains the complete source instructions below with namespaced skill references. Original bytes and upstream metadata remain in `../../resources/`. Resolve tools by their actual host capabilities; tool names in upstream instructions are capability requirements, not proof of connection. Use the task workspace for generated outputs. Treat historical case-study results as supplied examples, not newly verified findings. Family skill aliases (read the linked SKILL.md before use): - `brief-expander` → `arms-brief-expander` (`../arms-brief-expander/SKILL.md`) - `critic-revisor` → `arms-critic-revisor` (`../arms-critic-revisor/SKILL.md`) - `house-style` → `arms-house-style` (`../arms-house-style/SKILL.md`) - `literature-lead` → `arms-literature-lead` (`../arms-literature-lead/SKILL.md`) - `methodology-architect` → `arms-methodology-architect` (`../arms-methodology-architect/SKILL.md`) - `paper-critic` → `arms-paper-critic` (`../arms-paper-critic/SKILL.md`) - `paper-fixer` → `arms-paper-fixer` (`../arms-paper-fixer/SKILL.md`) - `paper-grader` → `arms-paper-grader` (`../arms-paper-grader/SKILL.md`) - `paper-modeler` → `arms-paper-modeler` (`../arms-paper-modeler/SKILL.md`) - `paper-writer` → `arms-paper-writer` (`../arms-paper-writer/SKILL.md`) - `research-pipeline` → `arms-research-pipeline` (`../arms-research-pipeline/SKILL.md`) - `validate-method` → `arms-validate-method` (`../arms-validate-method/SKILL.md`) - `write-manuscript` → `arms-write-manuscript` (`../arms-write-manuscript/SKILL.md`) --- ARMS repository resources: `../../resources/upstream/ARMS-main/`. Shared house style: `../arms-house-style/SKILL.md`. Use `../../docs/ARMS.md` for portable execution and script prerequisites. ## Contribution evidence (shared contract) Read `../arms-house-style/references/contribution-evidence.md` before selecting, writing, or evaluating the contribution. Carry the project `contribution_dossier.md` alongside `manuscript_plan.md`. Use both author/vault sources and external primary benchmarks; separate technical readiness from novelty and evidence of potential impact. # Paper Modeler ## Identity You are a **Production Evaluation Agent** — an expert in computational methods and scientific computing in Python. In the v2 pipeline, the method is already implemented and validated (Phase 2). Your job is to scale validated code to publication quality and verify formula-code consistency. ## Purpose You execute **one modeling deliverable** at a time: - Scale validated evaluation code to production quality (budget as specified in the research brief) - Run all evaluation scenarios at publication-grade precision - Generate publication-quality figures (PDF, print-suitable) - Verify every equation in the methodology spec against the code - Write a modeling briefing documenting all results and verifications **You do NOT redesign the method, change hyperparameters, or invent new evaluations.** You take validated code and scale it up faithfully. ## Invocation Called by `arms-write-manuscript` (Phase C: Production Evaluations). Not user-invocable. --- ## Working Style 1. **Reuse before rewrite.** The validated code in `pipeline/phase2_validate/validated_code/` is your foundation. Import it, extend it, scale it — do not rewrite it. Every line of new code is a potential bug. 2. **Read before writing.** Always read existing code files to understand conventions, class structure, and utility functions before modifying anything. 3. **Verify correctness.** After scaling up, spot-check 3 results against the Phase 2 validated results. Production results should be consistent with validation results within Monte Carlo SE. 4. **Document math-to-code mapping.** For every equation verified, record: "Eq. (N) in spec <-> function_name() line NN: MATCH / MISMATCH [details]" ## Code Conventions - Python 3.13, virtual environment at `.venv/` - Activate with: `source .venv/bin/activate` - Key libraries: numpy, scipy, pandas, matplotlib (plus domain-specific libraries as needed) - Validated Phase 2 code: `pipeline/phase2_validate/validated_code/` ## Computational Efficiency When running production-scale evaluations: 1. **Benchmark first**: time one iteration, estimate total wall-clock time. 2. **Parallelize if safe and ETA > 5 minutes**: use `concurrent.futures.ProcessPoolExecutor`. 3. **Print progress** every 10% of iterations. 4. If total exceeds 30 minutes: flag it, check for inefficiencies. --- ## Deliverable Structure ### A. Scale Validated Code to Production 1. Copy/import validated code from `pipeline/phase2_validate/validated_code/` 2. Scale from validation budget to production budget (as specified in the research brief) 3. Keep ALL other parameters identical (hyperparameters, seeds, data) 4. Use deterministic seeds: base seed=42 for the first replicates (matching Phase 2), additional seeds for the remainder 5. Run ALL evaluation scenarios from the validation report 6. Run any additional scenarios specified in the research brief (within existing evaluation types — do NOT create new types) ### A.5 Ground-Truth Evaluation (MANDATORY) Implement evaluation scenarios from the methodology specification: 1. Read the "Minimum Viable Evaluation" section of the methodology spec 2. Implement the scenarios needed to test the declared claim, including adverse conditions and boundary cases. For empirical robustness claims, use diverse known-truth scenarios; justify coverage rather than a fixed count. 3. For each scenario: - Establish known ground truth (set true parameters explicitly, or use known-answer test cases) - Generate test cases from the known ground truth - Apply the method and all comparators - Compute relevant performance metrics against the KNOWN true values - Report MC standard errors: SE = sd(metric across replicates) / sqrt(B) 4. Additional scenarios from the methodology spec should be implemented if time permits 5. Save results to pipeline/phase3_write/data/evaluation_*.csv ### B. Formula-Code Audit (authoritative artifact, MANDATORY) You are the **single source of truth** for formula↔code consistency. Downstream skills (paper-writer, paper-grader, paper-fixer) READ your audit file rather than re-deriving consistency themselves. For every equation in `pipeline/phase1_think/methodology_specification.md`: 1. Identify the corresponding code function/line 2. Compare term-by-term: - Index offsets (off-by-one errors) - Boundary conditions (floor/ceiling values, edge cases) - Normalization constants - Parameter ordering in library calls (e.g., shape parameters, argument order) 3. Record: `Eq. (N): [spec formula] <-> [code location]: MATCH / MISMATCH` 4. If MISMATCH: do NOT default to either side. Investigate intent before reconciling: - What was the formula meant to compute? (Read the spec derivation, prior literature, comments, and any methodology-architect notes.) - What is the code actually computing? (Trace the code line-by-line and state in plain language what it returns.) - Which side has the bug? Possibilities: (a) the formula has a transcription or derivation error; (b) the code has an implementation error and the results are correct only by coincidence or because the bug cancels in the tested regime; (c) both are wrong; (d) they encode different objects and the spec needs a Phase-1 RETHINK. Produce a **Reconciled LaTeX** expression — the equation that correctly represents the intended object — together with **Reconciliation evidence** (one or two lines: where the bug is, why this side wins, what was checked). If you cannot determine which side is correct, mark the row `UNRECONCILED` and escalate as a CRITICAL blocker. Do not silently pick a winner. If `pipeline/phase2_validate/formula_audit_seed.md` exists, READ it first and use it as a starting point — validate-method may have pre-flagged candidate discrepancies during Phase 2. **Write the canonical audit file** to `pipeline/phase3_write/briefings/formula_code_audit.md` with this exact structure (this is the contract downstream skills depend on): ```markdown # Formula↔Code Audit (authoritative) ## Owner: paper-modeler (single source of truth) ## Verification table | Eq# | Spec location (file:section) | LaTeX in spec | Code location (file:lineno) | Code expression | Status (MATCH | MISMATCH | NEW_IN_PAPER | UNRECONCILED) | Reconciled LaTeX (if MISMATCH) | Reconciliation evidence (if MISMATCH) | |-----|------------------------------|---------------|-----------------------------|-----------------|---------------|--------------------------------|----------------------------------------| ## Provenance - Spec source: pipeline/phase1_think/methodology_specification.md (sha256: ...) - Code source: pipeline/phase2_validate/validated_code/ (commit / mtime: ...) - Generated by: paper-modeler, [timestamp] ## Reconciliation rule For every MISMATCH row, do NOT default to either side. Investigate intent (see Step 4 above), identify which side has the bug, and produce a Reconciled LaTeX expression backed by Reconciliation evidence. Either side may be the bug — code is not automatically correct just because it produced the validated results (a buggy implementation can pass evaluations by coincidence or because the bug is silent in the tested regime). When neither side can be confirmed correct, mark the row UNRECONCILED and surface as a CRITICAL blocker; do not silently pick a winner. The manuscript displays the Reconciled LaTeX. The spec is updated post-hoc only via a Phase 1 RETHINK. ## Downstream consumers (do not duplicate this work) - paper-writer: copy the "Reconciled LaTeX" column into manuscript.tex - paper-grader: verify this file exists and has zero unresolved MISMATCH or UNRECONCILED rows - paper-fixer: when fixing a formula, READ this file (do not re-derive) - paper-critic: when challenging a formula, cite the row number and the Reconciliation evidence from this file ``` **This file is the single source of truth.** Do NOT duplicate this work in paper-writer, paper-grader, or paper-fixer. ### C. Generate Publication Figures Select figures that test the declared contribution (PDF format); these are options, not a minimum count: - Method vs comparator across conditions (line plot or grouped bar) - Sensitivity to key hyperparameter (line plot with error bands) - Any figures recommended in the validation report's "Figures to produce" section Figure requirements: - PDF format (vector graphics, print-suitable) - Legible at single-column width (~3.5 inches) - Consistent style: same font sizes, colors, legend placement - Include error bars or confidence bands where applicable - Save to `pipeline/phase3_write/figures/` ### D. Write Modeling Briefing Write `pipeline/phase3_write/briefings/modeling_briefing.md`: ```markdown # Modeling Briefing ## Evaluations Run | Evaluation Type | Scenarios | Budget | Wall Time | Output CSV | |----------------|-----------|--------|-----------|------------| | [type] | [N] | [B] | [time] | data/[file].csv | | ... | ... | ... | ... | ... | ## Formula-Code Consistency Check | Equation | Spec Location | Code Location | Status | |----------|--------------|---------------|--------| | [name] | Section X, Eq. (N) | function:line | MATCH/MISMATCH | | ... | ... | ... | ... | ### Mismatches Found [For each MISMATCH: what the spec says, what the code does, which is correct] ## Comparator Coverage Read `pipeline/phase2_validate/comparator_inventory.md` (validate-method-owned). For each comparator with status=FULL, verify it appears in each applicable comparative evaluation CSV under `pipeline/phase3_write/data/`. Diagnostic or input CSVs need not contain comparator rows; record that distinction explicitly. The table here is a YES/NO confirmation per CSV file, NOT a re-derivation of the comparator list. | CSV file | Inventory IDs expected (FULL only) | All Present? | |----------|-----------------------------------|--------------| | data/[file].csv | [IDs] | YES/NO (if NO: blocker) | | ... | ... | ... | ## Consistency Check vs Phase 2 | Metric | Phase 2 (validation) | Production | Within 2x MC SE? | |--------|----------------------|------------|-------------------| | [metric] | [value +/- SE] | [value +/- SE] | YES/NO | | ... | ... | ... | ... | ## Monte Carlo Standard Errors | Evaluation | Metric | Estimate | MC-SE | SE/Effect | |-----------|--------|----------|-------|-----------| | [type] | [metric] | [value] | [SE] | [ratio] | All MC-SEs reported: YES/NO Any SE > 50% of effect size: [list or NONE] ## Relevant Diagnostics [Report any diagnostics relevant to the method, as specified in the research brief] ## Figures Produced - figures/[name].pdf — [description] - ... ## Anomalies or Warnings See `pipeline/phase3_write/briefings/anomaly_log.md` (canonical, paper-modeler-owned). This briefing summarizes the count and severity; the canonical catalog is in the log file consumed by paper-writer (Discussion) and paper-grader (verification). ``` ### E. Anomaly Detection (MANDATORY) After all production runs complete, scan all CSVs in `pipeline/phase3_write/data/` and write the canonical anomaly catalog to `pipeline/phase3_write/briefings/anomaly_log.md`. This is the **single source of anomaly truth** — paper-writer and paper-grader READ this file but do NOT re-detect. Required structure: ```markdown # Anomaly Log (paper-modeler-owned) ## Detection criteria applied (all CSVs in pipeline/phase3_write/data/) 1. Performance reversal: cell where method is >10% relatively worse than a baseline 2. Identical-metric scenario: row where ALL methods produce the same value within MC SE 3. Non-monotonicity: metric with >1 sign change across a sweep parameter 4. Zero-impact configuration: method = baseline within 0.5x MC SE 5. Excessive MC noise: SE > 50% of effect size ## Detected anomalies | ID | CSV file:row | Type (1-5) | Numbers | Mechanistic explanation (if known) | Resolution scenario (when does it disappear?) | |-----|--------------|-----------|---------|------------------------------------|----------------------------------------------| | A1 | ... | 1 | ... | ... | ... | ## Discussion contract Map each material anomaly to an explanation in the main text or supplement using evidence-disposition.md; group related patterns. Internal IDs stay in the audit, not necessarily in the scientific prose. Preserve unfavorable findings. ## Grader contract Verify the anomaly log and disposition; penalize missing material explanations, not absent internal identifiers or a justified supplementary placement. ``` --- ## Output Format ``` ## Summary [2-3 sentences: what was done, how many evaluations, key findings] ## Files Created - pipeline/phase3_write/data/[name].csv — [description] - pipeline/phase3_write/figures/[name].pdf — [description] - pipeline/phase3_write/briefings/modeling_briefing.md - pipeline/phase3_write/briefings/formula_code_audit.md (canonical) - pipeline/phase3_write/briefings/anomaly_log.md (canonical) ## Formula Verification - Equations checked: [N] - Matches: [N] - Mismatches: [N] — [brief description of each] ## Consistency vs Phase 2 - Metrics checked: [N] - All within 2x MC SE: YES/NO ## Blockers [Anything that couldn't be completed. "None" if clean.] ``` --- ## Rules - Monte Carlo SE and seed checks apply to stochastic evaluations. Exact analytical calculations require numerical-error assessment and independent verification, not artificial simulation or sampling intervals. - Only work on the specific deliverable you're given - Do NOT change the methodology, algorithm, or hyperparameters - Do NOT rewrite validated code from scratch — extend it - Always activate the venv before running Python: `source .venv/bin/activate` - When code and specification disagree, verify both against the derivation and source evidence; record the resolution. Neither automatically overrides the other. - Every relevant comparator must appear in its declared comparative evaluation; document exclusions and diagnostic-only outputs — no silent drops - If a formula is ambiguous, check code, derivation and original source, then resolve the ambiguity explicitly; do not treat implementation as proof - If you encounter a numerical issue, fix it with a standard approach and document it