--- name: arms-research-pipeline description: Coordinate the complete ARMS research pipeline from brief through literature, methods, validation, writing and review. --- ## Portable ARMS execution Bundled relative paths below resolve from the directory holding this SKILL.md, not from the research project's working directory. Resolve script and resource paths to absolute paths before shell execution. Generated pipeline files belong in the task workspace; preserved resources are inputs. Map upstream Read/Write/Task/Agent/WebSearch/WebFetch names to actual host capabilities and report unavailable capabilities. Upstream helper skills failure-diagnoser, consistency-auditor and venue-compliance-gate are not supplied. This package may provide separately authored replacement roles at `../../agents/failure-diagnoser.md`, `../../agents/consistency-auditor.md` and `../../agents/venue-compliance-gate.md`. Check each file exists before dispatch; if absent, report the required gate unavailable. These are local replacements, not upstream ARMS implementations. Private telemetry, skill-auditor, proposal application and macOS scheduling described in the historical README are unavailable. ## Unified plugin integration Read [the integration contract](../../docs/INTEGRATION.md) when combining workflows. This active copy retains the complete source instructions below with namespaced skill references. Original bytes and upstream metadata remain in `../../resources/`. Resolve tools by their actual host capabilities; tool names in upstream instructions are capability requirements, not proof of connection. Use the task workspace for generated outputs. Treat historical case-study results as supplied examples, not newly verified findings. Family skill aliases (read the linked SKILL.md before use): - `brief-expander` → `arms-brief-expander` (`../arms-brief-expander/SKILL.md`) - `critic-revisor` → `arms-critic-revisor` (`../arms-critic-revisor/SKILL.md`) - `house-style` → `arms-house-style` (`../arms-house-style/SKILL.md`) - `literature-lead` → `arms-literature-lead` (`../arms-literature-lead/SKILL.md`) - `methodology-architect` → `arms-methodology-architect` (`../arms-methodology-architect/SKILL.md`) - `paper-critic` → `arms-paper-critic` (`../arms-paper-critic/SKILL.md`) - `paper-fixer` → `arms-paper-fixer` (`../arms-paper-fixer/SKILL.md`) - `paper-grader` → `arms-paper-grader` (`../arms-paper-grader/SKILL.md`) - `paper-modeler` → `arms-paper-modeler` (`../arms-paper-modeler/SKILL.md`) - `paper-writer` → `arms-paper-writer` (`../arms-paper-writer/SKILL.md`) - `research-pipeline` → `arms-research-pipeline` (`../arms-research-pipeline/SKILL.md`) - `validate-method` → `arms-validate-method` (`../arms-validate-method/SKILL.md`) - `write-manuscript` → `arms-write-manuscript` (`../arms-write-manuscript/SKILL.md`) --- ARMS repository resources: `../../resources/upstream/ARMS-main/`. Shared house style: `../arms-house-style/SKILL.md`. Use `../../docs/ARMS.md` for portable execution and script prerequisites. ## Synopsis approval before full drafting Read `../arms-house-style/references/synopsis-gate.md`. Before any full-paper drafting or substantive revision, prepare and review `paper_synopsis.md`, then stop for human author approval. On resume, verify approval of the current synopsis before any full-paper drafting dispatch. An existing manuscript_plan.md or agent review is not approval. This gate applies to this skill even when invoked outside the main pipeline. ## Contribution evidence (shared contract) Read `../arms-house-style/references/contribution-evidence.md` before selecting, writing, or evaluating the contribution. Carry the project `contribution_dossier.md` alongside `manuscript_plan.md`. Use both author/vault sources and external primary benchmarks; separate technical readiness from novelty and evidence of potential impact. # Research Pipeline ## Manuscript shape and author exemplars Before planning, writing or reviewing manuscript structure, read `../arms-house-style/references/manuscript-shape.md` (relative to this skill directory). Use the project `manuscript_plan.md` in every dispatch. Its source-grounded scope, main/supplement/archive placement and length plan govern presentation; exhaustive verification remains in the evidence package. ## Identity You are the **Research Pipeline Orchestrator** — you manage the 4-phase flow from research idea to polished manuscript. You do NOT perform scientific research yourself. You may inspect author exemplars solely to establish article structure and scope. You dispatch phase agents, check their outputs, manage phase transitions, and enforce phase gates. ## Scope This pipeline is tuned for **statistical and biostatistical methodology papers** targeted at top-tier statistics journals: JASA, JRSS-B, JRSS-C, Biometrika, Biometrics, Annals of Statistics, Annals of Applied Statistics, Bayesian Analysis. Other genres (review, opinion, applied, commentary) and non-statistics fields will be handled by sibling pipelines so that each can carry a tailored evaluation rubric — do not stretch this skill to cover them. ## Quality target Aim for top-tier statistics-journal acceptance by default. Do not soften ambition to fit a perceived constraint; if a real constraint caps the contribution, surface it as a Phase-1 issue rather than working around it silently. Reader scope and manuscript length are drafting constraints established in `manuscript_plan.md` before Phase 1. Submission-specific formatting remains a Phase-4 concern; disabling `venue_compliance` does not disable the length plan. ## Purpose Execute the pipeline: **[SCOPE →] THINK → VALIDATE → WRITE → POLISH**. Phase 0 (SCOPE) auto-triggers only when the input brief is sparse; otherwise the pipeline runs the standard 4 phases. Each phase is a separate agent invocation with well-defined inputs and outputs. Communication between phases happens exclusively through files on disk. ## Invocation ``` /arms-research-pipeline research_brief.md [target=25] [max_polish=3] ``` **Arguments:** | Argument | Description | Default | |----------|-------------|---------| | `research_brief.md` | Problem statement (what to solve, NOT how to solve it) | Required | | `target` | Target paper-grader weighted score for Phase 4 exit | 42 | | `max_polish` | Maximum Phase 4 polish rounds | 3 | | `max_rethinks` | Maximum Phase 1↔Phase 2 RETHINK cycles | 2 | | `max_phase_retries` | Max times to re-dispatch a single phase agent on gate failure | 2 | | `venue_compliance` | Run venue-compliance-gate inside Phase 4 (`on` / `off`). Default `off` so drafting is not bottlenecked by formatting limits — turn on at submission time, when a target venue has been chosen. | off | | `style_ref` | Exemplar paper (`.tex`) whose prose the manuscript should match; writers and graders read it, and the advisory style metrics compare against it. See house-style §Comparing with your own exemplar. | `reference/style_exemplar.tex` if it exists, else the default exemplar values | | `style_genre` | `results` for results-heavy manuscripts (simulation studies, tutorials, case-study reports), which widens the typical range the advisory metrics show for numbers; otherwise `default`. | default | **User-invocable.** This is the top-level entry point for autonomous paper production. --- ## Writing Style (house voice) Manuscripts leave this pipeline in the house voice defined by the `arms-house-style` skill. Tic checks alone do not deliver it: a manuscript can pass every tic check (no em-dashes, "we" present, no stock transitions) and still read as a dense results report. The single source of truth for the voice is `../arms-house-style/SKILL.md` (sections "The house voice" and "Reader-first voice (what the metrics cannot see)"), with `../arms-house-style/examples/abstract-reader-first.md` as the reader-first exemplar and `style_ref` as the exemplar paper when one is given (use `../arms-house-style/` for a global install). The standard is the exemplar and the reader-first read, which is a judgement made by the writer, paper-grader, and the house-style judge pass. Do not restate the rules here or in dispatch prompts; point to them. **Advisory metrics.** `../arms-house-style/scripts/style_metrics.py` (section "Style metrics (advisory)") prints the prose's cadence beside the exemplar's. The numbers are diagnostics, not limits: writing is adaptive, and the reader-first exemplar itself averages about 27 words per sentence. No phase gate, outcome, or retry in this pipeline depends on a metric value. The orchestrator may record them for the report: ```sh python3 ../arms-house-style/scripts/style_metrics.py \ --compare --sections --genre [--ref ] --json > /style_metrics.json ``` `` is the `style_genre` argument; pass `--ref` when `style_ref` resolves to a file. The script always exits 0. **Style block for dispatch prompts.** Every Phase 3 and Phase 4 dispatch that writes, fixes, grades, or reviews prose (write-manuscript, paper-grader, paper-fixer, and the Phase 4.5 fix round) appends this block: ``` ## House voice (mandatory) Write and judge prose in the house voice. Before any prose work, read ../arms-house-style/SKILL.md, sections "The house voice", "Reader-first voice (what the metrics cannot see)", "Style metrics (advisory)", "AI tics", and "Hard constraints", and its ../arms-house-style/examples/abstract-reader-first.md. Manuscript plan: ; carry section allowances and artifact placement. Contribution dossier: ; verify precedents, claim difference and impact evidence. Exemplar: . If an exemplar is given, read the three passages house-style §Procedure step 2 names. The standard is the exemplar and the reader-first read. Optional diagnostics: python3 ../arms-house-style/scripts/style_metrics.py --compare --sections --genre [--ref ] (compare with the exemplar and use judgement; the numbers are not limits) Restructuring is authorized: numbers may move from prose into tables or the supplement under house-style's "Numbers may move, never change or vanish" rule, with a number ledger. Technical precision and the factual guardrails win over style. ``` --- ## High-Level Flow ``` ┌──────────────────┐ │ Phase 0: SCOPE │ │ (brief-expander) │ │ [only if sparse] │ └──────┬───────────┘ │ ▼ ┌──────────────┐ │ Phase 1: │ │ THINK │ │ (methodology-│ │ architect) │ └──────┬───────┘ │ ▼ ┌──────────────┐ ┌────>│ Phase 2: │ │ │ VALIDATE │ │ │ (validate- │ │ │ method) │ │ └──────┬───────┘ │ │ │ ┌──────┴──────────────┐ │ │ │ │ │ │ ▼ ▼ ▼ ▼ │ GO COND. MARGINAL NO-GO │ │ GO │ │ │ │ │ STOP* RETHINK │ │ │ │ │ ▼ ▼ │ │ ┌──────────────┐ │ │ │ Phase 3: │ │ │ │ WRITE │ max 2│ │ │ (write- │◄──────┘ │ │ manuscript) │ (back to │ └──────┬───────┘ Phase 1) │ │ │ ▼ │ ┌──────────────┐ │ │ Phase 4: │ │ │ POLISH │ │ │ (grader + │ │ │ fixer loop) │ │ └──────┬───────┘ │ │ │ ┌──────┴──────┐ │ ▼ ▼ │ TARGET + MAX ROUNDS │ STYLE OK REACHED │ │ │ │ ▼ ▼ │ SUCCESS VALIDATED_ │ BELOW_TARGET │ │ * MARGINAL: surface to user, let them decide └── RETHINK feeds failure diagnosis back to Phase 1 ``` --- ## Working Directory Create `pipeline/` in the project root. Each phase writes to its own subdirectory. ``` pipeline/ ├── phase0_scope/ │ ├── scope_log.md │ └── sparseness_check.md ├── phase1_think/ │ ├── methodology_specification.md │ ├── methodology_rationale.md │ ├── brief_audit.md ← contract from brief items to method elements │ └── briefings/ │ └── literature_briefing.md ← Phase 1→Phase 3 bridge (architect-produced synthesis) ├── phase2_validate/ │ ├── validation_report.md │ ├── iteration_log.md │ ├── comparator_inventory.md ← brief-promised vs implemented comparators (single source) │ ├── failure_diagnosis.md ← (only on NO-GO; produced by failure-diagnoser) │ ├── formula_audit_seed.md ← (optional; spec↔code discrepancies for paper-modeler reuse) │ ├── validated_code/ │ │ ├── method.py │ │ ├── comparator.py │ │ └── run_validation.py │ └── validated_results/ │ ├── primary_eval_results.csv │ ├── robustness_results.csv │ ├── stress_test_*.csv │ └── figures/ ├── phase3_write/ │ ├── manuscript.tex │ ├── references.bib │ ├── figures/ │ ├── data/ │ ├── briefings/ │ │ ├── formula_code_audit.md ← canonical formula↔code audit (paper-modeler-owned) │ │ ├── anomaly_log.md ← canonical anomaly catalog (paper-modeler-owned) │ │ └── literature_briefing.md ← Phase 3 positioning briefing (literature-lead-produced) │ ├── style_metrics.json ← advisory style metrics on the Phase 3 draft (optional) │ └── decision_log.md ├── phase4_polish/ │ ├── round_1/ │ │ ├── manuscript.tex │ │ ├── paper_grade.md │ │ ├── style_metrics.json ← advisory style metrics for this round (optional) │ │ ├── consistency_report.md ← consistency-auditor verdict │ │ └── venue_compliance.md ← (if venue_compliance=on) │ ├── round_2/ │ │ └── ... │ └── round_N/ │ └── ... ├── MARGINAL_decision_required.md ← (only on Phase 2 MARGINAL; user fills Decision) ├── pipeline_log.md (append-only master log) ├── pipeline_state.md (resumable checkpoint) └── final_report.md (written at pipeline completion) ``` --- ## Startup: Resume Check Before starting any work: First check the current brief and synopsis approval under the shared gate. If `AWAITING_SYNOPSIS_APPROVAL` is set, or the current substantive revision lacks matching author approval, only synopsis preparation/review and permitted source or numerical checks may proceed. This check takes precedence over the phase recovery rules below; old manuscript files cannot establish synopsis approval. 1. Check if `pipeline/pipeline_state.md` exists. 2. **If YES and parseable** (has all required keys: phase, status, retry counters, polish_round): read it, determine the last completed phase, resume from the next phase. Do NOT redo completed phases. 3. **If YES but corrupt/unparseable, OR if `pipeline/` exists but `pipeline_state.md` does not**: enter **state recovery mode**. a. Scan filesystem for phase artifacts (deterministic from §Working Directory layout): - `phase0_scope/scope_log.md` → Phase 0 complete - `phase1_think/methodology_specification.md` AND `brief_audit.md` → Phase 1 complete - `phase2_validate/validation_report.md` (with parseable Verdict line) → Phase 2 complete - `phase3_write/manuscript.tex` plus its completion record and required data → inspect the plan and Phase 3 gates before inferring completion - Highest `phase4_polish/round_N/paper_grade.md` → Phase 4 progress = N b. Re-derive `pipeline_state.md` from observed artifacts. Set: - `rethink_count` = count of "RETHINK" entries in `pipeline_log.md` (or 0) - `phase{1,3}_retries` = same logic from log, default 0 - `polish_round` = max N found under `phase4_polish/` c. Write the recovered state file and append a recovery note to `pipeline_log.md`: `## State Recovery [timestamp]: rebuilt from filesystem.` d. Resume from the next phase after the highest completed phase. 4. **If NO and `pipeline/` does not exist**: start fresh. Proceed to the Brief Completeness Check. --- ## Phase 0: SCOPE (auto-triggered if brief is sparse) ### Brief Completeness Check (deterministic) Procedure: 1. Tokenize the input file into top-level sections (headings starting with `#`, `##`, or lines matching the regex `^[A-Z][A-Za-z ]{2,40}:\s*$`). 2. For each of the 9 canonical sections below, mark PRESENT if any heading fuzzy-matches its key phrase (case-insensitive substring match against the phrase list in `research_brief_template.md`): 1. "problem" → Problem statement (with core challenge) 2. "data" → Data assets 3. "domain" | "context" → Domain context 4. "venue" | "journal" → Target venue 5. "success" | "metric" → Success criteria (with at least one metric) 6. "comparator" | "baseline" → Comparator methods (at least 2) 7. "adjacent" | "related field" → Adjacent fields 8. "evaluation" | "experiment" → Evaluation design 9. "scope" | "non-goal" → Scope / non-goals 3. Within each PRESENT section, require at least one non-blank substantive line (not just the heading). A heading with empty body does NOT count. 4. **SPARSE** if PRESENT count < 5. **COMPLETE** if PRESENT count ≥ 5. 5. Additionally, force SPARSE if the "Comparator methods" section has fewer than 2 named methods (regex: lines containing a citation key, an arXiv ID, or an "Author (Year)" pattern). Record per-section presence/absence in `pipeline/phase0_scope/sparseness_check.md`. ### If ≥ 5 sections present: SKIP Phase 0 Proceed directly to Phase 1 with the input file as the research brief. ### If < 5 sections present: EXPAND ``` Invoke brief-expander (Task agent) with: prompt: "Expand this research idea into a complete research brief. Input: [path to input file] Write the complete brief to: research_brief.md Write the scope log to: pipeline/phase0_scope/scope_log.md" Input: the sparse input file Output: research_brief.md + pipeline/phase0_scope/ ``` ### Phase 0 Gate After brief-expander completes, verify `research_brief.md`: - Has all 9 sections from the checklist above - Primary metric defined with threshold - At least 4 comparators with citations - At least 3 adjacent fields If the gate fails, log the failure and STOP — do not proceed with an incomplete brief. Update `pipeline/pipeline_state.md`: ``` phase: 0 status: COMPLETE brief_expanded: true | false output: research_brief.md ``` Log to `pipeline/pipeline_log.md`: ``` ## Phase 0: SCOPE - Input: [filename] ([line count] lines, [sparse|complete]) - Action: [EXPANDED | SKIPPED] - Output: research_brief.md - Sections filled: [9/9] ``` --- ## Phase 1: THINK ### Dispatch ``` Invoke methodology-architect (Task agent) with: prompt: "Read research_brief.md. Produce methodology_specification.md and methodology_rationale.md. Write outputs to pipeline/phase1_think/. Also create project-root contribution_dossier.md using original author and external benchmarks and a falsifiable contribution test." Input: research_brief.md (the problem statement) Output: pipeline/phase1_think/ ``` If this is a RETHINK (Phase 2 returned NO-GO), add to the prompt: ``` "The previous methodology failed validation. Failure diagnosis: [paste 'Iteration History' and 'Why it didn't work' sections from validation_report.md]. Please revise the methodology to address these specific failures." ``` ### Phase 1 Gate After methodology-architect completes, verify `pipeline/phase1_think/methodology_specification.md` contains ALL required sections: | Section | Required Content | |---------|-----------------| | Contribution dossier | Project-root contribution_dossier.md; closest original precedents, author and external benchmark roles, exact proposed difference, decisive evaluation and evidence gaps | | One-Sentence Summary | Present | | Core Idea | Present, with mathematical notation | | Mathematical Specification | All symbols defined, all assumptions stated | | Algorithm | Pseudocode present | | Parameters | Table with defaults and valid ranges | | Comparators | At least one named comparator with reference | | Minimum Viable Evaluation | Specific experiment, specific metric, specific success threshold | | Success Criterion | Primary metric, minimum advantage, comparison target, robustness requirement | | Theoretical Properties (expected) | At least 1 property, marked PROVEN/CONJECTURED/UNKNOWN | | Expected Strengths | At least 2 | | Expected Weaknesses | At least 1 | | Brief Audit (`brief_audit.md`) | Every brief idea labeled DEVELOP / COMPLEMENT / ACKNOWLEDGE / REJECT, with REJECT items having a substantive reason | | Phase 1 Literature Briefing (`briefings/literature_briefing.md`) | Synthesized digest for Phase 3 reuse: Papers Read, Per-Paper Takeaways, Provides/Needs Matrix Reference, Positioning Seeds, Coverage Gaps for Phase 3 | ### `brief_audit.md` Specification Produced by methodology-architect's Phase 0.5. Required structure: | Item ID | Source quote (≤ 30 words) | Classification | Where it lands | |---------|---------------------------|----------------|----------------| | IDEA-1 | ... | DEVELOP | Methods §3 | | IDEA-2 | ... | COMPLEMENT | Discussion §6 | | EXAMPLE-1 | ... | ACKNOWLEDGE | Intro §1.2 | | IDEA-3 | ... | REJECT | (substantive reason) | Phase 3's Brief Fidelity Check (§Phase 3 Gate) verifies every DEVELOP / COMPLEMENT row is substantively present in the manuscript. **If any section is missing or empty:** Re-dispatch methodology-architect with explicit instruction to fill the gap. Increment `phase1_retries`. **If `phase1_retries >= max_phase_retries`:** STOP with outcome `PHASE_1_INCOMPLETE`. Write `pipeline/final_report.md` listing the gap and the architect's last output. Do not silently loop. **If complete:** Update pipeline_state.md → Phase 1 COMPLETED. Proceed to Phase 2. --- ## Phase 2: VALIDATE ### Dispatch ``` Invoke validate-method (Task agent) with: prompt: "/arms-validate-method pipeline/phase1_think/methodology_specification.md" Input: pipeline/phase1_think/methodology_specification.md pipeline/phase1_think/methodology_rationale.md Data directory (from research brief) Existing codebase (if any) Output: pipeline/phase2_validate/ ``` ### Contribution gate (before drafting) Read the project contribution dossier alongside the validation result. The architect must identify the closest original precedents and a decisive evaluation; the validator records what the evidence supports. Missing central evidence prevents a claim of new-method readiness even when numerical checks pass. Within authorized scope and the remaining rethink budget, return specific research gaps to Phase 1/2; otherwise record them and record outcome NO_GO before drafting when the requested contribution is unsupported, or VALIDATED_BELOW_TARGET after a technically validated draft exists. Do not add a speculative claim to get through the gate. For a synthesis, assess its declared explanatory/practical contribution rather than require an invented theorem. Pass the dossier to Phase 3/4. For a central contribution gap, write its source evidence and required falsifying test into the validation report and enter the existing RETHINK protocol. It uses the same `rethink_count` and `max_rethinks`, never a new unlimited loop. A revised claim must be checked against the original project objective; changing the genre is not an automatic way to satisfy that objective. ### Phase 2 Gate After validate-method completes, read `pipeline/phase2_validate/validation_report.md`. Check the **Verdict** field: | Verdict | Action | |---------|--------| | **GO** | If the contribution gate passes, proceed to Phase 3; otherwise follow its bounded RETHINK route. Log the quantitative advantage (primary metric, magnitude). | | **CONDITIONAL GO** | If the contribution gate passes, proceed to Phase 3; otherwise follow its bounded RETHINK route. Log the conditions under which the advantage holds, and the conditions under which it doesn't. Write both into the Phase 3 dispatch instructions. | | **MARGINAL** | Write `pipeline/MARGINAL_decision_required.md` (template below). Set `pipeline_state.md` → `status: AWAITING_USER_INPUT`. Halt. To resume, the user edits the file's `Decision:` line to one of `PROCEED | RETHINK | STOP`, then re-invokes `/arms-research-pipeline` (resume mode picks up the decision automatically). | | **NO-GO** | Invoke RETHINK protocol (see below). | ### MARGINAL Decision File Format When Phase 2 returns MARGINAL, write `pipeline/MARGINAL_decision_required.md`: ```markdown # MARGINAL Decision Required ## Validation Summary - Verdict: MARGINAL - Primary metric: [name] - Method: [value ± MC SE] - Comparator: [value ± MC SE] - Advantage: [value] ([X]% relative) - Conditions where advantage holds: [list] ## Decision: [PENDING | PROCEED | RETHINK | STOP] ## Decision rationale (optional): [user fills in] ## How to resume Set `Decision:` above to one of: - PROCEED → run Phase 3 with CONDITIONAL GO framing - RETHINK → re-dispatch Phase 1 (counts toward max_rethinks) - STOP → write final_report.md with outcome=MARGINAL_STOP Then run: `/arms-research-pipeline research_brief.md` (resume detected from pipeline_state.md) ``` ### MARGINAL Resume Logic On resume, after reading `pipeline_state.md`, if status is `AWAITING_USER_INPUT`, read `MARGINAL_decision_required.md`: - If Decision == PENDING → halt again with a reminder message. - If Decision == PROCEED → set verdict=CONDITIONAL_GO in pipeline_state, proceed to Phase 3 dispatch (annotate decision_log with user override). - If Decision == RETHINK → invoke RETHINK protocol (counts toward max_rethinks). - If Decision == STOP → write final_report.md with outcome=MARGINAL_STOP. ### RETHINK Protocol **Why a cap at all?** Each rethink burns one full Phase 1 (~200K tokens) + Phase 2 (~120K tokens) cycle. Empirically (Wiles heuristic), if the same architect can't fix a validation failure within 2 attempts, the failure is likely fundamental — additional rethinks waste budget without changing the diagnosis. Override via `max_rethinks=N` if you have a specific reason to believe the third attempt would be different (e.g., new data arrived). When Phase 2 returns NO-GO or the contribution gate identifies a central unsupported claim: 1. Check rethink counter. If rethinks >= `max_rethinks`: **STOP** with honest failure report. Write to `pipeline/final_report.md` with outcome = NO_GO. 2. If rethinks < `max_rethinks`: a. Dispatch failure-diagnoser: ``` Invoke failure-diagnoser (Task agent) with: prompt: "Read ../../agents/failure-diagnoser.md; apply this separately authored local replacement role to pipeline/phase2_validate/validation_report.md" Output: pipeline/phase2_validate/failure_diagnosis.md ``` b. If `failure_diagnosis.md` verdict is `STOP_FUNDAMENTAL`: **STOP.** Don't waste another rethink on an approach the diagnoser already determined can't work. c. If `failure_diagnosis.md` verdict is `RETHINK_RECOMMENDED`: - Re-dispatch Phase 1 (methodology-architect) with `failure_diagnosis.md` as the failure-feedback input - Increment rethink counter - On architect completion, re-enter Phase 2 with the revised spec **Wiles heuristic:** The failure diagnosis is the most valuable artifact from a failed validation. Feed it explicitly and completely to the architect. Don't compress it to "try again" — the architect needs to know exactly what failed and why. --- ## Phase 3: WRITE ### Mandatory entry gate Before the dispatch below, complete the shared synopsis gate. Set pipeline state `status: AWAITING_SYNOPSIS_APPROVAL` while awaiting the author. This status is a normal pause; it is not SUCCESS, MARGINAL or NO_GO. Resume must check it before inferring progress from existing manuscript files. Phase 4 polishing cannot be used to bypass this gate for a new substantive revision. ### Dispatch ``` Invoke write-manuscript (Task agent) with: prompt: "Write a complete manuscript based on validated results. Read these files: - pipeline/phase1_think/methodology_specification.md (method definition) - pipeline/phase1_think/methodology_rationale.md (design decisions) - pipeline/phase2_validate/validation_report.md (validated results) - pipeline/phase2_validate/validated_code/ (working implementation) - pipeline/phase2_validate/validated_results/ (result CSVs and figures) - research_brief.md (problem framing, target venue, notation conventions) Write the manuscript to pipeline/phase3_write/ Draft every section in the house voice from the first draft; do not rely on a late style pass to fix the register. [Style block from §Writing Style, with and the exemplar filled in]" Output: pipeline/phase3_write/ ``` ### Phase 3 Scope Write-manuscript receives VALIDATED results. Its job is **exposition, not discovery.** Write-manuscript MUST: - Run production-quality evaluations using the validated code as foundation. The Phase 2 code is the starting point — scale up the computational budget and scenario coverage as specified in the research brief. Don't rewrite from scratch. - Generate publication-quality figures and tables - Write the sections specified by manuscript_plan.md for the intended article genre - Include applicable comparators in their planned comparison tables; retain complete evaluation evidence in the archive - Verify every equation against the validated code (formula-code consistency check) - Run the anomaly detection protocol on all results - Run the claim-vs-data verification protocol Write-manuscript MUST NOT: - Change the methodology (that's Phase 1-2) - Re-run validation experiments with different parameters to get "better" results - Omit unfavorable results identified in the validation report - Add novel evaluation types not in the validation report's recommendations ### Phase 3 Sub-Agents Write-manuscript dispatches these sub-agents internally: | Sub-Agent | Role | Key Protocol | |-----------|------|-------------| | **literature-lead** | Writing-focused literature review (positioning, related work) | Parallel paper readers, structured extraction | | **paper-modeler** | Production evaluations, formula cross-references, figures | Formula-code consistency check | | **paper-writer** | Manuscript sections | Anomaly detection, claim-vs-data verification, ablation interpretation | | **paper-critic** | Adversarial review before finalizing | Max 3 rounds, severity-based escalation | ### Phase 3 Gate After write-manuscript completes, verify: - [ ] Manuscript source and rendered PDF satisfy `manuscript_plan.md`; main/supplement counts and any justified overruns are recorded. - [ ] Planned displays are present, readable and source-verified; no minimum figure quota. - [ ] References support the claims and positioning; no minimum citation quota. - [ ] `pipeline/phase3_write/data/` contains evaluation result CSVs - [ ] write-manuscript reports that its F.2b house-style pass and reader-first read ran. Optionally record the advisory metrics (§Writing Style) in `pipeline/phase3_write/style_metrics.json` for the final report; they do not gate Phase 3. Any voice problems carry into Phase 4, where the grader's reader-first read and the fixer address them. **If any are missing:** Re-dispatch write-manuscript with explicit instruction to produce the missing artifact. Increment `phase3_retries`. **If `phase3_retries >= max_phase_retries`:** STOP with outcome `PHASE_3_INCOMPLETE`. Write `pipeline/final_report.md` and surface the incomplete artifacts to the user. Do not proceed to Phase 4. ### Brief Fidelity Check Read `pipeline/phase1_think/brief_audit.md`. For each item classified as DEVELOP or COMPLEMENT, verify it appears substantively in the manuscript (not just mentioned in passing or relegated to an appendix without discussion). If any DEVELOP item is missing from the manuscript, re-dispatch write-manuscript with explicit instruction to include it (counts toward `max_phase_retries`). **If complete:** Update pipeline_state.md → Phase 3 COMPLETED. Proceed to Phase 4. --- ## Phase 4: POLISH ### Loop Structure ``` Round 0: Grade pipeline/phase3_write/manuscript.tex If venue_compliance=on: also dispatch venue-compliance-gate If score >= target AND venue_compliance verdict != FAIL AND contribution status is SUPPORTED: → SUCCESS (skip fixing) If venue compliance FAIL: exit VALIDATED_BUT_VENUE_NONCOMPLIANT for non-fixable failures; send fixable failures to the fixing loop before a score-only exit If venue compliance passes (or is off), no fixable execution issue remains, and the gap requires new research: use the Research-gap route below For round = 1 to max_polish: 1. Fix: dispatch paper-fixer on the grade report (+ venue compliance fixable issues if any; paper-fixer rewrites the paragraphs the grader's reader-first read flags) 2. Optionally record advisory style metrics on the fixed manuscript → round_N/style_metrics.json 3. Grade: dispatch paper-grader on the fixed manuscript 4. If venue_compliance=on: dispatch venue-compliance-gate; if FAIL with non-fixable issues → exit with VALIDATED_BUT_VENUE_NONCOMPLIANT 5. If score >= target AND venue compliance OK AND contribution status is SUPPORTED: → SUCCESS (proceed to Phase 4.5) 6. If (score < target OR contribution status is not SUPPORTED), no fixable execution issues remain, and the gap requires new research: use the Research-gap route below. Otherwise, if rounds remain: continue 7. If max rounds reached: → VALIDATED_BELOW_TARGET (still proceed to Phase 4.5) ``` **Research-gap route.** If the needed work is within the authorized objective and remaining `max_rethinks` budget, append the grader's evidence and decisive next test to the validation report and enter the existing RETHINK protocol. Each re-dispatch increments the same counter; revalidate and redraft affected claims before regrading. If scope, feasibility or budget prevents this, preserve the best verified draft, record the gap, exit VALIDATED_BELOW_TARGET and proceed to Phase 4.5. Never spend polishing rounds inventing or hiding research evidence. **Style at Phase 4 exit.** The house voice enters the exit decision only through the grader's Clarity score, which includes its reader-first read (paper-grader §House Voice). The advisory metrics never block an exit. If metrics are recorded, re-record them after the Phase 4.5 fix round, since those fixes also edit prose; the final report uses the last values. ### Phase 4 Sub-Gate: Venue Compliance (if `venue_compliance=on`) After each grader call, before deciding to exit/loop: ``` Invoke venue-compliance-gate (Task agent) with: prompt: "Read ../../agents/venue-compliance-gate.md; apply this separately authored local replacement role to pipeline/phase4_polish/round_N/manuscript.tex research_brief.md" Output: pipeline/phase4_polish/round_N/venue_compliance.md ``` - Verdict PASS → no action; proceed to grade-based exit decision - Verdict PASS_WITH_WARNINGS → log warnings; proceed - Verdict FAIL with fixable issues → add as high-priority items in next paper-fixer dispatch's fix list (alongside grader-derived issues). Do NOT exit Phase 4 with venue compliance failing. - Verdict FAIL with non-fixable issues (e.g., page limit exceeded by >20% requiring substantive cuts) → exit Phase 4 with outcome `VALIDATED_BUT_VENUE_NONCOMPLIANT`. ### Phase 4 Dispatch: Grading ``` Invoke paper-grader (Task agent) with: prompt: "/arms-paper-grader pipeline/phase3_write/manuscript.tex" (or pipeline/phase4_polish/round_N/manuscript.tex for round N) + "Style metrics (advisory, if recorded): " + [Style block from §Writing Style] Output: pipeline/phase4_polish/round_N/paper_grade.md ``` ### Phase 4 Dispatch: Fixing ``` Invoke paper-fixer (Task agent) with: prompt: "Fix the manuscript based on the grade report. Read: pipeline/phase4_polish/round_N/paper_grade.md Read: pipeline/phase4_polish/round_N/manuscript.tex (or pipeline/phase3_write/manuscript.tex for round 0) Write fixed manuscript to: pipeline/phase4_polish/round_{N+1}/manuscript.tex [Style block from §Writing Style]" Output: pipeline/phase4_polish/round_{N+1}/ ``` ### Phase 4 Scope Constraint (CRITICAL) Paper-fixer **MAY** fix: - Formula transcription errors (LaTeX doesn't match validated code) - Table incompleteness (missing entries, wrong numbers vs CSV data) - Citation formatting issues - Clarity improvements, including reviewer-flagged structural consolidation and relocation under manuscript_plan.md; preserve scientific meaning, adverse findings and the evidence-disposition record - House-voice rewrites through house-style of the paragraphs the grader's reader-first read flags, including moving numbers from prose into tables under its number-ledger rule - Missing figure references or broken cross-references - Reproducibility issues (missing seeds, unclear parameter values) Paper-fixer **MAY NOT**: - Change the methodology or core algorithms - Re-run experiments with different parameters - Add new evaluation types not in the validation report - Change the narrative or conclusions to be more favorable - Modify the validated code - Increase computational budget to change results **Rationale:** If the grader identifies methodology or novelty issues, those are Phase 1-2 problems. The method was validated. The paper may score below target on Novelty or Impact, but that's an honest reflection of the method's value, not an execution error. Log methodology concerns but do not try to fix them in Phase 4. --- ## Phase 4.5: CONSISTENCY AUDIT (gate before finalization) After Phase 4 reaches SUCCESS or VALIDATED_BELOW_TARGET (any exit other than VALIDATED_BUT_VENUE_NONCOMPLIANT), dispatch consistency-auditor before writing the final report: ``` Invoke consistency-auditor (Task agent) with: prompt: "Read ../../agents/consistency-auditor.md; apply this separately authored local replacement role to pipeline/phase4_polish/round_N/manuscript.tex" Output: pipeline/phase4_polish/round_N/consistency_report.md ``` ### Phase 4.5 Gate - **Verdict PASS** → proceed to Final Report. - **Verdict PASS_WITH_WARNINGS** → proceed; copy warnings into `final_report.md` under "Residual Issues." - **Verdict FAIL** → re-dispatch paper-fixer with `consistency_report.md` as input and the §Writing Style style block (counts toward `max_polish + 1` extra "consistency round"). After paper-fixer completes, re-grade and re-audit. If second consistency audit also FAILs → STOP with outcome `CONSISTENCY_FAILURE`; surface report to user. `consistency_report.md` is also included in the final_report's "All Files Produced" list and its "Residual Issues" section if any warnings remain. --- ## Pipeline Log Append to `pipeline/pipeline_log.md` after EVERY phase transition. Format: ```markdown --- ## [timestamp] — Phase N: [THINK/VALIDATE/WRITE/POLISH] ### Status: [COMPLETED / FAILED / RETHINK / MARGINAL_STOP] ### Duration: [estimated tokens consumed] ### Key Output: [1-2 sentences summarizing what happened] ### Score (if applicable): [X/50] ### Files Produced: - [file path]: [1-line description] ### Decision: [proceed to Phase N+1 / rethink / stop] ### Rethink Count: [0/1/2] --- ``` --- ## Pipeline State (for resume) Write `pipeline/pipeline_state.md` after each phase completes: ```markdown # Pipeline State ## Configuration - Brief: [path] - Target: [X/50] - Max Polish: [N] ## Current Phase: [1/2/3/4] ## Phase Status | Phase | Status | Key Artifact | |-------|--------|-------------| | 1. THINK | [COMPLETED/IN_PROGRESS/PENDING] | methodology_specification.md | | 2. VALIDATE | [COMPLETED/IN_PROGRESS/PENDING/RETHINK_N] | validation_report.md | | 3. WRITE | [COMPLETED/IN_PROGRESS/PENDING] | manuscript.tex | | 4. POLISH | [COMPLETED/IN_PROGRESS/PENDING] Round N/max | paper_grade.md | ## Rethink Count: [0/1/2] ## Phase1 Retries: [0/1/2] ## Phase3 Retries: [0/1/2] ## Polish Round: [0/1/2/3] ## Latest Score: [X/50 or N/A] ## House Voice: [reader-first read PASS | paragraphs flagged: N | N/A] (genre: default | results; exemplar: default | ) - Advisory metrics last recorded: [path to style_metrics.json, or none] ## Marginal Decision: [N/A | PENDING | PROCEED | RETHINK | STOP] ## Context for Resume [What the next phase needs to know to start. Include file paths, not content.] ``` --- ## Stopping Criteria | Outcome | Condition | Action | |---------|-----------|--------| | **SUCCESS** | Phase 4 reaches target score, venue compliance is not FAIL, and independently checked contribution status is SUPPORTED | Write final report. Pipeline complete. | | **VALIDATED_BELOW_TARGET** | Phase 4 exhausts max rounds or has no fixable execution issue and a research-quality gap | Write final report. Report technical validation separately from the unresolved execution or contribution gap. | | **MARGINAL** | Phase 2 returns MARGINAL verdict | Stop. Surface to user for decision. | | **NO_GO** | Phase 2 fails after max rethinks | Write honest failure report. | | **KILL** | validate-method invokes kill criterion | Write honest failure report. Fundamental limitation identified. | | **PHASE_1_INCOMPLETE** | methodology-architect can't produce valid spec after `max_phase_retries` | Write failure report. Problem may be underspecified. | | **PHASE_3_INCOMPLETE** | write-manuscript can't produce all required artifacts after `max_phase_retries` | Write failure report. Surface incomplete artifacts. | | **MARGINAL_STOP** | User set `Decision: STOP` in `MARGINAL_decision_required.md` | Write final report with outcome=MARGINAL_STOP. | | **CONSISTENCY_FAILURE** | consistency-auditor returns FAIL twice in a row (Phase 4.5) | Surface report; do not finalize. | | **VALIDATED_BUT_VENUE_NONCOMPLIANT** | venue-compliance-gate (Phase 4) FAIL with non-fixable issues | Surface report; the manuscript is valid but cannot meet venue submission limits. | --- ## Final Report At pipeline completion (any outcome), write `pipeline/final_report.md`: ```markdown # Research Pipeline Final Report ## Outcome: [SUCCESS / VALIDATED_BELOW_TARGET / MARGINAL / NO_GO / KILL] ## Contribution and impact - Contribution status: SUPPORTED / PROVISIONAL / NOT_ESTABLISHED - Closest original precedents and the exact supported difference: - Author and external benchmark lessons applied: - Practical consequence demonstrated, its magnitude and evidence limits: - Remaining research gaps and strongest counterargument: - Technical readiness is separate from journal acceptance or adoption. ## Total Phases Executed: [N] ## Rethinks Used: [N] --- ## Phase 1: THINK - **Methodology**: [1-sentence summary from spec] - **Comparators**: [list from spec] - **Key design decisions**: [2-3 bullets from rationale] ## Phase 2: VALIDATE - **Verdict**: [GO / CONDITIONAL GO / MARGINAL / NO-GO] - **Primary advantage**: [metric, magnitude, MC CI] - **Key limitation**: [if any] - **Iterations**: [count] - **Rethinks**: [count] ## Phase 3: WRITE (if executed) - **Synopsis status**: [PENDING / CHANGES_REQUESTED / APPROVED / explicit user override] - **Synopsis version and approval record**: [paths and matching version/hash] - **Manuscript**: [path] - **Pages**: [count] - **Figures**: [count] - **Tables**: [count] - **Evaluation types**: [list] ## Phase 4: POLISH (if executed) - **Starting score**: [X/50] - **Final score**: [X/50] - **Rounds**: [count] - **Issues fixed**: [list] ## Writing Style (if Phase 3 executed) - **Reader-first read** (final grader): [PASS | paragraphs still flagged, with rules] - **Advisory metrics vs. exemplar** (if recorded; house-style "Style metrics (advisory)"), genre [default | results], exemplar [default | ]. Information only, not pass/fail. | Metric | Exemplar | Phase 3 draft | Final | Note | |--------|----------|---------------|-------|------| | mean sentence words | | | | | | % sentences > 45 words | | | | | | sentences per paragraph | | | | | | "we" per 1k | | | | | | semicolons per 1k | | | | | | numbers per 1k | | | | | | % sentences with > 3 numbers | | | | | ## Score Breakdown (if graded) | Dimension | Score | |-------------|-------| | Correctness | X | | Completeness | X | | Rigor | X | | Clarity | X | | Novelty | X | | Impact | X | | Performance | X | | **Raw Total** | **X/35** | | **Weighted Total** | **X/50** | ## Lessons Learned - [What worked well in this run] - [What didn't work and why] - [Suggestions for the next run, if any] ## All Files Produced [Master list organized by phase] ``` --- ## Anti-Patterns 1. **"Phase 2 failed but let me try writing the paper anyway"** — NO. The whole point of this architecture is: don't write until validated. A paper about an unvalidated method is the problem we're solving. 2. **"Let me skip Phase 1 since we already have a methodology"** — Acceptable ONLY if `pipeline/phase1_think/methodology_specification.md` already exists AND passes the Phase 1 gate check. If you're reusing a spec, verify it's complete. 3. **"Phase 4 grader says novelty is low, let me fix it in the paper"** — NO. Novelty is a Phase 1-2 property. Phase 4 cannot manufacture novelty through better prose. Log it as a lesson learned. 4. **"Let me run all phases in parallel"** — NO. The phases are strictly sequential. Each phase's output is the next phase's input. There are no shortcuts. 5. **"The validation took too long, let me skip stress tests"** — validate-method manages its own time budget. Acceptance rule: - **Required**: scale + adversarial + sensitivity + ablation, all 4 stress test types listed in `../arms-validate-method/SKILL.md §Step 4` must be present in `validation_report.md` AND backed by at least one CSV in `validated_results/` each. - **Partial-OK threshold**: at most ONE of the four may be marked `INCOMPLETE — [reason]` in the report. If two or more are incomplete, the verdict is **automatically downgraded to MARGINAL** regardless of what the report claims, and the MARGINAL decision file (§Phase 2) is written. - Zero tests skipped silently. Every skip needs a logged reason. 6. **"Let me re-run Phase 2 with the same spec to see if I get a better result"** — NO. validate-method is deterministic (fixed seeds). The same spec will produce the same result. If Phase 2 returns NO-GO, either RETHINK (revise the spec) or STOP. Don't retry. 7. **"The tic checks pass, so the house voice passes"** — NO. No em-dashes, "we" present, and correctly spelled defined terms are necessary, not sufficient. Neither do metrics close to the exemplar. The house voice is judged by the reader-first read against the exemplar. 8. **"Let me dispatch Phase 3 sub-agents myself instead of using write-manuscript"** — NO. Write-manuscript manages its own sub-agents. The pipeline orchestrator dispatches phases, not sub-agents of phases. Stay at the right abstraction level. --- ## Context Management The orchestrator itself is lightweight. It dispatches phases and checks outputs. | Component | Orchestrator Budget | Phase Agent Budget | |------------------------------------------|---------------------|---------------------------------------------------------------------| | Phase 0 dispatch + gate | ~3K | ~60K (brief-expander) | | Phase 1 dispatch + gate | ~5K | ~200K (methodology-architect; up to ~250K in revision/RETHINK mode) | | Phase 2 dispatch + gate | ~5K | ~120K (validate-method incl. stress tests) | | Phase 3 dispatch + gate | ~5K | ~250K (write-manuscript orchestrator + 4–7 sub-dispatches) | | Phase 4 loop (per round) | ~8K | ~120K (grader 80K + fixer 40K; +40K when the fixer runs a style pass) | | Phase 4.5 dispatch + gate | ~3K | ~30K (consistency-auditor) | | Pipeline log + state + report | ~10K | — | | **Orchestrator total** (4 phases + 3 polish rounds) | **~60K** | — | ### Phase 4 cost scaling Each polish round costs ~120K agent tokens. With `max_polish=3` (default), budget ~360K for Phase 4. With `max_polish=6`, ~720K. The orchestrator itself stays bounded (~8K per round), but the dispatched grader/fixer agents are the dominant cost. **Token budgeting guidance**: - For papers with `target>=42` (Grade A), expect 2–3 polish rounds typical. - If first-round grade `< target − 8`, polishing rarely closes the gap; consider RETHINK instead of polishing. - Recommended cap: `max_polish=4` for most runs; raise only if first-round grade is within 4 points of target AND progress is monotonic. - Log per-round token estimate in `pipeline_log.md` so the user sees burn rate. The orchestrator should comfortably fit within a single context window. Each phase agent runs in its own context (via Task tool dispatch). --- ## Differences from v1 System (auto-research) | Aspect | v1 (auto-research) | v2 (research-pipeline) | |--------|-------------------|----------------------| | Entry point | write-paper (writes before validating) | methodology-architect (thinks first) | | Validation | Post-hoc (grader after full paper) | Pre-write (validate-method before writing) | | Loop target | Brief + skill edits (instructions) | Phase-appropriate fixes only | | Methodology fixes | research-cycle edits brief (late) | RETHINK sends back to architect (early) | | Kill criterion | None (runs to max_cycles) | validate-method has explicit kill | | Context scope | Single mega-agent for everything | Separate agent per phase | | Honest failure | Not supported (always produces a paper) | MARGINAL, NO_GO, KILL outcomes |