# template_prose_project

A configurable, reproducible **prose-review** pipeline built on
`infrastructure/prose/` (readability + structure + quality) and
`infrastructure/reference/` (BibTeX validation). Exemplar roster:
[`projects/AGENTS.md`](../../AGENTS.md#permanent-canonical-exemplars).

## When to use this template

Use this template for **manuscript-focused editorial pipelines**: readability
gates (Flesch-Kincaid bands), structural checking, citation/BibTeX
validation, and prose-quality review workflows where the document itself is
the artifact under test. If you are implementing algorithms with numerical
experiments, start from [`template_code_project`](../template_code_project/)
instead; for book-length structure see
[`template_textbook`](../template_textbook/). Full roster:
[`projects/AGENTS.md`](../../AGENTS.md#permanent-canonical-exemplars).

## Publication and rendering

The publishing metadata and per-platform status below are **compiled from
`manuscript/config.yaml`** by `infrastructure.publishing.status_report` — do not
hand-edit between the markers; update the config and regenerate (see the legend).

<!-- PUBLISHING-STATUS:START (generated by infrastructure.publishing.status_report) -->
**Editorial Quality at Scale: A Reproducible Prose-Review Pipeline** · v0.4.2 · MIT · Daniel Ari Friedman

Concept DOI: [10.5281/zenodo.20417104](https://doi.org/10.5281/zenodo.20417104) | Version DOI: [10.5281/zenodo.20932047](https://zenodo.org/records/20932047) | Repository: [docxology/template_prose_project](https://github.com/docxology/template_prose_project)

Publishing surface — 20 platforms, 9 published:

| Platform | Tier | Status | Reference | Credentials |
| --- | --- | --- | --- | --- |
| zenodo | first-class | ✅ published | [10.5281/zenodo.20417104](https://doi.org/10.5281/zenodo.20417104) | `ZENODO_API_TOKEN` |
| github | first-class | ✅ published | [docxology/template_prose_project](https://github.com/docxology/template_prose_project) | `GITHUB_TOKEN` |
| arxiv | first-class | ⚪ available | — | — |
| pypi | first-class | ✅ published | [https://test.pypi.org/project/template-prose-project/0.4.2/](https://test.pypi.org/project/template-prose-project/0.4.2/) | `PYPI_TOKEN`, `TESTPYPI_TOKEN` |
| ipfs_pinata | first-class | ✅ published | [https://gateway.pinata.cloud/ipfs/QmXXUgas1amP7Pd9rRoTdJQiNUzBqKQ4r3Jf85aaGEnjz4](https://gateway.pinata.cloud/ipfs/QmXXUgas1amP7Pd9rRoTdJQiNUzBqKQ4r3Jf85aaGEnjz4) | `PINATA_JWT` |
| ipfs_web3storage | first-class | ⚪ available | — | `WEB3_STORAGE_TOKEN` |
| software_heritage | first-class | ✅ published | [https://archive.softwareheritage.org/browse/origin/?origin_url=https://github.com/docxology/template_prose_project](https://archive.softwareheritage.org/browse/origin/?origin_url=https://github.com/docxology/template_prose_project) | — |
| github_pages | first-class | ✅ published | [https://docxology.github.io/template_prose_project/](https://docxology.github.io/template_prose_project/) | `GITHUB_TOKEN` |
| cloudflare_pages | first-class | ⚪ available | — | `CLOUDFLARE_API_TOKEN` |
| netlify | first-class | ✅ published | [https://6a44425bbd23b314b04b0154--tranquil-kleicha-0c9203.netlify.app](https://6a44425bbd23b314b04b0154--tranquil-kleicha-0c9203.netlify.app) | `NETLIFY_AUTH_TOKEN` |
| huggingface_hub | first-class | ✅ published | [https://huggingface.co/datasets/ActiveInference/template_prose_project](https://huggingface.co/datasets/ActiveInference/template_prose_project) | `HUGGINGFACE_TOKEN`, `HF_TOKEN` |
| osf | first-class | ✅ published | [https://osf.io/sh2r3/](https://osf.io/sh2r3/) | `OSF_TOKEN` |
| amazon_kdp | documented | 🟡 planned | — | `AMAZON_KDP_EMAIL`, `AMAZON_KDP_PASSWORD` |
| google_play_books | documented | 🟡 planned | — | `GOOGLE_PLAY_BOOKS_SERVICE_ACCOUNT_JSON` |
| gumroad | documented | 🟡 planned | — | `GUMROAD_ACCESS_TOKEN` |
| leanpub | documented | 🟡 planned | — | `LEANPUB_API_KEY` |
| lulu | documented | 🟡 planned | — | `LULU_CLIENT_KEY`, `LULU_CLIENT_SECRET` |
| draft2digital | documented | 🟡 planned | — | `DRAFT2DIGITAL_API_TOKEN` |
| stripe | documented | 🟡 planned | — | `STRIPE_SECRET_KEY`, `STRIPE_PUBLISHABLE_KEY` |
| ingramspark | documented | 🟡 planned | — | `INGRAMSPARK_CLIENT_ID`, `INGRAMSPARK_CLIENT_SECRET` |

_Keywords: prose analysis, readability, editorial review, reproducible research, manuscript quality._

_Status legend: ✅ published (durable identifier recorded in `config.yaml`) · 🔵 reserved (identifier reserved but not yet registered by final publication) · ⚪ available (adapter implemented and locally verifiable) · 🟡 planned. This block is generated — edit `manuscript/config.yaml`, then regenerate with `uv run python -m infrastructure.publishing.status_report --project <path> --write`._
<!-- PUBLISHING-STATUS:END -->

The 3 platforms still shown ⚪ available are not automatable to "published" with
current tooling/credentials, not an oversight: **arXiv** has no submission API
in this codebase (`infrastructure.publishing.arxiv` only prepares a local
tarball — a human must upload it via arxiv.org and the resulting `arxiv` URL
would then be added to `publication.published_artifacts`); **Cloudflare
Pages** needs a `CLOUDFLARE_ACCOUNT_ID` the configured API token cannot
auto-discover; **IPFS (Web3.Storage)** has no `WEB3_STORAGE_TOKEN` configured.

- Canonical renderer: [docxology/template](https://github.com/docxology/template) with `--project templates/template_prose_project`
- Tracked outputs: [`output/`](output/) in this project and `output/templates/template_prose_project/` in the monorepo; public output files above 50 MB stay out of git.

To regenerate this exemplar from the public monorepo:

```bash
git clone https://github.com/docxology/template
cd template
uv sync
./run.sh --project templates/template_prose_project --pipeline --core-only
uv run python scripts/pipeline/stage_04_validate.py --project templates/template_prose_project
uv run python scripts/pipeline/stage_05_copy.py --project templates/template_prose_project
```

Standalone repositories are publication mirrors for source, DOI metadata, and
tracked rendered artifacts. Use the monorepo above when you need the full shared
infrastructure, pipeline stages, or cross-template validation.

## Why this template — the transferable pattern

The bundled `manuscript/config.yaml` is intentionally permissive (FKGL band
10–18, citation floor 0.0) so the exemplar's *own abstract passes its own
gate*. This is a self-referential pipeline: the document describing the
gate runs *through* the gate. The transferable discipline is that the
explanatory prose and the threshold settings co-evolve — a forker who
tightens `prose.target_grade_level_max` to 14.0 without rewriting the
methodology section will discover that the project's own methodology fails
the methodology check. The pedagogy is "configuration drift is detected by
the artifact itself, not by an external reviewer". A stricter starting
point lives in
[`manuscript/config.yaml.example`](manuscript/config.yaml.example).

## What it does

```mermaid
flowchart LR
    CFG[manuscript/config.yaml] --> READ[src/pipeline/ · read manuscript]
    READ --> PROSE[infrastructure.prose<br/>metrics · structure · quality]
    READ --> BIB[infrastructure.reference<br/>BibTeX validation]
    PF[src/prose_facade.py<br/>parse_bib_keys · render_outline] --> CHECKS
    PROSE --> CHECKS[evaluate threshold checks]
    BIB --> CHECKS
    CHECKS --> JSON[manuscript_report.json<br/>checks.json]
    CHECKS --> RPT[src/report.py<br/>review_report.md]
    PF --> RPT
    PROSE --> FIG[src/figures.py<br/>word counts · readability ·<br/>citation density PNGs]
    JSON --> MV[src/manuscript_variables.py<br/>variable token substitution]
    MV --> SUB[output/manuscript/*.md<br/>tokens resolved]

    classDef io fill:#0f766e,stroke:#0f172a,color:#fff
    classDef proc fill:#1e3a8a,stroke:#0f172a,color:#fff
    classDef out fill:#7c2d12,stroke:#0f172a,color:#fff
    class CFG io
    class READ,PROSE,BIB,CHECKS,MV,PF proc
    class JSON,RPT,FIG,SUB out
```

## Quick start

```bash
# Run the full prose review (offline, no Ollama)
uv run python projects/templates/template_prose_project/scripts/run_prose_pipeline.py

# Strict mode: exit non-zero if any check fails
uv run python projects/templates/template_prose_project/scripts/run_prose_pipeline.py --strict

# Generate figures (after a pipeline run)
uv run python projects/templates/template_prose_project/scripts/y_generate_prose_figures.py

# Hydrate manuscript variables for the abstract
uv run python projects/templates/template_prose_project/scripts/z_generate_manuscript_variables.py
```

After the run, look in `output/`:

* `manuscript_report.json` — raw `ManuscriptReport`.
* `checks.json` — list of `CheckResult` (one per configured check).
* `review_report.md` — the human-readable review.
* `evidence_summary.json` — versioned diagnostic-only evidence separated into
  readability, citations, bibliography, structure, and quality flags.
* `figures/{section_word_counts,readability_metrics,citation_density}.png`.
* `data/manuscript_variables.json` — substitution variables for the abstract.
* `run_summary.json` — one-line metadata.

## Prerequisites & verification

**Combined-PDF rendering needs headless Chrome.** This manuscript embeds
Mermaid diagrams (`manuscript/05_pipeline_internals.md`); the combined PDF is
built with `mmdc`, which requires a pinned `chrome-headless-shell`. Install
it once (CI provisions it automatically; a fresh clone does not):

```bash
npm ci
npx --no-install puppeteer browsers install chrome-headless-shell
export PATH="$PWD/node_modules/.bin:$PATH"
```

Without it the **PDF Rendering** stage fails while per-section slides still
render — see [`docs/troubleshooting.md`](docs/troubleshooting.md#pdf-rendering-stage-fails-mmdc-could-not-find-chrome).

**Test/coverage gate (authoritative per-project command).** Exit code 0
alone is not proof; confirm tests collected > 0 and coverage ≥ 90%:

```bash
uv run pytest projects/templates/template_prose_project/tests/ \
  --cov=projects/templates/template_prose_project/src --cov-fail-under=90
# exemplar baseline → docs/_generated/COUNTS.md
```

Full end-to-end (tests → analysis → render → validate → copy):

```bash
uv run python scripts/runner/execute_pipeline.py --project template_prose_project --core-only
```

## Configuration

Every knob lives in `manuscript/config.yaml`:

| Section | Key | Default | Meaning |
|---|---|---|---|
| `prose` | `target_grade_level_min` / `_max` | `10.0 / 18.0` | Acceptable Flesch-Kincaid Grade Level band. |
| `prose` | `long_sentence_threshold` | `35` | Words per sentence above which sentences are flagged. |
| `prose` | `citation_density_min_per_1000` | `0.0` | Minimum citations per 1000 words. |
| `prose` | `require_h1_per_section` | `true` | Every file must have an H1. |
| `prose` | `forbid_skipped_levels` | `true` | Heading levels must be contiguous. |
| `prose` | `preset` | `lenient` (example: `strict`) | Named editorial profile seeding defaults for unset prose knobs. |
| `bibliography` | `references_path` | `manuscript/references.bib` | Path to BibTeX file. |
| `bibliography` | `fail_on_missing` | `true` | Fail if a cited `[@key]` is not in the bib. |
| `bibliography` | `fail_on_unused` | `false` | Fail if a bib entry is never cited. |
| `report` | `output_path` | `output/review_report.md` | Where `write_review_report` writes the markdown review. |
| `report` | `include_per_file_table` | `true` | Include the per-file words/sentences/FRE/FKGL/Fog table. |
| `report` | `include_outline` | `true` | Include the per-file heading outline section. |
| `report` | `include_quality_flags` | `true` | Include the long-sentence/passive/hedge quality-flags section. |

## Agentic research overlays

This exemplar includes two declarative overlays for advisory research controls:

- [`domain_profile.yaml`](domain_profile.yaml) declares the prose-research
  domain, review gates, citation/source policy, artifact expectations, and
  benchmark rubric weights.
- [`experiment_plan.yaml`](experiment_plan.yaml) declares the prose-validation
  conditions, primary metric direction, expected figures and tables, baseline,
  and strict-bibliography ablation.
- [`data/claim_ledger.yaml`](data/claim_ledger.yaml) registers sourced numeric
  claims that are intentionally tied to prose-review code, manuscript examples,
  or generated reports.

These files are validation and benchmark inputs only. They do not generate
experiments, mutate prompts, or run autonomous review agents.

## Architecture

* `src/config.py` — typed YAML loader.
* `src/pipeline/` — receives the pre-analysed `ManuscriptReport`, evaluates the configured checks (bibliography cross-check via `src/prose_facade.parse_bib_keys`), writes the JSON artefacts. **Pure orchestration; zero `infrastructure` imports — the thin `scripts/` call `infrastructure/` on its behalf.**
* `src/figures.py` — matplotlib renderers (no business logic).
* `src/manuscript_variables.py` — abstract substitution variables.
* `src/report.py` — markdown review-report assembly.
* `src/prose_facade.py` — project-owned report Protocols (`ManuscriptReportLike`, `FileReportLike`, …) plus `render_outline` and `parse_bib_keys`; decouples `src/` from `infrastructure.prose`/`infrastructure.reference` internals.
* `scripts/run_prose_pipeline.py` — thin orchestrator.
* `scripts/y_generate_prose_figures.py` — figure stage.
* `scripts/z_generate_manuscript_variables.py` — variable-hydration stage.

## Testing

```bash
uv run pytest projects/templates/template_prose_project/tests/ -v
```

All tests are offline; no mocks. The test suite uses real prose, real
BibTeX files, real `tmp_path` directories, real subprocess invocation
of the orchestrator scripts.

## See also

* [`AGENTS.md`](AGENTS.md) — agent-oriented walkthrough.
* [`manuscript/SYNTAX.md`](manuscript/SYNTAX.md) — Pandoc citation / cross-reference conventions for this project.
* [`../../docs/guides/manuscript-semantics.md`](../../../docs/guides/manuscript-semantics.md) — repository-wide canonical manuscript semantics.
* [`docs/architecture.md`](docs/architecture.md) — architectural diagram.
* [`docs/quickstart.md`](docs/quickstart.md) — getting started guide.
* [`infrastructure/prose/SKILL.md`](../../../infrastructure/prose/SKILL.md) —
  underlying prose-analysis API.

## Template integrity

- Forward backlog: [`TODO.md`](TODO.md).
- Copy-and-customize config: [`manuscript/config.yaml.example`](manuscript/config.yaml.example).
- Project validation: `uv run pytest projects/templates/template_prose_project/tests/ --cov=projects/templates/template_prose_project/src --cov-fail-under=90`.
- Repo drift validation: `uv run python scripts/audit/check_template_drift.py --strict`.

<!-- foam-orphan-nav:start (hand-maintained: links sub-docs so they are reachable; no generator refreshes or validates this block) -->

## Directory & sub-document map

Navigation links to in-tree documents (keeps them discoverable):

- [Data Directory Instructions](data/AGENTS.md)
- [template_prose_project/data](data/README.md)

<!-- foam-orphan-nav:end -->