# Generic Project Development Workflow: The Pipeline Orchestrator Paradigm

> **development workflow** ensuring source code, tests, and documentation coherence

**Quick Reference:** [How To Use](../core/how-to-use.md) | [Architecture](../core/architecture.md) | [Common Workflows](../reference/common-workflows.md)

This document explains the development workflow that ensures source code, tests, and documentation remain in coherence.

**For related information:**

- **[How To Use](../core/how-to-use.md)** - usage guide from basic to advanced
- **[Architecture](../core/architecture.md)** - System design overview
- **[Thin Orchestrator Summary](../architecture/thin-orchestrator-summary.md)** - Pattern implementation details
- **[Common Workflows](../reference/common-workflows.md)** - Step-by-step recipes for common tasks

## Overview

The generic project template implements a **unified test-driven development paradigm** where:

- **Source code** implements mathematical functionality
- **Tests** validate all functionality with coverage (60% infra, 90% project minimum)
- **Scripts** are **thin orchestrators** that import and use `projects/{name}/src/` methods
- **`scripts/runner/execute_pipeline.py`** orchestrates the declarative DAG pipeline

## Workflow Diagram

```mermaid
graph TB
    subgraph DEV["Development Components"]
        SRC["Source Code<br/>projects/{name}/src/"]
        TESTS["Tests<br/>projects/{name}/tests/"]
        SCRIPTS["Scripts<br/>projects/{name}/scripts/"]
        MANUSCRIPT["Manuscript<br/>projects/{name}/manuscript/"]
    end

    subgraph VALGEN["Validation and generation"]
        VALIDATION["Test Validation<br/>≥90% src/ coverage gate"]
        FIGURES["Figure Generation<br/>Using project src/ methods"]
        DATA["Data Generation<br/>Using project src/ methods"]
        MARKDOWN_VAL["Markdown Validation<br/>Images and references"]
    end

    subgraph PIPE["Build Pipeline"]
        RENDER["execute_pipeline.py<br/>Pipeline Orchestrator"]
        PDFS["PDF Generation<br/>Individual and combined"]
        LATEX["LaTeX Export<br/>For further processing"]
    end

    SRC --> VALIDATION
    TESTS --> VALIDATION
    SRC --> FIGURES
    SRC --> DATA
    SCRIPTS --> FIGURES
    SCRIPTS --> DATA
    MANUSCRIPT --> MARKDOWN_VAL
    FIGURES --> MARKDOWN_VAL
    DATA --> MARKDOWN_VAL

    VALIDATION --> RENDER
    FIGURES --> RENDER
    DATA --> RENDER
    MARKDOWN_VAL --> RENDER

    RENDER --> PDFS
    RENDER --> LATEX

    classDef component fill:#e1f5fe,stroke:#01579b,stroke-width:2px
    classDef validation fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
    classDef pipeline fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px

    class SRC,TESTS,SCRIPTS,MANUSCRIPT component
    class VALIDATION,FIGURES,DATA,MARKDOWN_VAL validation
    class RENDER,PDFS,LATEX pipeline
```

## How the Pipeline Orchestrator Works with Markdown and Code

The `scripts/runner/execute_pipeline.py` orchestrator (or `./run.sh --pipeline`) executes the pipeline stages sequentially, ensuring coherence between all components:

### 1. Code Validation Phase

- **Runs all generation scripts** - This validates that `projects/{name}/src/` code works correctly
- **Scripts import from project src/** - Ensures no code duplication and validates imports
- **Generates figures and data** - Creates outputs that markdown will reference

### 2. Markdown Validation Phase

- **Validates all image references** - Ensures figures referenced in markdown exist
- **Checks internal links** - Validates equation labels and section anchors
- **Validates equation formatting** - Ensures proper LaTeX equation environments

### 3. Documentation Generation Phase

- **Auto-generates glossary** - Creates API table from current `src/` code
- **Updates documentation** - Keeps code-doc sync automatically

### 4. Output Generation Phase

- **Builds individual PDFs** - Creates per-section PDFs from validated markdown
- **Builds combined PDF** - Creates unified document from all sections
- **Exports LaTeX** - Provides LaTeX source for further processing

## Test Suite and Code Connections

The test suite ensures coverage of all modules and validates the entire pipeline:

### What Tests Validate

- **Mathematical correctness** - All functions produce expected results
- **Import compatibility** - Scripts can successfully import from `projects/{name}/src/` modules
- **Output generation** - Figure and data generation works correctly
- **Deterministic execution** - All outputs are reproducible with fixed seeds
- **Path management** - Outputs go to correct directories

### Test-Driven Development Flow

```mermaid
flowchart TD
    START([Start Development]) --> TESTS[Write Tests First]
    TESTS --> IMPLEMENT[Implement Functionality]
    IMPLEMENT --> VALIDATE[Run Tests & Check Coverage]
    VALIDATE -->|Coverage below gate| ADD_TESTS[Add Missing Tests]
    ADD_TESTS --> VALIDATE
    VALIDATE -->|Coverage ≥90%| INTEGRATION[Test Script Integration]
    INTEGRATION --> DOCS[Update Documentation]
    DOCS --> PIPELINE[Run Pipeline]
    PIPELINE --> SUCCESS[Development]

    classDef process fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
    classDef decision fill:#fff3e0,stroke:#e65100,stroke-width:2px
    classDef success fill:#e8f5e8,stroke:#2e7d32,stroke-width:2px

    class TESTS,IMPLEMENT,VALIDATE,ADD_TESTS,INTEGRATION,DOCS,PIPELINE process
    class START,SUCCESS success
```

1. **Write tests first** - Define expected behavior before implementation
2. **Implement functionality** - Write code to pass tests
3. **Validate integration** - Ensure scripts can use the code
4. **Update documentation** - Reflect changes in markdown
5. **Run pipeline** - Use `uv run python scripts/runner/execute_pipeline.py --project {name} --core-only` to validate coherence

## Step-by-Step Workflow

### 1. Development Phase

```bash
# Always start with tests
uv run pytest projects/templates/template_code_project/tests/ --cov=projects/templates/template_code_project/src --cov-report=term-missing

# Check coverage (≥90% gate; live percentage per exemplar → docs/_generated/COUNTS.md)
coverage report

# Make code changes in projects/{name}/src/
# Update corresponding tests
# Update documentation if needed
```

### 2. Validation Phase

```bash
# Run tests again to ensure changes work
uv run pytest

# Generate figures and data
uv run python projects/templates/template_code_project/scripts/optimization_analysis.py
uv run python scripts/pipeline/stage_02_analysis.py --project template_code_project

# Validate markdown integrity
uv run python -m infrastructure.validation.cli markdown projects/templates/template_code_project/manuscript/
```

### 3. Integration Phase

```bash
# Run the core pipeline (no LLM stages)
uv run python scripts/runner/execute_pipeline.py --project {name} --core-only

# Or use unified interactive menu
./run.sh
```

With `--core-only`, `PipelineExecutor` runs the **core** path: clean outputs (unless disabled), environment setup, infrastructure tests (unless `--skip-infra`), project tests, analysis, PDF rendering, output validation, then copy outputs. That path is driven by scripts **`00`–`05`** (tests use **`01`**, which runs infrastructure + project suites).

**Full** pipeline (for example `./run.sh --pipeline` without `--core-only`) adds LLM review and translations (`scripts/pipeline/stage_06_llm_review.py`) before copy. **`scripts/pipeline/stage_07_executive_report.py`** is for multi-project / executive reporting, not the default single-project stage list.

### Canonical stage table (generated)

<!-- BEGIN:STAGE_TABLE -->
<!-- This block is generated from [`infrastructure/core/pipeline/pipeline.yaml`](../../infrastructure/core/pipeline/pipeline.yaml) by `scripts/docgen/stage_table.py`. Do not hand-edit. Stage indices are **0-based positions in the YAML** and intentionally do **not** match the `scripts/pipeline/stage_NN_*.py` numeric prefixes (for example, stage 11, "Copy Outputs", runs `scripts/pipeline/stage_05_copy.py`). -->

| Stage | Script | Tags | Failure mode |
| ----- | ------ | ---- | ------------ |
| **0** Clean Output Directories | built-in `_run_clean_outputs` | `core`, `clean` | soft fail |
| **1** Environment Setup | `scripts/pipeline/stage_00_setup.py` | `core` | hard fail |
| **2** Infrastructure Tests | `scripts/pipeline/stage_01_test.py --infra-only --verbose --infra-scope pipeline-smoke` | `core`, `tests` | configurable tolerance |
| **3** Project Tests | `scripts/pipeline/stage_01_test.py --project-only --verbose` | `core`, `tests` | configurable tolerance |
| **4** Project Analysis | `scripts/pipeline/stage_02_analysis.py` | `core` | hard fail |
| **5** Connector Search | `scripts/pipeline/stage_08_connector_search.py` | `science` | skipped if not configured |
| **6** Provenance Record | `scripts/pipeline/stage_09_provenance_record.py --stage Connector Search` | `provenance` | skipped if not configured |
| **7** PDF Rendering | `scripts/pipeline/stage_03_render.py` | `core` | hard fail |
| **8** Output Validation | `scripts/pipeline/stage_04_validate.py` | `core` | PDF/bookends and artifact/provenance failures block; optional-format structure remains a warning + report |
| **9** LLM Scientific Review | `scripts/pipeline/stage_06_llm_review.py --reviews-only` | `llm` | skipped if Ollama absent |
| **10** LLM Translations | `scripts/pipeline/stage_06_llm_review.py --translations-only` | `llm` | skipped if Ollama absent |
| **11** Copy Outputs | `scripts/pipeline/stage_05_copy.py` | `core` | soft fail |
| **12** Ebook Generation | `scripts/pipeline/stage_11_ebook.py` | `core`, `ebook` | soft fail |
| **13** Metadata Package | `scripts/pipeline/stage_12_metadata.py` | `core`, `metadata` | soft fail |
| **14** Executable Bundle | `scripts/runner/bundle_executable.py` | `bundle` | soft fail |
| **15** Archival Publication | `scripts/runner/archive_publication.py` | `archival` | soft fail |
<!-- END:STAGE_TABLE -->

## Key Components

### Source Code (`projects/{name}/src/`)

- **`optimizer.py`**, **`sweeps.py`**, **`invariants.py`**: Project-specific algorithms and computations
- Additional modules can be added for specific project needs

**Critical Principle**: ALL business logic and algorithms must live in `projects/{name}/src/` modules.

### Tests (`projects/{name}/tests/`)

- **90% minimum coverage** for `projects/{name}/src/` (live percentages per exemplar → [`../_generated/COUNTS.md`](../_generated/COUNTS.md))
- **60% minimum coverage** for `infrastructure/` (live percentage → `COUNTS.md`)
- **Real numerical examples** (no mocks)
- **Deterministic RNG seeds** for reproducibility
- **Fast and hermetic** execution

### Generation Scripts (`projects/{name}/scripts/`)

- **Import from project src/** modules (no code duplication)
- **Use project src/ methods for all computation** (never implement algorithms)
- **Generate figures and data** deterministically
- **Print output paths** to stdout for manifest collection
- **Use headless plotting** (MPLBACKEND=Agg)

### Documentation (`manuscript/`)

- **References source code** using inline code formatting
- **Displays generated figures** from `output/figures/`
- **Passes validation** for images, references, and equations
- **Auto-updated glossary** from source API

### Output Structure (`output/`)

```mermaid
flowchart LR
    OUT[output]
    OUT --> FIG[figures<br/>PNG · MP4 · SVG]
    OUT --> DATA[data<br/>CSV · NPZ · manifests]
    OUT --> PDF[pdf<br/>Individual + combined PDFs]
    OUT --> TEX[tex<br/>Exported LaTeX files]

    classDef d fill:#0f172a,stroke:#0f172a,color:#fff
    class OUT,FIG,DATA,PDF,TEX d
```

## Validation Rules

### Markdown Validation

- All images must exist and be properly referenced
- Internal links must have valid anchors
- Equations must have unique labels
- No bare URLs (use informative link text)

### Code Validation

- All public APIs must have type hints
- No circular imports
- Consistent formatting and naming
- Error handling for edge cases

### Test Validation

- statement and branch coverage (90% project, 60% infra minimum)
- All tests must pass
- No network or file-system writes outside output/
- Deterministic execution

## Development Commands

The individual commands (test, generate figures, validate markdown, build pipeline) are
the same ones already shown per-phase in [Step-by-Step Workflow](#step-by-step-workflow)
above — see that section for the full development/validation/integration sequence.

Two commands specific to this section, not covered above:

```bash
# Install dependencies (first-time setup)
uv sync

# Check coverage with per-line detail (vs. the summary `coverage report` shown earlier)
coverage report -m
```

Cleaning outputs is automatic — the pipeline removes `output/` before regenerating it
(see [Output Management](#output-management) below); there is no separate manual clean step.

## Output Management

The pipeline automatically manages outputs. All outputs are regenerated from markdown sources during the build process, ensuring consistency.

This script:

- Removes `output/` directory (all disposable)
- Preserves source code, tests, markdown, and scripts
- Provides clear instructions for regeneration

**Note**: All outputs are regeneratable from source, so cleaning is safe and often useful for troubleshooting or ensuring fresh builds.

### Output Directory Structure

Same layout as the [Output Structure diagram above](#output-structure-output) (`figures/`, `data/`, `pdf/`, `tex/`). All directories under `output/` are disposable and can be safely cleaned.

## Benefits of This Paradigm

1. **Coherence**: Source code, tests, and documentation stay synchronized
2. **Validation**: Automatic checking of all references and outputs
3. **Reproducibility**: Deterministic generation of all artifacts
4. **Maintainability**: Clear separation of concerns with unified workflow
5. **Quality**: test coverage enforced automatically
6. **Documentation**: Validation of references and outputs
7. **Thin Orchestrator Pattern**: Scripts use tested `projects/{name}/src/` methods, not duplicate logic

## Troubleshooting

### Common Issues

1. **Tests failing**: Check coverage and fix missing test cases
2. **Markdown validation errors**: Fix broken links, missing images, or duplicate labels
3. **Figure generation failures**: Ensure src/ modules work correctly
4. **PDF build errors**: Check pandoc and LaTeX installation

### Test Import Errors

**Symptom**: `ModuleNotFoundError: No module named 'project.src'`

**Solution**: Ensure tests/conftest.py adds src/ to sys.path:
```python
import os, sys
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
SRC = os.path.join(ROOT, "src")
if SRC not in sys.path:
    sys.path.insert(0, SRC)
```

### Coverage Below Threshold

**Symptom**: `CoverageWarning: 85% < 90% required`

**Solution**: Find uncovered lines and add tests:
```bash
uv run pytest --cov=src --cov-report=term-missing
```

### Thin Orchestrator Violation

**Symptom**: Business logic in scripts instead of src/

**Solution**: Move algorithms to src/ modules, scripts only handle I/O and visualization

### Validation Commands

```bash
# Check what's failing
uv run python -m infrastructure.validation.cli markdown projects/templates/template_code_project/manuscript/

# Regenerate specific figures
uv run python projects/templates/template_code_project/scripts/optimization_analysis.py

# Check test coverage gaps
coverage report -m
```

## Key Connections to Remember

1. **`projects/{name}/src/` modules → `projects/{name}/tests/` validation → `projects/{name}/scripts/` generation → `projects/{name}/manuscript/` documentation**
2. **The pipeline orchestrator ensures all connections are valid before building outputs**
3. **Changes in any component must be reflected in all connected components**
4. **The test suite validates the entire pipeline, not just individual modules**
5. **Documentation is validated against outputs to maintain coherence**
6. **Scripts are THIN ORCHESTRATORS that import and use `projects/{name}/src/` methods**
7. **Business logic lives ONLY in `projects/{name}/src/` - scripts handle orchestration and I/O**

## Thin Orchestrator Pattern

The workflow enforces a **thin orchestrator pattern** where:

- **`projects/{name}/src/`** contains ALL business logic, algorithms, and mathematical implementations
- **`projects/{name}/scripts/`** are lightweight wrappers that import and use `projects/{name}/src/` methods
- **`projects/{name}/tests/`** ensures coverage of all functionality
- **`scripts/runner/execute_pipeline.py`** orchestrates the entire pipeline

This ensures:

- **Maintainability**: Single source of truth for business logic
- **Testability**: tested core functionality
- **Reusability**: Scripts can use any `projects/{name}/src/` method
- **Clarity**: Clear separation of concerns
- **Quality**: Automated validation of the entire system

This workflow ensures that the generic project template maintains the highest standards of code quality, documentation coherence, and maintainability while providing a clear, scalable structure for development and collaboration.

For more details on architecture and implementation, see **[`../core/architecture.md`](../core/architecture.md)** and **[`thin-orchestrator-summary.md`](../architecture/thin-orchestrator-summary.md)**.