# Two-Layer Architecture Guide ## Overview This research template implements a clear two-layer architecture separating generic build infrastructure from project-specific scientific content. This document explains the architecture, design rationale, and how to work within this structure. ## Quick Reference: Layer 1 vs Layer 2 | Aspect | **Layer 1: Infrastructure** | **Layer 2: Project** | |--------|------------------------------|----------------------| | **Location** | `infrastructure/` (root level) | `projects/{name}/src/` (project-specific) | | **Purpose** | Generic, reusable build tools | Domain-specific research code | | **Scope** | Works with any project | Specific to this research | | **Test Coverage** | 60% minimum for `infrastructure/` | 90% minimum for `projects/{name}/src/` | | **Scripts** | `scripts/` (root, generic orchestrators) | `projects/{name}/scripts/` (project orchestrators) | | **Tests** | `tests/infra_tests/` (root level) | `projects/{name}/tests/` (project-specific) | | **Imports** | `from infrastructure.module import` | `from projects.{name}.src.module import` | | **Dependencies** | No project dependencies | Can import from infrastructure | | **Examples** | PDF generation, validation, figure management | Algorithms, simulations, analysis | ## Architecture Layers ### [LAYER 1: INFRASTRUCTURE] Generic Build & Validation Tools **Location:** `infrastructure/` (root level) **Purpose:** Reusable tools and utilities that apply to any research project using this template. These handle: - Build orchestration and PDF generation - Document validation and quality checking - Build artifact verification - Environment reproducibility tracking - Academic publishing assistance - Figure and image management - Markdown integration **Modules:** ```mermaid flowchart TB INFRA[infrastructure] INFRA --> CORE[core
exceptions · logging · config_loader] INFRA --> VAL[validation
pdf · markdown · integrity] INFRA --> DOC[documentation
figure · image · markdown integration · glossary] INFRA --> PUB[publishing
academic publishing tools] INFRA --> LLM[llm
LLM integration · literature workflows] INFRA --> REND[rendering
multi-format · PDF · HTML · slides · DOCX · EPUB] INFRA --> SCI[scientific
scientific dev tools] INFRA --> SEARCH[search
multi-source literature search] INFRA --> REF[reference
citation BibTeX I/O · verification reference-existence gate] INFRA --> REP[reporting
pipeline reports] INFRA --> STEG[steganography
secure PDF post-processing] INFRA --> AUTO[autoresearch
deterministic research loops] INFRA --> BENCH[benchmark
timing · resource summaries] INFRA --> CFG[config
repo-wide configuration] INFRA --> DOCKER[docker
containerization settings] INFRA --> DR[doctor
repo health diagnostics] INFRA --> METH[methods
DAG contracts · methods prose] INFRA --> ORCH[orchestration
pipeline/multi-project/secure CLI] INFRA --> PROJ[project
multi-project discovery] INFRA --> PROSE[prose
prose-manuscript analysis] INFRA --> SIA[sia
self-improvement harness] INFRA --> SK[skills
SKILL.md discovery] classDef root fill:#0f172a,stroke:#0f172a,color:#fff classDef pkg fill:#1e3a8a,stroke:#0f172a,color:#fff class INFRA root class CORE,VAL,DOC,PUB,LLM,REND,SCI,SEARCH,REF,REP,STEG,AUTO,BENCH,CFG,DOCKER,DR,METH,ORCH,PROJ,PROSE,SIA,SK pkg ``` **Key Characteristics:** - Generic and reusable across projects - Handles template infrastructure concerns - 60% minimum test coverage for infrastructure (see [`docs/_generated/COUNTS.md`](../_generated/COUNTS.md) for measured status) - No domain-specific logic - Interfaces with project files (manuscript/, output/) **Usage Pattern:** ```python # Infrastructure usage from scripts from infrastructure.documentation import FigureManager from infrastructure.documentation import MarkdownIntegration # These manage the document structure, not the science fm = FigureManager() fm.register_figure( filename="convergence_plot.png", caption="Algorithm convergence comparison", label="fig:convergence" ) ``` --- ### [LAYER 2: PROJECT] Project-Specific Algorithms & Analysis **Location:** `projects/{name}/src/` (project-specific code), `projects/{name}/scripts/` (project orchestrators) **Purpose:** Domain-specific code implementing the research project's scientific algorithms, data processing, analysis, and visualization. **Modules:** ```mermaid flowchart LR SRC[projects/<name>/src] SRC --> EX[example.py
basic operations] SRC --> OTHER[*.py
project-specific modules] classDef d fill:#0f172a,stroke:#0f172a,color:#fff classDef f fill:#0f766e,stroke:#0f172a,color:#fff class SRC d class EX,OTHER f ``` **Scripts (thin orchestrators):** > Each active exemplar has its own concrete `scripts/` layout — the names > below are the **template_code_project** canonical roster as of May 2026 > (template_prose_project uses a parallel set: `run_prose_pipeline.py`, > `y_generate_prose_figures.py`, `z_generate_manuscript_variables.py`, > `00_preflight.py`). ```mermaid flowchart LR SC[projects/templates/template_code_project/scripts/] SC --> PF[00_preflight.py
chrome-headless-shell preflight] SC --> OA[optimization_analysis.py
main analysis pipeline · thin wrapper around src/analysis/] SC --> BD[build_dashboard.py
numerical-invariants HTML dashboard] SC --> GD[generate_api_docs.py
API documentation generator] SC --> ZG[z_generate_manuscript_variables.py
variable token substitution · runs LAST] classDef d fill:#0f172a,stroke:#0f172a,color:#fff classDef f fill:#0f766e,stroke:#0f172a,color:#fff class SC d class PF,OA,BD,GD,ZG f ``` The May 2026 hardening pass split the former flat analysis module into `src/analysis/` (orchestration) and `src/figures/` (the six `generate_*` plot functions plus `apply_visualization_style` and `VIZ_CONFIG`). `src/analysis/__init__.py` re-exports the public analysis names so the existing `scripts/optimization_analysis.py` and infrastructure-dependent test classes keep working without changes. **Key Characteristics:** - Domain-specific and research-focused - Implements algorithms and computations - Calls infrastructure when needed - 90% minimum test coverage for project `src/` (measure locally or see [`docs/_generated/COUNTS.md`](../_generated/COUNTS.md)) - Follows thin orchestrator pattern **Usage Pattern:** ```python # Project-specific usage from scripts from projects.name.src.simulation import SimpleSimulation from projects.name.src.statistics import calculate_descriptive_stats from infrastructure.documentation import FigureManager # Science: Run simulation and analysis sim = SimpleSimulation() results = sim.run() stats = calculate_descriptive_stats(results) # Infrastructure: Manage figures fm = FigureManager() fm.register_figure( filename="results.png", caption="Simulation results", label="fig:results" ) ``` --- ## Layer Separation ### Architectural Boundaries ```mermaid graph TB subgraph L1["LAYER 1: INFRASTRUCTURE
(Build orchestration, validation, document management)"] subgraph SCRIPTS["Pipeline Orchestrators"] RUN_ALL[execute_pipeline.py
declared DAG pipeline] SCRIPT_LIST[scripts/pipeline/*.py
- stage_00_setup.py
- stage_01_test.py
- stage_02_analysis.py
- stage_03_render.py
- stage_04_validate.py
- stage_05_copy.py] end subgraph INFRA["infrastructure/"] INFRA_MODS[core/, validation/,
documentation/, publishing/,
llm/, rendering/,
scientific/, skills/, steganography/] end end subgraph L2["LAYER 2: SCIENTIFIC
(Algorithms, analysis, visualization, data)"] subgraph SRC["projects/{name}/src/"] SRC_MODS["optimizer.py · analysis/ · figures/
dashboard.py · manuscript_variables.py ·
project_paths.py · invariants.py · sweeps.py
— canonical exemplar layout; other exemplars vary"] end subgraph PROJ_SCRIPTS["projects/{name}/scripts
(thin orchestrators · see active_projects.md for the live exemplar roster)"] PROJ_SCRIPT_LIST[code: 00_preflight.py · optimization_analysis.py · build_dashboard.py · z_generate_manuscript_variables.py
prose: 00_preflight.py · run_prose_pipeline.py · y_generate_prose_figures.py · z_generate_manuscript_variables.py] end end subgraph MANUSCRIPT["manuscript
(research content)"] MANUSCRIPT_FILES[01_abstract.md through
99_references.md] end L1 -->|"Manages structure and
validates outputs"| L2 L1 -->|"Validates science"| L2 L2 -->|"Input: Manuscripts, configurations
Output: Figures, data, reports"| MANUSCRIPT classDef layer1 fill:#e1f5fe,stroke:#01579b,stroke-width:3px classDef layer2 fill:#f1f8e9,stroke:#33691e,stroke-width:3px classDef manuscript fill:#fff3e0,stroke:#e65100,stroke-width:2px class L1,SCRIPTS,INFRA,RUN_ALL,SCRIPT_LIST,INFRA_MODS layer1 class L2,SRC,PROJ_SCRIPTS,SRC_MODS,PROJ_SCRIPT_LIST layer2 class MANUSCRIPT,MANUSCRIPT_FILES manuscript ``` ### Import Guidelines **✅ Layer 1 → Layer 1:** Infrastructure modules can import from other infrastructure modules ```python from infrastructure.documentation import FigureManager from infrastructure.documentation import ImageManager ``` **✅ Layer 2 → Layer 1:** Project code can import infrastructure ```python from projects.name.src.visualization import plot_results from infrastructure.documentation import FigureManager # Use infrastructure for figure management fig = plot_results(data) fig.savefig("output/figures/results.png") fm = FigureManager() fm.register_figure( filename="results.png", caption="Results visualization", label="fig:results" ) ``` **✅ Layer 2 → Layer 2:** Project modules can import from other project modules ```python from projects.name.src.simulation import SimpleSimulation from projects.name.src.statistics import calculate_descriptive_stats ``` **❌ Layer 1 → Layer 2:** Infrastructure should NOT import project code ```python # BAD: Build tools shouldn't depend on project-specific code from infrastructure.validation.integrity.checks import verify_output_integrity from projects.name.src.simulation import SimpleSimulation # ❌ WRONG # This breaks the abstraction and makes infrastructure project-specific ``` --- ## Code Organization ### [LAYER 1] Infrastructure Structure For the full module roster, see the [Layer 1 diagram above](#layer-1-infrastructure-generic-build--validation-tools) — not repeated here to avoid maintaining the same 22-module list twice. What's new at this level of detail is the convention-file layout inside each module: ```mermaid flowchart TB INFRA[infrastructure
Layer 1 · importable packages
see COUNTS.md] INFRA --> ONE["<any module>/"] ONE --> META[__init__.py · AGENTS.md ·
README.md · SKILL.md] INFRA --> ROOT[mcp_server.py
top-level stdio MCP server] classDef root fill:#0f172a,stroke:#0f172a,color:#fff classDef pkg fill:#1e3a8a,stroke:#0f172a,color:#fff classDef meta fill:#0f766e,stroke:#0f172a,color:#fff class INFRA root class ONE,ROOT pkg class META meta ``` File-level layout inside each package: see [`infrastructure/AGENTS.md`](../../infrastructure/AGENTS.md). ### [LAYER 2] Project Structure ```mermaid flowchart TB PROJ[project
Project-specific code] PROJ --> SRC[src
Project scientific code] PROJ --> SC[scripts
Project orchestrators] PROJ --> T[tests
Project tests] SRC --> SRC_FILES[__init__.py · AGENTS.md · README.md ·
example.py · ...] SC --> SC_FILES[optimization_analysis.py · build_dashboard.py ·
z_generate_manuscript_variables.py · run_prose_pipeline.py ·
y_generate_prose_figures.py] T --> T_FILES[__init__.py · test_example.py ·
test_simulation.py · test_statistics.py · ...] classDef d fill:#0f172a,stroke:#0f172a,color:#fff classDef pkg fill:#1e3a8a,stroke:#0f172a,color:#fff classDef f fill:#0f766e,stroke:#0f172a,color:#fff class PROJ d class SRC,SC,T pkg class SRC_FILES,SC_FILES,T_FILES f ``` ### Test Structure ```mermaid flowchart TB ROOT_TESTS[tests
Root level · infrastructure tests] ROOT_TESTS --> INFRA_T[infra_tests
Layer 1 tests] ROOT_TESTS --> INTEG[integration
Cross-layer tests] ROOT_TESTS --> HELPERS[helpers
Test utilities] INFRA_T --> INFRA_F[__init__.py · test_build/ ·
test_validation/ · test_documentation/ · ...] INTEG --> INTEG_F[__init__.py · test_integration_pipeline.py · ...] PROJ_TESTS[projects/<name>/tests
Layer 2 · project tests] PROJ_TESTS --> PROJ_F[__init__.py · test_example.py ·
test_simulation.py · test_statistics.py · ...] classDef d fill:#0f172a,stroke:#0f172a,color:#fff classDef pkg fill:#1e3a8a,stroke:#0f172a,color:#fff classDef f fill:#0f766e,stroke:#0f172a,color:#fff class ROOT_TESTS,PROJ_TESTS d class INFRA_T,INTEG,HELPERS pkg class INFRA_F,INTEG_F,PROJ_F f ``` --- ## Execution Flow ### Build Pipeline - Layer Transitions ```mermaid flowchart TD START(["User runs:
uv run python scripts/runner/execute_pipeline.py --project {name} --core-only"]) --> CLEAN["STAGE 0: Clean Output Directories
- Remove old outputs
- Prepare fresh build"] CLEAN --> STAGE00["STAGE 00: LAYER 1
Environment Setup
- Validate Python, dependencies
- Check build tools"] STAGE00 --> PHASE1["PHASE 1: LAYER 1
Test Validation
- Run tests/infra_tests
- Run projects/{name}/tests
- Run tests/integration
- Validate coverage requirements
Report: LAYER-1-INFRASTRUCTURE Running"] PHASE1 --> PHASE2["PHASE 2: LAYER 2
Project Execution
- Run projects/{name}/scripts/*.py
- Generate figures
- Process data
- Create outputs
Report: LAYER-2-PROJECT Running"] PHASE2 --> PHASE2_5["PHASE 2.5: LAYER 1
Utilities
- Generate API glossary
- Validate markdown
- Check cross-references
Report: LAYER-1-INFRASTRUCTURE Running"] PHASE2_5 --> PHASE3_5["PHASE 3-5: LAYER 1
Document Generation
- Generate LaTeX preamble
- Build individual PDFs
- Build combined PDF
- Create HTML version
Report: LAYER-1-INFRASTRUCTURE Building"] PHASE3_5 --> PHASE6["PHASE 6: LAYER 1
Validation
- Validate PDF quality
- Check for rendering issues
Report: LAYER-1-INFRASTRUCTURE Done"] PHASE6 --> SUCCESS(["Success:
All PDFs generated,
all layers working"]) classDef layer1 fill:#e1f5fe,stroke:#01579b,stroke-width:2px classDef layer2 fill:#f1f8e9,stroke:#33691e,stroke-width:2px classDef success fill:#e8f5e8,stroke:#2e7d32,stroke-width:3px classDef start fill:#fff3e0,stroke:#e65100,stroke-width:2px class STAGE00,PHASE1,PHASE2_5,PHASE3_5,PHASE6 layer1 class PHASE2 layer2 class SUCCESS success class START start ``` ### Logging Output Example ``` ━━━ LAYER 1: Infrastructure Validation ━━━ [YYYY-MM-DD HH:MM:SS] [INFO] Running tests (infrastructure + scientific) ...tests output... [YYYY-MM-DD HH:MM:SS] [INFO] ✅ All tests passed with adequate coverage ━━━ LAYER 2: Project Computation ━━━ [YYYY-MM-DD HH:MM:SS] [INFO] Executing project scripts... [YYYY-MM-DD HH:MM:SS] [INFO] [LAYER-2-PROJECT] Starting analysis pipeline... ...script output... [YYYY-MM-DD HH:MM:SS] [INFO] ✅ ALL project scripts executed successfully ━━━ LAYER 1: Infrastructure Validation ━━━ [YYYY-MM-DD HH:MM:SS] [INFO] Running repository utilities (glossary + markdown validation) ...validation output... [YYYY-MM-DD HH:MM:SS] [INFO] ✅ Repository utilities completed ━━━ LAYER 1: Document Generation ━━━ [YYYY-MM-DD HH:MM:SS] [INFO] Step 3: Generating LaTeX preamble from markdown... [YYYY-MM-DD HH:MM:SS] [INFO] Step 4: Discovering and building ALL markdown modules... ...PDF generation output... [YYYY-MM-DD HH:MM:SS] [INFO] ✅ Combined document built successfully ``` --- ## Adding New Code ### Decision Tree: Where Should Code Go? ```mermaid flowchart TB Q1{Is this specific to
our research project?} Q1 -- yes --> L2[Layer 2
projects/<name>/src/] Q1 -- no --> Q2{Is it about
building / validating?} Q2 -- yes --> L1[Layer 1
infrastructure/] Q2 -- no --> RECONSIDER[Reconsider scope] L2 -.examples.-> L2EX[Simulation algorithms ·
Statistical analysis ·
Custom visualization ·
Parameter sweeps ·
Domain-specific processing] L1 -.examples.-> L1EX[PDF generation · Figure management ·
Document validation · Build verification ·
Generic utilities · Cross-project templates] Q3{Reusable across
projects?} -.tiebreaker.- Q1 Q3 -- yes --> L1 Q3 -- no --> L2 classDef q fill:#1e3a8a,stroke:#0f172a,color:#fff classDef l1 fill:#0f766e,stroke:#0f172a,color:#fff classDef l2 fill:#7c2d12,stroke:#0f172a,color:#fff class Q1,Q2,Q3 q class L1,L1EX l1 class L2,L2EX,RECONSIDER l2 ``` ### Adding a New Project Module 1. **Create the module:** ```bash vim projects/{name}/src/new_algorithm.py ``` 2. **Implement with type hints and docstrings:** ```python """New algorithm implementation.""" from typing import List, Optional def analyze_data(data: List[float]) -> Optional[float]: """Analyze data. Args: data: Input data Returns: Analysis result """ pass ``` 3. **Write tests:** ```bash vim projects/{name}/tests/test_new_algorithm.py ``` 4. **Add to projects/{name}/src/**init**.py:** ```python from .new_algorithm import analyze_data ``` 5. **Use in scripts:** ```python from projects.name.src.new_algorithm import analyze_data ``` 6. **Update documentation:** - Add to projects/{name}/src/AGENTS.md - Add to projects/{name}/src/README.md ### Adding a New Infrastructure Module 1. **Create the module:** ```bash vim infrastructure/validation/new_validator.py ``` 2. **Implement generic, project-independent logic:** ```python """New validation tool.""" def validate_output_structure(output_dir: str) -> bool: """Validate output directory structure.""" pass ``` 3. **Write tests:** ```bash vim tests/infra_tests/validation/test_pdf_validator.py ``` 4. **Document usage:** - Add to infrastructure/validation/AGENTS.md - Include usage examples 5. **Integrate with build pipeline:** - Update scripts/runner/execute_pipeline.py if needed - Update infrastructure modules if applicable --- ## Testing Strategy Test organization, per-layer commands, and coverage gates are documented once, canonically, in **[Testing Strategy](testing-strategy.md)** — see that file for `tests/infra_tests/`, `projects/{name}/tests/`, `tests/integration/`, and the full-suite/coverage-report commands. --- ## Best Practices ### For Infrastructure Development ✅ **Do:** - Write generic, reusable code - Document with project-independent examples - Test extensively with real scenarios - Handle errors gracefully - Provide clear logging ❌ **Don't:** - Import scientific modules - Assume specific research domain - Skip tests to ship features - Hardcode project-specific values - Mix concerns (building vs. computation) ### For Scientific Development ✅ **Do:** - Use infrastructure tools for document management - Follow thin orchestrator pattern in projects/{name}/scripts/ - Implement algorithms in projects/{name}/src/ modules - Test with data - Document domain-specific concepts ❌ **Don't:** - Duplicate build/validation logic - Implement document generation in scripts - Skip layer abstraction - Mix orchestration with computation - Depend on infrastructure internals ### Logging Best Practices ```python # In project scripts - mark layer transitions import logging logger = logging.getLogger(__name__) logger.info("[LAYER-2-PROJECT] Starting simulation...") logger.info("[LAYER-1-INFRASTRUCTURE] Using FigureManager for output...") ``` ```bash # In build scripts - mark phase transitions log_info "━━━ LAYER 1: Infrastructure Validation ━━━" log_info "━━━ LAYER 2: Scientific Computation ━━━" ``` --- ## Migration from Flat Structure If you have an old project with flat src/, migrating to the two-layer structure: 1. **Create packages:** ```bash mkdir -p infrastructure projects/{name}/src ``` 2. **Move modules:** - Infrastructure modules → infrastructure/ - Project modules → projects/{name}/src/ 3. **Update imports:** - `from example import` → `from projects.{name}.src.example import` - Build verification is handled by the validation module 4. **Update tests:** - Infrastructure tests → tests/infra_tests/ - Project tests → projects/{name}/tests/ - Update conftest.py if needed 5. **Validate:** ```bash uv run pytest tests/ projects/{name}/tests/ --cov=infrastructure --cov=projects/{name}/src uv run python scripts/runner/execute_pipeline.py --project {name} --core-only ``` --- ## Troubleshooting Import errors, layer violations, and mixed-concerns fixes are documented once, canonically, in **[Testing Strategy § Troubleshooting](testing-strategy.md#troubleshooting)**. --- ## References ### Architecture Documentation - [../core/architecture.md](../core/architecture.md) - system architecture overview - [decision-tree.md](../architecture/decision-tree.md) - Code placement flowchart - [thin-orchestrator-summary.md](../architecture/thin-orchestrator-summary.md) - Thin orchestrator pattern details ### Layer-Specific Documentation - [infrastructure/AGENTS.md](../../infrastructure/AGENTS.md) - Infrastructure layer documentation - [infrastructure/README.md](../../infrastructure/README.md) - Infrastructure quick reference - [template_code_project/src/AGENTS.md](../../projects/templates/template_code_project/src/AGENTS.md) - Project layer documentation - [template_code_project/src/README.md](../../projects/templates/template_code_project/src/README.md) - Project quick reference ### System Documentation - [../AGENTS.md](../AGENTS.md) - system documentation - [../README.md](../README.md) - Project overview - [../core/how-to-use.md](../core/how-to-use.md) - usage guide --- ## Key Takeaway **Layers separate concerns:** - **[LAYER 1: INFRASTRUCTURE]** handles *how* research is documented and built - **[LAYER 2: PROJECT]** focuses on *what* research is conducted This separation makes code more modular, reusable, and maintainable. ## Quick Navigation - **Understanding the architecture**: Start with the [Quick Reference](#quick-reference-layer-1-vs-layer-2) table above - **Adding code**: See [Decision Tree](#decision-tree-where-should-code-go) section - **Import patterns**: See [Import Guidelines](#import-guidelines) section - **Testing**: See [Testing Strategy](#testing-strategy) section