# Two-Layer Architecture Guide
## Overview
This research template implements a clear two-layer architecture separating generic build infrastructure from project-specific scientific content. This document explains the architecture, design rationale, and how to work within this structure.
## Quick Reference: Layer 1 vs Layer 2
| Aspect | **Layer 1: Infrastructure** | **Layer 2: Project** |
|--------|------------------------------|----------------------|
| **Location** | `infrastructure/` (root level) | `projects/{name}/src/` (project-specific) |
| **Purpose** | Generic, reusable build tools | Domain-specific research code |
| **Scope** | Works with any project | Specific to this research |
| **Test Coverage** | 60% minimum for `infrastructure/` | 90% minimum for `projects/{name}/src/` |
| **Scripts** | `scripts/` (root, generic orchestrators) | `projects/{name}/scripts/` (project orchestrators) |
| **Tests** | `tests/infra_tests/` (root level) | `projects/{name}/tests/` (project-specific) |
| **Imports** | `from infrastructure.module import` | `from projects.{name}.src.module import` |
| **Dependencies** | No project dependencies | Can import from infrastructure |
| **Examples** | PDF generation, validation, figure management | Algorithms, simulations, analysis |
## Architecture Layers
### [LAYER 1: INFRASTRUCTURE] Generic Build & Validation Tools
**Location:** `infrastructure/` (root level)
**Purpose:** Reusable tools and utilities that apply to any research project using this template. These handle:
- Build orchestration and PDF generation
- Document validation and quality checking
- Build artifact verification
- Environment reproducibility tracking
- Academic publishing assistance
- Figure and image management
- Markdown integration
**Modules:**
```mermaid
flowchart TB
INFRA[infrastructure]
INFRA --> CORE[core
exceptions · logging · config_loader]
INFRA --> VAL[validation
pdf · markdown · integrity]
INFRA --> DOC[documentation
figure · image · markdown integration · glossary]
INFRA --> PUB[publishing
academic publishing tools]
INFRA --> LLM[llm
LLM integration · literature workflows]
INFRA --> REND[rendering
multi-format · PDF · HTML · slides · DOCX · EPUB]
INFRA --> SCI[scientific
scientific dev tools]
INFRA --> SEARCH[search
multi-source literature search]
INFRA --> REF[reference
citation BibTeX I/O · verification reference-existence gate]
INFRA --> REP[reporting
pipeline reports]
INFRA --> STEG[steganography
secure PDF post-processing]
INFRA --> AUTO[autoresearch
deterministic research loops]
INFRA --> BENCH[benchmark
timing · resource summaries]
INFRA --> CFG[config
repo-wide configuration]
INFRA --> DOCKER[docker
containerization settings]
INFRA --> DR[doctor
repo health diagnostics]
INFRA --> METH[methods
DAG contracts · methods prose]
INFRA --> ORCH[orchestration
pipeline/multi-project/secure CLI]
INFRA --> PROJ[project
multi-project discovery]
INFRA --> PROSE[prose
prose-manuscript analysis]
INFRA --> SIA[sia
self-improvement harness]
INFRA --> SK[skills
SKILL.md discovery]
classDef root fill:#0f172a,stroke:#0f172a,color:#fff
classDef pkg fill:#1e3a8a,stroke:#0f172a,color:#fff
class INFRA root
class CORE,VAL,DOC,PUB,LLM,REND,SCI,SEARCH,REF,REP,STEG,AUTO,BENCH,CFG,DOCKER,DR,METH,ORCH,PROJ,PROSE,SIA,SK pkg
```
**Key Characteristics:**
- Generic and reusable across projects
- Handles template infrastructure concerns
- 60% minimum test coverage for infrastructure (see [`docs/_generated/COUNTS.md`](../_generated/COUNTS.md) for measured status)
- No domain-specific logic
- Interfaces with project files (manuscript/, output/)
**Usage Pattern:**
```python
# Infrastructure usage from scripts
from infrastructure.documentation import FigureManager
from infrastructure.documentation import MarkdownIntegration
# These manage the document structure, not the science
fm = FigureManager()
fm.register_figure(
filename="convergence_plot.png",
caption="Algorithm convergence comparison",
label="fig:convergence"
)
```
---
### [LAYER 2: PROJECT] Project-Specific Algorithms & Analysis
**Location:** `projects/{name}/src/` (project-specific code), `projects/{name}/scripts/` (project orchestrators)
**Purpose:** Domain-specific code implementing the research project's scientific algorithms, data processing, analysis, and visualization.
**Modules:**
```mermaid
flowchart LR
SRC[projects/<name>/src]
SRC --> EX[example.py
basic operations]
SRC --> OTHER[*.py
project-specific modules]
classDef d fill:#0f172a,stroke:#0f172a,color:#fff
classDef f fill:#0f766e,stroke:#0f172a,color:#fff
class SRC d
class EX,OTHER f
```
**Scripts (thin orchestrators):**
> Each active exemplar has its own concrete `scripts/` layout — the names
> below are the **template_code_project** canonical roster as of May 2026
> (template_prose_project uses a parallel set: `run_prose_pipeline.py`,
> `y_generate_prose_figures.py`, `z_generate_manuscript_variables.py`,
> `00_preflight.py`).
```mermaid
flowchart LR
SC[projects/templates/template_code_project/scripts/]
SC --> PF[00_preflight.py
chrome-headless-shell preflight]
SC --> OA[optimization_analysis.py
main analysis pipeline · thin wrapper around src/analysis/]
SC --> BD[build_dashboard.py
numerical-invariants HTML dashboard]
SC --> GD[generate_api_docs.py
API documentation generator]
SC --> ZG[z_generate_manuscript_variables.py
variable token substitution · runs LAST]
classDef d fill:#0f172a,stroke:#0f172a,color:#fff
classDef f fill:#0f766e,stroke:#0f172a,color:#fff
class SC d
class PF,OA,BD,GD,ZG f
```
The May 2026 hardening pass split the former flat analysis module into
`src/analysis/` (orchestration) and `src/figures/` (the six `generate_*`
plot functions plus `apply_visualization_style` and `VIZ_CONFIG`).
`src/analysis/__init__.py` re-exports the public analysis names so the existing
`scripts/optimization_analysis.py` and infrastructure-dependent test
classes keep working without changes.
**Key Characteristics:**
- Domain-specific and research-focused
- Implements algorithms and computations
- Calls infrastructure when needed
- 90% minimum test coverage for project `src/` (measure locally or see [`docs/_generated/COUNTS.md`](../_generated/COUNTS.md))
- Follows thin orchestrator pattern
**Usage Pattern:**
```python
# Project-specific usage from scripts
from projects.name.src.simulation import SimpleSimulation
from projects.name.src.statistics import calculate_descriptive_stats
from infrastructure.documentation import FigureManager
# Science: Run simulation and analysis
sim = SimpleSimulation()
results = sim.run()
stats = calculate_descriptive_stats(results)
# Infrastructure: Manage figures
fm = FigureManager()
fm.register_figure(
filename="results.png",
caption="Simulation results",
label="fig:results"
)
```
---
## Layer Separation
### Architectural Boundaries
```mermaid
graph TB
subgraph L1["LAYER 1: INFRASTRUCTURE
(Build orchestration, validation, document management)"]
subgraph SCRIPTS["Pipeline Orchestrators"]
RUN_ALL[execute_pipeline.py
declared DAG pipeline]
SCRIPT_LIST[scripts/pipeline/*.py
- stage_00_setup.py
- stage_01_test.py
- stage_02_analysis.py
- stage_03_render.py
- stage_04_validate.py
- stage_05_copy.py]
end
subgraph INFRA["infrastructure/"]
INFRA_MODS[core/, validation/,
documentation/, publishing/,
llm/, rendering/,
scientific/, skills/, steganography/]
end
end
subgraph L2["LAYER 2: SCIENTIFIC
(Algorithms, analysis, visualization, data)"]
subgraph SRC["projects/{name}/src/"]
SRC_MODS["optimizer.py · analysis/ · figures/
dashboard.py · manuscript_variables.py ·
project_paths.py · invariants.py · sweeps.py
— canonical exemplar layout; other exemplars vary"]
end
subgraph PROJ_SCRIPTS["projects/{name}/scripts
(thin orchestrators · see active_projects.md for the live exemplar roster)"]
PROJ_SCRIPT_LIST[code: 00_preflight.py · optimization_analysis.py · build_dashboard.py · z_generate_manuscript_variables.py
prose: 00_preflight.py · run_prose_pipeline.py · y_generate_prose_figures.py · z_generate_manuscript_variables.py]
end
end
subgraph MANUSCRIPT["manuscript
(research content)"]
MANUSCRIPT_FILES[01_abstract.md through
99_references.md]
end
L1 -->|"Manages structure and
validates outputs"| L2
L1 -->|"Validates science"| L2
L2 -->|"Input: Manuscripts, configurations
Output: Figures, data, reports"| MANUSCRIPT
classDef layer1 fill:#e1f5fe,stroke:#01579b,stroke-width:3px
classDef layer2 fill:#f1f8e9,stroke:#33691e,stroke-width:3px
classDef manuscript fill:#fff3e0,stroke:#e65100,stroke-width:2px
class L1,SCRIPTS,INFRA,RUN_ALL,SCRIPT_LIST,INFRA_MODS layer1
class L2,SRC,PROJ_SCRIPTS,SRC_MODS,PROJ_SCRIPT_LIST layer2
class MANUSCRIPT,MANUSCRIPT_FILES manuscript
```
### Import Guidelines
**✅ Layer 1 → Layer 1:** Infrastructure modules can import from other infrastructure modules
```python
from infrastructure.documentation import FigureManager
from infrastructure.documentation import ImageManager
```
**✅ Layer 2 → Layer 1:** Project code can import infrastructure
```python
from projects.name.src.visualization import plot_results
from infrastructure.documentation import FigureManager
# Use infrastructure for figure management
fig = plot_results(data)
fig.savefig("output/figures/results.png")
fm = FigureManager()
fm.register_figure(
filename="results.png",
caption="Results visualization",
label="fig:results"
)
```
**✅ Layer 2 → Layer 2:** Project modules can import from other project modules
```python
from projects.name.src.simulation import SimpleSimulation
from projects.name.src.statistics import calculate_descriptive_stats
```
**❌ Layer 1 → Layer 2:** Infrastructure should NOT import project code
```python
# BAD: Build tools shouldn't depend on project-specific code
from infrastructure.validation.integrity.checks import verify_output_integrity
from projects.name.src.simulation import SimpleSimulation # ❌ WRONG
# This breaks the abstraction and makes infrastructure project-specific
```
---
## Code Organization
### [LAYER 1] Infrastructure Structure
For the full module roster, see the [Layer 1 diagram above](#layer-1-infrastructure-generic-build--validation-tools) —
not repeated here to avoid maintaining the same 22-module list twice. What's new at this
level of detail is the convention-file layout inside each module:
```mermaid
flowchart TB
INFRA[infrastructure
Layer 1 · importable packages
see COUNTS.md]
INFRA --> ONE["<any module>/"]
ONE --> META[__init__.py · AGENTS.md ·
README.md · SKILL.md]
INFRA --> ROOT[mcp_server.py
top-level stdio MCP server]
classDef root fill:#0f172a,stroke:#0f172a,color:#fff
classDef pkg fill:#1e3a8a,stroke:#0f172a,color:#fff
classDef meta fill:#0f766e,stroke:#0f172a,color:#fff
class INFRA root
class ONE,ROOT pkg
class META meta
```
File-level layout inside each package: see [`infrastructure/AGENTS.md`](../../infrastructure/AGENTS.md).
### [LAYER 2] Project Structure
```mermaid
flowchart TB
PROJ[project
Project-specific code]
PROJ --> SRC[src
Project scientific code]
PROJ --> SC[scripts
Project orchestrators]
PROJ --> T[tests
Project tests]
SRC --> SRC_FILES[__init__.py · AGENTS.md · README.md ·
example.py · ...]
SC --> SC_FILES[optimization_analysis.py · build_dashboard.py ·
z_generate_manuscript_variables.py · run_prose_pipeline.py ·
y_generate_prose_figures.py]
T --> T_FILES[__init__.py · test_example.py ·
test_simulation.py · test_statistics.py · ...]
classDef d fill:#0f172a,stroke:#0f172a,color:#fff
classDef pkg fill:#1e3a8a,stroke:#0f172a,color:#fff
classDef f fill:#0f766e,stroke:#0f172a,color:#fff
class PROJ d
class SRC,SC,T pkg
class SRC_FILES,SC_FILES,T_FILES f
```
### Test Structure
```mermaid
flowchart TB
ROOT_TESTS[tests
Root level · infrastructure tests]
ROOT_TESTS --> INFRA_T[infra_tests
Layer 1 tests]
ROOT_TESTS --> INTEG[integration
Cross-layer tests]
ROOT_TESTS --> HELPERS[helpers
Test utilities]
INFRA_T --> INFRA_F[__init__.py · test_build/ ·
test_validation/ · test_documentation/ · ...]
INTEG --> INTEG_F[__init__.py · test_integration_pipeline.py · ...]
PROJ_TESTS[projects/<name>/tests
Layer 2 · project tests]
PROJ_TESTS --> PROJ_F[__init__.py · test_example.py ·
test_simulation.py · test_statistics.py · ...]
classDef d fill:#0f172a,stroke:#0f172a,color:#fff
classDef pkg fill:#1e3a8a,stroke:#0f172a,color:#fff
classDef f fill:#0f766e,stroke:#0f172a,color:#fff
class ROOT_TESTS,PROJ_TESTS d
class INFRA_T,INTEG,HELPERS pkg
class INFRA_F,INTEG_F,PROJ_F f
```
---
## Execution Flow
### Build Pipeline - Layer Transitions
```mermaid
flowchart TD
START(["User runs:
uv run python scripts/runner/execute_pipeline.py --project {name} --core-only"]) --> CLEAN["STAGE 0: Clean Output Directories
- Remove old outputs
- Prepare fresh build"]
CLEAN --> STAGE00["STAGE 00: LAYER 1
Environment Setup
- Validate Python, dependencies
- Check build tools"]
STAGE00 --> PHASE1["PHASE 1: LAYER 1
Test Validation
- Run tests/infra_tests
- Run projects/{name}/tests
- Run tests/integration
- Validate coverage requirements
Report: LAYER-1-INFRASTRUCTURE Running"]
PHASE1 --> PHASE2["PHASE 2: LAYER 2
Project Execution
- Run projects/{name}/scripts/*.py
- Generate figures
- Process data
- Create outputs
Report: LAYER-2-PROJECT Running"]
PHASE2 --> PHASE2_5["PHASE 2.5: LAYER 1
Utilities
- Generate API glossary
- Validate markdown
- Check cross-references
Report: LAYER-1-INFRASTRUCTURE Running"]
PHASE2_5 --> PHASE3_5["PHASE 3-5: LAYER 1
Document Generation
- Generate LaTeX preamble
- Build individual PDFs
- Build combined PDF
- Create HTML version
Report: LAYER-1-INFRASTRUCTURE Building"]
PHASE3_5 --> PHASE6["PHASE 6: LAYER 1
Validation
- Validate PDF quality
- Check for rendering issues
Report: LAYER-1-INFRASTRUCTURE Done"]
PHASE6 --> SUCCESS(["Success:
All PDFs generated,
all layers working"])
classDef layer1 fill:#e1f5fe,stroke:#01579b,stroke-width:2px
classDef layer2 fill:#f1f8e9,stroke:#33691e,stroke-width:2px
classDef success fill:#e8f5e8,stroke:#2e7d32,stroke-width:3px
classDef start fill:#fff3e0,stroke:#e65100,stroke-width:2px
class STAGE00,PHASE1,PHASE2_5,PHASE3_5,PHASE6 layer1
class PHASE2 layer2
class SUCCESS success
class START start
```
### Logging Output Example
```
━━━ LAYER 1: Infrastructure Validation ━━━
[YYYY-MM-DD HH:MM:SS] [INFO] Running tests (infrastructure + scientific)
...tests output...
[YYYY-MM-DD HH:MM:SS] [INFO] ✅ All tests passed with adequate coverage
━━━ LAYER 2: Project Computation ━━━
[YYYY-MM-DD HH:MM:SS] [INFO] Executing project scripts...
[YYYY-MM-DD HH:MM:SS] [INFO] [LAYER-2-PROJECT] Starting analysis pipeline...
...script output...
[YYYY-MM-DD HH:MM:SS] [INFO] ✅ ALL project scripts executed successfully
━━━ LAYER 1: Infrastructure Validation ━━━
[YYYY-MM-DD HH:MM:SS] [INFO] Running repository utilities (glossary + markdown validation)
...validation output...
[YYYY-MM-DD HH:MM:SS] [INFO] ✅ Repository utilities completed
━━━ LAYER 1: Document Generation ━━━
[YYYY-MM-DD HH:MM:SS] [INFO] Step 3: Generating LaTeX preamble from markdown...
[YYYY-MM-DD HH:MM:SS] [INFO] Step 4: Discovering and building ALL markdown modules...
...PDF generation output...
[YYYY-MM-DD HH:MM:SS] [INFO] ✅ Combined document built successfully
```
---
## Adding New Code
### Decision Tree: Where Should Code Go?
```mermaid
flowchart TB
Q1{Is this specific to
our research project?}
Q1 -- yes --> L2[Layer 2
projects/<name>/src/]
Q1 -- no --> Q2{Is it about
building / validating?}
Q2 -- yes --> L1[Layer 1
infrastructure/]
Q2 -- no --> RECONSIDER[Reconsider scope]
L2 -.examples.-> L2EX[Simulation algorithms ·
Statistical analysis ·
Custom visualization ·
Parameter sweeps ·
Domain-specific processing]
L1 -.examples.-> L1EX[PDF generation · Figure management ·
Document validation · Build verification ·
Generic utilities · Cross-project templates]
Q3{Reusable across
projects?} -.tiebreaker.- Q1
Q3 -- yes --> L1
Q3 -- no --> L2
classDef q fill:#1e3a8a,stroke:#0f172a,color:#fff
classDef l1 fill:#0f766e,stroke:#0f172a,color:#fff
classDef l2 fill:#7c2d12,stroke:#0f172a,color:#fff
class Q1,Q2,Q3 q
class L1,L1EX l1
class L2,L2EX,RECONSIDER l2
```
### Adding a New Project Module
1. **Create the module:**
```bash
vim projects/{name}/src/new_algorithm.py
```
2. **Implement with type hints and docstrings:**
```python
"""New algorithm implementation."""
from typing import List, Optional
def analyze_data(data: List[float]) -> Optional[float]:
"""Analyze data.
Args:
data: Input data
Returns:
Analysis result
"""
pass
```
3. **Write tests:**
```bash
vim projects/{name}/tests/test_new_algorithm.py
```
4. **Add to projects/{name}/src/**init**.py:**
```python
from .new_algorithm import analyze_data
```
5. **Use in scripts:**
```python
from projects.name.src.new_algorithm import analyze_data
```
6. **Update documentation:**
- Add to projects/{name}/src/AGENTS.md
- Add to projects/{name}/src/README.md
### Adding a New Infrastructure Module
1. **Create the module:**
```bash
vim infrastructure/validation/new_validator.py
```
2. **Implement generic, project-independent logic:**
```python
"""New validation tool."""
def validate_output_structure(output_dir: str) -> bool:
"""Validate output directory structure."""
pass
```
3. **Write tests:**
```bash
vim tests/infra_tests/validation/test_pdf_validator.py
```
4. **Document usage:**
- Add to infrastructure/validation/AGENTS.md
- Include usage examples
5. **Integrate with build pipeline:**
- Update scripts/runner/execute_pipeline.py if needed
- Update infrastructure modules if applicable
---
## Testing Strategy
Test organization, per-layer commands, and coverage gates are documented once,
canonically, in **[Testing Strategy](testing-strategy.md)** — see that file for
`tests/infra_tests/`, `projects/{name}/tests/`, `tests/integration/`, and the
full-suite/coverage-report commands.
---
## Best Practices
### For Infrastructure Development
✅ **Do:**
- Write generic, reusable code
- Document with project-independent examples
- Test extensively with real scenarios
- Handle errors gracefully
- Provide clear logging
❌ **Don't:**
- Import scientific modules
- Assume specific research domain
- Skip tests to ship features
- Hardcode project-specific values
- Mix concerns (building vs. computation)
### For Scientific Development
✅ **Do:**
- Use infrastructure tools for document management
- Follow thin orchestrator pattern in projects/{name}/scripts/
- Implement algorithms in projects/{name}/src/ modules
- Test with data
- Document domain-specific concepts
❌ **Don't:**
- Duplicate build/validation logic
- Implement document generation in scripts
- Skip layer abstraction
- Mix orchestration with computation
- Depend on infrastructure internals
### Logging Best Practices
```python
# In project scripts - mark layer transitions
import logging
logger = logging.getLogger(__name__)
logger.info("[LAYER-2-PROJECT] Starting simulation...")
logger.info("[LAYER-1-INFRASTRUCTURE] Using FigureManager for output...")
```
```bash
# In build scripts - mark phase transitions
log_info "━━━ LAYER 1: Infrastructure Validation ━━━"
log_info "━━━ LAYER 2: Scientific Computation ━━━"
```
---
## Migration from Flat Structure
If you have an old project with flat src/, migrating to the two-layer structure:
1. **Create packages:**
```bash
mkdir -p infrastructure projects/{name}/src
```
2. **Move modules:**
- Infrastructure modules → infrastructure/
- Project modules → projects/{name}/src/
3. **Update imports:**
- `from example import` → `from projects.{name}.src.example import`
- Build verification is handled by the validation module
4. **Update tests:**
- Infrastructure tests → tests/infra_tests/
- Project tests → projects/{name}/tests/
- Update conftest.py if needed
5. **Validate:**
```bash
uv run pytest tests/ projects/{name}/tests/ --cov=infrastructure --cov=projects/{name}/src
uv run python scripts/runner/execute_pipeline.py --project {name} --core-only
```
---
## Troubleshooting
Import errors, layer violations, and mixed-concerns fixes are documented once,
canonically, in **[Testing Strategy § Troubleshooting](testing-strategy.md#troubleshooting)**.
---
## References
### Architecture Documentation
- [../core/architecture.md](../core/architecture.md) - system architecture overview
- [decision-tree.md](../architecture/decision-tree.md) - Code placement flowchart
- [thin-orchestrator-summary.md](../architecture/thin-orchestrator-summary.md) - Thin orchestrator pattern details
### Layer-Specific Documentation
- [infrastructure/AGENTS.md](../../infrastructure/AGENTS.md) - Infrastructure layer documentation
- [infrastructure/README.md](../../infrastructure/README.md) - Infrastructure quick reference
- [template_code_project/src/AGENTS.md](../../projects/templates/template_code_project/src/AGENTS.md) - Project layer documentation
- [template_code_project/src/README.md](../../projects/templates/template_code_project/src/README.md) - Project quick reference
### System Documentation
- [../AGENTS.md](../AGENTS.md) - system documentation
- [../README.md](../README.md) - Project overview
- [../core/how-to-use.md](../core/how-to-use.md) - usage guide
---
## Key Takeaway
**Layers separate concerns:**
- **[LAYER 1: INFRASTRUCTURE]** handles *how* research is documented and built
- **[LAYER 2: PROJECT]** focuses on *what* research is conducted
This separation makes code more modular, reusable, and maintainable.
## Quick Navigation
- **Understanding the architecture**: Start with the [Quick Reference](#quick-reference-layer-1-vs-layer-2) table above
- **Adding code**: See [Decision Tree](#decision-tree-where-should-code-go) section
- **Import patterns**: See [Import Guidelines](#import-guidelines) section
- **Testing**: See [Testing Strategy](#testing-strategy) section