# Development Standards and Guidelines — `docs/rules/`

## Overview

This directory contains development standards, coding guidelines, and best practices for the Research Project Template system. All code must follow these standards.

> **Note**: Cursor reads the repository root **[`.cursorrules`](../../.cursorrules)** file for IDE/agent routing. Extended editorial standards live in **`docs/rules/`** (this directory).

> **Project paths in examples**: default to [`projects/templates/template_code_project/`](../../projects/templates/template_code_project/). Authoritative active `projects/` names → [_generated/active_projects.md](../_generated/active_projects.md) (see [_generated/README.md](../_generated/README.md)).

## Files

| File | Purpose | Best For |
|------|---------|----------|
| [AGENTS.md](AGENTS.md) | This file - overview and navigation | Understanding the system |
| [README.md](README.md) | Quick reference guide | Finding quick answers |
| [folder_structure.md](folder_structure.md) | Folder structure documentation standard | Understanding folder organization |
| [error_handling.md](error_handling.md) | Error handling and exception patterns | Writing error handling code |
| [security.md](security.md) | Security standards and guidelines | Writing secure code |
| [python_logging.md](python_logging.md) | Logging standards and best practices | Adding logging to code |
| [infrastructure_modules.md](infrastructure_modules.md) | Infrastructure module development standards | Creating new infrastructure modules |
| [testing_standards.md](testing_standards.md) | Testing patterns and coverage standards | Writing tests with pytest |
| [documentation_standards.md](documentation_standards.md) | AGENTS.md and README.md writing guide | Writing documentation |
| [type_hints_standards.md](type_hints_standards.md) | Type annotation patterns | Adding type hints |
| [llm_standards.md](llm_standards.md) | LLM/Ollama integration patterns | Using local LLMs for research |
| [code_style.md](code_style.md) | Code formatting and style standards | Writing consistent, readable code |
| [git_workflow.md](git_workflow.md) | Git workflow and commit standards | Managing version control |
| [api_design.md](api_design.md) | API design and interface standards | Creating clean, consistent APIs |
| [manuscript_style.md](manuscript_style.md) | Manuscript formatting and style standards | Writing research manuscripts |
| [reporting.md](reporting.md) | Reporting module standards and outputs | Using reporting utilities |
| [refactoring.md](refactoring.md) | Refactoring and modularization standards | Refactoring code with clean breaks |
| [memory_and_decision_records.md](memory_and_decision_records.md) | Decision memory, ADRs, local agent memory, and RedTeam negative controls | Capturing rationale without stale parallel ledgers |

## Key Principles

### 1. Two-Layer Architecture

**Layer 1: Infrastructure** (Generic, reusable)

- Location: `infrastructure/` (root level)
- Domain-independent tools
- Reusable across projects
- 60% minimum test coverage required (current % — see [coverage-gaps.md](../development/coverage-gaps.md))

**Layer 2: Project** (Specific, customizable)

- Location: `projects/{name}/src/` (project-specific code)
- Research-specific code
- Uses generic infrastructure utilities
- 90% minimum test coverage required (current % — see [COUNTS.md](../_generated/COUNTS.md))
- Tests: `projects/{name}/tests/`
- Scripts: `projects/{name}/scripts/` (thin orchestrators)

### 2. Thin Orchestrator Pattern

- **Business logic** → `projects/{name}/src/` or `infrastructure/`
- **Orchestration** → `scripts/` (root, generic) or `projects/{name}/scripts/` (project-specific)
- **No logic in scripts** - only coordination

### 3. Quality Standards

- **Testing**: 90%+ coverage required (60% infra, 90% project), data only
- **Documentation**: AGENTS.md + README.md for each directory
- **Type Safety**: Type hints on all public APIs
- **Error Handling**: Use custom exception hierarchy
- **Logging**: Unified logging system throughout

### 4. Pipeline Entry Points

The template provides **two pipeline orchestrators** with different scope:

**Interactive Menu (`./run.sh`)**

- **Use for**: Full pipeline with optional LLM stages
- **Stages**: 1-9 (displayed as [1/9] to [9/9], with an initial clean step shown as [0/9])
- **Features**: Interactive menu, research templates, LLM reviews, translations
- **When to use**: Full builds, LLM features needed

**Python Orchestrator (`uv run python scripts/runner/execute_pipeline.py`)**

- **Use for**: Core or full pipeline, programmatic execution (`--core-only` skips LLM stages)
- **Stages**: Default **core** run is **eight** executor stages (clean outputs, setup, infrastructure tests, project tests, analysis, PDF rendering, validation, copy outputs). With `--skip-infra`, infrastructure tests are omitted (seven stages). Full pipeline adds LLM review and translations before copy; see [`executor.py`](../../infrastructure/core/pipeline/executor.py) (`PipelineExecutor._build_stage_list`).
- **Features**: Checkpoint/resume; delegates to canonical `scripts/pipeline/stage_NN_*.py` entrypoints, including optional LLM, bundle, archival, science, and provenance stages where applicable
- **When to use**: Automated environments, CI/CD, or when you want explicit flags instead of the menu

## Quick Start

### For AI Agents (New to the System)

1. **Start here**: Read [AGENTS.md](AGENTS.md) (this file) for overview
2. **Pick your task** - Read the appropriate guide:
    - **Writing code**: [error_handling.md](error_handling.md), [python_logging.md](python_logging.md), [type_hints_standards.md](type_hints_standards.md)
    - **Creating modules**: [infrastructure_modules.md](infrastructure_modules.md), [testing_standards.md](testing_standards.md)
    - **Writing docs**: [documentation_standards.md](documentation_standards.md)
    - **Writing manuscripts**: [manuscript_style.md](manuscript_style.md)
    - **Writing tests**: [testing_standards.md](testing_standards.md)
    - **Capturing rationale**: [memory_and_decision_records.md](memory_and_decision_records.md)
3. **Follow the standards** - Apply the patterns and examples from the guide
4. **Cross-reference** - See "See Also" sections for related guides

### For Developers (Quick Reference)

1. **Quick lookup**: Use [README.md](README.md) for fast pattern references
2. **Deep dive**: Read specific guide files as needed
3. **Copy patterns**: Use code examples from appropriate guide
4. **Reference**: Keep this directory open during development

## Development Workflow

### Before Starting

1. ✅ Read relevant rules files in `docs/rules/`
2. ✅ Understand two-layer architecture
3. ✅ Review existing similar code
4. ✅ Plan with thin orchestrator pattern

### During Development

1. ✅ Write tests first (TDD)
2. ✅ Follow type hints standards
3. ✅ Use unified logging
4. ✅ Handle errors properly
5. ✅ Document as you go

### Before Committing

1. ✅ Test coverage requirements met (60% infra, 90% project)
2. ✅ All tests pass
3. ✅ No linter errors
4. ✅ Documentation updated
5. ✅ AGENTS.md and README.md updated

## Standards by Topic

### Error Handling

See [error_handling.md](error_handling.md)

**Key Points**:

- Use custom exception hierarchy from `infrastructure/core/exceptions.py`
- Always chain exceptions with `from`
- Provide context in exceptions
- Log before raising critical errors

**Example**:

```python
from infrastructure.core.exceptions import ValidationError

try:
    result = process_data(input)
except ValueError as e:
    raise ValidationError("Invalid input", context={"input": input}) from e
```

### Logging

See [python_logging.md](python_logging.md)

**Key Points**:

- Use `infrastructure.core.logging.utils.get_logger(__name__)`
- Log at appropriate levels (DEBUG, INFO, WARNING, ERROR, CRITICAL)
- Include context in log messages
- Use structured logging where possible

**Example**:

```python
from infrastructure.core.logging.utils import get_logger

logger = get_logger(__name__)
logger.info(f"Processing {count} items")
```

### Infrastructure Development

See [infrastructure_modules.md](infrastructure_modules.md)

**Key Points**:

- Generic and domain-independent only
- 60% minimum test coverage required (current % — see [coverage-gaps.md](../development/coverage-gaps.md))
- AGENTS.md + README.md
- Public API in `__init__.py`
- Type hints on all functions
- Follow thin orchestrator pattern
- Use custom exceptions from `core.exceptions`
- Use unified logging from `core.logging.utils`

**Structure**:

```mermaid
flowchart LR
    M[infrastructure module]
    M --> INIT[__init__.py public API]
    M --> CORE[core.py core logic]
    M --> CLI[cli.py optional]
    M --> CFG[config.py optional]
    M --> DOCS[AGENTS README SKILL]
```

**Related Documentation:**

- [infrastructure/AGENTS.md](../../infrastructure/AGENTS.md) - Module organization
- [infrastructure_modules.md](infrastructure_modules.md) - Development standards
- [error_handling.md](error_handling.md) - Exception patterns
- [python_logging.md](python_logging.md) - Logging standards

### Manuscript Writing

See [manuscript_style.md](manuscript_style.md)

**Key Points**:

- Use isolated Pandoc `$$` display blocks or labelled equation environments,
  depending on the project renderer and reference scheme.
- Always label figures, tables, and equations
- Use descriptive labels (e.g., `fig:convergence_plot`, `eq:objective`)
- Reference using `\ref{}` for sections/figures/tables, `\eqref{}` for equations
- Place citations before punctuation
- Use relative paths for figures: `../output/figures/`

**Example**:

```markdown
\begin{equation}
\label{eq:objective}
f(x) = \sum_{i=1}^{n} w_i \phi_i(x)
\end{equation}

As shown in \eqref{eq:objective}, the objective function...
```

**Related Documentation:**

- [manuscript_style.md](manuscript_style.md) - manuscript formatting guide
- [projects/templates/template_code_project/manuscript/](../../projects/templates/template_code_project/manuscript/) - Example manuscript (active project)
- [docs/usage/markdown-template-guide.md](../usage/markdown-template-guide.md) - Markdown guide

## Testing Standards

### Coverage Requirements

- **Infrastructure**: 60% minimum (current % — see [coverage-gaps.md](../development/coverage-gaps.md))
- **Project code**: 90% minimum (current % — see [COUNTS.md](../_generated/COUNTS.md))
- **Integration tests**: All critical workflows covered

### Test Organization

```mermaid
flowchart TB
    T[tests]
    T --> INFRA[infra_tests Layer 1]
    T --> INTEG[integration cross-layer]
    T --> HELP[helpers utilities]
    INFRA --> SUB[test_module per package]
```

### Test Principles

1. **No mocks** for core logic - use data
2. **Test behavior**, not implementation
3. **Clear names** - test should be self-documenting
4. **Fast execution** - unit tests < 1s each
5. **Isolated** - tests should not depend on each other

## Documentation Standards

### Directory Documentation

Every directory must have:

1. **AGENTS.md** - Technical documentation
    - Purpose and architecture
    - Usage examples
    - Configuration options
    - Testing instructions
    - Best practices

2. **README.md** - Quick reference
    - Quick start
    - Key features
    - Common commands
    - Links to detailed docs

### Code Documentation

**Module docstrings**:

```python
"""Module description.

Detailed explanation of module purpose and contents.
"""
```

**Function docstrings**:

```python
def function(arg: str) -> str:
    """One-line summary.

    Detailed description.

    Args:
        arg: Description

    Returns:
        Description

    Raises:
        ExceptionType: When this happens

    Example:
        >>> function("test")
        'result'
    """
```

## Common Patterns

### Import Standards

```python
# Standard library
import os
from pathlib import Path

# Third party
import requests
import pytest

# Local infrastructure (Layer 1)
from infrastructure.core.logging.utils import get_logger
from infrastructure.core.exceptions import TemplateError
from infrastructure.documentation import FigureManager

# Local project (Layer 2)
from project.src import module
from project.src.simulation import SimpleSimulation
```

### Configuration Pattern

```python
from dataclasses import dataclass
import os

@dataclass
class ModuleConfig:
    """Module configuration."""

    setting: str = "default"

    @classmethod
    def from_env(cls) -> "ModuleConfig":
        """Load from environment variables."""
        return cls(
            setting=os.getenv("MODULE_SETTING", "default")
        )
```

### Error Handling Pattern

```python
from infrastructure.core.logging.utils import get_logger
from infrastructure.core.exceptions import TemplateError

logger = get_logger(__name__)

try:
    result = risky_operation()
except SpecificError as e:
    logger.error(f"Operation failed: {e}")
    raise TemplateError("High-level description") from e
```

## File Organization

### Project Structure

```mermaid
flowchart TB
    ROOT[template repo]
    ROOT --> INFRA[infrastructure Layer 1]
    ROOT --> PROJ[projects Layer 2]
    ROOT --> SCR[scripts Layer 1]
    ROOT --> T[tests Layer 1]
    ROOT --> DOC[docs]
    ROOT --> RULES[docs rules]
    INFRA --> INFRA_SUB[infra packages]
    PROJ --> P_NAME[project name]
    P_NAME --> P_SUB[src tests scripts manuscript]
```

## Checklist for New Code

### Before Writing

- [ ] Understand two-layer architecture
- [ ] Identify correct layer (infrastructure vs project)
- [ ] Review relevant rules files in `docs/rules/`
- [ ] Plan with thin orchestrator pattern

### While Writing

- [ ] Write tests first (TDD)
- [ ] Add type hints to all functions
- [ ] Use unified logging system
- [ ] Handle errors with custom exceptions
- [ ] Document as you go

### Before Committing

- [ ] Test coverage requirements met (60% infra, 90% project)
- [ ] All tests pass
- [ ] No linter errors
- [ ] AGENTS.md - [ ] README.md updated
- [ ] Code reviewed against standards
- [ ] Integration tests pass

## References

### Internal Documentation

- [Root AGENTS.md](../AGENTS.md) - System overview
- [Infrastructure AGENTS.md](../../infrastructure/AGENTS.md) - Infrastructure layer
- [Projects AGENTS.md](../../projects/AGENTS.md) - Projects layer
- [Documentation Hub](../AGENTS.md) - docs hub guide

### External Resources

- [Python Type Hints](https://docs.python.org/3/library/typing.html)
- [pytest Documentation](https://docs.pytest.org/)
- [PEP 8 Style Guide](https://peps.python.org/pep-0008/)

## All Development Guidelines

This directory provides modular development standards. Each file covers specific aspects:

| File | Purpose | Best For |
|------|---------|----------|
| [AGENTS.md](AGENTS.md) | Overview & navigation | Understanding the system |
| [README.md](README.md) | Quick reference | Finding patterns fast |
| [folder_structure.md](folder_structure.md) | Folder organization standard | Setting up documentation |
| [error_handling.md](error_handling.md) | Exception patterns | Writing error handling |
| [security.md](security.md) | Security standards | Writing secure code |
| [python_logging.md](python_logging.md) | Logging standards | Adding logging to code |
| [infrastructure_modules.md](infrastructure_modules.md) | Infrastructure development | Creating modules |
| [testing_standards.md](testing_standards.md) | Testing patterns | Writing tests |
| [documentation_standards.md](documentation_standards.md) | Documentation guidelines | Writing docs |
| [type_hints_standards.md](type_hints_standards.md) | Type annotation patterns | Adding type hints |
| [code_style.md](code_style.md) | Code formatting standards | Writing consistent code |
| [git_workflow.md](git_workflow.md) | Git workflow standards | Managing version control |
| [api_design.md](api_design.md) | API design standards | Creating clean APIs |
| [llm_standards.md](llm_standards.md) | LLM/Ollama patterns | Using local LLMs |
| [manuscript_style.md](manuscript_style.md) | Manuscript formatting standards | Writing research manuscripts |
| [reporting.md](reporting.md) | Reporting module standards | Reporting outputs and dashboards |

## Integration with Main Documentation

The `docs/rules/` standards align with and support the main documentation:

| Development Aspect | Rules File | Main Documentation |
|---|---|---|
| System Design | [AGENTS.md](AGENTS.md) | [Root AGENTS.md](../AGENTS.md) |
| Documentation Structure | [folder_structure.md](folder_structure.md) | [Root AGENTS.md](../AGENTS.md) |
| Architecture | [AGENTS.md](AGENTS.md) | [docs/core/architecture.md](../core/architecture.md) |
| Infrastructure | [infrastructure_modules.md](infrastructure_modules.md) | [infrastructure/AGENTS.md](../../infrastructure/AGENTS.md) |
| Error Handling | [error_handling.md](error_handling.md) | [docs/operational/error-handling-guide.md](../operational/error-handling-guide.md) |
| Security | [security.md](security.md) | [docs/development/security.md](../development/security.md) |
| Logging | [python_logging.md](python_logging.md) | [docs/operational/logging/README.md](../operational/logging/README.md) |
| Testing | [testing_standards.md](testing_standards.md) | [tests/AGENTS.md](../../tests/AGENTS.md) |
| Code Style | [code_style.md](code_style.md) | [docs/best-practices/best-practices.md](../best-practices/best-practices.md) |
| Git Workflow | [git_workflow.md](git_workflow.md) | [docs/best-practices/version-control.md](../best-practices/version-control.md) |
| API Design | [api_design.md](api_design.md) | [docs/core/architecture.md](../core/architecture.md) |
| Reporting | [reporting.md](reporting.md) | [docs/modules/modules-guide.md](../modules/modules-guide.md) |
| Documentation | [documentation_standards.md](documentation_standards.md) | [docs/core/workflow.md](../core/workflow.md) |
| Type Safety | [type_hints_standards.md](type_hints_standards.md) | [docs/core/architecture.md](../core/architecture.md) |
| LLM Integration | [llm_standards.md](llm_standards.md) | [infrastructure/llm/AGENTS.md](../../infrastructure/llm/AGENTS.md) |
| Manuscript Writing | [manuscript_style.md](manuscript_style.md) | [projects/templates/template_code_project/manuscript/](../../projects/templates/template_code_project/manuscript/) |
| Refactoring | [refactoring.md](refactoring.md) | [docs/best-practices/best-practices.md](../best-practices/best-practices.md) |

## Cross-Reference Guide

**For Architecture Decisions:**
→ Check [AGENTS.md](AGENTS.md) for decision criteria
→ Read [docs/core/architecture.md](../core/architecture.md) for full architectural overview
→ Consult [docs/architecture/thin-orchestrator-summary.md](../architecture/thin-orchestrator-summary.md) for pattern details

**For Infrastructure Development:**
→ Check [infrastructure_modules.md](infrastructure_modules.md) for standards
→ Study [infrastructure/AGENTS.md](../../infrastructure/AGENTS.md) for module organization
→ Review [infrastructure/README.md](../../infrastructure/README.md) for quick patterns

**For Code Quality:**
→ Check [error_handling.md](error_handling.md) and [python_logging.md](python_logging.md)
→ Read [docs/best-practices/best-practices.md](../best-practices/best-practices.md) for practices
→ See [docs/operational/error-handling-guide.md](../operational/error-handling-guide.md) for detailed patterns

## Quick Navigation Guide

### By Development Activity

| Activity | Start Here | Then Read |
|----------|-----------|-----------|
| Setting up folders | [folder_structure.md](folder_structure.md) | [documentation_standards.md](documentation_standards.md) for writing docs |
| Writing functions | [type_hints_standards.md](type_hints_standards.md) | [error_handling.md](error_handling.md), [python_logging.md](python_logging.md) |
| Code formatting | [code_style.md](code_style.md) | [type_hints_standards.md](type_hints_standards.md) for consistency |
| Git workflow | [git_workflow.md](git_workflow.md) | [testing_standards.md](testing_standards.md) for commit testing |
| API design | [api_design.md](api_design.md) | [type_hints_standards.md](type_hints_standards.md) for type safety |
| Handling errors | [error_handling.md](error_handling.md) | [python_logging.md](python_logging.md) for context |
| Writing secure code | [security.md](security.md) | [error_handling.md](error_handling.md) for secure error handling |
| Adding logging | [python_logging.md](python_logging.md) | [error_handling.md](error_handling.md) for error logging |
| Writing tests | [testing_standards.md](testing_standards.md) | [error_handling.md](error_handling.md) for error testing |
| Creating modules | [infrastructure_modules.md](infrastructure_modules.md) | All of the above standards |
| Writing docs | [documentation_standards.md](documentation_standards.md) | Specific guide for your doc type |
| Writing manuscripts | [manuscript_style.md](manuscript_style.md) | [projects/templates/template_code_project/manuscript/](../../projects/templates/template_code_project/manuscript/) for manuscript structure |
| Adding type hints | [type_hints_standards.md](type_hints_standards.md) | [documentation_standards.md](documentation_standards.md) for docstrings |
| Using LLM/Ollama | [llm_standards.md](llm_standards.md) | [infrastructure_modules.md](infrastructure_modules.md) for module patterns |
| Generating reports | [reporting.md](reporting.md) | [docs/modules/modules-guide.md](../modules/modules-guide.md) for module details |

### By File Size & Detail Level

**Quick Reference (< 100 lines)**

- [README.md](README.md) - Fast patterns lookup

**Medium Details (100-300 lines)**

- [error_handling.md](error_handling.md) - Exception patterns
- [python_logging.md](python_logging.md) - Logging standards
- [type_hints_standards.md](type_hints_standards.md) - Type annotation patterns
- [code_style.md](code_style.md) - Code formatting standards
- [git_workflow.md](git_workflow.md) - Git workflow standards

**Guides (300-600 lines)**

- [api_design.md](api_design.md) - API design standards
- [testing_standards.md](testing_standards.md) - testing guide
- [infrastructure_modules.md](infrastructure_modules.md) - Module development guide
- [documentation_standards.md](documentation_standards.md) - Documentation writing guide
- [llm_standards.md](llm_standards.md) - LLM/Ollama integration guide
- [manuscript_style.md](manuscript_style.md) - Manuscript formatting and style guide

**System Overview (400+ lines)**

- [AGENTS.md](AGENTS.md) - This file - system overview

## Maintenance

This directory is maintained as part of the template repository. All updates should:

1. Maintain consistency across files
2. Update this AGENTS.md when adding new rule files
3. Keep README.md in sync with new guides
4. Follow the same standards described here
5. Ensure cross-references to main documentation are accurate
6. Test all code examples in guides
7. Keep "See Also" sections current

### Adding New Guidelines

When creating a new rules file:

1. Add entry to Files table in this file
2. Add entry to Files section in README.md
3. Create "See Also" section linking to related guides
4. Include code examples for all patterns
5. Test all examples before committing
6. Update integration table with main docs

## See Also

**Development Standards:**

- [`README.md`](README.md) - Quick reference guide
- [`../AGENTS.md`](../AGENTS.md) - system documentation
- [`infrastructure/AGENTS.md`](../../infrastructure/AGENTS.md) - Infrastructure layer documentation
- [`projects/AGENTS.md`](../../projects/AGENTS.md) - Projects layer documentation

**Related Documentation:**

- [`docs/core/architecture.md`](../core/architecture.md) - System architecture overview
- [`docs/core/workflow.md`](../core/workflow.md) - Development workflow
- [`tests/AGENTS.md`](../../tests/AGENTS.md) - Testing philosophy and guide

---

**Version**: 3.3.1
**Last Updated**: 2026-06-08
**Files**: 18 (AGENTS.md + README.md + 16 guideline files)
**Status**: All 16 guideline files cross-referenced
**Updates**: All development rules and standards synchronized with docs/