# Testing Standards and Patterns

## Overview

All code in this repository requires test coverage and should exercise real
behavior. Prohibited mock frameworks remain forbidden, and CI separately
enforces zero semantic dependency replacements. Environment/path isolation is
permitted and separately inventoried. Tests must be fast, deterministic, and
self-documenting.

## Coverage Requirements

### Mandatory Standards

- **Infrastructure modules**: 60% minimum coverage (current % — see [coverage-gaps.md](../development/coverage-gaps.md))
- **Project code**: 90% minimum coverage (current % — see [COUNTS.md](../_generated/COUNTS.md))
- **Integration tests**: All critical workflows covered
- **Edge cases**: All error paths tested

### Coverage Verification

```bash
# Run tests with coverage report
uv run pytest tests/ --cov=infrastructure --cov=projects/{name}/src --cov-report=html

# View coverage report
open htmlcov/index.html

# Verify coverage meets requirements
uv run pytest tests/ --cov=infrastructure --cov-fail-under=60
uv run pytest projects/{name}/tests/ --cov=projects/{name}/src --cov-fail-under=90
```

## Test Organization

### Directory Structure

```mermaid
flowchart TB
    T[tests]
    T --> META[conftest shared fixtures]
    T --> INFRA[infra_tests]
    T --> INTEG[integration]
    T --> REG[regression]
    INFRA --> CORE[core tests]
    INFRA --> VAL[validation tests]
```

### Module Test Organization

For each infrastructure module:

```mermaid
flowchart LR
    T[tests infra_tests module]
    T --> INIT[__init__.py]
    T --> CFG[conftest.py fixtures]
    T --> CORE[test_core.py]
    T --> CLI[test_cli.py optional]
    T --> ERR[test_errors.py]
    T --> INTEG[test_integration.py]
```

## Testing Principles

### 1. Mock-framework prohibition and semantic stand-ins

Do not introduce `MagicMock`, `mocker.patch`, `unittest.mock`, or another
mocking framework. Environment/path isolation through `monkeypatch.setenv`,
`delenv`, and `chdir` is permitted when real code still runs.
`monkeypatch.setattr`/`setitem` and deletion variants are different: the
repository inventories them as semantic dependency replacements that must be
migrated, not as evidence of no-mock compliance.

```bash
# Enforced lexical gate: prohibited framework imports/calls only
uv run python scripts/audit/verify_no_mocks.py

# Blocking semantic inventory
uv run python scripts/audit/verify_no_mocks.py --inventory --max-dependency-replacements 0
```

```python
# ✅ GOOD: Test with data
def test_validation_passes():
    data = {"name": "Alice", "age": 30}
    assert validate_data(data) is True

# ❌ ABSOLUTELY FORBIDDEN: NEVER use any mocking
# def test_validation_passes():
#     mock_data = MagicMock()  # NEVER ALLOWED
#     mocker.patch("module.function")  # NEVER ALLOWED
#     # This would break the testing philosophy
```

### Network-Dependent Modules

For modules requiring external services (LLM, Publishing APIs):

1. **Pure Logic Tests**: Test configuration, validation, data handling without network
2. **Integration Tests**: Mark with `@pytest.mark.requires_ollama` (or similar marker)
3. **Fail with setup guidance**: Default-selected tests should not silently skip unavailable services

```python
# ✅ GOOD: Pure logic test (no network needed)
def test_config_from_env(clean_llm_env):
    os.environ["OLLAMA_HOST"] = "http://test:11434"
    config = OllamaClientConfig.from_env()
    assert config.base_url == "http://test:11434"

# ✅ GOOD: Integration test with marker
@pytest.mark.requires_ollama
class TestLLMIntegration:
    @pytest.fixture(autouse=True)
    def check_ollama(self, ensure_ollama_for_tests):
        assert ensure_ollama_for_tests

    def test_query(self):
        client = LLMClient()
        response = client.query("Hello")
        assert response is not None

# Run commands:
# pytest -m "not requires_ollama"  # Deselect local Ollama tests
# pytest -m requires_ollama        # Only network tests
```

### 2. Test Behavior, Not Implementation

```python
# ✅ GOOD: Test the observable behavior
def test_sort_returns_sorted_list():
    result = sort_numbers([3, 1, 2])
    assert result == [1, 2, 3]

# ❌ BAD: Testing implementation details
# Testing "how" something works (implementation) is brittle and not valuable.
# Only test "what" the function does (observable behavior).
```

### 3. Clear, Self-Documenting Names

```python
# ✅ GOOD: Name clearly describes what's tested
def test_validation_fails_when_email_is_missing():
    data = {"name": "Alice"}  # No email
    with pytest.raises(ValidationError):
        validate_user_data(data)

# ❌ BAD: Unclear what's tested
def test_validation_error():
    data = {}
    with pytest.raises(ValidationError):
        validate_user_data(data)
```

### 4. Fast Execution

```python
# ✅ GOOD: Unit tests < 1 second
def test_format_string():
    result = format_date(datetime(2025, 1, 1))
    assert result == "2025-01-01"

# ❌ BAD: Slow integration tests in unit test suite
def test_format_string():
    # Writes to file, reads from API, etc.
    # Takes 10 seconds
```

### 5. Isolated Tests

```python
# ✅ GOOD: Each test is independent
def test_add():
    assert add(2, 2) == 4

def test_subtract():
    assert subtract(4, 2) == 2

# ❌ BAD: Tests depend on each other
def test_initialization():
    global calculator
    calculator = Calculator()

def test_add():
    global calculator
    assert calculator.add(2, 2) == 4
```

## Test Patterns

### Arrange-Act-Assert (AAA)

```python
def test_user_creation():
    # Arrange: Set up test data
    user_data = {"name": "Alice", "email": "alice@example.com"}

    # Act: Perform the action
    user = create_user(**user_data)

    # Assert: Verify the result
    assert user.name == "Alice"
    assert user.email == "alice@example.com"
```

### Testing Error Conditions

```python
def test_validation_error_has_context():
    """Test that validation errors include helpful context."""
    try:
        validate_email("invalid-email")
    except ValidationError as e:
        assert "email" in str(e).lower()
        assert "invalid" in str(e).lower()
    else:
        pytest.fail("ValidationError not raised")

# Or using pytest.raises
def test_validation_error_with_pytest_raises():
    """Cleaner approach using pytest.raises."""
    with pytest.raises(ValidationError) as exc_info:
        validate_email("invalid-email")

    assert "invalid" in str(exc_info.value).lower()
```

### Testing with Fixtures

```python
# conftest.py - Shared fixtures
import pytest

@pytest.fixture
def sample_data():
    """Provide sample test data."""
    return {
        "name": "Alice",
        "email": "alice@example.com",
        "age": 30
    }

@pytest.fixture
def temp_file(tmp_path):
    """Create a temporary file."""
    file = tmp_path / "test.txt"
    file.write_text("test content")
    return file

# test_module.py - Use fixtures
def test_user_creation(sample_data):
    """Test using fixture."""
    user = create_user(**sample_data)
    assert user.name == sample_data["name"]

def test_file_reading(temp_file):
    """Test using temporary file."""
    content = read_file(temp_file)
    assert content == "test content"
```

### Parameterized Tests

```python
import pytest

@pytest.mark.parametrize("input,expected", [
    ("123", 123),
    ("456", 456),
    ("0", 0),
    ("-123", -123),
])
def test_parse_integer(input, expected):
    """Test parsing various integer strings."""
    assert parse_integer(input) == expected

@pytest.mark.parametrize("invalid_input", [
    "abc",      # Not a number
    "12.34",    # Float, not int
    "",         # Empty
])
def test_parse_integer_invalid(invalid_input):
    """Test that invalid inputs raise errors."""
    with pytest.raises(ValueError):
        parse_integer(invalid_input)
```

### Testing Logging

```python
def test_operation_is_logged(caplog):
    """Test that operation is properly logged."""
    import logging
    caplog.set_level(logging.INFO)

    perform_operation()

    assert "Operation started" in caplog.text
    assert "Operation completed" in caplog.text
```

## Integration Testing

### End-to-End Workflows

```python
# tests/integration/test_validation_pipeline.py
def test_full_validation_pipeline():
    """Test validation workflow."""
    # 1. Load data from file
    data = load_test_data("sample.csv")

    # 2. Validate all records
    results = validate_all(data)

    # 3. Check results
    assert results.valid_count == 95
    assert results.error_count == 5

    # 4. Generate report
    report = generate_report(results)
    assert "95 valid" in report
```

### Testing Script Execution

```python
# tests/integration/test_script_execution.py
def test_analysis_script_generates_output(tmp_path):
    """Test that analysis script produces expected output."""
    import subprocess

    output_dir = tmp_path / "output"
    output_dir.mkdir()

    # Run script
    result = subprocess.run(
        ["python3", "projects/templates/template_code_project/scripts/optimization_analysis.py"],
        capture_output=True,
        text=True
    )

    # Check execution
    assert result.returncode == 0

    # Check output files
    assert (output_dir / "figure.png").exists()
```

## Common Fixtures

### conftest.py - Project-Wide Fixtures

```python
import pytest
from pathlib import Path
import tempfile

@pytest.fixture
def test_data_dir():
    """Path to test data directory."""
    return Path(__file__).parent / "data"

@pytest.fixture
def sample_data():
    """Load sample test data."""
    return {
        "count": 100,
        "items": list(range(100))
    }

@pytest.fixture
def temp_dir():
    """Create temporary directory."""
    with tempfile.TemporaryDirectory() as tmpdir:
        yield Path(tmpdir)

@pytest.fixture
def logger_fixture(caplog):
    """Set up logging capture for testing logging."""
    import logging
    caplog.set_level(logging.DEBUG)
    return caplog
```

## Running Tests

### Basic Commands

```bash
# Run all tests
uv run pytest tests/

# Run specific test file
uv run pytest tests/infra_tests/core/test_agent_memory.py

# Run specific test function
uv run pytest tests/infra_tests/core/test_agent_memory.py::test_empty_memory_payload_has_required_keys

# Run with verbose output
uv run pytest tests/ -v

# Run with coverage
uv run pytest tests/ --cov=infrastructure --cov=projects/{name}/src

# Run with coverage and HTML report
uv run pytest tests/ --cov=infrastructure --cov-report=html

# Stop at first failure
uv run pytest tests/ -x

# Show print statements
uv run pytest tests/ -s
```

### Filtering Tests

```bash
# Run only tests matching a pattern
uv run pytest tests/ -k "validation"

# Run all error tests
uv run pytest tests/ -k "error"

# Run excluding certain tests
uv run pytest tests/ --ignore=tests/integration/
```

## CI/CD Integration

### GitHub Actions Example

```yaml
name: Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: uv sync

      - name: Run tests
        run: |
          uv run pytest tests/infra_tests/ --cov=infrastructure --cov-fail-under=60
          uv run pytest projects/{name}/tests/ --cov=projects/{name}/src --cov-fail-under=90

      - name: Upload coverage
        uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
```

## Debugging Tests

### Print Debugging

```bash
# Show print statements in passing tests
uv run pytest tests/ -s

# Show print statements only for failures
uv run pytest tests/ --tb=short
```

### Interactive Debugging

```bash
# Use pdb (Python debugger)
uv run pytest tests/ --pdb

# Drop to debugger on failure
uv run pytest tests/ --pdb --lf
```

### Viewing Test Output

```bash
# Show full traceback
uv run pytest tests/ --tb=long

# Short traceback
uv run pytest tests/ --tb=short

# No traceback
uv run pytest tests/ --tb=no
```

## Quality Checklist

Before committing tests:

- [ ] Coverage requirements met (60% infra, 90% project) verified
- [ ] All tests pass (`uv run pytest tests/`)
- [ ] No skipped tests (`-k "not skip"`)
- [ ] Tests run in < 30 seconds total
- [ ] Test names are clear and descriptive
- [ ] No prohibited mock-framework imports/calls used
- [ ] New tests avoid dependency-replacement stand-ins; inventory impact reviewed
- [ ] data used in all tests
- [ ] Edge cases tested
- [ ] Error conditions tested
- [ ] Documentation added to AGENTS.md and README.md

## See Also

- [error_handling.md](error_handling.md) - Exception patterns for tests
- [documentation_standards.md](documentation_standards.md) - Documenting tests
- [docs/guides/testing-and-reproducibility.md](../guides/testing-and-reproducibility.md) - Test-driven development guide
- [docs/development/testing/testing-guide.md](../development/testing/testing-guide.md) - Testing best practices
- [tests/AGENTS.md](../../tests/AGENTS.md) - Test framework setup
- [pytest Documentation](https://docs.pytest.org/)