# End-to-End Multi-Agent Lifecycle & Crash Recovery Demo This document provides a detailed walkthrough of the complete end-to-end multi-agent scenario implemented in [`tests/e2e/continuity.test.ts`](file:///Users/kuba/Documents/Github/dsh-continuum/tests/e2e/continuity.test.ts). It demonstrates how **`dsh-continuum`** maintains unified project memory, enforces separation of duties, spills large outputs, and guarantees flawless crash recovery across distinct agent roles. --- ## Architecture Flow Overview ```mermaid sequenceDiagram autonumber actor Orch as Orchestrator Agent actor Res as Researcher Agent actor Imp as Implementer Agent actor Rev as Reviewer Agent participant Cont as Continuum Storage (events.jsonl) Orch->>Cont: continuum_project(action: 'init', name: 'Distributed Consensus') Orch->>Cont: continuum_project(action: 'update_northstar', purpose: 'Zero-downtime consensus') Orch->>Cont: continuum_task(action: 'create', title: 'Implement Raft Log Engine') Res->>Cont: continuum_memory(action: 'record_finding', statement: 'Append-only fsync log is required') Imp->>Cont: continuum_task(action: 'claim', taskId: 'TASK-001') Note over Imp,Cont: Lease issued (10-min TTL) Imp->>Cont: continuum_memory(action: 'attach_evidence', content: 15KB log data) Note over Cont: Auto-spilled to artifacts/ART-EVD-*.bin (hash & snippet in event) Imp->>Cont: continuum_task(action: 'submit', taskId: 'TASK-001') Rev->>Cont: continuum_task(action: 'verify', verdict: 'rejected', notes: 'Missing fsync verification') Note over Cont: Task transitions to 'rejected' & 'ready' (attempt counter incremented to 2) Imp->>Cont: continuum_task(action: 'claim', taskId: 'TASK-001') Imp->>Cont: continuum_task(action: 'submit', taskId: 'TASK-001', deliverables: ['src/raft.ts with fsync']) Rev->>Cont: continuum_task(action: 'verify', verdict: 'passed', notes: 'Verified fsync safety') Note over Cont: Task completed! Orch->>Cont: continuum_checkpoint(action: 'create', label: 'Milestone 1 Complete') Note over Cont: [SIMULATED PROCESS CRASH / SIGKILL] actor FreshAgent as Incoming Fresh Agent FreshAgent->>Cont: Load state from events.jsonl Note over FreshAgent,Cont: Replays event stream, recovers full state, loads spilled 15KB artifact FreshAgent->>Cont: continuum_context(action: 'render') Note over FreshAgent: Injects rendered Context Pack with North Star, completed tasks, & evidence ``` --- ## Step-by-Step Scenario Execution ### Step 1: Project Initialization & North Star Alignment **Agent:** `agent-orchestrator` (Role: `orchestrator`) The orchestrator initializes the continuous project context and records the guiding North Star: ```typescript await engine1.initProject('proj-demo', 'Distributed Consensus Engine', 'High-reliability consensus implementation', { id: 'agent-orchestrator', role: 'orchestrator' }) await engine1.setNorthStar( 'Implement zero-downtime leader election and log compaction', ['Linearizable reads', 'Crash fault tolerance'], { id: 'agent-orchestrator', role: 'orchestrator' }, ['Byzantine fault tolerance'], ['Local reliable NVMe storage'], ['All unit tests pass with fast-check fuzzing'] ) ``` **Result:** `events.jsonl` records `project.created` (seq 1) and `northstar.updated` (seq 2). --- ### Step 2: Task Creation with Independent Verification Policy **Agent:** `agent-orchestrator` (Role: `orchestrator`) The orchestrator breaks down work into an actionable task. Notice the `verificationPolicy: 'independent'` — this activates strict separation of duties: ```typescript const taskRes = await engine1.createTask({ id: 'TASK-001', title: 'Implement Raft Log Engine', description: 'Write serialized log appender with fsync write barriers', priority: 'high', acceptanceCriteria: ['Log compaction', 'Fsync flush guarantee'], verificationPolicy: 'independent' // Four-eyes principle enforced! }, { actor: { id: 'agent-orchestrator', role: 'orchestrator' } }) ``` --- ### Step 3: Domain Research & Fact Accumulation **Agent:** `agent-researcher` (Role: `researcher`) Before writing code, the researcher analyzes requirements and records architectural findings: ```typescript await engine1.recordFinding({ id: 'FIND-001', statement: 'Direct fsync on every append drops throughput by 80%. Batch commits with interval flushes are necessary.', confidence: 'high', scope: 'architecture', sources: ['https://raft.github.io/raft.pdf'], relatedTaskId: 'TASK-001' }, { actor: { id: 'agent-researcher', role: 'researcher' } }) ``` --- ### Step 4: Claiming Task & Execution Lease Issuance **Agent:** `agent-implementer` (Role: `implementer`) The implementer claims the task. The engine issues a renewable execution lease with an active expiration timestamp: ```typescript const claimRes = await engine1.claimTask('TASK-001', { actor: { id: 'agent-implementer', role: 'implementer' } }) // Returns: // { // taskId: 'TASK-001', // leaseId: 'LEASE-9a1b2c3d', // expiresAt: 1725289200000, // claimedBy: 'agent-implementer' // } ``` --- ### Step 5: Large Evidence Output Spilling **Agent:** `agent-implementer` (Role: `implementer`) The implementer runs benchmark tests producing 15,000 bytes of output. Because this exceeds the 10KB threshold: 1. The full 15,000 bytes are saved to disk at `.continuum/artifacts/ART-EVD-EV-001-v1.bin`. 2. The SHA-256 digest is computed and stored. 3. The event log payload only contains a concise snippet and pointer to the artifact. ```typescript const largePayload = 'A'.repeat(15000) const evRes = await engine1.attachEvidence({ id: 'EV-001', type: 'test_run', summary: '15KB test log output from benchmark suite', content: largePayload, taskId: 'TASK-001' }, { actor: { id: 'agent-implementer', role: 'implementer' } }) assert.strictEqual(evRes.data.spilledArtifactId, 'ART-EVD-EV-001') ``` --- ### Step 6: Task Submission & Verification Rejection **Agent:** `agent-implementer` submits $\to$ `agent-reviewer` reviews. The implementer submits the task. Reviewer red-teams the submission and rejects it due to missing unit tests: ```typescript // 1. Implementer submits await engine1.submitTask('TASK-001', { deliverables: ['src/raft/log.ts'], notes: 'Initial version complete' }, { actor: { id: 'agent-implementer', role: 'implementer' } }) // 2. Reviewer inspects and rejects await engine1.verifyTask('TASK-001', { verdict: 'rejected', notes: 'Benchmarking indicates missing batch flush under load. Requires test case.', evidenceIds: ['EV-001'] }, { actor: { id: 'agent-reviewer', role: 'reviewer' } }) ``` **State Transition:** The task transitions to `rejected` and automatically resets to `ready` with `attempt: 2`. The failed approach and reviewer feedback are permanently logged. --- ### Step 7: Remediation & Approval **Agent:** `agent-implementer` re-claims and re-submits $\to$ `agent-reviewer` approves. ```typescript // Re-claim under attempt 2 await engine1.claimTask('TASK-001', { actor: { id: 'agent-implementer', role: 'implementer' } }) // Re-submit with fix await engine1.submitTask('TASK-001', { deliverables: ['src/raft/log.ts', 'tests/raft/batch.test.ts'], notes: 'Added batch flush buffer with timer trigger' }, { actor: { id: 'agent-implementer', role: 'implementer' } }) // Reviewer verifies and approves await engine1.verifyTask('TASK-001', { verdict: 'passed', notes: 'Verified throughput and recovery guarantees under crash test.', evidenceIds: ['EV-001'] }, { actor: { id: 'agent-reviewer', role: 'reviewer' } }) ``` --- ### Step 8: Milestone Checkpointing **Agent:** `agent-orchestrator` (Role: `orchestrator`) The orchestrator creates an immutable checkpoint snapshot: ```typescript await engine1.createCheckpoint({ label: 'Phase 1: Raft Core Verified', actor: { id: 'agent-orchestrator', role: 'orchestrator' } }) ``` --- ### Step 9: Abrupt Process Crash & State Resumption **Simulation:** The agent runtime process is killed abruptly. A fresh engine instance starts up pointing to the project directory: ```typescript // Cold start with a brand new engine instance const engine2 = new ContinuumEngine(storage2) const restoredState = await engine2.loadState() // All entities restored faithfully: assert.strictEqual(restoredState.project.name, 'Distributed Consensus Engine') assert.strictEqual(restoredState.tasks.get('TASK-001')?.status, 'completed') assert.strictEqual(restoredState.tasks.get('TASK-001')?.attempt, 2) assert.strictEqual(restoredState.tasks.get('TASK-001')?.verification?.verdict, 'passed') // Spilled artifact loaded from disk: const artifactBytes = await storage2.readArtifact(evRes.data.spilledArtifactId!, 1) assert.strictEqual(artifactBytes.length, 15000) ``` --- ### Step 10: Automatic Context Pack Generation for Incoming Agent The incoming agent automatically receives a token-budgeted, cryptographically hashed Context Pack injected into its system prompt: ```text # PROJECT CONTEXT: Distributed Consensus Engine Tagline: Agents are temporary. The project is continuous. ## North Star - Purpose: Implement zero-downtime leader election and log compaction - Acceptance Criteria: All unit tests pass with fast-check fuzzing ## Active Task: None (Task TASK-001 completed) - Recently Completed: TASK-001: Implement Raft Log Engine (Attempt 2) Verified By: agent-reviewer (Verdict: passed) ## Key Findings & Decisions - [FIND-001] Direct fsync on every append drops throughput by 80%... (Confidence: high) ## Evidence & Artifacts - [EV-001] 15KB test log output from benchmark suite (Spilled artifact: ART-EVD-EV-001) ``` The incoming agent requires zero manual onboarding prompt text. It immediately continues the project with complete awareness of past decisions, rejected attempts, and verified deliverables.