# RFC-0006 — Temporal Design Patterns for OpenCompany > **Current-state reading rule:** defect lists, measurements and code sketches below are anchored to their cited baseline commits. Some fixes have since landed. Use the [documentation audit](docs-internal/DOCUMENTATION_AUDIT.md), current runtime references and recorded generation flags for shipped behavior; proposed paths and later rollout phases remain designs until implemented. This audit does not change the accepted owner decisions. - **Status:** Draft, 2026-10-24. Owner decisions T1-T20 are in §3.1. The amendments to RFC-0005 listed in §20 and the corrections to RFC-0004 listed in §21 were made in the same commit. On 2026-10-25 the Phase 1b plan was simplified (§22.2), and RFC-0007 amended §9.6 and §22.5. Phase 1b's docs pass (1b.45, 2026-11-02 and 2026-11-03) added notes wherever Phase 1b built something differently. Fixes after Phase 1b (2026-11-04 and 2026-11-05) are listed in §22.2's Done list and RFC-0005 §24.4. - **Created:** 2026-10-24 - **Scope:** - every Temporal workflow and activity in `server/services/temporal/` and `server/nodes/scheduler/cron_scheduler/`; - the code that starts, signals, updates and cancels them: `server/services/deployment/`, `server/services/events/`, `server/services/chat/`, `server/services/approvals/`, `server/services/employees/`; - the plugin framework's activity behaviour: `server/services/plugin/`, and the nodes whose behaviour depends on Temporal semantics (Timer, approvalGate, every sending node); - worker topology and configuration (`worker.py`, the standalone worker), and the replay and identity tests in `server/tests/temporal/`. - **Relationship:** - Complements RFC-0004 (using the SDK and platform fully). It covers what RFC-0004 skipped: a pattern-by-pattern review of Temporal's design-pattern catalogue, AI cookbook, best practices, Python samples and Nexus pages. §21 corrects the RFC-0004 statements that verification found wrong or that a decision superseded. - Amends RFC-0005 (the event OS) wherever a pattern or an owner decision changes a phase's design (§20). RFC-0005 stays the plan of record; this RFC adds detail and fixes gaps. - Adds Phase 1b to RFC-0005's rollout: fixes that need no workflow versioning and should land before Phase 2 (§22). - **Baseline:** HEAD 4f73cb10 (2026-10-24), temporalio 1.30.0, Temporal CLI pinned to v1.9.1 (server 1.32.0) for new installs. Line numbers are as of that commit and drift. - **Sources:** docs.temporal.io as of 2026-10-24 and temporalio/samples-python at 33f7dc29 (2026-10-22). See §2. ## Architecture diagrams Source baseline: `56f0b397`. [Complete diagram suite](docs/diagrams/architecture/README.md). **Event OS control plane and resource lifecycle — Proposed target · accepted roadmap.** [![Event OS control plane and resource lifecycle (Proposed target · accepted roadmap)](docs/diagrams/architecture/14-event-os-control-plane-v1.png)](docs/diagrams/architecture/14-event-os-control-plane-v1.png) [Editable Mermaid](docs/diagrams/architecture/14-event-os-control-plane.mmd) · [Source evidence and limitations](docs/diagrams/architecture/14-event-os-control-plane.brief.md) ## How to read this - **Owner:** §1 (summary), §3 (decisions), §24 (defaults chosen and what is still open). - **Reviewer:** §4 (the catalogue at a glance), §5 (rules every workflow follows), and §20 (what changes in RFC-0005). - **Implementer:** §6 to §19, one section per pattern category or source. Each pattern has the same parts: what the docs prescribe, SDK support, what OpenCompany does today with file and line, a verdict, recommendations with their versioning and phase, remarks on pitfalls, and a Python sketch where something is missing or misused. Then §22 (rollout) and §23 (acceptance gates). - **Quotations of RFC-0005:** where an entry in §6 to §19 describes "the RFC-0005 plan", it quotes RFC-0005 as reviewed, before the amendments in §20. Verdicts use six words: - **Adopted correctly:** the code follows the pattern. - **Adopted with deviations:** the code follows it with gaps, listed. - **Misused:** the code uses the pattern's mechanism in a way the docs warn against, or that breaks a rule OpenCompany has set. - **Missing and beneficial:** OpenCompany should adopt it. - **Planned in RFC-0005:** RFC-0005 already designs it, and this RFC says whether that design follows the docs. - **Not applicable:** it solves a problem OpenCompany does not have, with the reason. Priorities: - **P1:** fix before Phase 2: in Phase 1b, or in Phase 1 for T17's fairness and partition. These fixes need no workflow versioning. - **P2:** fix with the RFC-0005 phase named. - **P3:** later or never. ## 1. Summary Temporal publishes a catalogue of design patterns for durable workflows: orchestration, messaging, entities, external waits, transactions, retries, batches, quality of service, latency and worker configuration. It also publishes an AI cookbook, best practices, Python samples and the Nexus pages. This RFC checks OpenCompany against every one of them, page by page, and says what to change. **What we found.** OpenCompany's Temporal code follows the documented patterns in shape. Of the 45 design-pattern entries: - 26 are adopted with deviations; - 4 are misused; - 5 are missing and would help; - 3 are already planned in RFC-0005; - 7 do not apply. None is adopted with no deviation at all. The deviations sit exactly where the pages put their warnings: what happens on a cancel, a retry, a restart or a long wait. Verified examples: - **Side effects can happen twice.** Every message a graph node sends retries up to three times with no guard, including the reply after the approval gate in every employee that Hire builds. Tool calls such as an HTTP POST, a GitHub pull request or a Vercel deploy run again after a broken-off attempt. Both contradict E12. (§11.2) - **Agent children are terminated, not cancelled.** Four agent-child starts take the default parent close policy, so a parent that closes early kills its agents without their cleanup. Browser claims, permits and job rows are left behind. (§6.2) - **A bug in workflow code makes its own fix fail replay.** Eight workflow types run cleanup activities on any exception, bugs included, and three swallow bugs per event. (§18.2) - **Waits do not survive restarts.** The Timer node sleeps inside an activity, and a graceful shutdown fails it. Reviewed team-job delivery strands its draft when a retry lands within its lease. The approval gate polls inside a 24-hour activity. (§9.5, §9.6) - **Some calls have no deadline:** every client-side Update call, a streamed LLM call (the 600 s client timeout is per read), and every Gemini call. (§7.3, §16.3) - **Some failures are fatal or silent.** - The controller fails within seconds to about 20 seconds of database trouble while it holds a backlog. - Compensations give up after 3 or 10 attempts (until 1b.20 made them retry without limit). - The Start path carries saved credentials into Temporal input. - No Temporal metric is exported, so no alert can fire. (§12.3, §10.2, §17.8, §17.10) **What changes.** The owner settled twenty questions on 2026-10-24 (§3.1). In short: - **Fresh start confirmed.** Old Temporal databases are deleted at the pinned release (T5). - **Polls become retried activities** (T7). - **Owner chat starts at once,** with no debounce (T9). - **Owner actions write an intent** and are delivered in the background (T10). - **Stuck turns park** instead of failing (T11). - **Stop does not wait for drafts** (T12). - **Side effects are redone only when idempotent** (T13). - **Agent transcripts stay in history** (T6). - **Hosted instances run workers in a second process** (T8). - **Fairness ships with the fresh start** (T17). - **Lost supervisors are rebuilt in place** (T20). **How it ships.** - **Phase 1b** (§22.2) is a new step before RFC-0005's Phase 2: fixes that need no workflow versioning (each leaves the replayed command sequence unchanged), one commit each, simplified on 2026-10-25, with RFC-0007's approvals track inside it. Side effects come first, then lifecycle, compensation, retries, platform, and tests and docs. - **Design-level changes** land in the RFC-0005 phases they belong to (§22.3). RFC-0005 was amended in the same commit (§20), and 34 RFC-0004 statements that verification found wrong or that a decision superseded were corrected (§21). - **What does not apply:** Nexus, eager workflow start, delayed start, asynchronous completion and MapReduce trees, each for a stated reason (§19, §14.4, §9.4, §18.15, §12.5). ## 2. Method and sources - **Docs** (all fetched as raw markdown on 2026-10-24 by appending `.md` to each page URL): - the design-pattern catalogue (https://docs.temporal.io/design-patterns): the index, 10 category overviews and 35 patterns, 46 pages. The same pages existed on 2026-10-16; only the Saga page changed since. - the AI section and its cookbook, 19 pages; - the best-practices section (13 pages, four of them new since 2026-10-16) and the Python SDK best practices (6 pages); - the Nexus section and the standalone activity and operation pages, 15 pages. - **Code samples:** temporalio/samples-python at commit 33f7dc29 (2026-10-22), 57 sample folders. It targets temporalio 1.34.0 or later; every call it makes was checked against the installed 1.30.0 source. - **Research:** twelve parallel passes, one per pattern category or source. Each read its pages in full and audited the code at HEAD. - **Verification:** four independent passes re-checked every P1 claim, every Misused verdict, every defect statement and every figure this RFC quotes. They tried to refute each one against the code, the 1.30.0 and 1.34.0 SDK sources, the CLI and dev-server source, and the recorded 0.3.1 replay histories. Of 493 claims checked: - 388 were confirmed; - 88 were corrected; - 4 were refuted; - 13 could not be verified without a live server. This RFC states only confirmed or corrected findings. - **Owner questions:** four rounds on 2026-10-24 (§3.1). - **Drafting:** six passes wrote §6 to §19 from the verified notes. Six more passes applied the owner's last decisions, shared numbering and cross-references. - **Not done:** nothing was run against a live server. Timings and event counts come from the recorded replay histories, which were made with stub activities, so they are upper bounds on orchestration cost, not measurements of a live instance. ## 3. Decisions ### 3.1 Owner decisions (2026-10-24, four rounds) | # | Decision | Changes | |---|---|---| | T1 | **Sources.** The review covers the design-pattern catalogue plus the AI cookbook, the best practices, the Python samples and Nexus. | §2 | | T2 | **Role.** This RFC amends RFC-0005 in the same commit wherever a pattern changes a phase's design. | §20 | | T3 | **Depth.** A full entry per pattern, with Python sketches. | §6-§19 | | T4 | **Priority.** When two patterns pull in different directions, reliability wins: no lost events and no stuck or duplicated runs, even at some cost in latency or money. | throughout | | T5 | **Fresh start confirmed.** RFC-0005 E16 stands: the first pinned release deletes old Temporal databases, and every install runs server 1.32. The code that keeps an install's old database and server (commit 49177427) is reverted in Phase 1. Server 1.31 and 1.32 features need no per-install gating once that release ships. | RFC-0005 §3.1 E16, §24.4 | | T6 | **Agent transcripts stay in history.** They keep travelling in LLM-step inputs, bounded by the byte budget, compaction and continue-as-new. RFC-0005's rule that nothing over 64 KB enters Temporal (I7) gets an exception for agent loops. | RFC-0005 §6.2 I7 | | T7 | **Polls are retried activities.** Each poll trigger gets one small workflow that runs a single activity. The activity fetches, stores the new items and the cursor in the database in one transaction, then ends with a BENIGN error so Temporal retries it after the interval. History stays near zero. This replaces one Schedule per poll. | RFC-0005 §3.1 E5, §15 | | T8 | **Hosted tier S splits workers out.** Hosted instances run Temporal workers in a second process; the desktop stays one process. First, activities bound to the backend process (the WhatsApp bridge, the phone emulator, the Claude session pool, the socket broadcaster, the encryption key) are routed to a queue only that process polls. | RFC-0005 §20 | | T9 | **No debounce for owner chat.** The turn starts at once; messages sent while it runs merge at the agent's next step (merge point B1). The docs say timers are accurate only from one second up. | RFC-0005 §11 | | T10 | **Owner actions write an intent first.** Stop, Resume, approval decisions and Interrupt write a database row and answer at once (§7.3 applies the same to Start and Apply). A background deliverer sends the Update with a stable id until it is applied, and the UI shows "applying". For approval decisions, RFC-0007 (2026-10-25) replaces the Update with a nudge Signal, sent again by the approvals watchdog; the workflow re-reads the row. | RFC-0005 §10.2, §13.4, §17.5; RFC-0007 §4.6 | | T11 | **Stuck turns park.** A turn that cannot continue for a reason that may clear (a provider outage past the step's time budget, an exhausted key or allowance, an account problem) suspends where it stopped. It holds no worker slot, raises an attention item, and continues on Resume, top-up, or when the outage ends. | RFC-0005 §10.3, §14 | | T12 | **Stop does not wait for drafts.** Stop completes while an approval draft waits. A decision taken while stopped is sent on Resume. | RFC-0005 §10.2, §17.5 | | T13 | **Side effects are redone only when idempotent.** A side-effecting tool call that broke off is run again only if its plugin declares itself idempotent or uses a provider idempotency key. Otherwise it is recorded as `unknown` and the employee is told. This extends E12 from sends to every side-effecting tool. | RFC-0005 §3.1 E12, §17.3 | | T14 | **A release that cannot replay an open run starts anyway.** The affected generations are marked "needs Reset". This applies to replay failures only; a release that fails its readiness check still rolls back (RFC-0005 §21.2). | RFC-0005 §21.2, §24.1 | | T15 | **Outgoing content rules.** Every outgoing message passes built-in rules (a saved credential, a secret-shaped token, a recipient outside the conversation) and the owner's own rules, even with Ask first off. A match holds the message for the owner. Amended by the owner on 2026-10-25: no key checks ("the LLM router will handle the API keys"), so the saved-credential and secret-token rules and 1b.7 are dropped. The recipient rule and owner rules ship in RFC-0005 Phase 7, where the approval gate calls the approvals service in one line (RFC-0007 §3). | RFC-0005 §17 | | T16 | **Phase 1 is finished after this RFC,** by the same session. | RFC-0005 §24.4 | | T17 | **The fresh-start release adds fairness and one partition.** The dev server gets `matching.enableFairness=true` and one task-queue partition (read and write) in the release that creates every Temporal database empty, because partitions can be lowered safely only on a new database. Retention and search attributes at namespace creation stay deferred, as decided on 2026-10-23 (RFC-0004 D4). The Start-snapshot secret fix ships in Phase 1b. | RFC-0005 §13.2, §20, §24.2 | | T18 | **The step budget and parking ship with Phase 5, for new generations only** (the generation flag `scheduler_version=1`). Existing generations keep today's unlimited LLM-step retries and fail-and-pause. A budget without parking would fail turns and lose their input. | RFC-0005 §14, §24.2 | | T19 | **A parked turn waits at most 7 days.** Then its input is dead-lettered with an attention item. RFC-0005's per-class turn deadlines become "slow turn" alerts that do not count parked time. (A 24-hour backstop for turns that are stuck without being parked is a default, §24.1.) | RFC-0005 §13.1, §18.1, §23 | | T20 | **A lost supervisor is rebuilt in place.** A terminated or failed supervisor is started again from the database (`ALLOW_DUPLICATE_FAILED_ONLY`), keeping chat and Context. After 3 rebuilds within an hour it stops and asks the owner to Reset. | RFC-0005 §10.1 | Decisions already in force: RFC-0005 §3.1 (E1-E16) and RFC-0004 §17, including Q6: a total time budget per LLM step. §11 of this RFC proposes the values, and T18 places them in Phase 5. ## 4. The catalogue at a glance One row per entry in §6 to §19. Priority P1 means Phase 1b (the ids are listed in §22), P2 means the RFC-0005 phase named, and P3 means later or never. ### 4.1 Design patterns | § | Pattern | Verdict | P | Action | |---|---|---|---|---| | 6.1 | Task orchestration (overview) | Adopted with deviations | P1 | Explicit child options, named bounds, named loser cleanup; lint (1b.40, dropped) | | 6.2 | Child Workflows | Adopted with deviations | P1 | Agent children request cancellation (1b.12, moved to the cancellation lane with the re-raise); cancel-and-drain and durable Claude Code and RLM delegations (cancellation lane) | | 6.3 | Parallel Execution | Adopted with deviations | P2 | One failure policy (running nodes finish, nothing new starts); graph by reference; delegation-window fix | | 6.4 | Pick First (race) | Adopted with deviations; RFC-0005 turn-deadline race planned (deviates) | P1 | A chat run past 24 h cancels its workflow (1b.14); for Phase 4, slow-turn alerts (T19) and a 24-hour no-progress backstop (a default, §24.1) | | 7.1 | Workflow messaging (overview) | Adopted with deviations | P1 | The handler checklist (§5 rules 12-15) | | 7.2 | Signal with Start | Missing and beneficial (as start-with-state); RFC-0005 relay planned (deviates) | P2 | Triggers in the controller's start input; the stream cursor as the entity's start input | | 7.3 | Request-Response via Updates | Adopted with deviations; RFC-0005 owner Updates planned (follows) | P1 | A deadline on every Update call (1b.15); stable ids, slim results, the T10 deliverer | | 7.4 | Event Accumulator | Adopted with deviations; RFC-0005 debounce planned (follows) | P2 | Dead-letter poison events; no owner-chat debounce (T9); absolute debounce deadline; carry cursors | | 8.1 | Entity and lifecycle (overview) | Not applicable (selection guide) | P3 | The database-as-truth rules (§5.1) | | 8.2 | Entity Workflow | Adopted with deviations; RFC-0005 plan deviates | P2 | Decommission the Workspace controller; turn numbers from admission; a seeded `applied_seq`; supervisor rebuilt in place (T20) | | 8.3 | Continue-As-New | Adopted with deviations | P2 | Slim controller carry, cancellation fix, rollover at the first boot of each release; slim agent carry (1b.19) | | 8.4 | Updatable Timer | Planned in RFC-0005 (follows, misses details) | P2 | One timer over absolute deadlines; no zero timeouts; no sub-second windows; one instance-wide upkeep sweep | | 9.1 | External interaction (overview) | Not applicable (selection guide) | P3 | "Where each wait goes" (§5 rule 2) | | 9.2 | Polling External Services | Adopted with deviations | P1 | Bounded attempts, without heartbeats (1b.17); one retried poll activity per trigger in Phase 3 (T7) | | 9.3 | Long-Running Activity | Adopted with deviations | P1 | Activities re-raise cancellation (1b.2); long waits fit one attempt (1b.4, dropped); checkpoint external job ids | | 9.4 | Delayed Start | Not applicable | P3 | No adoption; never `start_delay` on ids addressed by Signal-With-Start | | 9.5 | Delayed Callback (webhooks) | Misused (Timer node; webhooks until Phase 2) | P1 | Timer node fixes (1b.8); a durable Timer behind a flag; the Phase 2 inbox for webhooks | | 9.6 | Approval | Misused (the wait) | P1 | Reviewed delivery re-attaches (1b.9); approved sends wait while stopped (1b.5); stranded drafts expire (1b.10); ApprovalWorkflow for held tool calls (RFC-0007, AP.4a and AP.4b), then the gate and delivery (Phase 7) | | 10.1 | Distributed transactions (overview) | Adopted with deviations | P1 | The transaction rule in DESIGN.md (1b.45) | | 10.2 | Saga | Adopted with deviations | P1 | Compensations retry until they land (1b.20); Reset closes everything it started (1b.21); Start's compensation in reverse (1b.22); interrupted Resets finish (1b.23) | | 10.3 | Early Return | Adopted with deviations | P1 | A bounded Workspace Update-with-Start (1b.15); later behind the T10 deliverer | | 11.1 | Error handling (overview) | Adopted with deviations | P1 | Shared retry constants (with 1b.9, 1b.16 and 1b.20); the lint (1b.41) dropped | | 11.2 | Fixed Count of Retries | Adopted with deviations | P1 | Effect classes and the redo guard (1b.1); the legacy tool path honours declared policies (1b.3) | | 11.3 | Fixed Wall-Time Retries | Missing and beneficial | P2 | A per-step budget of 15 min (cloud) and 45 min (local); expiry parks the turn (Phase 5, new generations, T18) | | 11.4 | Non-Retryable Errors | Misused | P1 | LLM-step bugs and `PayloadsTooLarge` non-retryable (1b.26); compaction on the step's policy (1b.27) | | 11.5 | Delayed Retry | Adopted with deviations | P2 | A provider wait longer than the remaining budget parks the turn (Phase 5) | | 11.6 | Fast/Slow Retries | Adopted with deviations | P2 | Parking is the slow phase (Phase 5); controller spawn backoff and dead letter behind a patch | | 11.7 | Retry Alerting via Metrics | Missing and beneficial | P1 | The attention item and a quieter log now (1b.30); the metric counter in Phase 8 | | 11.8 | Resumable Activity | Adopted with deviations | P2 | One park mechanism (Phase 5; T11, T18, T19) | | 12.1 | Batch processing (overview) | Adopted with deviations | P1 | A real-server test of the 2,000-Update limit (1b.43) | | 12.2 | Fan-Out with Child Workflows | Adopted with deviations | P2 | Admission caps for new generations; a cap on open team tasks | | 12.3 | Batch Iterator | Adopted with deviations | P1 | Controller paging retries without limit (1b.16); bounded poll pages (1b.18) | | 12.4 | Sliding Window | Missing and beneficial | P2 | Permit waits inside the workflow with a wake Signal; Reset releases permits | | 12.5 | MapReduce Tree | Not applicable | P3 | Do not adopt | | 13.1 | QoS and throughput (overview) | Overview | P2 | One enforcement point per resource; "reserved share" means admitted turns | | 13.2 | Downstream Rate Limiting | Misused | P2 | Frozen class-queue routes for LLM steps; per-credential buckets; never cap `machina-tasks` | | 13.3 | Priority | Planned in RFC-0005 (follows, misses details) | P2 | An explicit key on every top-level start; entities keyed by lane | | 13.4 | Fairness | Planned in RFC-0005 (follows, misses details) | P1 | Fairness and one partition in the fresh-start release (T17); the employee as fairness key | | 14.1 | Performance and latency (overview) | Overview | P2 | A timing mode with production settings; a pinned metrics port | | 14.2 | Local Activities | Missing and beneficial | P2 | AgentWorkflow progress broadcasts as best-effort local activities (`progress_local_version=1`, after SDK 1.34.0); new types schedule no UI-only activity | | 14.3 | Early Return + Local Activities | Adopted with deviations (Workspace tasks); Not applicable (chat) | P3 | Only if measured | | 14.4 | Eager Workflow Start | Not applicable | P3 | Do not adopt | | 15.1 | Worker configuration (overview) | Adopted with deviations | P1 | A placement axis; one main-queue registration (1b.36) | | 15.2 | Worker-Specific Task Queues | Adopted with deviations; process affinity missing | P1 | The pool always polls (1b.34); the standalone guard (1b.35); an instance queue before the T8 split | | 15.3 | Activity Dependency Injection | Adopted with deviations | P1 | One main-queue registration (1b.36); no worker builder | ### 4.2 AI agents, best practices, samples and Nexus | § | Entry | Verdict | P | Action | |---|---|---|---|---| | 16.1 | Durable AI and the cookbook | Adopted with deviations | P2 | Priority and fairness keys (Phase 5); summaries without user text (Phase 8) | | 16.2 | Agentic loop with tool calls | Adopted with deviations | P1 | Non-retryable failures (1b.26); compaction's policy (1b.27) | | 16.3 | LLM call timeouts | Adopted with deviations | P1 | 30-minute attempts for local endpoints and a classified request timeout (1b.25; no in-attempt deadline, no Gemini timeout); the step budget in Phase 5 (T18) | | 16.4 | Hello world (OpenAI Responses) | Adopted with deviations | P3 | Settle node-level SDK retries with RFC-0004 Q9 | | 16.5 | Retry policy from HTTP responses | Adopted with deviations | P2 | Honour `x-should-retry`, retry 409, cap Retry-After (Phase 5) | | 16.6 | Deep research | Adopted with deviations | P3 | Summaries (Phase 8) | | 16.7 | Structured outputs | Adopted with deviations | P3 | Stop offering a tool after 3 refusals in a turn | | 16.8 | Claim check | Adopted with deviations | P1 | Slim the carry (1b.19); relief before every step (Phase 4); the I7 exception and its bounds (T6) | | 16.9 | Guardrails | Adopted with deviations | P1 | Built-in content rules (1b.7, dropped by the owner); owner and recipient rules (Phase 7) (T15) | | 16.10 | Human in the loop | Misused | P1 | See §9.6 | | 16.11 | Agent memory and state | Adopted with deviations | P1 | The redo guard (1b.1) | | 16.12 | Durable MCP server | Not applicable | P3 | A durable route only if Claude Code employees should send | | 16.13 | Multi-agent with ADK | Adopted with deviations | P3 | Tool errors built with `json.dumps` (Phase 4) | | 16.14 | Framework cookbook pages | Not applicable | P3 | Keep the native loop | | 16.15 | Framework plugin samples | Not applicable | P3 | Ideas only | | 16.16 | OpenRouter samples | Planned in RFC-0005 (deviates) | P2 | Provider exhaustion and long outages park the turn (Phase 5) | | 16.17 | Bedrock and AgentCore samples | Planned in RFC-0005 (follows) | P2 | No change | | 16.18 | Google GenAI sample | Not applicable | P3 | None | | 16.19 | Tracing and metrics samples | Not applicable | P2 | gen_ai metric names, no content (Phase 8) | | 16.20 | Workflow Streams sample | Adopted with deviations | P3 | Keep the chat hub | | 17.1 | Best practices (index) | Not applicable | P3 | None | | 17.2 | Namespace best practices | Adopted with deviations | P2 | Retention and search attributes at creation, deferred per RFC-0004 D4 (T17) | | 17.3 | Multi-tenant patterns | Planned in RFC-0005 (follows) | P2 | See §13.4; no ids in metric labels | | 17.4 | Managing APS limits | Missing and beneficial | P1 | A history-cost counting script (1b.44, dropped); jittered budget timers; no UI-only activities | | 17.5 | Workflow cost optimization | Adopted with deviations | P2 | No UI-only activities in new types; graphs by reference | | 17.6 | Cost governance | Not applicable | P3 | Release tags in fleet telemetry | | 17.7 | Cloud access control | Not applicable | P3 | Telemetry key hygiene | | 17.8 | Security controls | Misused (Start path) | P1 | Sanitized Start snapshot (1b.31, 1b.32); named ports and netpolicy (1b.33) | | 17.9 | Worker deployment and performance | Adopted with deviations | P1 | Graceful shutdown (1b.37); the T8 worker split | | 17.10 | Alerting on worker metrics | Missing and beneficial | P1 | A stuck-runs attention item (1b.38b); the SDK metrics and starter alerts in RFC-0005 Phase 8 | | 17.11 | Error handling | Adopted with deviations | P1 | Explicit chain-unique activity ids; cancellation (1b.2); the send guard (1b.1) | | 17.12 | Pre-production testing | Missing and beneficial | P2 | Replay open runs before polling and mark "needs Reset" (T14, Phase 9a) | | 17.13 | Knowledge hub | Adopted with deviations | P1 | Stale comments (1b.45) | | 17.14 | Python best practices (index) | Not applicable | P3 | None | | 17.15 | Python debugging | Adopted with deviations | P2 | Core logs at WARN; opt-in span exporter (Phase 8) | | 17.16 | Python error handling | Adopted with deviations | P1 | The bug guard (1b.13); bugs stay stuck and visible | | 17.17 | Python testing suite | Adopted correctly | P1 | Node-cancel and per-type bug tests | | 17.18 | Python SDK sandbox | Misused | P1 | The determinism lint (1b.40, dropped); new types sandboxed (Phase 4) | | 17.19 | Python sync vs async | Adopted with deviations | P1 | Loop-lag watchdog (1b.39); the T8 second process (Phase 9) | | 18.1 | Pattern-to-sample map | - | - | 12 of 35 patterns have no Python sample | | 18.2 | Exit cleanup and handler waits | Misused | P1 | The bug guard (1b.13) | | 18.3 | Message handlers and Update-with-Start | Adopted correctly | P3 | None | | 18.4 | Workflow basics (`hello`) | Adopted correctly | P3 | None | | 18.5 | Observability samples | Missing and beneficial | P1 | Task failures at ERROR (1b.29); BENIGN retries (1b.28) | | 18.6 | Polling | Adopted with deviations | P2 | T7's poll workflow (Phase 3) | | 18.7 | Timers | Planned in RFC-0005 (follows) | P2 | Versioned `alarms_changed` (Phase 6) | | 18.8 | Schedules | Adopted correctly | P3 | Cron jitter (planned) | | 18.9 | Patching | Adopted with deviations | P2 | Four-stage retirement; dormant patches after SDK 1.34.0 | | 18.10 | Replay | Adopted with deviations | P2 | Replay open runs at upgrade (Phase 9a) | | 18.11 | Worker-specific task queues | Adopted with deviations | P2 | The instance queue (T8) | | 18.12 | Worker processes | Planned in RFC-0005 (follows) | P2 | Second process on hosted tier S (T8) | | 18.13 | Worker versioning | Not applicable | P3 | Declined for topology | | 18.14 | Long-running activities | Adopted with deviations | P1 | Cancellation (1b.2); heartbeat interval from the timeout | | 18.15 | Asynchronous activity completion | Not applicable | P2 | Human waits move to ApprovalWorkflow, a nudge Signal plus a timer: held tool calls since RFC-0007, the gate and delivery in Phase 7 | | 18.16 | Resource pool | Adopted with deviations | P3 | Reset releases permits (planned for 1b.21, not built there; the age reclaim still frees them) | | 18.17 | Interpreters and batches | Adopted correctly | P3 | None | | 18.18 | Codecs, external storage, converters | Not applicable | P3 | References first; a codec only with an owner decision (RFC-0004 Q13) | | 18.19 | Eager start and local activities | Not applicable | P3 | Drop RFC-0004's eager-start experiment | | 18.20 | Test techniques | Adopted correctly | P3 | None | | 18.21 | Samples that do not apply | Not applicable | P3 | None | | 19.1-19.10 | Nexus, its operations, patterns, security and standalone activities | Not applicable | P3 | Keep out; correct RFC-0004 §4.2, §6.6, §9.6 and §15 | ### 4.3 Counts Across the 45 design-pattern entries (10 category overviews and 35 patterns; the catalogue's index page has no entry), counting each entry's main verdict: - Adopted correctly: none. - Adopted with deviations: 26. - Misused: 4 (Delayed Callback, Approval, Non-Retryable Errors, Downstream Rate Limiting). - Missing and beneficial: 5. - Planned in RFC-0005: 3. - Not applicable, or an overview with no verdict of its own: 7. Most of OpenCompany's Temporal code follows the documented patterns in shape. The deviations are in the details the pages call out: cancellation, retries, waits, and what happens on a restart. ## 5. Rules every workflow and activity follows These rules distil the patterns in §6 to §19 into checks a reviewer can apply to any change in `server/services/temporal/`, the plugin framework or a new workflow type. Each names the section that explains it. Where a lint or test enforces a rule, the rule says so (§22 schedules them). ### 5.1 The database holds the truth 1. **Workflows hold handles, cursors and timers, not facts** (RFC-0005 I10). - Anything the owner sees, or that a restart, a Reset or a fresh Temporal database must recover, lives in the app database first. That covers control state, approvals, sends, budgets, alarms and conversation cursors. - Turn numbers and any id that feeds an idempotency or spend key live there too, and every entity start (the first, a restart after passivation, a rebuild) seeds its state from the database. - A Signal is a hint to look (I1). (§8.1, §8.2) 2. **A wait goes where it can survive what it waits through.** (§9.1) - Waiting seconds for an answer: an activity, within one attempt. - Waiting for a person or a date: an Update plus a durable workflow timer. - Waiting for a provider to recover: activity retries, then a parked turn (T11). - Never sleep inside an activity for longer than one attempt can last. - Never put `start_delay` on an id that the relay addresses with Signal-With-Start: any delivery would bypass the delay (§9.4). ### 5.2 Workflow code 3. **Determinism is checked, not assumed.** - The 14 existing types and RFC-0007's ApprovalWorkflow run unsandboxed (`sandboxed=False`). A determinism lint covers their workflow code: the methods of `@workflow.defn` classes and the helpers they call, not activities that share the module. It checks for no I/O, no wall clock, no unseeded randomness, no `asyncio.wait` or `asyncio.as_completed` (their ordering is not deterministic; `workflow.wait` and `workflow.as_completed` are), and no command-issuing code in a signal handler. That lint was 1b.40, dropped on 2026-10-25, so today only the replay gate checks this. - New RFC-0005 types run in the sandbox, passing through only side-effect-free leaf modules. (§17.18) 4. **Cleanup runs only when the workflow is really exiting.** - Code that issues commands in `finally`, `except Exception` or `except BaseException` first re-raises a workflow bug (`is_workflow_bug`). Otherwise a bug records cleanup commands, marks rows failed, and makes the fixed release fail replay. - `asyncio.TimeoutError` counts as a failure, not a bug. (§18.2) 5. **Every child start states its policies.** - Each child start passes `parent_close_policy`, `cancellation_type` and `id_reuse_policy` explicitly. - Agent children use `REQUEST_CANCEL` with `WAIT_CANCELLATION_COMPLETED`; per-event runs use `ABANDON`. - A lint was to reject a child start without them (1b.40, dropped on 2026-10-25). (§6.2) 6. **Know what replay compares.** - On replay Temporal compares only a command's kind, its activity or child type, and its id. - Changing timeouts, retry policies, task queues or a parent close policy needs no patch. - Changing a cancellation type does need a patch whenever a replayed history can contain a cancel of that activity or child, because the cancellation type decides whether a cancel emits a command. (§6.2; corrects RFC-0004 §8.5) 7. **Every fan-out names its bound, and every race names its loser's fate.** That means who cancels the loser, whether the workflow waits for its cleanup, and how long that wait may last. (§6.3, §6.4) 8. **Timers run over absolute deadlines.** - One live timer per workflow over named absolute deadlines, through a shared helper. - Never `wait_condition` with a timeout of zero or less. - No sub-second timers: the docs say timers are accurate only from one second up (https://docs.temporal.io/workflow-execution/timers-delays). (§8.4) 9. **Close cleanly.** - Continue-as-new and completion happen only in the main method, after `await workflow.wait_condition(workflow.all_handlers_finished)`, with no `await` between the final buffer check and the close. - A long-lived type rolls over when the server suggests it, or at 1,800 Updates (90% of the 2,000-per-run limit), whichever comes first. (§7.4, §8.3, §12.1) 10. **Activity ids are idempotency keys.** Wherever an id feeds an idempotency or spend key, it is explicit and unique across the continue-as-new chain. The SDK's default activity id is a per-run counter that restarts after continue-as-new. (§17.11) 11. **No UI-only activities in new types.** Progress broadcasts become best-effort local activities or ride an adjacent activity. Today they are 35-52% of an AgentWorkflow history. (§14.2, §17.4) ### 5.3 Messages 12. **Handlers stay small.** Signal handlers only buffer or set flags. Rejections that need no I/O live in Update validators, which reject without writing history. (§7.1) 13. **Every client-side Update call that a request or a loop waits on runs under a deadline** (`update_calls.bounded_update`, 1b.15); a background transition's own Updates may wait. `rpc_timeout` bounds one request only: after the Update is accepted, `execute_update` waits for the outcome without it. (§7.3; corrects RFC-0004 §8.2) 14. **Owner actions go through intents** (T10). - Stop, Resume, approval decisions, Interrupt, Start and Apply write an intent row and answer at once. - A background deliverer sends the Update by workflow id, with a stable id and a per-attempt deadline, until it is applied. An approval decision is the exception (RFC-0007 §4.6): the request writes the row and nudges the waiting `ApprovalWorkflow` with a Signal, and the chat watchdog nudges again until the decision is read. - No request path waits on Temporal (RFC-0005 I2). (§7.3, §10.2) 15. **Updates carry absolute values and a revision.** - Handlers are idempotent by revision across continue-as-new, because Update-id dedupe works only within one run. - A stale revision is rejected in the validator. (§7.3, §8.2) ### 5.4 Activities, retries and side effects 16. **Named retry policies only.** - Every activity call site uses a shared policy constant. A lint rejects inline policies in the workflow modules `server/tests/test_retry_policies.py` lists; the lint for every module was 1b.41, dropped on 2026-10-25. - A paid or side-effecting call never retries without a time budget. - One exception: the LLM step and compaction on generations without `scheduler_version=1` (RFC-0005 Phase 5) keep unlimited retries, under the retry alert (1b.30). A budget without parking would fail their turns (T18). - An unlimited policy is otherwise allowed only for idempotent bookkeeping, compensations, queue paging, permit waits, the approval gate's wait for the owner, the Pi driver's turn, T7's poll activities (each retry is the next poll), and sends and deliveries made at-most-once by their claim (the egress outbox; reviewed delivery's `inflight` mark, 1b.9). Each is declared, and the retry alert covers all of them except the planned retries: T7's polls, and the permit waits, approval gate and Pi driver that `PLANNED_WAITS` lists (1b.30). (§11.1; rewords RFC-0004 AG9) 17. **Classify every failure.** - User and configuration errors fail fast as non-retryable: `NodeUserError`, permission errors, `PayloadsTooLarge` (Core's type since SDK 1.31; `PayloadSizeError` before) and bugs inside the LLM step. - Rate limits retry as BENIGN with the provider's `next_retry_delay`. - Planned retries (T7's poll attempts, permit waits, the approval gate, the Pi driver) are BENIGN and excluded from retry alerts. (§11.4, §11.5, §11.7) 18. **Side effects say whether they may repeat** (T13, E12). - One service table, `services/approvals/repeats.py`, lists the calls that may run again: they change nothing outside, or make the same change again. Every other call that changes something outside OpenCompany is `unsafe`. No plugin declares anything (1b.1a, on the owner's direction of 2026-10-25). - An `unsafe` call runs under a claim guard: if an attempt broke off after the provider might have acted, the next attempt records `unknown` and tells the employee instead of calling again. - This applies to graph nodes and tool calls alike, sends included. (§11.2, §16) 19. **Cancellation reaches the code.** - Activities re-raise cancellation instead of returning a failure envelope (RFC-0004 H2). - A worker shutdown becomes a retryable error, so the next worker resumes the work. (§9.3) 20. **Long activities heartbeat.** - The heartbeat interval is a third of `heartbeat_timeout`, at most 30 s. - They record external job ids in a database row before waiting on the job, and also in heartbeat details. A retry then re-attaches to the job instead of starting it again. Heartbeat details alone are lost when the activity is scheduled again after a rollover. (§9.3) ### 5.5 Capacity and order 21. **One enforcement point per resource.** - Turns are limited by admission (RFC-0005 §13.3), providers by per-credential buckets. - Queue-wide rate caps go only on a queue with one downstream and one fixed limit, and never on `machina-tasks`, where a cap turns off eager activity dispatch. (§13.1, §13.2) 22. **Every top-level workflow start sets an explicit priority key.** An unkeyed start runs at 3, which is RFC-0005's internal class; unkeyed children and activities inherit their parent's key. An AST test enforces this. (§13.3) ### 5.6 Payloads and secrets 23. **References, not bytes.** - Media travels as `FileRef`, and events over 64 KB travel by reference (RFC-0005 I7). - Agent transcripts are the one exception (T6). They stay within its bounds: at most 393,216 bytes of messages before every step, each step input under 1 MiB, the continue-as-new carry under 1.9 MB and the stored conversation under 1 MB. (§16.8) 24. **No secrets in Temporal inputs.** Snapshots of node parameters are sanitized on every path, Start included, and activities read credentials from the saved row. (§17.8) ### 5.7 Workers and tests 25. **One worker builder.** - Every Worker, embedded, standalone, test or recorder, comes from one function. - That function makes dependencies ready before polling, registers from one list (`workflow_registry.py`), and checks the registration against what workflows schedule. - Built narrower in 1b.36 (d379ab3e): one main-queue registration (`_main_queue_activities`, `temporal_plugins()`) serves the embedded manager, the standalone worker and `create_worker`. There is no builder function and no registration check. - The pool always polls the queues frozen runs can target. - Activities bound to the backend process run on the instance queue (T8). (§15) 26. **Every limit we rely on has a real-server test.** That includes Updates per run, history size and continue-as-new suggestions. Every change to workflow code passes the N-1 replay gate and the golden identity test. (§12.1, §17.17) ### 5.8 Changes that span systems 27. **One local transaction where possible; a saga only across systems.** - A change that touches the database and Temporal, or a provider, writes its intent in one local transaction, and a worker carries it out (an outbox). - Irreversible effects go last. - A saga only where several systems change: compensations run last in, first out, each on its own. - An effect that cannot be compensated (a sent message, a refund) is guarded before it happens by an Undo window and a claim at a revision; after it, an unclear outcome is reported as `unknown`, never undone. - An owner's action writes an intent and answers; a deliverer applies it (T10). (§10.1; `docs-internal/DESIGN.md` §5) ## 6. Task orchestration Three pages cover how one workflow splits, parallelizes and races work. OpenCompany uses all three mechanisms where the docs put them: agents run as child workflows, graph nodes run as parallel activities, and races happen inside activities. The gaps are in what happens to work that is still running when its parent stops or fails. Terms used below: - A **Child Workflow** is a workflow started by another workflow, its parent, with its own id and history. - A **command** is what workflow code asks the server to do: schedule an activity, start a child, start a timer, record a patch marker. On replay the SDK checks the code's commands against the recorded history, so a change to which commands a workflow issues needs versioning: `workflow.patched(...)` inside a type that old and new runs share, or a per-generation `resource_manifest` flag frozen into the workflow's input (RFC-0005 §24.1 rule 2). - The **parent close policy** decides what the server does to a child that is still running when its parent closes: TERMINATE (kill it; none of its code runs), REQUEST_CANCEL (send it a cancellation request, so its cleanup runs) or ABANDON (leave it running). - The **cancellation type** decides what the parent's handle does when the parent itself cancels the child or activity: TRY_CANCEL (request the cancel and resolve at once; the default for activities), WAIT_CANCELLATION_COMPLETED (resolve only when the child or activity has finished cancelling; the default for children) or ABANDON (resolve at once and send no cancel). - The **id reuse policy** decides whether a new execution may take the id of a closed one. ### 6.1 Task orchestration patterns (overview) **Docs:** https://docs.temporal.io/design-patterns/task-orchestration-patterns. A chooser for the three patterns of this section: - "A process is large or reused across Workflows": Child Workflows "with independent histories". - "Independent work can run at the same time": Parallel Execution "with a concurrency bound". - "Several approaches compete and you want the first to finish": Pick First "and cancel the losers". - Scale-out goes to Batch Processing, latency to Performance & Latency. **SDK:** none of its own (§6.2 to §6.4). **OpenCompany today:** - Agents are child workflows. MachinaWorkflow schedules graph nodes as parallel activities. AgentWorkflow runs a turn's tools one at a time, delegations as a window of children, and Task Manager work as ABANDON runners. The controller starts one child run per event. Races exist only inside activities. (§6.2 to §6.4.) - No rule says which close policy, cancellation type and id reuse policy a child start uses. Four agent-child starts, at six call lines, leave the parent close policy at the SDK default, TERMINATE (table in §6.2). - No fan-out names its bound. MachinaWorkflow schedules every ready node (`server/services/temporal/workflow.py:640-874`); the controller starts one child per event without a cap (`server/services/temporal/workflow_control_workflow.py:431-483`). - Work that should stop is either killed without cleanup or left running: an uncontrolled run's failure abandons its running nodes (`workflow.py:907-908`), a cancelled root's agent children are terminated (§6.2), and a chat run past 24 hours leaves its workflow running (§6.4). - Nothing enforces determinism. All 14 workflow definitions pass `sandboxed=False`, so the SDK's sandbox warning for `asyncio.wait` can never fire, and no test forbids it. **Verdict:** Adopted with deviations. The three mechanisms sit where the docs put them; bounds and the cleanup of losers are stated nowhere. **Recommendation:** 1. **State three orchestration rules and lint them.** §5 adopts them as rules: (a) every child start passes `parent_close_policy`, `cancellation_type` and `id_reuse_policy` explicitly; (b) every fan-out names its bound; (c) every race names who cancels the loser and whether the workflow waits for its cleanup. The checks ship in the determinism lint, 1b.40, which §17.18 specifies. The orchestration part this entry adds to it: a child start without the three explicit options; `asyncio.wait` and `asyncio.as_completed` in workflow modules; a direct `execute_activity` or `start_activity` call inside a `@workflow.signal` handler. (Pi `pause` and `resume` reach an activity indirectly, through `_forward`; §7.3 R6 removes them.) The same commit writes out the current defaults at the child starts that rely on them; passing a default explicitly changes nothing on replay. The retry-policy lint over the same workflow modules is 1b.41 (§11.1). - Versioning: none (a test, and explicit defaults). - When: Phase 1b (1b.40), after 1b.13. Dropped on 2026-10-25 (§22.2), so the lint waits. Until the cancellation lane ships 1b.12, the agent-child starts state today's policies explicitly. Effort S. Risk: none at run time. **Remarks:** The overview points to Batch Processing. OpenCompany has no record-batch workload, so those pages map only to the controller's one child per event, its queue paging and the delegation window (§12.2 to §12.4). ### 6.2 Child Workflows **Docs:** https://docs.temporal.io/design-patterns/child-workflows. A child "executes as an independent Workflow with its own Workflow ID, event history (51,200 event limit), and lifecycle". Use a child for a separate id, work that "may outlive the parent", reuse, another task queue or an independent history; use an activity for external calls ("Consider Activities first"). - Python: `workflow.execute_child_workflow(...)` waits for the child to finish; `workflow.start_child_workflow(...)` "returns a handle once the child has started". - Parent close policy: TERMINATE for "Tightly coupled processes", ABANDON for "Fire-and-forget, long-running tasks", REQUEST_CANCEL for "Graceful cleanup". "The default policy is TERMINATE, which kills children when the parent closes." The cancellation concept page adds: "Anything a terminated Workflow would have cleaned up is left as it was: reserved inventory stays reserved, held locks" stay held (https://docs.temporal.io/encyclopedia/workflow/cancellation-and-termination). - "You must wait for the child to start before the parent completes." The Python guide adds: from a non-main coroutine, "make sure the Parent Workflow doesn't complete before that call resolves". - Best practices: unique ids, "Generate deterministic IDs based on business keys" (the page's own samples use `workflow.uuid4()`); choose the policy; handle `ChildWorkflowError`; "Limit parallelism" (a hard limit of 2,000 pending children per parent, and at most 1,000 children per parent in total is recommended); "Set timeouts"; typed stubs; "Monitor child executions". - Pitfalls: "Treating Child Workflows like Activities", unbounded children in a loop, "Ignoring the Parent Close Policy", omitting ids, not handling child failures. The comparison table rates an "Async Lambda" as "Independent history: No; Can outlive parent: No". **SDK:** - 1.30.0: `workflow.start_child_workflow(..., cancellation_type=WAIT_CANCELLATION_COMPLETED, parent_close_policy=TERMINATE, id_reuse_policy=ALLOW_DUPLICATE, execution_timeout, run_timeout, task_timeout, retry_policy, memo, search_attributes, static_summary, static_details, priority)` (`temporalio/workflow/_workflow_ops.py:291-316`). Children have no `id_conflict_policy`. - 1.34.0 adds only the experimental `event_groups` parameter; the enums and defaults are unchanged. - In both versions, cancelling a workflow cancels only its main task (`temporalio/worker/_workflow_instance.py:598-606` in 1.30.0, `:660-668` in 1.34.0). A child that the code is not awaiting at that moment gets no cancel; the server applies its parent close policy when the parent closes. - Cancellation fixes after 1.30.0, per the release notes: 1.31 "cancellations no longer swallowed in some circumstances"; 1.32 no nondeterminism error when a try-cancel child finishes after its cancel; 1.33 duplicate cancel commands de-duplicated (`_request_cancel`, present in the 1.34.0 source and absent in 1.30.0). **OpenCompany today:** Every child start in production code. The 27 recorded scenarios in `server/tests/temporal/fixtures/0.3.1/` and `fixtures/0.3.1-cancel/` record the same values. "Default" is the SDK default. No child start sets an execution or run timeout. | Parent -> child | Start | Parent close | Cancellation type | Id reuse | |---|---|---|---|---| | MachinaWorkflow -> AgentWorkflow or PiCodingAgentWorkflow | `server/services/temporal/workflow.py:826` (execution control on), `:828` (off); id `-agent-` (`:170-172`) | TERMINATE (default) | default | ALLOW_DUPLICATE (default) | | DelegatedTaskWorkflow -> agent | `server/services/temporal/agent_workflow.py:595-598` | TERMINATE (default) | default | default | | AgentWorkflow -> agent (`delegate_to_*`) | `agent_workflow.py:1740-1744` | TERMINATE (default) | default | default | | AgentWorkflow -> DelegatedTaskWorkflow (Task Manager runner) | `agent_workflow.py:1894-1899` (options `:1878-1891`, with `EventWorkflowId`) | ABANDON | default | default | | EmployeeJobWorkflow -> lead agent | `server/services/temporal/employee_job_workflow.py:117` (on), `:120` (off) | TERMINATE (default) | default | default | | PiInlineDelegationWorkflow -> DelegatedTaskWorkflow | `server/services/temporal/pi_inline_delegation.py:365-369` | REQUEST_CANCEL | WAIT_CANCELLATION_COMPLETED (explicit) | default | | WorkspaceTaskControllerWorkflow -> NodeInvocationWorkflow | `server/services/temporal/workspace_tasks_workflow.py:114-121` | REQUEST_CANCEL | explicit | REJECT_DUPLICATE | | NodeInvocationWorkflow -> Workspace agent | `server/services/temporal/node_invocation.py:34-39` | REQUEST_CANCEL | explicit | default | | Controller (push and poll) -> MachinaWorkflow per event | `server/services/temporal/trigger_listener_workflow.py:495`, `:497`; `polling_trigger_workflow.py:398-402`; entered from `workflow_control_workflow.py:755-765`, `:888-895` | ABANDON | default | ALLOW_DUPLICATE_FAILED_ONLY | | CronTriggerWorkflow -> MachinaWorkflow | `server/nodes/scheduler/cron_scheduler/_workflow.py:203` (options `:176-189`) | ABANDON | default | ALLOW_DUPLICATE_FAILED_ONLY | - **What follows the docs.** - Agent loops are children, with their own history, continue-as-new (`agent_workflow.py:2982-3069`) and control Updates; tools are activities. NodeInvocationWorkflow runs a plain node as an activity and only an agent as a child (`node_invocation.py:31-49`). No child is used as a function call. - Ids are deterministic and built from business keys (table). - Starts resolve before a parent can finish. Cron and listeners return after the start resolves (`cron_scheduler/_workflow.py:186-214`; `trigger_listener_workflow.py:493-497`). Task Manager runner starts run in asyncio tasks that are awaited in call order before the turn ends (`agent_workflow.py:2015-2021`, `:2374-2381`), and the rollover guard refuses continue-as-new while any is live (`:2993-2996`). An exception that escapes the tool loop leaves runner-start tasks unawaited. - Child failures: MachinaWorkflow turns a finished child's exception into a node failure (`workflow.py:1082-1133`); AgentWorkflow returns a failed delegation to the model (`agent_workflow.py:2574-2628`); DelegatedTaskWorkflow and EmployeeJobWorkflow record a terminal state and re-raise (`agent_workflow.py:692-702`; `employee_job_workflow.py:121-126`). - Timeouts: none on children, on purpose (`workflow.py:148-155`; `agent_workflow.py:186-191`): runs may stay paused for months and timeout timers keep running through a pause. This departs from "Set timeouts" deliberately (RFC-0005 §27). - Parallelism: delegations per turn are capped by `max_concurrent_subagents` (default 3, `agent_workflow.py:1451-1453`) and per root by a database permit (`server/services/temporal/agent_activities.py:2247-2339`). - Monitoring: runner and child ids are recorded by `agent.register_task_execution` (`agent_workflow.py:553-561`, `:601-609`, `:1748-1757`). Workflow types travel as strings on purpose, guarded by `server/tests/temporal/test_workflow_identity_golden.py` and `server/services/temporal/workflow_registry.py`. - **Where a parent closes early, so TERMINATE kills an agent without cleanup:** 1. *An uncontrolled run fails.* The first failed node ends the loop (`workflow.py:907-908`); the run completes with `success: False` (`:913-929`) and the server terminates its agent children. Manual canvas Runs are uncontrolled: execution-control fields are copied only for an admitted generation (`server/services/temporal/executor.py:152-173`; `server/routers/websocket.py:968-975`). 2. *Chat Stop outlasts its grace period.* Stop moves the run to `stopping` (`server/services/chat/ledger.py:598-639`). The watchdog cancels the run's MachinaWorkflow 45 to about 75 seconds later (`ledger.py:714-728`; `stop_grace_s` 45 and `watchdog_interval_s` 30 in `server/config/chat_defaults.json`). MachinaWorkflow waits in `workflow.wait_condition` (`workflow.py:1120`) with no `CancelledError` handler, closes Cancelled, and the server terminates its AgentWorkflow. A run that stops cooperatively inside the grace closes normally. 3. *Apply with "stop work".* `server/services/employees/safe_apply.py:112-116` cancels running executions found by `EventWorkflowId`. Agent children carry no search attributes (`workflow.py:820-822`), so only the parent's TERMINATE reaches them. Task Manager agents are the exception: their runner carries `EventWorkflowId` (`agent_workflow.py:1882-1891`), is cancelled directly and awaits its agent (`:610`), so that agent gets a real cancel. A plain agent in a tool call does not stop on it: it turns the cancelled tool call into a tool error and keeps working (`agent_workflow.py:2574-2582`; only Workspace and browser-runtime agents re-raise). `fixtures/0.3.1-cancel/task_runner_cancel` records this: the assignee answers and completes, and the runner records the delegation as finished. 4. *A child-start collision.* `WorkflowAlreadyStartedError` at `workflow.py:826` or `:828` is a `FailureError` (`temporalio/exceptions.py:53`) and fails the whole run instead of one node. Any other start failure becomes a `RuntimeError`, which fails the Workflow Task instead (`temporalio/worker/_workflow_instance.py:918-920`). 5. *An AgentWorkflow cancelled while its delegations run.* asyncio passes the cancel only to the handle being awaited. The other started children in `delegation_handles` (`agent_workflow.py:1467`) are terminated when the agent closes. `_cleanup_cancelled_delegations` (`:1518-1566`) settles the task rows and permits but cancels no handle. - **What termination skips.** AgentWorkflow's `fail_job(cancelled=True)` (employee runtime-v2 leads only, `agent_workflow.py:826-836`), its browser-claim release (`:842-857`), its delegation cleanup, and the cleanup of its own children. Consequences: - *Browser claims.* A claim left by a terminated agent keeps the profile `BrowserBusy` for every other task (`server/services/browser_owners.py:245-258`). Reset releases it only while the terminated execution is still retained (24 hours on the dev server) and only if it belongs to the generation being reset (`server/services/browser_reset.py:23-45`, `:60-75`). A manual canvas Run's claim, or one older than retention, is never released, and it also defers the profile cleanup of a deleted employee (`server/nodes/browser/__init__.py:94-98`). - *Subagent permits.* A leaked permit holds one of the root's delegation slots (default limit 3) for up to 3 hours; the next acquire for that root then releases it (`server/core/database.py:4143-4187`). - *Employee job rows.* A runtime-v2 lead started by MachinaWorkflow records its own failure only from workflow code (`agent_workflow.py:826-829`; `employee_job_workflow.py:28-66`), so its job stays non-terminal; no automatic repair exists. EmployeeJobWorkflow leads are settled by the parent's shielded `employee.job.failed` (`employee_job_workflow.py:121-126`) unless the parent itself is terminated. - Reset is not on this list: it terminates the whole tree on purpose (`server/services/deployment/handlers.py:974-1017`) after expanding by `RootWorkflowId` (`:862-900`); RFC-0004 H12 and D9 cover its cancel-first variant. - **Async Lambda: delegations from Claude Code and RLM.** `_execute_delegated_agent` runs a delegated agent as `asyncio.create_task(run_child_agent())` (`server/services/handlers/tools.py:889-890`) under a process-local semaphore (`:78-81`, `:475`). - From Claude Code, this path is reached only through Task Manager `assign_task` to a non-Coding teammate: the MCP bridge injects `ai_service` and `database` only for `coding_agent`, `taskManager` and `_builtin_check_delegated_tasks` with `pi_harness_version == 1` (`server/services/cli_agent/workflow_tools.py:337-355`). A Claude Code `delegate_to_*` call to any other agent already fails with "Agent delegation requires ai_service and database in config" (`tools.py:401-405`), a separate defect. - From RLM, both direct delegate tools and Task Manager reach it (`server/services/rlm/adapters.py:269-272`; Task Manager's in-process branch, `server/nodes/tool/task_manager/__init__.py:198-209`). - A Coding Agent target becomes durable through PiInlineDelegationWorkflow, but only when `pi_harness_version == 1` (`tools.py:378-386`). - The task runs in the process that handled the call: the API process for the `/mcp/ide` bridge (`server/main.py:406`, `:668`), the worker process of the RLM activity for RLM. It is lost on restart (the status lookup then answers `not_found`, `tools.py:1061-1062`), invisible to Temporal, untouched by Stop and Reset (`server/services/workflow.py:833-858` cancels CLI sessions only), its TeamTask row stays `running` (`server/core/database.py:3698`), and its permit self-heals only after 3 hours. **Verdict:** Adopted with deviations. Ids, awaited starts, ABANDON for per-event runs and the child-versus-activity split follow the docs. Two deviations: four agent-child starts keep TERMINATE although agents own cleanup (browser claims, permits, job rows), and delegations from Claude Code and RLM run as Async Lambdas. **Recommendation:** 1. **R1. Agent children get REQUEST_CANCEL.** Pass `parent_close_policy=workflow.ParentClosePolicy.REQUEST_CANCEL` and an explicit `cancellation_type=workflow.ChildWorkflowCancellationType.WAIT_CANCELLATION_COMPLETED` at the six call lines: `workflow.py:826` and `:828` (through `child_start_kwargs`, `:820-822`), `agent_workflow.py:595-598` and `:1740-1744`, `employee_job_workflow.py:117` and `:120`. The Workspace agent child already does this (`node_invocation.py:34-39`). - Versioning: none. On replay, core compares only a child's id and type (and an activity's id and type); the parent close policy is never compared. The checks are in the core sources shipped in the 1.30.0 wheel, under `temporalio/bridge/sdk-core/crates/sdk-core/src/worker/workflow/machines/`: `child_workflow_state_machine.rs:233-258` and `activity_state_machine.rs:371-394`. The docs agree: changing "The Child Workflow Options in a call to spawn a Child Workflow Execution" is listed among changes "that would not result in non-determinism errors" (https://docs.temporal.io/workflow-definition). The golden identity test is unchanged, because it captures only the `id` and `activity_id` keywords (`server/tests/temporal/workflow_identities.py:36-40`). `test_replay_corpus.py` must stay green. - When: the cancellation lane (§22.4), together with a re-raise of a cancelled tool call and LLM step in AgentWorkflow (`agent_workflow.py:2574-2578`, `:1260-1264`) under `agent-delegation-drain-v1`. The owner moved it out of Phase 1b on 2026-10-25 (1b.12). Without the re-raise, a plain agent that receives the cancel keeps calling the model and tools after its parent closed, where TERMINATE stops it today. RFC-0004 decided R1 (D9, RFC-0004 §8.4). Effort S, plus the re-raise. - Risk: a closed parent now leaves its child running its cleanup for a while (see Remarks on id reuse). 2. **R2. Cancel and drain.** On cancellation, MachinaWorkflow cancels every handle in `running` and waits for them, with a bound (sketch); AgentWorkflow's `_cleanup_cancelled_delegations` does the same for every started handle in `delegation_handles` before its database cleanup. With the same patch, node activities get `cancellation_type=WAIT_CANCELLATION_COMPLETED`, so the drain waits for the activity's own cleanup. §6.3 rec 1 adds the failure half of the same patch. - Versioning: `workflow.patched("machina-close-and-drain-v1")` in MachinaWorkflow, read once before the scheduling loop, and `workflow.patched("agent-delegation-drain-v1")` in AgentWorkflow. The cancellation-type change must ship inside the same patch: a cancellation type is not compared on replay, but it decides whether a cancel emits a RequestCancel command and when the workflow sees the outcome, and core raises a nondeterminism error when a non-ABANDON activity or child records a start or completion after its cancel ("Non-Abandon cancel mode activities cannot be completed after being cancelled", `activity_state_machine.rs:779-781`; the child equivalent at `child_workflow_state_machine.rs:146-149`). Here core is stricter than the workflow-definition page, which lists activity and child option changes as safe. The repo already gated the same change for Workspace nodes behind `workspace-node-cancellation-v1` (`workflow.py:856-861`). - When: P2, in a cancellation lane after Phase 1 (which ends with the 1.34.0 upgrade and its 1.31 and 1.33 cancellation fixes) and Phase 1b, and before RFC-0005 Phase 4, whose Interrupt and 24-hour no-progress backstop (a default, §24.1; §6.4 rec 2) cancel a turn and expect its agents to clean up. 1b.11 recorded six cancel-path scenarios from 0.3.1 into `server/tests/temporal/fixtures/0.3.1-cancel/`: - a root cancelled while its agent child runs, uncontrolled (the chat Stop grace path) and controlled; - an uncontrolled branch failure (a manual Run with two branches, one failing); - a cancelled Task Manager runner, whose assignee swallows the cancel; - the same runner cancelled while its generation is stopped, whose assignee ends cancelled; - a stopped team job cancelled before its lead starts. The corpus still has no ActivityTaskFailed, because node failures complete with a failure envelope. Effort M. - Risk: a node that never sees the cancel holds the run until the bound. The drain assumes node activities report cancellation; today `NodeExecutor` turns `CancelledError` into a `{success: False, error: "Cancelled"}` result (RFC-0004 H2), which resolves the handle but records the activity as Completed. 1b.2 (§9.3) fixes H2 in Phase 1b, before this lane. 3. **R3. Durable delegations from Claude Code and RLM** (default chosen; §24 lists the alternative). When Temporal is on, every delegation reached through `_execute_delegated_agent` starts a durable runner instead of an asyncio task: a new workflow type `InlineDelegationWorkflow` that generalizes PiInlineDelegationWorkflow's admission (an admission row with a request digest, a deterministic runner id from root, parent, target and call, `REJECT_DUPLICATE`, `memo` with the digest; `pi_inline_delegation.py:120-230`) to every agent type in `AGENT_WORKFLOW_TYPES`, starts `DelegatedTaskWorkflow` as its child with REQUEST_CANCEL (as `pi_inline_delegation.py:365-369` does), and is started with `EventWorkflowId` so Reset and Apply find it (RFC-0004 H12 notes the Pi inline runner lacks it). It must cover both entry paths: Task Manager's in-process branch and the direct `delegate_to_*` tools; the Claude Code bridge then passes `database` and the trusted scope for those calls too. - Versioning: none (a new workflow type; old runs never meet it). - When: P2, an independent lane after Phase 1, before hosted tier S splits workers into a second process (T8): process-local tasks and semaphores do not survive that split. Effort L. - Risk: admission and permission checks (`assert_runtime_access`) must run on the trusted scope, not on model-supplied arguments. 4. **R4. The Task Manager runner rejects duplicates.** Start it with `id_reuse_policy=WorkflowIDReusePolicy.REJECT_DUPLICATE` (`agent_workflow.py:1878-1881`), like every other do-once start (`server/services/employees/jobs.py:240-243`, `pi_inline_delegation.py:219-226`, `workspace_tasks_workflow.py:118`), and treat `WorkflowAlreadyStartedError` as "already queued". Versioning: the policy needs none; the new exception handling changes commands only after a failed start, so ship it under the next AgentWorkflow patch. When: P3. Effort S. Risk: none known; duplicates are near impossible because runner ids include the iteration and the task id. 5. **R5. Agent children carry `EventWorkflowId`.** Pass `search_attributes=event_workflow_search_attributes(workflow_id)` on agent child starts. Only safe-apply's stop-work sweep then reaches them directly (`safe_apply.py:112-116`); the manager's sweep filters to the two listener types (`server/services/deployment/manager.py:1176-1185`). Versioning: none (an option). When: P3. Effort S. Risk: an agent child may get two cancel requests (direct and from its parent), which is harmless. 6. **R6. A failed child start fails only its node.** Catch `WorkflowAlreadyStartedError` around `workflow.py:824-829` and record it as that node's failure. Versioning: it changes commands only after a failed start, which fails the run today, so put it under `machina-close-and-drain-v1`. When: P2 with R2. Effort S. Risk: only `WorkflowAlreadyStartedError` is caught; other start failures still fail the Workflow Task and stay visible. **Remarks:** - R1 and R2 are both needed. REQUEST_CANCEL sends the cancel when the parent closes and does not wait, so the parent's result cannot reflect the child's cleanup. The drain alone does not cover parents that close without running code: termination, or a failure path that skips the handler. - Replay never re-sends a recorded command. Children started before R1 ships keep TERMINATE; only children started afterwards get REQUEST_CANCEL; an in-flight run can hold both. - AgentWorkflow already runs its cleanup after catching `CancelledError`, shielding the activities (`agent_workflow.py:826-859`). That is Python's form of "cleanup outside the canceled scope". It runs only when a cancel reaches the workflow while it is not in a tool call or LLM step that turns the cancel into an error result. R1 sends the cancel; the cancellation lane's re-raise makes it reach the cleanup. In `fixtures/0.3.1-cancel/`, `task_runner_cancel` shows the swallow, and `task_runner_paused_cancel` shows an assignee that waits for Resume ending cancelled. - Id reuse after R1. Per-event roots use ALLOW_DUPLICATE_FAILED_ONLY, so a Cancelled or Terminated root id can run again. If its old agent child is still cleaning up, the new start of `-agent-` raises `WorkflowAlreadyStartedError` and, until R6, fails the new run. The controller treats a duplicate start as accepted when the accumulator is on (`workflow_control_workflow.py:468-472`), so repeats are rare; R2's drain closes the window. - The parent close policy also applies when a parent continues as new. AgentWorkflow's rollover guard refuses to continue while delegations or runner starts are live (`agent_workflow.py:2982-2996`); keep it after R1, or a rollover would cancel live delegations. - Pending-child limit: only the controller's count follows the event rate. Whether ABANDON children count as pending while the parent runs was not verified; the docs' "incomplete Child Workflows" suggests they do. At 2,000 the start command fails and the workflow task retries until a child closes, stalling the employee's triggers. Not reachable at today's volumes (unmeasured); RFC-0005's one turn per conversation bounds it by design. - Tests that pin today's policies: `server/tests/temporal/test_agent_workflow.py:394` (runner ABANDON) and `server/tests/temporal/test_browser_task_dispatch.py:28` (REQUEST_CANCEL). Unit tests that assert agent-child start options may need updating. - If a release ever ships without a needed patch, T14 starts it anyway and marks the affected generations "needs Reset". The patches above keep open runs resumable instead. **Sketch** (R2 in MachinaWorkflow, SDK 1.30.0; AgentWorkflow's delegation drain has the same shape): ```python # server/services/temporal/workflow.py, MachinaWorkflow._run_graph DRAIN_PATCH = "machina-close-and-drain-v1" _DRAIN_BOUND = timedelta(minutes=2) drain_v1 = workflow.patched(DRAIN_PATCH) # read once, before the scheduling loop ... if drain_v1: # Same patch: a replayed history can only hold a cancel of this activity if it has the marker. start_kwargs["cancellation_type"] = workflow.ActivityCancellationType.WAIT_CANCELLATION_COMPLETED ... try: while True: ... # the existing scheduling loop (workflow.py:647-911) except asyncio.CancelledError: if drain_v1 and running: for handle in running.values(): handle.cancel() # activity: RequestCancelActivityTask; agent child: cancel request, then wait try: await workflow.wait_condition( lambda: all(h.done() for h in running.values()), timeout=_DRAIN_BOUND, # a node that never sees the cancel must not hold the run ) except asyncio.TimeoutError: workflow.logger.warning("Cancel drain bound reached; closing with work still cancelling") raise # controlled runs then unregister in run()'s finally (workflow.py:503-513) ``` ### 6.3 Parallel Execution **Docs:** https://docs.temporal.io/design-patterns/parallel-execution. Start every independent operation, then wait for the results. - Python: `asyncio.gather(*[workflow.execute_activity(...) for item in items])`; "Avoid premature blocking. Schedule all Activities before waiting for any results." - Controlled parallelism: chunks of `max_parallel`, each awaited before the next ("Limit concurrency"). - Errors: `asyncio.gather(..., return_exceptions=True)`; waiting for all "fails on the first error by default"; "Support partial results." - Limits and pitfalls: at most 2,000 pending activities ("Scheduling more causes Workflow Task failures"), counted separately from children, Signals and cancellation requests; "Blowing the 4 MB gRPC message limit ... if their combined inputs are large. Batch scheduling across multiple Workflow Tasks"; history growth toward 51,200 events; Workflow memory for many futures; "Parallel execution makes tracing more difficult"; "Rate limit". - Not a fit for dependent operations or strict ordering. - The in-flight figure "aim for 500" is on the Batch Processing page, not this one (https://docs.temporal.io/design-patterns/batch-processing-patterns). **SDK:** activity and child handles are asyncio tasks (`temporalio/workflow/_activities.py:61`, `_workflow_ops.py:53`), so `asyncio.gather` works. `workflow.wait` and `workflow.as_completed` are the deterministic replacements for the asyncio versions (`temporalio/workflow/_asyncio.py`, the same in 1.34.0 apart from line endings). Paginated workflow-task completions (SDK 1.33 with server 1.32, behind `history.enableWorkflowTaskCompletionPagination`, default false) are the only server-side relief for the 4 MB limit; the recombined size must still fit, and an over-limit completion then fails the Workflow Task and retries forever instead of terminating the execution. T5 removes the server-version gate, not the size rule. 1.34.0's experimental Event Groups could group a turn's commands in the UI; RFC-0004 Q12 keeps preview features off. **OpenCompany today:** - **MachinaWorkflow schedules continuously.** Ready nodes come from `_find_ready_nodes` in graph order (`server/services/temporal/workflow.py:1244-1264`), all start at once (`:674-874`), and completions are taken one at a time by `wait_condition` and an ordered scan (`_wait_any_complete`, `:1101-1133`). This beats the docs' chunks (no wait for a chunk's slowest member) and is deterministic: insertion order, never `asyncio.wait`. - **No in-flight bound.** `running` grows to every ready node (`:640-874`); only the graph's width limits it. - **Inputs carry the graph.** Every node input embeds `nodes`, `edges`, the whole `parameter_snapshot` and upstream outputs (`workflow.py:708-724`); agent tool inputs embed them too (`server/services/temporal/agent_workflow.py:2170-2195`). K ready nodes in one task send K copies in one completion. Recorded toy graphs are small: AgentWorkflow start 1,738 bytes and the `node.chatReply.v1` input 1,612 bytes (`server/tests/temporal/fixtures/0.3.1/machina_chat_agent.json`); real sizes are unmeasured. Already RFC-0004 M6 and D7; RFC-0005's turn loads the graph by reference (RFC-0005 §10.2). - **Three failure policies.** Uncontrolled Temporal runs (manual canvas Runs and generations without execution control) stop at the first failure and abandon running siblings (`workflow.py:907-908`). Controlled runs close admission and let in-flight branches finish, so independent nodes not yet started never start (`:656-657`). The in-process executor cancels the remaining tasks and waits for them (`server/services/execution/executor.py:441-458`). - **A turn's tool calls run one at a time** (`agent_workflow.py:2035-2369`; the `await _execute_plugin_tool_activity(...)` at `:2363-2369`). Concurrency exists only for Task Manager `assign_task` preflights (`start_activity` plus `asyncio.gather(..., return_exceptions=True)`, `:1903-1992`, the page's error-handling shape) and `delegate_to_*` children. The in-process loop is sequential too (`server/services/agent_runtime.py:359-388`). 39 lines under `server/nodes` declare `"readonly": True` in plugin `annotations` (for example `server/nodes/filesystem/file_read/__init__.py:45`), but `agent.prepare_payload` records no annotations. - **The delegation window is ordered, not sliding,** although its comment calls it "a bounded sliding window" (`agent_workflow.py:1442-1446`). Handles are awaited in model order. On the uncontrolled path the next delegation starts only after the earliest call finishes without raising (`:2259-2270`, `:2343-2348`); the controlled path starts it after any completion (`:2666-2668`). Most child failures come back as `success: False` results and do start the next one; a child that raises (terminated, cancelled, or failed by an uncaught error) blocks it. The never-started delegation's call then reports `KeyError` (`:2259`, caught at `:2574`). For a team lead, `agent.finish_delegation` is then called for a task that was never queued; it raises "Delegated team task does not exist" three times (`server/services/temporal/agent_activities.py:2383-2384`), and the lead AgentWorkflow fails. Uncontrolled path only. - **Controller poll loops.** One asyncio task per polling trigger (`server/services/temporal/workflow_control_workflow.py:332`, `:526-528`) catches and logs its own errors (`:849-866`); rollover cancels and gathers them (`:612-615`). This is not the docs' isolation: a cancel that lands while a poll awaits its activity arrives as `ActivityError`, which `except Exception` swallows, so the gather can wait forever and stall the rollover (RFC-0004 H8). Legacy controllers are fully exposed and v1 controllers in a narrower window (§8.3). Under T7, polls leave the controller for new generations. - Results are ordered deterministically: tool messages in model order, preflight results zipped in order (`:1980-1989`), delegations awaited in order. **Verdict:** Adopted with deviations. The scheduling loop is better than the docs' sample; the deviations are three failure policies, no bound, inputs that copy the graph, a delegation window that can fail a team lead, and poll loops that swallow a cancel. **Recommendation:** 1. **One failure policy for graph runs** (default chosen; §24 lists the alternative): when a node fails, running siblings finish and no new node starts. This is what controlled runs already do. Uncontrolled MachinaWorkflow runs adopt it under the same marker as §6.2 R2 (`machina-close-and-drain-v1`): with the marker, a failure sets `ready = []` instead of `break` (sketch). The in-process executor stops cancelling (`executor.py:441-458`) and waits for running tasks without starting new ones. - Versioning: the MachinaWorkflow change is inside §6.2 R2's patch; the in-process change needs none. Ship both in the same release. - When: P2, the cancellation lane with §6.2 R2. Effort M. - Why this default: T4 puts reliability first, and T13 records a side-effecting call that broke off as `unknown` (§11.2, 1b.1). Cancelling running siblings when an unrelated node fails turns known outcomes into unknown ones; starting nothing new stops further side effects. - Risk: a failed run keeps spending on branches whose result is no longer used. 2. **Graph by reference before any added parallelism.** RFC-0004 M6 and RFC-0005 §10.2 already plan it for turns. Every concurrent command multiplies the embedded graph and parameter snapshot, so items 4 and 5 wait for it. Versioning: the turn's frozen input flag (`event_os_version=1`, RFC-0005 §10.4). When: P2, RFC-0005 Phase 4. Effort M (planned). Risk: activities read the graph from the database, so a generation's snapshot must stay immutable while its runs are open. 3. **Fix the delegation window on the uncontrolled path.** Start the next delegation after any outcome of the one being awaited, failure included, as the controlled path does (`agent_workflow.py:2666-2668`); never call `agent.finish_delegation` for a delegation that was never queued; correct the "sliding window" comment. The same patch refills on any completion (§12.4 rec 3): wait with `workflow.wait(..., return_when=asyncio.FIRST_COMPLETED)` on the started delegations and start the next as soon as any of them finishes, while still appending results in call order. Today both paths start the next delegation only after the one awaited in call order completes (`agent_workflow.py:2343-2348`, `:2666-2668`). - Versioning: `workflow.patched("agent-delegation-window-v2")` (command order changes after a failed delegation, and whenever a later delegation finishes first). - When: P2, the cancellation lane, not later: for any team lead it fails the whole lead AgentWorkflow. Effort S. Risk: low. §12.4 rec 3 refers here. 4. **Cap in-flight nodes per run.** Take at most N ready nodes (for example 100, under the batch page's "aim for 500"). It changes commands only for graphs wider than N; gate it with a frozen input flag. When: P3. Effort S. Risk: a very wide graph runs slower; no graph today comes near the bound (unmeasured). 5. **Read-only batches run concurrently** (default chosen; §24 lists the alternative). When every call in one model response is a tool whose plugin declares `annotations["readonly"] is True`, has no approval spec and parsed cleanly, start them together (at most 8) and append results in call order; any other batch runs one at a time as today. `agent.prepare_payload` records each tool's annotations in the prepared tool index, and a per-generation flag, `tool_concurrency_version=1` in the generation's `resource_manifest`, frozen into the run's input, turns the batching on (RFC-0005 §24.1 rule 2). - Versioning: `tool_concurrency_version=1` (command order changes for such batches). No run-level version is recorded in `agent.prepare_payload`'s result, since T18 leaves rule 2 unamended; only new generations batch. - When: P3, after item 2. Effort M. - Risk: the batch is only as safe as the plugins' `readonly` annotations; each concurrent call must also be admitted by execution control (`begin_action`) so Stop's checkpoint counts it. **Remarks:** - Keep the FIRST_COMPLETED loop: the docs' chunked "controlled parallelism" makes every chunk wait for its slowest member. - Several completions in one activation are handled in insertion order, one per loop pass, with new starts in between; replay produces the same order. - RFC-0004 §8.1 item 3 (replace `_wait_any_complete` with `workflow.wait(..., return_when=FIRST_COMPLETED)`) changes no commands; do not bundle it with a command change. - Pending activities stay far below 2,000 (graph width; sequential tools). Preflights are one per `assign_task` call in a turn, which the model controls; a per-turn cap (for example 20) is cheap insurance (model behaviour unverified). - Per-call error isolation already holds in AgentWorkflow: a failed tool becomes an error tool message (`agent_workflow.py:2574-2628`). - Tracing: activity ids are node ids (`workflow.py:847`) and stable tool ids (`agent_workflow.py:2350-2354`). - An execution whose task completion exceeds 4 MB is terminated ("the Temporal Service terminates the Workflow Execution", https://docs.temporal.io/troubleshooting/blob-size-limit-error). T6 keeps agent transcripts in LLM-step inputs, but only one LLM step per agent runs at a time; item 5 is where copies multiply. **Sketch** (items 1 and 5, SDK 1.30.0; helper names are illustrative): ```python # (1) MachinaWorkflow: one failure policy, under §6.2 R2's marker (drain_v1) if errors and (self._execution_control.enabled or drain_v1): ready = [] # start nothing new; running nodes finish ... if not result.get("success"): errors.append(error_info) if not self._execution_control.enabled and not drain_v1: break # pre-patch histories: abandon running siblings # (5) AgentWorkflow: a model batch that is entirely read-only runs at once readonly_batch = ( run_input.get("tool_concurrency_version") == 1 # the generation's flag, frozen at admission and 1 < len(calls) <= _MAX_READONLY_BATCH # 8 and all(_is_readonly_call(tool_index, call) for call in calls) ) if readonly_batch: handles = [] for index, call in enumerate(calls): info = tool_index[call["name"]] await self._execution_control.begin_action() # Stop's checkpoint counts each call handles.append(workflow.start_activity( _tool_activity_name(info), args=[_tool_payload(index, call)], activity_id=_tool_activity_id(info["tool_node_id"], iteration, index), **_tool_options(info))) outcomes = await asyncio.gather(*handles, return_exceptions=True) for index, (call, outcome) in enumerate(zip(calls, outcomes)): self._execution_control.end_action() _append_tool_outcome(messages, index, call, outcome) # model order; today's error text else: ... # today's sequential loop ``` ### 6.4 Pick First (Race) **Docs:** https://docs.temporal.io/design-patterns/pick-first. Run several activities that reach the same goal, take the first result and cancel the rest. - Python: `asyncio.create_task(workflow.execute_activity(...))` per candidate; `done, pending = await workflow.wait([task1, task2], return_when=asyncio.FIRST_COMPLETED)`; cancel each task in `pending`; return `done.pop().result()`. - Activities heartbeat and re-raise `asyncio.CancelledError`: "Activities do not cancel instantly (they wait for the next heartbeat)". - Cleanup variant: `cancellation_type=workflow.ActivityCancellationType.WAIT_CANCELLATION_COMPLETED`, then await each pending task, swallowing `CancelledError`. - Pitfalls: missing heartbeats; "Not waiting for cancellation cleanup"; "Ignoring errors from the winning Activity"; "Forgetting to cancel remaining Activities", which "continue running indefinitely". - Not a fit when "Activities have side effects that should not be cancelled". **SDK:** `workflow.wait` returns ordered lists and rejects coroutines ("Passing coroutines is forbidden, use tasks explicitly", `temporalio/workflow/_asyncio.py:89-118`). Handles have `.cancel()`. `ActivityCancellationType` is TRY_CANCEL (default), WAIT_CANCELLATION_COMPLETED or ABANDON (`temporalio/workflow/_activities.py:71-82`), unchanged in 1.34.0, which also de-duplicates a repeated activity cancel command. The worker throttles heartbeats to min(0.8 x heartbeat timeout, 60 s) by default (`temporalio/worker/_worker.py:125-126`); OpenCompany does not override it. **OpenCompany today:** - **No race in workflow code:** no `workflow.wait` or `asyncio.wait` in any workflow. The one hit in a workflow module (`server/services/temporal/employee_job_workflow.py:20`) is inside the activity `deliver_employee_job`. - **In-activity races follow the docs.** `_until_chat_run_stops` races the model call against a 1-second database stop watcher, cancels the loser in `finally` and gathers both, so the winner's error surfaces through `task.result()` (`server/services/temporal/agent_activities.py:419-444`). Its docstring states the docs' trade-off: "a heartbeat-borne cancel arrives too late to stop a stream mid-sentence". `_await_with_llm_heartbeats` (`:378-398`) and `deliver_employee_job` (`employee_job_workflow.py:11-27`) clean up the same way. - **Chat Stop** is a cooperative race against a grace timer: the run moves to `stopping`, the model step returns its partial answer and unstarted tools are skipped; 45 to about 75 seconds later the watchdog cancels the run's workflow (`server/services/chat/ledger.py:598-639`, `:714-728`). The shape is right; the cancel reaches only the root, whose agent child is then terminated (§6.2). - **The chat run timeout cancelled nothing** (until 1b.14, e6fb360a, which cancels the run's workflow first). A running run past `max_running_s` (86,400 s) is failed as `timed_out` (`ledger.py:766-767`), and `_sweep_one` is not even given `temporal_cancel` (`:701-703`; compare `:722-726`). `is_stopping` matches only `stopping` and `stopped` (`:642-645`), so the LLM stop watcher and the tool skip never see the `error` run, and `post_reply` still writes for a terminal run (`:334-370`). The workflow keeps running, may keep billing model calls, and the freed lane admits a second run on the same conversation: the page's "Forgetting to cancel remaining Activities", at run level. Detection takes 24 hours plus up to one 30-second sweep; the clock restarts at Resume and paused controlled runs are skipped (`:742-765`). - **Uncontrolled failure:** the first failure "wins" and siblings are not cancelled (§6.3). - **Pi driver, the docs' cleanup variant:** `_await_request` waits for "operation recorded or driver done"; `_stop_driver(cancel=True)` cancels the driver activity, which runs with WAIT_CANCELLATION_COMPLETED, and awaits it (`server/services/temporal/pi_workflow.py:180-216`). - **Not races, correctly:** the Undo window is settled by a database claim at the approval revision, and a started send is never cancelled (`server/services/temporal/approved_tool_call_workflow.py:41-75`), as the page advises for side effects. Workspace Take over is a process-local lease (`server/nodes/browser/_live_control.py:300-352`, `server/nodes/mobile/_control.py:30-50`). `wait_for_delegation` lets only the waiter time out and never cancels the child (`server/services/handlers/tools.py:946-976`). - **Planned in RFC-0005:** turn deadlines per class (RFC-0005 §13.1: 15, 15, 30 and 60 minutes), the turn as a child with REQUEST_CANCEL and WAIT_CANCELLATION_COMPLETED (RFC-0005 §10.1), Interrupt that cancels a turn "at its next merge point" with its inputs returned to pending (RFC-0005 §13.4), and "Turn hangs" (RFC-0005 §23). Missing: 1. Turn n+1 must not start until turn n's handle is done, or two turns share one conversation key (I4). 2. A bound on that wait, and a last resort when cleanup hangs. A workflow can only request cancellation of another workflow; termination needs an activity that calls the client. 3. The winner may be a failure: a failing turn and a firing deadline in one activation need a deterministic rule. 4. A turn's clocks must stop while the generation is paused (the chat watchdog restarts its window at Resume; Temporal timers keep running through a pause, `workflow.py:148-155`) and while the turn is parked (T11). That makes each an updatable timer. T19 also turns the per-class deadlines into "slow turn" alerts; cancelling is left to a 24-hour no-progress backstop, a default (§24.1; recommendation 2). 5. Cancelling a turn helps only if the turn passes the cancel to its agents (§6.2 R1 and R2). 6. A cancelled turn's agent can still finish a model step and save after turn n+1 started; the consumption transaction must check that the saving turn still holds the conversation. **Verdict:** Adopted with deviations. In-activity races follow the docs; the run-level timeout left its loser running until 1b.14. The RFC-0005 turn as a child follows the docs; its turn-cancel race is Planned in RFC-0005 (deviates) on the six points above. **Recommendation:** 1. **A chat run that hits its 24-hour limit cancels its workflow** (default chosen; §24 lists the alternative). Done (1b.14, e6fb360a). Pass `temporal_cancel` to `_sweep_one` and, in its `timed_out` branch, cancel `run.temporal_workflow_id` and `run.temporal_run_id` as `_sweep_stopping` does (`ledger.py:714-728`). This 24-hour limit covers today's chat runs; time waiting on an approval does not count (RFC-0007 D2, §4.7). RFC-0005 turns are bounded by the 24-hour no-progress backstop (a default, §24.1; recommendation 2), which never counts time parked, paused or waiting for the owner; from RFC-0005 Phase 5 (`scheduler_version=1`) a parked turn waits at most 7 days (§11.8; T18, T19). - Versioning: none (client code). - When: Phase 1b (1b.14). Until the cancellation lane ships 1b.12, the server terminates the root's agent children when the cancelled root closes; the lane's re-raise then stops a plain agent on the cancel. Effort S. - Why this default: T4. A run that keeps going after its run row ended is a second writer on the conversation and an unbounded spend. - Risk: an answer that would have arrived after 24 hours is lost; the owner sees `timed_out`, as today. 2. **Specify the turn's clocks and its cancel race in RFC-0005** (RFC-0005 §10.2, §13.1, §13.4, §18.1, §23; sketch): - T19: the per-class turn deadlines (15, 15, 30 and 60 minutes for C1 to C4) become "slow turn" alerts. The entity raises an attention item and a metric, and cancels nothing. - A backstop cancels a turn after 24 hours with no completed step (default chosen, §24.1). It never counts time the turn is parked, paused or waiting for the owner (an approval draft, the owner's browser). Each completed step restarts it. - Both clocks are named deadlines on §8.4's one live timer, and both stop while the generation is paused or the turn is parked, so parked time never counts toward either. Parking exists only for generations with `scheduler_version=1` (RFC-0005 Phase 5, T18); a parked turn waits at most 7 days, then its input is dead-lettered with an attention item (§11.8, T19). - A turn that finished, even by failing, in the same activation as the backstop wins. - Otherwise the entity cancels the turn, waits up to `_TURN_CANCEL_GRACE` (2 minutes) for it to close, then terminates the turn and its agent children by id through an activity; only then may turn n+1 start. Interrupt uses the same wait. - Versioning: none (ConversationWorkflow is a new type). - When: P2, RFC-0005 Phase 4, after §6.2 R2; the parked-time rules arrive with parking in Phase 5 (T18). Effort M. 3. **Fence consumption by the turn's lease.** The transaction that saves a transcript and advances `consumed_seq` (RFC-0005 §9.5 step 5) checks that the saving turn still holds its `sched_leases` row (RFC-0005 §13.3); a cancelled or terminated turn's late save is refused. Versioning: none (a database transaction). When: P2, RFC-0005 Phase 4 design text. Effort S. Risk: the refused step's model spend is already settled; only its transcript is dropped. **Remarks:** - Heartbeat delivery bounds every cancel. Node activities heartbeat every 30 seconds from a background task when start-to-close exceeds the heartbeat timeout (`server/services/plugin/base.py:1248-1263`), and the 2-minute heartbeat timeout gives a 60-second throttle. A cancel rides the next heartbeat response, so it reaches a node activity about 0-60 seconds after it is sent (about 30 seconds on average), and an LLM step (1-minute heartbeat timeout, `server/services/temporal/agent_workflow.py:99`) within about 48 seconds. That is why the chat path polls a 1-second stop flag. - The 24-hour start-to-close (`workflow.py:167`) also ends a wedged node attempt, after which the retry policy may run it again (3 attempts by default). A drain or a turn-cancel wait still needs its own bound. - Side effects: on Stop, Interrupt or the no-progress backstop, an in-flight send is cancelled like any other activity. T13 then records it as `unknown` (1b.2, §9.3; the redo guard is 1b.1, §11.2), unless its plugin is idempotent or uses a provider idempotency key. - Record a replay fixture of the chat Stop grace path before changing any cancel path (1b.11). **Sketch** (the turn's clocks inside ConversationWorkflow: RFC-0005 Phase 4, parking from Phase 5; SDK 1.30.0): ```python # self._deadlines is §8.4's helper: one live timer for all of the entity's named deadlines. turn = await workflow.start_child_workflow( "MachinaWorkflow", turn_input, id=f"turn-{self._conv_id}-{self._turn_no}", # number from admission (§8.2) parent_close_policy=workflow.ParentClosePolicy.REQUEST_CANCEL, cancellation_type=workflow.ChildWorkflowCancellationType.WAIT_CANCELLATION_COMPLETED, ) now = workflow.now() self._deadlines.set("slow_turn", now + _SLOW_TURN[self._turn_class]) # 15/15/30/60 min: alert only self._deadlines.set("no_progress", now + _NO_PROGRESS) # 24 h, no completed step (§24.1) while True: due = await self._deadlines.wait(turn.done) if turn.done(): # checked first: a turn that ended, even by failing, wins outcome = self._turn_outcome(turn) break if "slow_turn" in due: # a latency target (T19): attention item and metric, no cancel self._deadlines.set("slow_turn", None) await self._report_slow_turn() if "no_progress" in due: turn.cancel() # its agents clean up only with §6.2 R1 and R2 try: await workflow.wait_condition(turn.done, timeout=_TURN_CANCEL_GRACE) except asyncio.TimeoutError: # a workflow cannot terminate another one: an activity does await workflow.execute_activity( "conversation.terminate_turn", {"turn_id": turn.id}, # the turn and its agent children start_to_close_timeout=timedelta(seconds=60), retry_policy=DEFAULT_ACTIVITY_RETRY) outcome = "no_progress" # RFC-0005 §23, "Turn hangs": the turn is cancelled, then the retry policy break self._deadlines.set("slow_turn", None) self._deadlines.set("no_progress", None) self._release_turn(outcome) # only now may turn n+1 start (I4) # Handlers only move these deadlines; the helper re-arms on the new minimum (§8.4): # - a completed step (the progress RFC-0005 §18.1's stuck-turn alarm watches): no_progress = now + 24 h; # - a pause, or a park (T11, from Phase 5): both are removed and their remaining time is kept; # - a wait for the owner (an approval draft, the owner's browser): no_progress alone, the same way; # - when the pause, park or wait ends: each is set again at now + its remaining time. ``` ## 7. Workflow messaging Three pages cover how callers talk to running workflows. Terms used below: - A **Signal** is a one-way message to a workflow. The server records it and returns; the caller gets no answer and the workflow may handle it later. - An **Update** is a message that the workflow validates, records and answers. A **validator** is a read-only check that runs before the Update is written to history; when it raises, the Update is rejected and leaves no history. - **Signal-With-Start** (SWS) starts a workflow if none is running under the id and delivers a Signal to it, in one call. **Update-With-Start** does the same with an Update. - **Continue-as-new** closes the current run and starts a fresh one under the same id with new input, to keep history bounded. ### 7.1 Workflow messaging patterns (overview) **Docs:** https://docs.temporal.io/design-patterns/workflow-messaging-patterns. A chooser: "You want to send a message without checking whether the Workflow is running": Signal with Start. "You need a result back from the Workflow, with validation": Request-Response via Updates. "You receive many events and want to process them in batches": Event Accumulator. The rules come from the three pattern pages and the concept pages: - **Order.** In each activation, handlers run before the main method, including the first activation after Signal-With-Start or continue-as-new (https://docs.temporal.io/handling-messages). - **Exceptions.** A Signal handler that raises `ApplicationError`, or lets an activity exhaust a finite retry policy, fails the whole workflow; any other exception fails the Workflow Task, and "the Workflow will get stuck". In an Update handler, a validator exception rejects the Update; an `ApplicationError` or an exhausted activity or child fails only that Update; in the Python SDK any other exception fails the Workflow Task too (only a `FailureError` or `asyncio.TimeoutError` fails just the Update). - **Dedupe.** Updates are deduplicated by Update id "per Workflow run"; Signals need an application key. Neither request ids nor Update ids survive continue-as-new. - **Updates need a worker.** With none polling, "Your request will be retried by the SDK Client indefinitely. You can use asyncio.timeout to impose a timeout" (https://docs.temporal.io/develop/python/workflows/message-passing). - **Limits.** 10 in-flight and 2,000 total Updates per run; the server suggests continue-as-new at 90% of the 2,000. `history.maximumSignalsPerExecution` defaults to 10,000; the dynamic-config reference says further Signals fail with InvalidArgument, while the event page says the execution is terminated. The docs disagree; either way, continue as new first. Throughput: "below 5/sec per workflow instance" (accumulator page). - **Closing.** "Complete all handlers before using Continue-as-New"; "Use Continue-as-New from your main Workflow Definition method". **SDK:** everything this section needs exists in 1.30.0: `Client.start_workflow(..., start_signal=, start_signal_args=, id_reuse_policy=, id_conflict_policy=, priority=, rpc_timeout=)` (`temporalio/client/_client.py:527-565`); `execute_update_with_start_workflow` and `WithStartWorkflowOperation`; `WorkflowUpdateStage`; `workflow.all_handlers_finished`; `HandlerUnfinishedPolicy`; `workflow.current_update_info`; `Info.is_continue_as_new_suggested`, `get_current_history_length` and `get_current_history_size`; `Priority(priority_key, fairness_key, fairness_weight)`. 1.33.0 made continue-as-new from an Update handler fail the Workflow Task (OpenCompany never does it) and moved Update-With-Start to Core's long-poll timeout. 1.34.0 removes the private `start_workflow` keywords `callbacks`, `links` and `request_id` (unused here) and adds `WorkflowAlreadyStartedError.first_run_id` (server 1.32 or later) and `Info.original_execution_run_id`. **OpenCompany today:** - **Events are Signals routed through Visibility.** `dispatch.emit` runs one Visibility query per event and sends a plain `on_event` Signal by workflow id (`server/services/events/dispatch.py:155-165`, `:251-258`); it drops the event when the query fails or finds nothing (`:163-165`, `:186-191`). The consumer is the per-generation controller (`server/services/temporal/workflow_control_workflow.py:336-359`). - **Control Updates with validators** on eight workflow types: the controller, MachinaWorkflow (`server/services/temporal/workflow.py:249-263`), DelegatedTaskWorkflow (`server/services/temporal/agent_workflow.py:500-514`), AgentWorkflow (`:759-773`), EmployeeJobWorkflow (`server/services/temporal/employee_job_workflow.py:78-92`), PiInlineDelegationWorkflow (`server/services/temporal/pi_inline_delegation.py:329-343`), CronTriggerWorkflow (`server/nodes/scheduler/cron_scheduler/_workflow.py:88-101`) and PiCodingAgentWorkflow (`server/services/temporal/pi_workflow.py:49-77`). - **Updates without validators:** Workspace `submit` and `reset` (`server/services/temporal/workspace_tasks_workflow.py:82-193`); Pi `register_driver`, `submit_operation` and `assistant_ready` (`pi_workflow.py:79-107`); the controller's `pause_admissions` (`workflow_control_workflow.py:255-264`). - **Update-With-Start:** Workspace tasks only (`server/services/node_invocations.py:44-74`). - **Other Signals:** the controller's `register_trigger`, `reset` and legacy `pause` and `resume`; Pi `pause` and `resume`, which run an activity (`pi_workflow.py:64-73`); MachinaWorkflow's `on_event`, which buffers into `_matched_events` that nothing reads (`workflow.py:427-447`). - **Signal-With-Start:** not found in `server/`. - **`@workflow.init`:** 9 decorators in 8 production files. - **Request-response outside Temporal:** chat admission (`server/services/chat/ledger.py:252-331`); approval decisions by compare-and-swap on a revision (`server/services/approvals/decisions.py`); Stop and Resume intent rows (`server/services/deployment/handlers.py:2295-2310`); Apply claims, re-driven every 10 seconds by `recover_applies` (`server/services/employees/safe_apply.py:285-289`, `server/services/employees/activation.py:55-72`). **Verdict:** Adopted with deviations. Control needs an answer, so it uses Updates; events need none and must queue while paused, so they use Signals. That matches the chooser. Delivery is at-most-once through Visibility (RFC-0004 H1, replaced in RFC-0005 Phases 2 and 4), and the client-side rules are missing (§7.3). **Recommendation:** 1. **Adopt a handler checklist** as rules in §5, for every new or changed handler: (a) Signal handlers only buffer or set flags, and drop a malformed payload with a log line instead of raising; (b) every rejection that needs no I/O lives in a validator; (c) every client Update call runs under a deadline (§7.3 R1, 1b.15) and passes a stable id where it is retried and the handler cannot fail transiently after acceptance; (d) continue-as-new or completion happens only in the main method, after `wait_condition(all_handlers_finished)`, with no `await` between the final buffer check and the close; (e) every Signal payload carries an idempotency key, carried across continue-as-new; (f) Signals and Updates are counted per run, not only events; (g) owner actions (Start, Stop, Resume, Apply, approval decisions, Interrupt) go through intent rows and the deliverer (T10, §7.3 R7), and any other request path that sends an Update (Workspace submit) does so under a deadline and answers "pending" when it expires. - Versioning: none. The lintable parts were 1b.40 (§6.1 rec 1, §17.18), dropped on 2026-10-25 (§22.2). When: Phase 1b. Effort S. Risk: none at run time. **Remarks:** - The accumulator page says "Signal handlers must not call activities or yield to the scheduler"; the Python page allows `async def` handlers that run activities. Both hold: Python permits it, but such a Signal handler fails the whole workflow if its activity exhausts a finite retry policy. Pi `pause` and `resume` are that case (`DEFAULT_ACTIVITY_RETRY`, 3 attempts, `server/services/temporal/_retry_policies.py:62-67`). - RFC-0004 §8.2 already lists deadlines, stable ids, validators for Pi and Workspace, and counting Signals and Updates; none of it has landed. ### 7.2 Signal with Start **Docs:** https://docs.temporal.io/design-patterns/signal-with-start. One atomic call starts the workflow if it is not running and delivers the Signal; if it runs, only the Signal is delivered. Python: `client.start_workflow(Wf.run, id=..., task_queue=..., start_signal="add_item", start_signal_args=[...])`. The call returns when the server has accepted the Signal, not when it is processed. - Best practices: derive the workflow id from the entity; "Implement Signal idempotency. Track processed operation IDs"; "Use WorkflowInit. Initialize state before Signals are delivered (... Python's `__init__`)"; "Handle unbounded execution. Use Continue-As-New"; "Use ALLOW_DUPLICATE_FAILED_ONLY for entity Workflows" (while the same page says both ALLOW_DUPLICATE and ALLOW_DUPLICATE_FAILED_ONLY "work well"); include operation ids; return early on duplicates. - Pitfalls: no idempotency ("Signals can be delivered more than once (for example, client retries)"); "Unbounded history growth ... will hit the 51,200 event or 10K Signal limit"; "Losing pending Signals on Continue-As-New. Drain all pending Signals before calling Continue-As-New, and pass unprocessed ones as input to the new execution"; expecting a return value. On a Signal that races continue-as-new, "Temporal prevents this race". - Not a fit for "one-time operations (use REJECT_DUPLICATE policy instead)" or for request-response that needs synchronous confirmation ("use Update with Start"). **SDK:** present in 1.30.0 as listed in §7.1. Each call gets a fresh `request_id` (`temporalio/client/_impl.py:327`) that the caller cannot set, in 1.30.0 or 1.34.0, so the server deduplicates only the SDK's own RPC retries. `id_conflict_policy` is passed through as UNSPECIFIED (`_impl.py:332-335`); the server's default for Signal-With-Start is to use the running execution. `priority` travels with the start request (`_impl.py:355-356`), so it most likely applies only when the call starts a run (the server side was not verified). `rpc_timeout` bounds the single RPC; a deadline error is not retried. Signal-With-Start cannot use eager start. The workflow-side `workflow.signal_with_start_workflow` is experimental; RFC-0004 Q12 keeps preview features off. **OpenCompany today:** 1. **Controllers correctly avoid Signal-With-Start.** A controller's existence is lifecycle state, so an event must never create one (RFC-0004 §10.1 item 2); `dispatch.emit` signals by id only (`server/services/events/dispatch.py:251-258`). 2. **The controller is started empty, then signalled: the "start, then signal" shape the accumulator page warns about.** - `_start_controller` starts `WorkflowControlWorkflow` with state, revision and scope but no triggers (`server/services/deployment/handlers.py:614-657`). The deployment manager then sends one `register_trigger` Signal per trigger node (`server/services/deployment/manager.py:952-982`). - Until a trigger is registered, `on_event` finds no matching spec and drops the event without queueing it (`server/services/temporal/workflow_control_workflow.py:336-359`). Before the first registration the `ControlEventTypes` search attribute is absent, so dispatch treats the controller as match-all and does signal it (`dispatch.py:175-184`, `:217-248`). - Nothing guards the window. The generation stays `starting` until after `_await_deployment_setup` (`handlers.py:2106-2156`), and producers deliver to `starting` generations: chat answers "now" for `starting` (`server/services/chat_thread.py:72`, `:84-92`), and webhook routes read every row in `WORKFLOW_CONTROL_ACTIVE_STATES`, which includes `starting` (`server/nodes/trigger/webhook_trigger/_routes.py:36-46`). Events in that window are lost. - A crash between the start and the Signals leaves a live but empty controller. `_converge_interrupted_start` fails the generation when no trigger registered but treats a partial registration as complete (`handlers.py:1708-1745`, the test `if triggers_registered or not graph_has_triggers`); the boot re-arm then registers the rest. - The controller already supports start-with-state: `run` seeds `triggers` from its input (`workflow_control_workflow.py:488-533`), and for execution-control v1 `@workflow.init` seeds the same carried state before any handler runs (`:62-92`). Every continued run already takes this path. 3. **Legacy controllers seed nothing in `__init__`** (`workflow_control_workflow.py:90-92`), so an `on_event` in the first activation after continue-as-new is dropped. Every generation started on current code is v1 (`server/services/deployment/control.py:160`), and T5's fresh start fails every older generation, which owners Reset and Start as v1. No live path remains after the first pinned release. 4. **RFC-0005 relay (planned, RFC-0005 §9.4).** `client.start_workflow("ConversationWorkflow", id=, start_signal="deliver", start_signal_args=[batch], id_reuse_policy=ALLOW_DUPLICATE, priority=Priority(priority_key=, fairness_key=))`; at most one Signal-With-Start in flight per stream and 4 per entity per second; `delivered_seq` is committed after acceptance; the entity drops `seq <= applied_seq`, carried across continue-as-new (RFC-0005 §9.5). Ids `conv--g--`; the supervisor `emp--g` is started at Start (RFC-0005 §10.1) and rebuilt in place when lost (rec 5, T20). Passivation: "owner 24 h, customer 30 min, others 10 min", and "the server rejects completion while new Signals are unhandled" (RFC-0005 §11). Subscriptions are compiled "in the same transaction that moves the generation to `running`" (RFC-0005 §9.3). **Verdict:** - Today: Missing and beneficial, as start-with-state rather than Signal-With-Start. It gives the atomicity Signal-With-Start provides without letting events create controllers. - RFC-0005: Planned in RFC-0005 (deviates). It follows the docs on deterministic ids, a stable idempotency key (`seq`, a gap-free high-water mark, which is better than a seen-set), batching, a rate below 5 per second, buffer-only handlers, `@workflow.init`, a continue-as-new trigger well below 10,000 Signals (4,000) and commit after acceptance. It misses the start input after passivation, the Reset race, the instant EG12 measures from, and a deadline on the call. **Recommendation:** 1. **Start the controller with its triggers in the start input.** The deployment manager builds every trigger spec first and `_start_controller` passes them as `triggers` (sketch). `register_trigger` stays only for the boot re-arm. For a listener id it already knows, it changes nothing today (`workflow_control_workflow.py:327-329`); after 1b.32 (§17.8) it refreshes only that listener's sanitized `parameter_snapshot`. - Versioning: none in workflow code. Old histories replay with their recorded input; the input shape is the one every continued run already uses. New first runs move their search-attribute upsert and poll starts into the first task, which the golden identity test does not lock. - When: P2, the controller-hardening lane, with H10's slim carry. It is not needed for EG12: RFC-0005 delivers events only from `running` (RFC-0005 §9.3; rec 3), which comes after the Signals were accepted, and Signals are applied in arrival order. It closes the window where today's Visibility path loses chat messages, webhooks and unscoped events sent to a `starting` controller, and the crash window. Effort M. - Risk: specs embed the graph and parameter snapshot (RFC-0004 H10). N specs in one start input can exceed the 2 MB payload limit, so Start fails at once instead of at the first rollover (`ControllerContinuationTooLarge`, `:705-711`). Land the slim carry with it. Each spec's snapshot must be the one 1b.31 (§17.8) sanitizes on the Start path, so no secret enters the start input; rec 1 lands after it. 2. **Pass the stream cursor as the entity's start input.** The Signal-With-Start call in RFC-0005 §9.4 passes no workflow argument. Pass only the stream cursor, `consumed_seq` (as `{"applied_seq": consumed_seq}`, §8.2 rec 2), read in the same transaction as the batch, and seed `applied_seq` from it in `@workflow.init` on every start, a restart after passivation included, because the `deliver` handler of a new run runs before `run`. Workflow arguments take effect only when the call starts a run, which is exactly the passivated case. Without it, a restarted run cannot tell consumed inputs from new ones. The turn number is not in the start input: §8.2 rec 2 allocates it in the admission transaction (RFC-0005 §13.3), which no restart can reuse, so turn ids, agent child ids and spend keys stay unique after passivation. Versioning: none (ConversationWorkflow is a new type). When: P2, RFC-0005 Phase 4. Effort S. Risk: the relay must read the cursor in the batch's transaction, or a stale `consumed_seq` re-applies inputs. 3. **Define EG12 by the Start commit.** Compile subscriptions in the transaction that creates the `starting` generation, and hold its streams with reason `starting` until the move to `running` releases them. An event accepted after Start committed is then delivered once the generation runs; today's RFC text would mark it `no_consumer`. Versioning: none (database and router). When: P2, RFC-0005 Phase 2 (design text and router). Effort S. Risk: a Start that fails must release or dead-letter its held streams. 4. **Close the Reset race.** The relay's generation check (RFC-0005 §9.4 step 1) and its Signal-With-Start are not atomic; a Reset committed between them lets ALLOW_DUPLICATE start an entity for a dead generation. `admission.acquire` (RFC-0005 §13.3) also checks the generation, and a dead generation's entity completes without a turn. The alternative is for Reset to take the stream leases. Versioning: none (new types). When: P2, RFC-0005 Phase 4. Effort S. Risk: none known. 5. **State the reuse policies.** ALLOW_DUPLICATE is right for the entity: passivation completes the run, and the page's ALLOW_DUPLICATE_FAILED_ONLY would refuse to restart a completed entity. The supervisor follows T20 (RFC-0005 §10.1): - A lost supervisor (terminated or failed) is rebuilt in place from the database, keeping chat and Context. - Start starts it with `REJECT_DUPLICATE` under the new generation's id (RFC-0005 §10.1). Each rebuild passes `id_reuse_policy=WorkflowIDReusePolicy.ALLOW_DUPLICATE_FAILED_ONLY`, so the id can be taken again only after a terminated, cancelled, timed-out or failed run, and `id_conflict_policy=WorkflowIDConflictPolicy.USE_EXISTING`, so a rebuild that meets a live run attaches to it. - After 3 rebuilds within an hour it stops rebuilding and raises an attention item that asks the owner to Reset. - Today's v1 controller has no such rebuild: a vanished one fails its generation and only Reset recovers it (`handlers.py:780-835`, `:1912-1914`). - Versioning: none (new types). When: P2, RFC-0005 Phase 4. Effort M. - Risk: the policy also accepts a start after a cancelled run, so the rebuild first checks in the database that the generation is still live. 6. **Priority applies at start.** An entity started by a C4 input would keep priority 4 for its own workflow tasks after a C1 input arrives. §13.3 rec 3 settles this: entities are keyed by lane, not by their first input, and turn children carry their inputs' class (RFC-0005 §11 step 5). The fairness key in the same `Priority` is the employee id, which never changes; fairness itself is on from the fresh-start release (T17, §13.4). Versioning: none (a start option). When: P2, RFC-0005 Phase 4, with §13.3. Effort S. 7. **Put a deadline on the relay's calls.** `rpc_timeout=timedelta(seconds=10)` on every Signal-With-Start (Phase 4) and every plain Signal to today's controllers (Phase 2). A timed-out call commits nothing; the relay re-sends and the entity deduplicates by `seq`. Versioning: none (client code). Effort S. Risk: none known. **Remarks:** - A client-retried Signal-With-Start that straddles the entity's continue-as-new reaches both runs; only the carried `applied_seq` makes that harmless. The controller's carried `seen_event_ids` play the same role (`workflow_control_workflow.py:695`). - Passivation: "Temporal prevents this race" is stated for continue-as-new. For completion it is documented elsewhere: when new Signals arrive while a Workflow Task runs and the workflow tries to close, "The Temporal Service fails the Workflow Task and schedules a new one" (https://docs.temporal.io/references/workflow-task-errors, Unhandled Command). It protects the entity only if it checks its buffer after `wait_condition(all_handlers_finished)` and closes with no `await` in between (§7.4 A3). A `deliver` accepted after the close starts a new run, which is correct with ALLOW_DUPLICATE and recommendation 2. - Signal budget: the 4,000-Signal trigger must count every Signal type (`deliver`, `inputs_consumed`, `turn_suspended`, `wake`) in the handlers; the SDK has no Signal-count API. The relay window (32 inputs or 512 KB, RFC-0005 §9.4 step 3) keeps a stuck entity from absorbing Signals toward the 10,000 server limit. - EG7 ("10,000 messages ... cost fewer than 1,000 Signals per entity run") holds only for bursts: at one message per second every message is its own Signal. Define it as a burst test: the 10,000 messages are appended before the relay runs, so batches are full (16 inputs, about 625 Signals). - Testing: run Signal-With-Start against `WorkflowEnvironment.start_local()` with the pinned CLI. Test the passivation race by sending `deliver` while the entity's completing task is in flight; time skipping cannot interleave it. Add an entity history to the replay corpus as soon as Phase 4 records one. **Sketch** (SDK 1.30.0): ```python # (1) Today's controller: triggers travel in the start input (caller-only change) specs = {spec["listener_id"]: spec for spec in await manager.build_trigger_specs(control)} await client.start_workflow( "WorkflowControlWorkflow", args=[{**control_args, "triggers": specs}], # v1 @workflow.init seeds them before any handler id=control.controller_workflow_id, task_queue=queue, search_attributes=attrs, ) # (2) RFC-0005 relay: the workflow argument is used only when this call starts a run seed = {"applied_seq": consumed_seq} # the stream cursor, read in the same transaction as the batch await client.start_workflow( "ConversationWorkflow", seed, id=entity_id, task_queue="machina-tasks", start_signal="deliver", start_signal_args=[batch], id_reuse_policy=WorkflowIDReusePolicy.ALLOW_DUPLICATE, # passivation completes the run priority=Priority(priority_key=lane_key, fairness_key=employee_id), # keyed by lane (§13.3 rec 3) rpc_timeout=timedelta(seconds=10), ) @workflow.defn class ConversationWorkflow: @workflow.init def __init__(self, seed: dict) -> None: self._applied_seq = int(seed.get("applied_seq") or 0) # deliver runs before run(): seed here self._pending: list[dict] = [] # turn numbers come from admission (§8.2 rec 2), not from here @workflow.signal def deliver(self, batch: dict) -> None: # buffer only for row in batch["rows"]: if row["seq"] > self._applied_seq: self._pending.append(row) self._applied_seq = row["seq"] ``` ### 7.3 Request-Response via Updates **Docs:** https://docs.temporal.io/design-patterns/request-response-via-updates. `@workflow.update` plus `@.validator`: the validator runs first and, if it raises, the Update is rejected with nothing written to history; otherwise the handler changes state and returns a typed result, and "The Update is recorded in Workflow history before the response is returned". - Best practices: "Validate early"; typed exceptions; "Return quickly"; "Track Update IDs only across Continue-As-New ... Update ID deduplication is scoped to a single Workflow Run"; "Set timeouts"; atomic state changes. - Pitfalls: long operations in the handler ("Offload long-running work to Activities"); exceeding 2,000 total Updates; "Not setting Update timeouts. Without a client-side timeout, the caller blocks indefinitely if the Worker is unavailable"; set "a stable, business-meaningful `updateId` when you want a retried Update-with-Start to attach to an existing in-flight Update"; "Updates require a Worker to be online and responsive. For fire-and-forget operations, use Signals instead". - Limits: about 2 MB per argument or result; each Update adds events toward 51,200; "a maximum of 10 in-flight Updates per Workflow execution and a maximum of 2,000 total Updates". - The page's Python sample calls `uuid.uuid4()` inside the handler; workflow code must use `workflow.uuid4()`. **SDK:** everything is in 1.30.0, including `start_update(..., wait_for_stage=ACCEPTED)` for long handlers and `WorkflowUpdateRPCTimeoutOrCancelledError`. A `WithStartWorkflowOperation` is single-use ("WithStartWorkflowOperation cannot be reused", `temporalio/client/_client.py:1185-1187`), so a retried Update-With-Start builds a new one. One trap, read in the source: `rpc_timeout` does not bound `execute_update`. It bounds each request while the Update is sent, but after acceptance the SDK polls for the outcome without it (1.30.0 `temporalio/client/_workflow.py:840` and `temporalio/client/_impl.py:802`; 1.34.0 `:846` and `:889`), and both loops repeat while the server answers without an outcome. Only `asyncio.timeout` (or `asyncio.wait_for`) around the whole call bounds it. The SDK turns the resulting cancellation into `WorkflowUpdateRPCTimeoutOrCancelledError` whose cause is `CancelledError`, which looks the same as the caller being cancelled; check the timeout scope's `expired()` to tell them apart. **OpenCompany today:** - **What follows the docs.** - Revision-idempotent control Updates whose validators raise non-retryable `ApplicationError` (`server/services/temporal/workflow_control_workflow.py:161-174`, `:196-198`, `:234-253`, `:311-323`; `server/services/temporal/execution_control.py:58-106`), and chain identity checked by `first_execution_run_id` (`server/services/temporal/execution_control.py:72-74`). - Stable ids for retried control calls: `control:{revision}:admission`, `:checkpoint` and `:release` (`server/services/deployment/execution_control.py:80-81`, `:120-121`, `:150-152`, `:181-183`); reconcile and pause or resume ids (`server/services/deployment/handlers.py:1263`, `:2313-2315`, `:2405`); safe-apply ids derived from a database claim (`server/services/employees/safe_apply.py:107`, `:199`, `:231`, `:254-255`, `:264`). - Cross-workflow Updates only through activities, as the Python guide requires (`server/services/temporal/execution_control_activities.py:17-27`; `server/services/temporal/pi_activities.py:106-112`). - Continue-as-new only from main loops after `all_handlers_finished` (`workflow_control_workflow.py:595-716`; `server/services/temporal/workspace_tasks_workflow.py:195-215`; `server/services/temporal/agent_workflow.py:2992-3031`; `server/services/temporal/pi_workflow.py:382-387`); the controller requests rollover from Update handlers instead of continuing there (`workflow_control_workflow.py:588-593`). - Update-With-Start for Workspace tasks uses `USE_EXISTING` and `@workflow.init`, because "Update-With-Start can invoke a handler before run()" (`server/services/node_invocations.py:52-59`; `workspace_tasks_workflow.py:39-53`). - Update-id dedupe across continue-as-new is done with business keys, as the page advises: revisions, bounded `submissions` and `resets` carried across the Workspace rollover (`workspace_tasks_workflow.py:43-46`, `:210-215`), and Pi digests in Update ids (`pi_activities.py:263`, `:297-298`). - **Deviations.** 1. *No deadline on any app-process Update call:* Workspace Update-With-Start (`node_invocations.py:52`); `_update_controller_state` (`handlers.py:708-712`); every call in `transition_generation` (`server/services/deployment/execution_control.py:80`, `:108`, `:120`, `:150`, `:181`, `:187`); safe-apply (`safe_apply.py:107`, `:199`, `:231`, `:254`, `:255`, `:264`). The socket waiter awaits `asyncio.shield(task)` with no timeout (`handlers.py:134-139`). Only `_finish_versioned_transition` handles `WorkflowUpdateRPCTimeoutOrCancelledError` (`handlers.py:1423-1430`); its "bounded RPC poll" comment is wrong without a deadline. Worker-side calls are bounded by their activity's 60-second start-to-close (`server/services/temporal/execution_control.py:210-229`). 2. *Random Update ids on retried calls.* `execution_control.register` and `unregister` pass no `id=` (`execution_control_activities.py:17-27`), so each of the 3 attempts of `DEFAULT_ACTIVITY_RETRY` is a new Update; `transition_generation`'s `unregister_execution` and its final controller `set_control_state` pass none either (`server/services/deployment/execution_control.py:108-111`, `:187-190`). The handlers are idempotent (`workflow_control_workflow.py:219-247`), so this costs history and in-flight slots, not correctness. 3. *Rejections inside handlers.* Workspace `submit` checks admission before and after its lock and `reset` checks its request id (`workspace_tasks_workflow.py:83-97`, `:147-148`), so every refusal ("queue full", "resetting", "admission changed") is an accepted-then-failed Update in history. The caller compensates with a fresh random Update id per submit (`node_invocations.py:246-252`). Pi `register_driver`, `submit_operation` and `assistant_ready` have no validators (`pi_workflow.py:79-107`). RFC-0004 L3. 4. *Update results copy the whole controller status.* Every controller Update returns `self.status()`, including the full `live_roots` map (`workflow_control_workflow.py:369-388`), into the controller's history and again into each child's activity result. A recorded entry costs about 135 bytes with a 25-character id (`server/tests/temporal/fixtures/0.3.1/controller_push.json`); real `-