--- name: "runbook-writing-stinger" license: AGPL-3.0-or-later description: "Write and audit operational runbooks. Use for break-fix, scheduled tasks, exact commands, escalation, rollback, or game days. Read README.md for the guide map." --- # runbook-writing-stinger Operational runbook craft: the exact-command discipline, the no-implied-context rule, escalation path architecture, rollback procedures, runbook-as-test methodology, and postmortem-to-runbook linkage. **Read this file first** to orient. Then open the guide that matches your task. --- ## The five core principles These govern every decision in this stinger. Full justification with failure modes per principle in `guides/00-principles.md`. 1. **No implied context.** Every command is copy-pasteable. Every URL is absolute. Every env var is defined. Every decision point is explicit. A runbook written for "someone who knows the system" is not a runbook. 2. **Exact-command discipline.** No "something like `npm run embeddings:status`." Exact flags, exact dataset paths, exact daemon names. Vague commands create incident-time variance. 3. **Explicit escalation paths.** Every runbook names its escalation contact (person, team, channel) with a response-time expectation. "Escalate if needed" is not an escalation path. 4. **Rollback before you ship.** Every state-changing step has an undo step or an explicit irreversibility acknowledgment. Rollback is never improvised during an incident. 5. **Runbook-as-test mandate.** An untested runbook is a hypothesis. Exercise runbooks quarterly (game day) and on every postmortem action item. Mark untested runbooks prominently. **Research anchor:** Google SRE Book Chapter 11 defines on-call as requiring co-equal resources: clear escalation paths, well-defined procedures, and blameless postmortem culture. All five principles map directly to this triad. See `research/external/2026-sre-google-being-on-call-chapter.md`. --- ## Quick reference: which guide to read | Task | Guide | |---|---| | Learn the five principles and their failure modes | `guides/00-principles.md` | | Choose which runbook type to write | `guides/01-runbook-types.md` | | Audit an existing runbook for no-implied-context violations | `guides/02-no-implied-context-audit.md` | | Structure escalation paths correctly | `guides/03-escalation-path-architecture.md` | | Write rollback sections | `guides/04-rollback-procedures.md` | | Plan and execute a game day / runbook exercise | `guides/05-runbook-as-test.md` | | Link a postmortem action item to a runbook | `guides/06-postmortem-linkage.md` | | Validate a runbook before marking it ready | `guides/07-done-checklist.md` | | See a full worked example | `examples/happy-path-break-fix.md` | | Audit an existing runbook end-to-end | `examples/audit-existing-runbook.md` | | Start a new runbook from a blank template | `templates/` | --- ## When this stinger activates This stinger is pre-loaded by `runbook-writing-wasp-drone`. Do not load it independently unless you are the Drone. Trigger contexts: - "Write a runbook for the [service/alert]" - "Our runbook for [X] is outdated" - "Audit this runbook" + paste of existing doc - "Turn this postmortem action item into a runbook" - "We need to run a game day / exercise our runbooks" - "Our on-call docs are weak / missing / wrong" - Postmortem with action item: "Write runbook for [failure mode]" - PR or code change that introduces a new failure mode without a runbook Do NOT activate for: - PagerDuty / OpsGenie configuration (ci-release-wasp-drone) - Infrastructure provisioning decisions embedded in runbooks (ci-release-wasp-drone owns the what; this stinger owns the how-to-document-it) - Incident culture or postmortem process design beyond the document format (library-wasp-drone) --- ## Runbook types at a glance Three types, three templates. Details in `guides/01-runbook-types.md`. | Type | When to use | Template | |---|---|---| | **Break-fix** | Alert fires, service degraded, on-call responds | `templates/break-fix-runbook.md` | | **Scheduled operation** | Planned maintenance, deployment window, DR drill | `templates/scheduled-operation-runbook.md` | | **Diagnostic** | Root-cause investigation, "it's slow but not paged" | `templates/diagnostic-runbook.md` | --- ## Open questions from research (flags for the user) The following were surfaced by `scripture-historian` and were not resolved by the Command Brief. Flag to the user before finalizing any guide that touches these areas: 1. **Runbook-as-code scope**: Should this stinger cover automation hooks (Rundeck, AWS SSM, Jupyter notebooks)? Research shows 2026 SRE practice increasingly blurs manual vs. automated runbooks. Current stance: manual runbooks only; automation is an advanced pattern flagged in `guides/01-runbook-types.md` as "out of scope, see ci-release-wasp-drone." 2. **Security attribute**: The SRE School quality model adds "no exposed secrets, least privilege commands" as a ninth attribute. Added to `guides/07-done-checklist.md` as a checklist item; flag to user if their environment has PCI/HIPAA compliance requirements. 3. **Alert-links-to-runbook principle**: Added as Principle 6 in `guides/00-principles.md` ("Alert linking"), the alert payload must directly link to the specific runbook, not a runbook index. 4. **Freshness KPIs**: Added postmortem action item completion rate as a KPI in `guides/07-done-checklist.md`. User should decide whether to track in a dashboard. 5. **Storage tooling**: This stinger is tool-agnostic (Notion, Confluence, Slab, Git/Backstage all work). Tool-specific tips are callouts in `guides/00-principles.md`. > TODO: open question, if the user's org uses runbook automation tools (Rundeck, Shoreline, AWS SSM), a future `runbook-writing-wasp-drone` would own the integration layer. Flag this need if it surfaces. --- ## Critical directives (verbatim from Command Brief) - **Never use implied commands.** See `guides/02-no-implied-context-audit.md`. - **Never skip the escalation path.** See `guides/03-escalation-path-architecture.md`. - **Always include rollback for every state-changing step.** See `guides/04-rollback-procedures.md`. - **Mark untested runbooks prominently.** See `guides/05-runbook-as-test.md`. - **Apply the five-minute rule.** A runbook requiring more than five minutes to understand enough to execute is too long. Split it or add a prominent TL;DR summary at the top. --- ## Folder layout ``` runbook-writing-stinger/ +- SKILL.md (this file, read first) +- README.md (one-page overview) +- guides/ | +- 00-principles.md (six core principles with failure modes) | +- 01-runbook-types.md (break-fix vs scheduled-operation vs diagnostic) | +- 02-no-implied-context-audit.md (audit protocol, step-by-step) | +- 03-escalation-path-architecture.md (naming, formatting, SLA tiering) | +- 04-rollback-procedures.md (reversible vs irreversible, undo templates) | +- 05-runbook-as-test.md (game day methodology, quarterly cadence) | +- 06-postmortem-linkage.md (closed loop: incident -> postmortem -> runbook) | +- 07-done-checklist.md (validation pass before marking ready) +- examples/ | +- happy-path-break-fix.md (end-to-end worked example: embeddings daemon stall alert) | +- audit-existing-runbook.md (worked audit: before and after with violations called out) +- templates/ | +- break-fix-runbook.md (canonical template with all required sections) | +- scheduled-operation-runbook.md (planned maintenance template) | +- diagnostic-runbook.md (root-cause investigation template) +- reports/ | +- README.md (what the Drone's audit reports look like) +- research/ (READ ONLY - authored by scripture-historian) +- research-plan.md +- research-summary.md +- index.md +- internal/command-brief-notes.md +- external/ (8 source notes) ``` --- *Part of The Wasp Nest, curated by [Mario Aldayuz a.k.a @thenotoriousllama](https://github.com/thenotoriousllama).*