---
name: devops-delivery
description: 'Audits deployability and operability: twelve-factor compliance, CI/CD pipeline gates, monitoring, SLOs, releases, rollbacks, incidents. Use for deploy pipelines, Dockerfiles, environment config, observability setup, release runbooks, or "is this production-ready".'
license: MIT
metadata:
  author: lammanhhoang
  version: "1.0.0"
---

# DevOps Delivery Audit

Audit a system's path to production: how it's configured, built, shipped, observed, rolled back, and recovered. Findings cite rule numbers ("violates rule #7"). Verdicts are PASS / FAIL per rule — no "mostly fine".

## When to use

- Reviewing CI/CD pipelines, Dockerfiles, deploy scripts, Helm charts, GitHub Actions / GitLab CI configs.
- Auditing environment config, secrets handling, log setup, health checks.
- Designing or reviewing release processes, rollback plans, incident response.
- Answering "is this production-ready?" — run the full audit loop below.

**Do NOT use for:** application code review (logic bugs, style), infrastructure provisioning design (Terraform module architecture, network topology), or cloud cost optimization. Those are separate concerns.

## Audit workflow

1. **Twelve-factor pass.** Walk rules #1–#12 against the repo: Dockerfile, config loading, process model, logging. Record PASS/FAIL per factor.
2. **Pipeline pass.** Check gate order, caching, timing, and artifact promotion against rules #13–#17.
3. **Observability pass.** Check dashboards and alerts against rules #18–#22.
4. **Release/rollback pass.** Check the release procedure against rules #23–#29. If no written runbook exists, that alone is a finding — generate one from assets/release-runbook-template.md.
5. **Incident readiness pass.** Rules #30–#33.
6. **Deep audit (optional).** For a full production-readiness review (config, resilience, security, data, capacity), read references/CHECKLIST.md — load when the user asks "is this production-ready" or requests a launch review.
7. **Report.** List findings as `[FAIL rule #N] <one-line evidence> → <one-line fix>`, ordered by blast radius: data loss > outage risk > slow recovery > hygiene.

## Rules: Twelve-factor audit (#1–#12)

One question per factor. Answer from evidence in the repo, not from what the team says.

| # | Factor | Audit question | PASS signal | FAIL signal |
|---|--------|----------------|-------------|-------------|
| 1 | I. Codebase | Is there exactly one repo per app, deployed to all environments from the same history? | One repo, env differences are config only | Per-env branches/forks; copy-paste "prod" repo |
| 2 | II. Dependencies | Are all dependencies declared in a manifest with a lockfile, and nothing assumed from the host? | `package-lock.json` / `poetry.lock` / `go.sum` committed; Docker base pinned by digest or exact tag | `latest` base image; "install X on the server first"; lockfile gitignored |
| 3 | III. Config | Does anything env-specific (URLs, credentials, flags) live in code or baked into the image? | All env-specific values read from environment variables or a secret manager at runtime | Hardcoded hostnames, `config.prod.js` compiled in, secrets in the image |
| 4 | IV. Backing services | Can you swap a database/queue/cache/SMTP endpoint by changing config alone? | Services addressed by URL/DSN from config; no code change to repoint | Connection details in code; local vs. remote services handled by different code paths |
| 5 | V. Build, release, run | Are build, release, and run strictly separated — with releases immutable and identifiable? | Build once → tag with version/SHA → run that exact artifact everywhere | Editing code on servers; rebuilding per environment; mutable `latest` deploys |
| 6 | VI. Processes | Is the app stateless — no sticky sessions, no local-disk state that must survive a restart? | Session/state in Redis/DB/object storage; any instance can serve any request | In-memory sessions, uploads written to local disk, "don't restart node 3" |
| 7 | VII. Port binding | Does the app self-host by binding a port from config, rather than living inside an external server? | `PORT` env var respected; app is its own web server | Requires a hand-configured Apache/Tomcat on the host to be reachable |
| 8 | VIII. Concurrency | Does it scale by adding processes/replicas rather than growing one giant process? | Horizontal scaling works; workload types (web/worker) run as separate process types | Only vertical scaling; threads-in-one-process is the sole model; singleton assumptions |
| 9 | IX. Disposability | Can any instance be killed at any moment without dropping work, and start in seconds? | Handles SIGTERM: drains requests, re-queues jobs; startup < ~10s | Long warmup, lost in-flight jobs on restart, deploys need "quiet hours" |
| 10 | X. Dev/prod parity | Are dev, staging, and prod the same OS, backing services, and deploy method? | Same containers, same Postgres (not SQLite-in-dev), staging deploys via the same pipeline | SQLite dev / Postgres prod; staging deployed by hand; "works on my machine" class bugs |
| 11 | XI. Logs | Does the app write logs as an event stream to stdout/stderr, leaving routing to the platform? | Structured (JSON) lines to stdout; aggregation handled outside the app | App writes/rotates its own log files; logs only on the container's local disk |
| 12 | XII. Admin processes | Do migrations and one-off tasks run as one-off processes from the same release artifact? | `kubectl run`/`heroku run`-style one-offs using the release image and its config | SSH-in-and-run-SQL; admin scripts living outside the repo; migrations run from a laptop |

Source: 12factor.net. Full per-factor remediation notes: references/CHECKLIST.md, section "Config & twelve-factor".

## Rules: CI pipeline (#13–#17)

**#13 — Gate order MUST be cheapest-first, fail-fast:**

```
lint → typecheck → unit tests → build → integration tests
     → security scan (SCA + secret scan) → artifact publish
```

Lint and typecheck (seconds) run before unit tests (minutes) before build and integration tests. A pipeline that builds a Docker image before running lint burns 5 minutes to discover a missing semicolon. Security scan = SCA on dependencies (default: `osv-scanner scan source -r .`; also `npm audit`, Dependabot/Snyk) **plus** secret scanning (e.g. `gitleaks git`) — both, not either. Publish is last and only on green.

**#14 — Fail fast MUST be on.** No `continue-on-error` on gates; parallelize independent gates (lint ∥ typecheck ∥ unit tests) but never let a later gate start masking an earlier failure.

**#15 — Dependencies MUST be cached** (e.g. `actions/cache` keyed on the lockfile hash; Docker layer cache with `--cache-from` or BuildKit). An uncached pipeline re-downloading node_modules every run is a FAIL.

**#16 — Total pipeline SHOULD complete in under 10 minutes** commit-to-verdict. Over 10 minutes, developers batch commits and stop trusting CI. Fixes in order of leverage: cache deps, parallelize gates, split slow integration suites, shard tests.

**#17 — Build ONE artifact and promote it.** The exact image/binary tested in staging is the one deployed to prod — identified by immutable tag (git SHA or semver, never `latest`). Rebuilding per environment is a FAIL: it deploys an artifact nobody tested. Environment differences enter via config (rule #3), never via rebuild.

## Rules: Monitoring & SLOs (#18–#22)

**#18 — Minimum dashboard = the four golden signals** (Google SRE book): **latency** (p50/p95/p99, successful and failed requests separately), **traffic** (requests/sec), **errors** (rate of failed requests — explicit 5xx and wrong-answer 200s), **saturation** (how full the constrained resource is: memory, connection pool, queue depth). A service without these four visible on one dashboard FAILS.

**#19 — Every user-facing service MUST have an SLO with an error budget.** One paragraph version: pick a target (e.g. 99.9% of requests succeed in <500ms over a 30-day window). The complement is the error budget — 99.9% leaves 0.1%, i.e. **43.2 minutes of full downtime per 30-day window**. While budget remains, ship features. When the budget is exhausted, **feature work freezes and the team works only on reliability** until the budget recovers. This converts "how reliable is reliable enough" from an argument into arithmetic.

**#20 — Alert on symptoms, not causes.** Page on user-facing SLO burn ("error rate 3× budget burn rate for 10 min"), never on `CPU > 80%`. High CPU with happy users is not an incident; happy CPU with failing users is. Cause-based signals (CPU, disk, GC pauses) belong on dashboards for diagnosis, not in the pager.

**#21 — Every page MUST be actionable.** If the response to an alert is "it usually resolves itself", delete or demote the alert. Alert fatigue is how real pages get missed.

**#22 — Track the four DORA metrics as the improvement dashboard** (not for individual performance review):

| Metric | Measures | Elite reference point |
|--------|----------|----------------------|
| Deployment frequency | How often you ship to prod | On demand, multiple/day |
| Lead time for changes | Commit → running in prod | < 1 day |
| Change failure rate | % of deploys causing degradation/rollback/hotfix | ~5% |
| Failed deployment recovery time | Failed deploy → restored | < 1 hour |

The first two measure throughput, the last two stability. They improve together, not in trade-off — that is DORA's core empirical finding.

## Rules: Release & rollback (#23–#29)

**#23 — Releases follow: version → tag → staging → verify → prod → verify.** Full procedure with commands: assets/release-runbook-template.md — instantiate it per service.

**#24 — Version bump and changelog derive from conventional commits:** `fix:` → patch, `feat:` → minor, `BREAKING CHANGE:`/`!` → major. Automate with `semantic-release` or `changesets` (default: `semantic-release`; escape hatch: `changesets` for monorepos needing independent package versions). Hand-written version bumps drift; generated changelogs don't.

**#25 — Staging verification = smoke tests + golden signals**, not "it deployed without errors". Hit the critical user paths; watch latency/errors on the staging dashboard for a defined soak (10–30 min).

**#26 — Prod deploys default to canary or rolling.** Canary: route 1–5% of traffic to the new version, compare golden signals against baseline, then promote. Escape hatch: **blue-green** when you need instant full cutover and instant full rollback (e.g. lockstep API changes) and can afford double capacity. Big-bang 100% deploys FAIL the audit for any service with an SLO.

**#27 — Rollback trigger criteria MUST be written BEFORE the deploy**, in the runbook, as a mechanical condition — e.g. "error rate > 2% for 5 minutes, or p99 > 2× baseline for 10 minutes → roll back, no meeting required." Deciding thresholds during the incident guarantees the sunk-cost debate ("let's give it five more minutes") that turns a blip into an outage. Automate the trigger where the platform supports it (Argo Rollouts analysis, spinnaker ACA); otherwise the on-call executes it without asking permission.

**#28 — Every deploy MUST have a tested rollback path.** "Tested" means actually exercised — in staging, or routinely in prod via the previous release. Rollback = redeploy the previous immutable artifact (rule #17 is what makes this possible). If rollback has never been run, you don't have one; you have a hope.

**#29 — Schema migrations follow expand-and-contract** so code rollback NEVER requires schema rollback:

1. **Expand:** add the new column/table/index (backward compatible — old code ignores it). Deploy.
2. **Migrate:** new code writes both/reads new with fallback; backfill data. Deploy.
3. **Contract:** once no running code depends on the old shape (and a full release cycle has passed), drop it. Deploy.

Each step is independently deployable and independently rollback-safe. A migration that renames a column in one step couples code and schema: rolling back the code breaks against the new schema, and rolling back the schema loses writes. That is a FAIL regardless of how good the rest of the pipeline is.

## Rules: Incidents (#30–#33)

**#30 — Triage on user impact first.** First question in any incident: "are users affected, how many, how badly?" — that sets severity and urgency. Not "what changed" — that comes later.

**#31 — Stop the bleeding before diagnosing.** Mitigation order: **rollback** the latest deploy (most incidents are self-inflicted; check deploy timeline first) → **flag off** the offending feature → **scale up / shed load**. Root-causing a live outage while users bleed is the classic on-call failure mode. It is always acceptable to mitigate without understanding why the mitigation works.

**#32 — During the incident: one incident commander, one comms channel, timestamps recorded.** The IC coordinates and communicates; they do not debug. Notes taken now are the postmortem's raw material.

**#33 — Blameless postmortem within 48 hours** for any user-impacting incident. Blameless = name systems and processes, never people; "the deploy pipeline allowed an untested artifact" not "Alice skipped the tests". Every postmortem produces action items with owners and due dates, tracked like any other work. Template: assets/postmortem-template.md.

## Gotchas

- **Health checks that lie:** `/healthz` returning 200 unconditionally. Liveness = process is up; readiness = dependencies reachable and warm. Wire readiness into the load balancer or you'll route traffic to instances that can't serve it.
- **`:latest` anywhere** — base image, deploy tag, or cache key — silently breaks rules #2, #5, #17 and makes rollback (#28) impossible, since "previous version" no longer names anything.
- **Secrets in build args or image layers:** `docker history` exposes `ARG API_KEY`. Secrets enter at runtime (env/secret manager) or via BuildKit `--mount=type=secret`, never at build time.
- **SIGTERM ignored because PID 1 is a shell:** `CMD ["sh", "-c", "node server.js"]` means signals hit `sh`, not node — Kubernetes waits `terminationGracePeriodSeconds` then SIGKILLs mid-request. Use exec-form `CMD ["node", "server.js"]` or `tini`.
- **Migrations auto-run on app startup:** N replicas racing the same migration on deploy, and rollback re-triggering it. Run migrations as a one-off release step (rule #12), gated before the rollout.
- **Staging that isn't:** different instance sizes, no real data volume, missing backing services. It still catches config errors; it stops predicting performance. Say which of the two claims your staging supports.
- **Canary without comparison:** deploying to 5% and eyeballing "seems fine" is not canary analysis. Compare canary vs. baseline golden signals over a fixed window with a pre-set threshold (rule #27).

## What NOT to do

- Do NOT pass an audit "with notes". Every rule is PASS or FAIL; FAILs get a fix line.
- Do NOT accept "we can't reproduce prod locally" as a rule #10 waiver — containers made parity cheap; the audit question is whether the team chose it.
- Do NOT recommend a menu of tools. One default per job (stated in the rules above), escape hatches only when the constraint is named.
- Do NOT let a beautiful pipeline excuse a missing rollback path. Rules #27–#29 outrank everything else in this skill: shipping fast is optional, recovering fast is not.
- Do NOT write a postmortem that blames a person. If a human error caused it, the finding is the system that let one human error reach users.