--- name: release-engineer description: Structured release and ship workflow skill. Use when the user wants to ship code, create a PR, merge to main, deploy to production, run pre-ship checks, bootstrap a test framework, or needs a repeatable release process from final review to live deployment. --- # Release Engineer Approach every release as the engineer who owns it end-to-end — from the last line of code to verified traffic in production. A release isn't done when the PR is merged. It's done when the system is healthy, metrics are normal, and you could roll back in under five minutes if they weren't. --- ## Step 0: Triage the Release Path Before executing any steps, determine which path applies: 1. **Normal release** — feature or fix developed on a branch, PR into `main`, standard CI gate, deploy to production on merge. 2. **Hotfix release** — production is broken right now. Branch from the production tag or `main`, fix, fast-track review, deploy, backport. 3. **Release candidate** — a named, versioned artifact that will be promoted through staging → production with a manual gate. If the user hasn't said which path, ask. Choosing the wrong path (treating a hotfix like a normal release) adds minutes you don't have when production is down. Choosing the wrong path in the other direction (treating a normal feature like a hotfix) bypasses the quality gates that exist for a reason. State which path you're on at the start of every release workflow, and why. --- ## Step 1: Pre-Ship Checklist Run this checklist before touching `git` for any release. Every item has a reason — don't skip any without documenting why. ### 1.1 Sync with Base Branch ```bash git fetch origin git checkout main && git pull --ff-only origin main git checkout git rebase origin/main # Resolve any conflicts. Rebasing (not merging) keeps history linear and # makes the diff in the PR exactly what you changed — not mixed with merge noise. ``` If the rebase surfaces conflicts, resolve them now. A PR with conflicts doesn't merge cleanly and signals to reviewers that the branch is stale. ### 1.2 Automated Quality Gates Run the full local quality suite — don't skip steps because "CI will catch it." CI feedback takes minutes; local feedback takes seconds. A red CI on a PR you just opened is a social and velocity cost. ```bash # Lint — catches style, potential bugs, and enforces consistency npm run lint # or: ruff check . / golangci-lint run / cargo clippy # Type check — catches contract violations that tests might not cover npm run type-check # or: mypy . / tsc --noEmit # Unit tests with coverage npm run test:unit -- --coverage # or: pytest --cov=src --cov-report=term-missing # or: go test ./... -coverprofile=coverage.out # Integration tests (if applicable locally) npm run test:integration ``` **If any step fails, stop.** Fix the failure, re-run from the top of this section. Shipping broken tests is shipping a broken trust baseline — the next engineer can't tell if a new test failure is their fault or yours. ### 1.3 Coverage Audit Coverage isn't a target to game — it's a signal. If you're adding a significant new path and coverage dropped, ask yourself whether the untested paths are acceptable risks. ```bash # Check coverage against your project's threshold npx nyc check-coverage --lines 80 --functions 80 --branches 70 # or: pytest --cov=src --cov-fail-under=80 # or: go tool cover -func=coverage.out | tail -1 ``` If coverage is below threshold: - Add tests for the new code paths you introduced, not random tests to inflate the number. - If the uncovered path is intentionally untestable (e.g., a panic recovery branch), add a `// coverage: ignore` annotation with a comment explaining why. ### 1.4 Secrets Scan ```bash # Scan for accidentally committed secrets before they leave your machine git diff origin/main --name-only | xargs grep -l -E "(api_key|secret|password|token|BEGIN (RSA|EC|OPENSSH))" 2>/dev/null # or use a dedicated tool: trufflehog git file://. --since-commit origin/main --only-verified # or: gitleaks detect --source . --verbose ``` If secrets are found: 1. Do NOT proceed with the PR. 2. Remove the secret from the code. 3. If the secret was ever committed (even briefly), rotate it immediately — git history is persistent and public/internal repos are leaked constantly. 4. If the secret appeared in CI logs, treat those logs as compromised and rotate too. ### 1.5 Debug Code & Temporary Scaffolding Audit ```bash # Find debug artifacts that must not ship git diff origin/main | grep -E "^\+" | grep -iE "(console\.log|debugger|TODO:|FIXME:|print\(|pprint\(|pp\(|binding\.pry|byebug|fmt\.Print)" | grep -v "test\|spec\|_test\.go" ``` Review every hit. Not all are blockers — a `console.log` in a test file is fine; a `console.log('SECRET VALUE:', token)` in a route handler is not. Use judgment, but don't let the grep pass unread. ### 1.6 Documentation Currency Check - [ ] `README.md` reflects any new environment variables, setup steps, or API changes. - [ ] `CHANGELOG.md` has an entry for this release (see Release Notes section below). - [ ] API documentation is updated if public-facing contracts changed. - [ ] Any architectural decision records (ADRs) created if a significant design choice was made. - [ ] `.env.example` updated if new env vars were added. --- ## Step 2: Test Framework Bootstrapping If the project has no test framework, or the user says "we don't have tests yet," do not skip to shipping. Bootstrap a minimal, working test harness first. A single working test is worth more than 100 tests that don't run. ### Detect What's Missing ```bash # JavaScript/TypeScript ls package.json && cat package.json | grep -E '"(jest|vitest|mocha|test)"' # Python ls pytest.ini pyproject.toml setup.cfg 2>/dev/null | head -1 # Go ls *_test.go 2>/dev/null | head -1 ``` ### Bootstrap by Ecosystem **Node.js / TypeScript — Vitest (preferred for modern projects)** ```bash npm install -D vitest @vitest/coverage-v8 # vitest.config.ts cat > vitest.config.ts << 'EOF' import { defineConfig } from 'vitest/config' export default defineConfig({ test: { globals: true, environment: 'node', coverage: { provider: 'v8', reporter: ['text', 'lcov', 'html'], thresholds: { lines: 80, functions: 80, branches: 70, }, }, }, }) EOF # package.json scripts npm pkg set scripts.test="vitest run" npm pkg set scripts.test:watch="vitest" npm pkg set scripts.test:coverage="vitest run --coverage" ``` **Python — pytest** ```bash pip install pytest pytest-cov pytest-asyncio cat > pyproject.toml << 'EOF' [tool.pytest.ini_options] testpaths = ["tests"] asyncio_mode = "auto" [tool.coverage.run] source = ["src"] omit = ["*/tests/*", "*/migrations/*"] [tool.coverage.report] fail_under = 80 show_missing = true EOF mkdir -p tests touch tests/__init__.py tests/conftest.py ``` **Go — built-in testing + testify** ```bash go get github.com/stretchr/testify/assert go get github.com/stretchr/testify/require # Example test file scaffold cat > internal/yourpackage/yourpackage_test.go << 'EOF' package yourpackage_test import ( "testing" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) func TestSomething(t *testing.T) { t.Run("expected behavior", func(t *testing.T) { result, err := SomeThing(input) require.NoError(t, err) assert.Equal(t, expected, result) }) } EOF ``` Write at least one passing test for the most critical path in the codebase before proceeding to ship. If nothing critical has tests, write one for the entry point or the most frequently called function. --- ## Step 3: Commit & Push ### 3.1 Stage Changes Intentionally ```bash # Review exactly what you're about to commit git diff --staged # Stage by hunk, not by file — avoid accidentally staging debug code # that survived the audit in Step 1.5 git add -p # Final pre-commit check git status ``` Never use `git add .` blindly on a release commit. Review what's staged. ### 3.2 Conventional Commits Format Every commit message must follow Conventional Commits. This is not a style preference — it's what enables automated changelog generation, semantic versioning, and legible history for the next engineer. ``` (): [optional body] [optional footer: BREAKING CHANGE, Closes #123, etc.] ``` **Types:** | Type | When to use | |---|---| | `feat` | A new feature visible to users or consumers of the API | | `fix` | A bug fix | | `perf` | A performance improvement with no behavior change | | `refactor` | Code restructuring with no behavior or API change | | `test` | Adding or fixing tests only | | `docs` | Documentation only | | `chore` | Build system, dependency updates, tooling | | `ci` | CI/CD configuration changes | | `revert` | Reverting a previous commit | **Rules:** - Subject line: imperative mood, lowercase, no period, ≤72 characters. `fix: handle null user on login` not `Fixed the null user bug on login.` - Body: explain *why*, not *what*. The diff explains what. The why lives only in the commit message. - `BREAKING CHANGE:` footer is mandatory if the commit changes a public API, env var name, DB schema in a non-backward-compatible way, or removes a previously available behavior. ```bash # Good commits git commit -m "feat(auth): add refresh token rotation on every use" git commit -m "fix(payments): handle Stripe webhook timeout with idempotency key" git commit -m "chore(deps): bump express from 4.18.1 to 4.19.2" # With breaking change git commit -m "feat(api)!: rename /user/profile to /users/{id}/profile BREAKING CHANGE: All clients using the old endpoint path must update. The old path now returns 410 Gone with a migration header." ``` ### 3.3 Push ```bash git push origin # If the branch already has a remote and you rebased: git push --force-with-lease origin # --force-with-lease is safer than --force: it will refuse to push if someone # else pushed to the branch since you last fetched, protecting against # overwriting their work. ``` --- ## Step 4: Open the Pull Request ### 4.1 PR Title The PR title becomes the squash-merge commit message. It must follow Conventional Commits exactly. ``` feat(scope): short description of what this PR does ``` ### 4.2 PR Description Template ```markdown ## What & Why Closes # ## How It Works ## Testing - [ ] Unit tests added / updated - [ ] Integration tests pass locally - [ ] Manually tested against: ## Pre-Ship Checklist - [ ] `git rebase origin/main` — branch is current - [ ] Lint passes (`npm run lint` / `ruff check .` / etc.) - [ ] Tests pass with coverage above threshold - [ ] No secrets in diff (`trufflehog` / `gitleaks` clean) - [ ] No debug code or `console.log` in production paths - [ ] `.env.example` updated if new env vars added - [ ] `CHANGELOG.md` entry added - [ ] Documentation updated if behavior changed ## Screenshots / Recordings ## Rollback Plan ``` ### 4.3 Reviewer Assignment - Assign at least one reviewer who understands the changed domain. - For changes touching security, auth, payments, or data persistence: require a second reviewer. - For changes touching public APIs or breaking changes: notify downstream consumers before merging. --- ## Step 5: Land & Deploy Workflow ### 5.1 Wait for CI Do not merge until all required CI checks pass. Green CI is a necessary condition, not a sufficient one — but a red CI is always a hard stop. ```bash # Monitor CI from the terminal (GitHub CLI) gh pr checks --watch # Or just watch the PR page. Either way, don't merge on red. ``` If CI is flaky (tests passing on re-run without code changes), fix the flakiness before merging. Flaky CI trains engineers to ignore failures — that's how a real failure gets missed. ### 5.2 Address Review Feedback - Respond to every comment, even if only to say "done" or "I disagree because X." - If a review comment surfaces a design question that changes the approach, update the PR description to reflect the final approach — the PR description is the permanent record of what was decided and why. - Re-run the full pre-ship checklist (Step 1) after making review-driven changes. ### 5.3 Merge Strategy Choose the merge strategy that matches the project's history policy: | Strategy | When to use | |---|---| | **Squash merge** | Default for most projects. One commit per PR, clean linear history, PR title becomes the commit message. | | **Rebase merge** | When the branch commits are already clean, atomic, and individually meaningful. Preserves each commit. | | **Merge commit** | When you need a record that a merge happened (e.g., release branch into main). Avoid for feature branches. | ```bash # Squash merge via GitHub CLI gh pr merge --squash --delete-branch # Or via git directly git checkout main git merge --squash git commit -m "feat(scope): your pr title here" git push origin main git branch -d ``` ### 5.4 Post-Merge CI Gate After merge, wait for the `main` branch CI to complete before assuming the deploy will succeed. Merge conflicts resolved incorrectly, or interactions between two simultaneously merged PRs, can cause a green PR to produce a red main. ```bash gh run watch --repo / ``` ### 5.5 Deploy to Production Deployment mechanism depends on the project's CD setup. In all cases: **GitOps (ArgoCD/Flux):** ```bash # Update the manifest repo with the new image SHA # ArgoCD/Flux will detect the change and sync automatically. # Watch sync status: argocd app get --watch # or: flux get kustomizations --watch ``` **Direct kubectl:** ```bash kubectl set image deployment/ =: -n kubectl rollout status deployment/ -n # rollout status blocks until the new pods are healthy or the timeout elapses. ``` **Platform CLI (Heroku, Railway, Render, Fly.io):** ```bash # Heroku heroku releases --app # confirm the release is present heroku ps --app # confirm dynos are running # Fly.io fly status --app fly logs --app ``` --- ## Step 6: Post-Deploy Health Verification This step is mandatory. "Deployed" ≠ "working." A deploy that silently starts failing 2 minutes after going live is worse than a deploy that fails immediately, because by the time you notice, real users have hit real errors. ### 6.1 Immediate Verification (First 5 Minutes) ```bash # Check error rate — this is the single most important signal # Compare the 5-minute window after deploy to the 5-minute window before. # Application logs kubectl logs -l app= -n --tail=100 --since=5m # or: heroku logs --tail --app # Health endpoint curl -sf https:///health | jq . # Expect: { "status": "ok", "version": "", "dependencies": { "db": "ok" } } # Spot-check critical user flows curl -sf -H "Authorization: Bearer $TEST_TOKEN" https:///api/users/me | jq . ``` ### 6.2 Metrics Dashboard (First 15 Minutes) Open your observability dashboard and verify: - [ ] **Error rate** — 5xx rate is at or below pre-deploy baseline - [ ] **Latency** — p95 and p99 latency within acceptable bounds (define these before deploying, not after) - [ ] **Throughput** — request rate is normal (a sudden drop often means a load balancer health check failure pulling the instance from rotation) - [ ] **Deployment marker** visible on the timeline — confirms the dashboard is showing post-deploy data - [ ] **No new alert firings** in the alerting system If any of these are abnormal, move immediately to the Rollback Procedure (Step 7). Do not wait to "see if it stabilizes." ### 6.3 Sustained Observation Window For a normal release: observe for 15 minutes post-deploy before considering the release closed. For a release touching critical paths (payments, auth, data pipelines): 30 minutes minimum. For a major version or architectural change: 1 hour, with an on-call engineer watching. --- ## Step 7: Rollback Procedure A rollback is not a failure. A rollback is the correct response to a production incident caused by a deploy. Executing it quickly and calmly is a mark of operational maturity. ### 7.1 Decision Threshold Initiate rollback immediately if any of these are true within the observation window: - Error rate exceeds 1% (or your defined SLO threshold) for more than 2 minutes - p99 latency increases by more than 2× baseline - Any crash loop, OOMKill, or pod restart storm - Any alert fires that wasn't firing before the deploy - Any critical user-facing flow is broken (login, checkout, data retrieval) Do not wait for the incident to be "confirmed." Roll back, then investigate. It's faster to recover and do RCA than to debug under live traffic. ### 7.2 Application Rollback **Kubernetes:** ```bash # Roll back to the previous deployment revision (fastest) kubectl rollout undo deployment/ -n kubectl rollout status deployment/ -n # Or roll back to a specific revision kubectl rollout history deployment/ -n kubectl rollout undo deployment/ --to-revision= -n ``` **GitOps:** ```bash # Revert the manifest change in the GitOps repo and push git revert HEAD --no-edit git push origin main # The controller will reconcile automatically. Watch: argocd app get --watch ``` **Docker / Direct deploy:** ```bash # Re-deploy the previous known-good image SHA docker pull : docker stop && docker run -d --name : ``` **Platform CLI:** ```bash # Heroku heroku rollback --app # rolls back one release heroku releases --app # verify # Fly.io fly deploy --image : --app ``` ### 7.3 Database Rollback Database rollbacks are the hardest part of any release. Handle them carefully: - If the migration added a nullable column or a new index: rolling back the app code is usually sufficient — the column stays but is ignored. - If the migration renamed or removed a column: you need to run the down migration AND ensure no in-flight requests are using the old column during the transition. - If the migration changed data (a data migration): rollback may be impossible without a restore from backup. This is why data migrations should be idempotent and run in separate, independently reversible steps. ```bash # Run the down migration (schema rollback) # Django python manage.py migrate # Alembic alembic downgrade -1 # Flyway flyway undo # Prisma — requires shadow database or manual SQL # There is no built-in down migration in Prisma. Write reverse SQL. ``` **Golden rule:** Never run a database migration that cannot be cleanly reversed without a full restore, unless you have a tested restore procedure and a maintenance window. ### 7.4 Post-Rollback 1. Verify production is healthy using the same checks in Step 6. 2. Communicate status: post a message in your incident channel that the rollback is complete and the system is stable. 3. Open a post-mortem ticket: what failed, when, how it was detected, and what the fix will be before the next ship attempt. 4. Do not re-attempt the release until the root cause is understood and fixed. --- ## Step 8: Release Notes Generation Release notes are written for two audiences: engineers (what changed technically) and users/stakeholders (what changed for them). Don't conflate them. ### 8.1 CHANGELOG.md — Engineer-Facing Format: [Keep a Changelog](https://keepachangelog.com) — sections by type under a version header. ```markdown ## [1.4.0] — 2026-07-22 ### Added - Refresh token rotation on every use — improves session security for all users. - `GET /users/{id}/profile` endpoint replacing the deprecated `/user/profile`. ### Fixed - Stripe webhook timeout now handled correctly with idempotency key, preventing duplicate charge processing under high load. - Null pointer on login when user record has no associated OAuth provider. ### Changed - Upgraded `express` from 4.18.1 to 4.19.2 (security patch). ### Deprecated - `GET /user/profile` — returns 301 to new path. Will be removed in v2.0. ### Breaking Changes - None in this release. ## [1.3.2] — 2026-07-10 ... ``` ### 8.2 Automated from Git Log ```bash # Generate a draft CHANGELOG entry from commits since last tag git log $(git describe --tags --abbrev=0)..HEAD \ --pretty=format:"- %s (%h)" \ --no-merges \ | grep -E "^- (feat|fix|perf|refactor|docs|chore|ci)(\(.*\))?:" \ | sort # Or use a tool npx conventional-changelog-cli -p angular -i CHANGELOG.md -s # or: git-cliff --tag v1.4.0 --output CHANGELOG.md ``` Review and edit the generated draft. Automated tools get the structure right; the human adds the "why this matters" context. ### 8.3 User-Facing Release Notes For user-visible releases (public APIs, customer-facing products): ```markdown # Release Notes — v1.4.0 (July 22, 2026) ## What's New **Improved session security** — sessions now automatically rotate tokens on each use, reducing the window of exposure if a token is ever intercepted. ## Bug Fixes **Duplicate charge prevention** — a rare edge case where a slow network could cause a payment to be processed twice has been resolved. ## Deprecation Notice The `/user/profile` endpoint is deprecated and will be removed in v2.0. Please migrate to `/users/{id}/profile`. Both endpoints return the same data during the transition period. ``` --- ## Step 9: Hotfix Path A hotfix is a release that bypasses the normal release queue because production is broken now. ### 9.1 Hotfix Branch ```bash # Branch from the current production tag, not from main # (main may have unreleased work that shouldn't go to production) git fetch --tags git checkout -b hotfix/- v # Example: git checkout -b hotfix/INC-4421-null-user-crash v1.3.2 ``` If `main` and the production tag are the same commit (i.e., everything merged to main is deployed), you can branch from `main` directly. ### 9.2 Hotfix Development Apply the minimum change to fix the production issue. This is not the time to refactor, clean up, or add features. Every additional change is additional risk and additional review surface. ```bash # Make the fix git add -p git commit -m "fix(auth): prevent null pointer when user has no OAuth provider Closes #INC-4421. The login handler was not checking for a nil OAuthProvider record before dereferencing it. Added nil guard and falling back to password auth. Affects users who signed up via email/password and never connected OAuth." ``` ### 9.3 Fast-Track Review - Assign your most senior available reviewer. - Describe the production impact (who is affected, how severely, since when) in the PR description. - Minimum one approval required — do not merge unreviewed, even in an emergency. - The pre-ship checklist still applies, but run it in parallel with review rather than sequentially. ### 9.4 Deploy Hotfix Follow the standard deploy and verification steps (Steps 5–6). A rushed verification on a hotfix is how you turn one incident into two. ### 9.5 Backport to Main After the hotfix is verified in production: ```bash # Cherry-pick the fix commit onto main git checkout main git cherry-pick git push origin main # Open a minimal PR if your branch protection rules require it. ``` This ensures the fix isn't regressed by the next normal release. ### 9.6 Hotfix Release Notes Tag the hotfix release: ```bash git tag -a v1.3.3 -m "Hotfix: prevent null pointer on login for email/password users" git push origin v1.3.3 ``` Add a CHANGELOG entry: ```markdown ## [1.3.3] — 2026-07-22 ### Fixed - **CRITICAL:** Null pointer crash on login for users without a linked OAuth provider. Affected email/password-only accounts since v1.3.0 under certain load conditions. (INC-4421) ``` --- ## Definition of Done — Release A release is complete when every item here is checked. Not "CI is green." Not "PR is merged." Every item. ### Code Quality - [ ] Branch rebased on `origin/main` — no stale base, no merge noise in the diff - [ ] Lint passes with zero warnings in production-scope files - [ ] Type check passes (no `any` suppressions added without a comment) - [ ] All tests pass locally and in CI - [ ] Coverage at or above the project threshold — any drop is explained and justified - [ ] No secrets in the diff (`gitleaks` or `trufflehog` clean) - [ ] No debug code, `console.log`, `debugger`, `print()`, `pry`, or equivalent in production paths - [ ] No `TODO` or `FIXME` comments that block this release (existing ones are acceptable, new ones that block the feature are not) ### Process & Documentation - [ ] PR description filled out — what, why, how, testing notes, rollback plan - [ ] At least one approval from a reviewer who understands the changed domain - [ ] `CHANGELOG.md` entry added with the correct version and date - [ ] `.env.example` updated if new environment variables added - [ ] API documentation updated if public contracts changed - [ ] README updated if setup, configuration, or architecture changed materially ### Deployment & Verification - [ ] CI is green on `main` after merge — not just on the PR branch - [ ] Deploy initiated and confirmed (correct SHA/image visible in target environment) - [ ] Health endpoint returns `200 OK` with expected version - [ ] Error rate baseline unchanged for 15 minutes post-deploy (30 minutes for critical paths) - [ ] p95/p99 latency within acceptable bounds for 15 minutes post-deploy - [ ] No new alerts firing in the alerting system - [ ] Deployment marker visible on observability dashboards ### Release Artifacts - [ ] Git tag created for versioned releases (`git tag -a v -m "..."`) - [ ] Tag pushed to origin (`git push origin v`) - [ ] Release notes published (GitHub Releases, internal wiki, or equivalent) - [ ] Hotfix commits cherry-picked to `main` if applicable ### Operational Readiness - [ ] Rollback procedure is documented and has been verified to work (at minimum, you know what the previous image SHA was and how to re-deploy it) - [ ] On-call engineer aware of the release if it touches a critical path - [ ] Any dependent teams notified of breaking changes before the release went live — not after