--- name: soak-test description: "Soak test protocol for extended play — what to observe and log for slow leaks, fatigue, late-appearing edge cases." argument-hint: "[duration: 30m | 1h | 2h | 4h] [focus: memory | stability | balance | all]" user-invocable: true allowed-tools: Read, Glob, Grep, Write, Bash(bash "*/.claude/skills/soak-test/../../hooks/yaml-helper.sh" resolve_config *) model: sonnet --- !`bash "${CLAUDE_SKILL_DIR}/../../hooks/yaml-helper.sh" resolve_config --keys automation` **Automation mode**: Resolve `modes.automation` (`project.local.yaml` → `project.yaml` → default `collaborative`). Every `AskUserQuestion` call and every file write follows `.claude/docs/automation-modes.md` (collaborative asks always · guided major-only · autonomous logs and proceeds; `automation_always_ask` categories always prompt). # Soak Test A soak test (also called an endurance test) is an extended play session run with specific observation goals. Unlike a smoke check (broad critical path, ~10 min) or a single-feature playtest (~30 min), a soak test runs for **30 minutes to several hours** to surface: - **Memory leaks** — gradual heap growth that only appears after scene transitions - **Performance drift** — frame time degradation that worsens over time - **State accumulation bugs** — issues that only appear after N repetitions of a mechanic (inventory full, score overflow, AI state corruption) - **Fun fatigue** — mechanics that feel good in a first session but grow repetitive over extended play - **Content exhaustion** — the point where players run out of novel content **This skill generates the observation protocol and analysis harness — the human does the actual playing.** **Output:** `production/qa/soak-test-[date]-[duration].md` **When to run:** - Polish phase — before `/gate-check release` - After fixing a memory or stability issue (regression soak) - When extended play has not been formally tracked --- ## 1. Parse Arguments **Duration** (default: `1h`): - `30m` — short soak; suitable for testing a single mechanic or scene - `1h` — standard soak; covers most common leak categories - `2h` — extended soak; recommended for first full Polish soak - `4h` — deep soak; required for games with long session design (RPGs, sims) **Focus** (default: `all`): - `memory` — focus on heap size, object count, leak patterns - `stability` — focus on crash/freeze/hang detection - `balance` — focus on fun fatigue, content exhaustion, difficulty perception - `all` — all of the above --- ## 2. Load Context Read: - `project.yaml` — `engine.name` (for engine-specific memory monitoring guidance) and `performance.*` budgets (memory ceiling, target FPS); for any key absent or empty (including when `project.yaml` has no `performance` or `engine` block), fall back to `.claude/docs/technical-preferences.md` - `design/gdd/game-concept.md` — intended session length (for comparison against soak duration), core loop description - Most recent file in `production/qa/playtests/` — prior playtest findings (to avoid re-documenting known issues) - Most recent file in `production/qa/qa-plan-*.md` — current sprint test coverage (to understand what has been formally tested vs. what the soak covers) Note any performance budget targets (`performance.*` from `project.yaml`, else `.claude/docs/technical-preferences.md`): - Memory ceiling: [N MB, or "not set"] - Target FPS: [N, or "not set"] - Frame budget: [N ms, or "not set"] --- ## 3. Define Observation Checkpoints Based on duration, generate timed checkpoints: **30m soak**: T+0, T+10, T+20, T+30 **1h soak**: T+0, T+15, T+30, T+45, T+60 **2h soak**: T+0, T+20, T+40, T+60, T+80, T+100, T+120 **4h soak**: T+0, T+30, T+60, T+90, T+120, T+180, T+240 At each checkpoint, the observer records the observation items defined in Phase 4. --- ## 4. Generate the Soak Test Protocol ### Memory / Stability observation items (if focus = memory or all) Engine-specific monitoring guidance. > **Record the unit the tool shows; never convert, and never assume one.** > A soak test looks for **growth**, so every threshold below is a *ratio or a > delta against this session's own T+0 baseline* — which is unit-agnostic and > stays correct however the editor reports the number. Write the unit down at > T+0 exactly as displayed and use it consistently for the rest of the run. > > This replaces a note asserting the return units of > `Performance.get_monitor` — **NOT SOURCEABLE from `docs/engine-reference/`**, > the identifier appears nowhere in it — a claim that sat one line under a row asking > the tester to record "Static Memory (**KB**)". A wrong units claim in a leak > detector is off by 1024× in the one measurement the protocol exists to take, > and it would read as a plausible instruction throughout. Deltas need no such > claim, so the safest fix was to stop needing it. **Godot 4:** - Open Debugger → Monitors tab; track `Memory → Static Memory` and `Object Count → Objects` across checkpoints - Record: Static Memory (**unit as displayed**), Object Count, Orphan Nodes count - Alert threshold: Memory growth > 20% from T+0 after the first 15 minutes (some growth on load is expected; sustained growth indicates a leak) - **Orphan Nodes is the one absolute number worth watching**: it should return to its T+0 value after a scene unload. A ratio hides that; a non-zero floor that keeps rising is a leak regardless of units **Unity:** - Open Memory Profiler (Window → Analysis → Memory Profiler) - Record: Total Reserved Memory, GC Allocated, Object Count at each checkpoint (**units as displayed**) - Alert threshold: GC Allocated growing monotonically across 3+ checkpoints — a monotonicity check, deliberately unit-free **Unreal Engine:** - Use `stat memory` console command at each checkpoint - Record: Physical Memory Used, Physical Memory Available (**units as displayed**) - Alert threshold: Physical Memory Used growth **> 20% over the full soak**, measured against this run's T+0. The previous threshold was an absolute "> 50MB", which silently assumes both the unit *and* a project scale — 50MB is a rounding error for one game and a catastrophe for another ### Stability observation items (if focus = stability or all) At each checkpoint, note: - [ ] No crash, hang, or freeze occurred since last checkpoint - [ ] Frame rate still within target budget ([target FPS] fps) - [ ] Audio still playing correctly (no desync or silence) - [ ] All HUD elements still rendering correctly - [ ] Input responding as expected (no input loss or lag spike) ### Balance / fatigue observation items (if focus = balance or all) Collect subjective observations at each checkpoint: - [ ] Core mechanic still feels rewarding (Y/N) - [ ] Perceived difficulty level: [too easy / appropriate / too hard] - [ ] Any "I've seen this before" moments since last checkpoint? (novel content exhaustion) - [ ] Any moment of frustration since last checkpoint? Note cause. - [ ] Any moment of peak engagement since last checkpoint? Note cause. --- ## 5. Generate the Protocol Document ```markdown # Soak Test Protocol > **Date**: [date] > **Duration**: [duration] > **Focus**: [memory | stability | balance | all] > **Engine**: [engine] > **Generated by**: /soak-test --- ## Pre-Session Setup Before starting the soak: - [ ] Game is running from a **fresh launch** (not resumed from a prior session) - [ ] All background applications closed (minimise OS memory interference) - [ ] Performance monitoring tool open and recording: - **Godot**: Debugger → Monitors tab → Memory section visible - **Unity**: Memory Profiler window open - **Unreal**: `stat memory` ready in console - [ ] Soak target confirmed: [session design intent from game concept] - [ ] Prior known issues to watch for: [from most recent playtest / qa-plan] --- ## Baseline (T+0) — Record Before Playing | Metric | Baseline Value | |--------|---------------| | Memory / Heap | [record before first frame of gameplay] | | Object Count | [record] | | FPS (first 30 seconds) | [record] | | [Engine-specific metric] | [record] | --- ## Checkpoint Log ### T+[N] minutes **Memory / Stability** *(if applicable)*: | Metric | Value | Δ from Baseline | Alert? | |--------|-------|-----------------|--------| | Memory / Heap | | | | | Object Count | | | | | FPS | | | | | Crashes / Hangs | | | | **Stability checks**: - [ ] No crash or hang since last checkpoint - [ ] Frame rate within budget ([N] fps target) - [ ] Audio correct - [ ] HUD rendering correctly - [ ] Input responding correctly **Balance / Fatigue** *(if applicable)*: - Core mechanic still rewarding: Y / N - Difficulty perception: too easy / appropriate / too hard - Notable moments: [note any peak engagement or frustration] - Content exhaustion signs: Y / N — [describe] **Free observations**: *(Note anything unexpected observed since the last checkpoint)* --- [Repeat Checkpoint Log section for each timed checkpoint] --- ## Post-Session Analysis ### Memory Trend | Checkpoint | Memory | Δ/hr extrapolated | |------------|--------|-------------------| | T+0 | | | | [T+N] | | | **Leak detected?** Y / N **Estimated time to OOM at current rate**: [N hours / not applicable] ### Stability Summary Total crashes: [N] Total hangs: [N] Worst FPS observed: [N] fps at [checkpoint] Performance degradation: stable / mild / severe ### Balance / Fatigue Summary Fun curve: [engaged throughout / fatigue onset at T+N / repetitive from start] Content exhaustion point: [never / at T+N / early] Difficulty arc: [appropriate / too easy throughout / difficulty spike at T+N] ### Issues Found | ID | Severity | Checkpoint | Description | |----|----------|------------|-------------| | SOAK-001 | S[1-4] | T+[N] | [description] | --- ## Verdict: PASS / PASS WITH CONCERNS / NOT ASSESSED / FAIL **PASS**: No leaks detected, stability maintained, fun factor consistent **PASS WITH CONCERNS**: Minor drift or fatigue noted; addressable in Polish **NOT ASSESSED**: The soak did not run to a length that could show what it looks for — say how far it got and which checkpoints were never reached **FAIL**: Memory leak confirmed, stability breach, or severe fun fatigue > **A short soak cannot return PASS.** Everything this protocol exists to detect > — slow leaks, fatigue, late-appearing edge cases — is by definition invisible > early, so a session that ended before the checkpoints it was built around has > not shown stability; it has shown nothing yet. Record `NOT ASSESSED — reached > T+[N] of [duration]; checkpoints [list] not reached`. Ranks **above both pass > values** and **below FAIL**: a crash observed at T+20 is a real finding no > matter how short the run, and must not be demoted behind the run's length. > The same applies when the build crashed for reasons unrelated to the soak, when > no memory instrumentation was available (nothing was measured, so "no leaks > detected" means "no leaks could have been detected"), or when the protocol was > written but never executed — a protocol document is not a result. --- ## Sign-Off - **Tester**: [name] — [date] - **QA Lead review**: [name] — [date] ``` --- ## 6. Write Output Present the protocol summary in conversation, then ask: "May I write this soak test protocol to `production/qa/soak-test-[date]-[duration].md`?" Write only after approval. After writing: "Protocol written. To run the soak: 1. Open the file and follow the Pre-Session Setup checklist 2. Record each checkpoint as you play 3. Complete the Post-Session Analysis section when done 4. File bugs from 'Issues Found' to `production/qa/bugs/` 5. Run `/bug-triage sprint` after the session to integrate any S1/S2 issues If the verdict is FAIL, run `/smoke-check` again after fixing the issues." --- ## Collaborative Protocol - **This skill generates a protocol — humans run it** — never attempt to run a soak test automatically. The observations require a human observer. - **Duration should match the game's session design** — a 5-minute game doesn't need a 4h soak; a city-builder might. Use judgment and ask if unclear. - **First soak should be `all` focus** — narrow focus (memory-only) is for regression soaks after a specific fix, not the first pass - **Ask before writing** — always confirm before creating the protocol file