---
name: systematic-debugging
description: Debugs failures with a reproduce-hypothesize-instrument-root-cause loop instead of guess-and-patch. Use when facing a bug, failing test, crash, wrong output, flaky behavior, or regression - especially after one failed fix attempt or when tempted to "just try" changes.
license: MIT
metadata:
  author: lammanhhoang
  version: "1.0.0"
---

# Systematic Debugging

## When to use

Use for any gap between expected and observed behavior: failing test, crash, exception, wrong output, regression, intermittent/flaky failure, performance cliff. Especially when one fix attempt has already failed, or you notice yourself about to "just try" a change.

Do NOT use for: writing new features (no defect exists yet), a compile/import error whose message names the exact missing symbol (fix it directly), or pure style/lint cleanup. If that "direct fix" fails even once, this skill activates.

## The Iron Law

**NO FIX WITHOUT A REPRODUCTION AND A ROOT CAUSE YOU CAN STATE IN ONE SENTENCE.**

The sentence has a required shape: "X fails because Y." Example: "checkout_total is off by one cent because the discount is applied after per-line tax rounding instead of before." If you cannot produce that sentence with evidence behind both clauses, you are guessing. Guess-fixes create new bugs, and the guess that "works" usually masks the real defect — it returns later, in production, harder to find.

## The loop

1. **REPRODUCE deterministically.** A script or test that fails on demand. Record the exact command and its exit code. If you cannot reproduce, your job is data collection, not fixing: add logging, capture inputs, snapshot the environment. Never edit product code you cannot watch fail.
2. **READ the actual error.** The full message, the full stack, the exact file:line — not a skim. Take the FIRST error in the output, not the last (later errors are usually cascade). Open the named file at the named line before theorizing. Half of bugs die at this step.
3. **HYPOTHESIZE — one cause.** The single most likely cause given evidence you actually have, written as "X fails because Y." No hypothesis? You lack evidence — go back to step 2 or add instrumentation blind-spot coverage first.
4. **INSTRUMENT to test THAT hypothesis.** A targeted log line, breakpoint, assert, or bisect whose output will differ between hypothesis-true and hypothesis-false. Change ONE variable at a time. Instrumentation that cannot disconfirm your hypothesis is noise.
5. **CONFIRM the root cause.** Evidence must show Y happening — not merely be consistent with Y. If refuted: log the hypothesis, instrumentation, and evidence in the debug journal (assets/debug-log-template.md), return to step 3. After the SECOND refuted hypothesis, run the assumption audit below before hypothesis #3.
6. **WRITE the failing regression test** that encodes the root cause. Run it. Watch it fail for the right reason (the root-cause reason, not a typo in the test).
7. **FIX.** The minimal change that addresses Y. One logical change. No drive-by refactoring in the same commit.
8. **VERIFY all three:** the regression test passes; the original step-1 repro command is clean; the full suite is green. Paste all three outputs. Missing any one = not fixed.

## Assumption audit (mandatory after 2 refuted hypotheses)

Stop. Do not form hypothesis #3 from the same mental model that produced #1 and #2. The bug lives in a wrong assumption, not in the code you keep re-reading. List every assumption and verify each in under a minute:

| Assumption | 30-second verification |
|---|---|
| "My code is even executing" | Print/log at function entry; add a version string to startup logs |
| "The file I edit is the file that runs" | `python -c "import mod; print(mod.__file__)"`; checksum the deployed artifact |
| "The config in the repo is the config loaded" | Log the loaded values at startup, not the file contents |
| "The data is what I think it is" | Print `repr()` / `JSON.stringify` of the actual runtime value |
| "Dependency versions match" | `pip freeze \| grep pkg` / `npm ls pkg` in the FAILING environment |
| "The cache is not involved" | Clear it once; if behavior changes, cache is a suspect to investigate — not a fix |
| "This test is independent" | Run it alone; run it after only its suspected polluter |

Escalate instrumentation, never guess rate: more logging, a real debugger, a bisect — not faster random edits.

## Binary search tactics

**git bisect** — for "it worked before" regressions:

```bash
git bisect start
git bisect bad HEAD
git bisect good v2.3.0            # last commit known to work
git bisect run ./repro.sh         # exit 0 = good, 125 = skip, 1-127 otherwise = bad (126/127 count as bad; >127 aborts — see tactics.md)
# git prints "<sha> is the first bad commit"
git bisect reset
```

`repro.sh` is anything whose exit code encodes pass/fail, e.g. `pytest tests/test_totals.py::test_report -x -q`. Bisect is log2: ~13 automated runs search 8,000 commits. Use it before reading any diffs by hand.

**Delete-half** — for "this input breaks it": split the failing input in half, run the repro on each half, recurse into the half that still fails. Converges in log2(n) runs. If neither half fails alone, the failure needs interaction across the split — shrink by quarters instead.

**Minimal reproduction shrinking:** remove one element at a time (a dependency, a fixture, a flag, a line); re-run the repro after EVERY removal; keep the removal if it still fails, revert if it passes. Target: under 20 lines, single file, no project dependencies. You may not blame a framework or library until a shrunk repro that uses only that library still fails.

For the full tactic pack — bisecting flaky tests, exit-code traps, shrink scripts, environment diffing — read references/tactics.md (load when running a bisect, shrinking a repro, or chasing a flaky/heisenbug).

## Heisenbugs and flaky tests

Measure before touching anything — one green run of a 30% flake proves nothing:

```bash
for i in $(seq 20); do pytest tests/test_flaky.py -x -q > /dev/null 2>&1 || echo "FAIL run $i"; done
```

Record the failure count. After the fix, run the SAME command 20x again and paste both counts side by side.

Suspects, in order of prior probability:

1. **Test order / shared state** — run the failing test alone. Passes alone but fails in the suite? It's pollution: bisect the set of preceding tests.
2. **Concurrency** — races, missing await/join, thread-pool reuse. A `sleep()` that "fixes" it is a flag marking the exact location of the missing synchronization; find the real ordering guarantee.
3. **Time** — timezone, DST, token expiry, midnight boundaries, `now()` captured twice. Freeze the clock in tests.
4. **External state** — leftover DB rows, temp files, occupied ports, network.

## Rationalization table

| You are thinking | Reality |
|---|---|
| "Let me just try changing this" | Stop. That's rationalization, not a hypothesis. Name the "because Y" first — or go gather evidence. |
| "It's probably the framework / library / compiler" | It's your code until a <20-line standalone repro using only the library still fails. That is the entry fee for blaming upstream. |
| "Adding a sleep fixes it" | It hides a race. You now own the same bug plus a slower suite. Find the missing synchronization. |
| "It works on my machine" | Then a machine difference IS your evidence. Diff versions, env vars, locale, and input data between the two machines (references/tactics.md, section 6). |
| "One more tweak and it'll work" | You are past two refuted hypotheses. The assumption audit is mandatory now. |
| "The fix is obvious, no need to reproduce" | Then reproducing costs 2 minutes and proves you right. If reproducing is hard, the fix was not obvious. |
| "No time to write the regression test" | The test is the only proof the fix works and the only barrier to round 2. It is not optional overhead; it is step 6. |
| "Cannot reproduce — closing" | Closing without captured data means the bug returns with less patience attached. Add the logging that would catch the next occurrence, THEN close. |
| "Restarting / clearing cache fixed it" | Nothing was fixed. Something leaked, grew, or expired — find what, or schedule the rediscovery. |

## Red flags — stop immediately if you catch yourself

- Editing product code before a reproduction exists
- Making a 3rd distinct change with no confirmed hypothesis
- A debug commit touching 3+ unrelated locations
- Landing a fix with no new test that failed before and passes after
- Wrapping the failing line in try/except or catch-and-ignore — that is suppression, not repair
- Deleting an assert, widening a tolerance, or marking a test skip/flaky to get green
- Writing "cannot reproduce" without having added any instrumentation
- Declaring a flaky test fixed off a single green run

## Definition of fixed

Never claim fixed without pasting, verbatim:

1. Regression test output: failing before the fix, passing after.
2. The original repro command from step 1 running clean.
3. The full suite summary line, green. For flaky bugs: the 20x re-run count, before and after.

"Should be fixed now" without these three outputs is a guess wearing a suit.

## Debug journal

For any bug that survives one refuted hypothesis, keep a journal: copy assets/debug-log-template.md next to your work and fill it as you go — symptom, repro command, every hypothesis with its instrumentation and verdict, root-cause sentence, fix, regression test, verification outputs. The journal is what makes the assumption audit possible and stops you from re-testing refuted hypotheses.

## What NOT to do

- Never change two things between two observations — you learn nothing from the result.
- Never fix and refactor in the same commit.
- Never suppress a symptom (retry, sleep, try/except, tolerance bump) and call it a fix.
- Never trust a green run you did not watch fail first.
- Never skip step 1 because the bug "only happens in production" — capture production inputs and replay them; that IS the reproduction.