--- name: crucible-investigation-methodology description: actively investigating a hypothesis — running a sweep, dispatching multi-agent analysis, designing serial adversarial gates, allowed-tools: Read, Write, Edit, Bash, Grep, Glob, Agent --- # Investigation Methodology — 6 execution patterns > **Self-Evolving Skill**: If any pattern here fails in practice (wrong results, wasted compute), update the section AND append to `references/evolution-log.md`. Don't defer. These 6 patterns executed in service of `a-research-foundations`. They are the "how" to the foundations' "why". Apply in roughly this order for a new hypothesis. --- ## 1. LLM-native data representation — quintile tokens Before asking an agent to "look at" numerical market data, encode as per-bar token sequences using **rolling quintile ranks** within a causal window. **Canonical schema**: ``` idx dir body_q range_q dur_q uwick_q lwick_q loc sess fwd+H... ``` Each quintile is `1..5` in a causal 200-bar rolling window (see Skill A §1). Agents can spot motifs like `+1:5:1|+1:5:1|+1:5:1` (three consecutive fast big-up bars) that are invisible in float-space. Context-budget rule: 60 KB tokenized stats-table fits in agent context; 67 MB raw bars don't. Full reference: `findings/methodology/01-llm-native-data-representation.md`. --- ## 2. Serial adversarial gates (A/B/C/D/E protocol) Before trusting any in-sample positive, survive 4-5 independent gates in series. | Gate | Question | Catches | | ---------------------------------------- | -------------------------------------------- | ----------------------------------------------------- | | A — Directional breakdown | Is edge from long/short/both? | Diffusive-looking edges that are actually directional | | B — Mirror symmetry | Does the inverse trigger show mirror edge? | Sample-window drift inflating one side | | C — OOS time-split (80/20 chronological) | Does finding survive on held-out later data? | In-sample overfit | | D — Cross-asset replay | Does it replicate on other symbols? | Asset-specific overfit | | E — Full-history per-year | Is it positive in ≥60% of years? | Single-year-luck | **Gate C is non-negotiable.** Never report a positive without time-split OOS. **Classify verdicts**: - All pass → `validated-cross-asset` (promote) - A+B+C pass, D fails → `validated-asset-specific` (narrow scope, don't kill) - C fails → kill and archive with `resurrect_if:` conditions Full reference: `findings/methodology/05-serial-adversarial-gates.md`. --- ## 3. Multi-lens agent synthesis (4-5 parallel specialists) When brute force is infeasible, launch 4-5 specialist agents in parallel with distinct analytical lenses. Convergence = validation; divergence = diagnostic. **Canonical lens set**: 1. Pattern-motif / n-gram 2. Regime / trend classifier 3. Morphology / wick hunter 4. Information theorist / surprisal 5. **Hidden-signal hunter / skeptic critic** — ALWAYS include; prevents confirmation bias **Rules**: - Same raw data, different lens prompts - Each agent briefed: "test against shuffled-null, estimate multiple-testing burden, require null-test z > 2 AND per-year stability" - 5th agent explicitly asks: "prior 4 agreed; do you concur or dissent?" - ≥3/5 convergence = likely real; only 1 agent = likely lens-specific artifact **Disagreement is information**. If two agents find different signals, test both independently. Full reference: `findings/methodology/04-multi-lens-agent-synthesis.md`. --- ## 4. Per-trade enrichment for loss postmortem When a signal works sometimes and fails sometimes, convert aggregate question → row-level question. **Pipeline**: ``` Step 1 — Run signal across full history → collect N trade outcomes Step 2 — Compute 20-30 causal features at each trigger bar Step 3 — Emit parquet (per-trade × feature matrix) Step 4 — Ship to multi-lens agents (§3) Step 5 — Each agent hunts filters Step 6 — Evaluate filters via shuffled-null + OOS ``` **Must-have feature categories** (at least one from each): - Trend/regime (SMAs, slopes, autocorrelations) - Volatility regime (rv, range compression, Kaufman ER) - Pattern context (quintiles at trigger + lags) - Exhaustion/cluster (bars-since-last-signal, signal-count-in-window) - Position (dist-to-swing-high ATR-normalized) - Calendar (hour, weekday, month, session) Session example: turned +0.178 bps baseline → +0.514 bps filtered (2.9× lift). Full reference: `findings/methodology/06-per-trade-enrichment-postmortem.md`. --- ## 5. Agnostic-null cascade (orthogonal null retest) When a signal passes one null type, retest with an **orthogonal** null before trust. Example from session: - Phase F-B shuffled the vol-forecast column → +0.370 signal "lost" to z=−5.15 under that null - Phase C shuffled the trigger mask (proper null) → +0.439 signal passed at z=+5.74 If Phase C had been the only test, we'd have trusted a potentially wrong null. If Phase F-B had been the only test, we'd have missed a real signal. **Rule**: whenever a positive emerges, identify AT LEAST ONE orthogonal null and retest. Common orthogonal pairs: - Feature-shuffle AND mask-shuffle - Permutation AND block-bootstrap - Shuffled-selection AND parametric (e.g., binomial tail) If signal passes both, trust more. If it passes one but fails the other, investigate why — the divergence tells you something about the signal's mechanism. --- ## 6. Compute orchestration (pueue + wrapper scripts) Long parallel sweeps go through pueue on BigBlack with a specific discipline. **Anti-pattern** (silently fails, 0-second "successes"): ```bash ssh bigblack '~/.local/bin/pueue add -- bash -c "VAR=1 python script.py"' ``` **Correct pattern**: ```bash cat > /tmp/run.sh <<'EOF' #!/usr/bin/env bash set -euo pipefail export VAR="${VAR:-default}" cd /tmp exec uv run --python 3.14 --with numpy --with pandas python -u /tmp/script.py EOF scp /tmp/run.sh bigblack:/tmp/ ssh bigblack 'chmod +x /tmp/run.sh && ~/.local/bin/pueue add --group mygroup --label "job" -w /tmp -- /tmp/run.sh' ssh bigblack '~/.local/bin/pueue wait --quiet; tail -50 ~/.local/share/pueue/task_logs/.log' ``` Key points: - Wrapper script (never inline `bash -c`) — pueue's `sh -c` wrapping strips inline quotes - `python -u` for unbuffered stdout (otherwise logs stay empty for 40 min) - `pueue restart --in-place ` (not bare `restart`, which creates new task IDs) - Use `fork` multiprocessing context to share pre-computed arrays via COW Full reference: `findings/methodology/08-compute-orchestration-pueue.md`. See also: `Skill(devops-tools:pueue-job-orchestration)` for extended patterns. --- ## Confirmation counts (provisional) | Pattern | Confirmed | Notes | | -------------------------- | ------------------------------ | --------------------------------------------------------- | | 1. quintile-tokens | 3 | Qualitative scan, Phase B stats, per-trade CSV | | 2. serial-gates | 1 major (NGRAM3FU 4-gate pass) | Proven template | | 3. multi-lens agents | 3 | Act-2 qualitative, Phase C stats, Phase L loss-postmortem | | 4. per-trade-enrichment | 1 | Phase L — needs re-confirmation | | 5. orthogonal null cascade | 1 | F-B vs C — the discovery of the principle itself | | 6. pueue orchestration | 30+ tasks | Very high confidence; standard infra | --- ## Post-Execution Reflection After invoking this skill: 1. Did a pattern catch a bug, save time, or produce a validated finding? Increment `confirmed` in the table above; note the session/audit folder in `references/evolution-log.md`. 2. Did a pattern fail (misled you, wasted compute)? Demote in the table with a brief note; add to evolution log with link to the failing session. 3. New execution pattern emerged? Draft a new section + append to evolution log. 4. Compute anti-pattern caught you again? Re-check §6 wording — clarify the trap.