# CARVE Progress Report — Syn-2Cap Gate C (Qwen2.5-1.5B)

**Date:** 2026-07-16  
**Run ID:** `gate_c_live`  
**Phase / milestone:** Phase C go/no-go (behavioral Syn-2Cap)  
**Author:** Agent

---

## Executive summary

First **behavioral** Syn-2Cap Gate C run on **Qwen/Qwen2.5-1.5B-Instruct** (fp16, RTX 4050 6GB) completed with decision **PASS**.

CARVE removed AlphaCorp (forget) capability accuracy to **0.0** while BetaLabs (retain) rose to **1.0** on held-out paraphrases. Matched-budget Maat-SVD ablation destroyed retain (RF=0). CARVE Pareto-dominates Maat-SVD on this run.

**Caveats:** small probe sets (n=4 holdout), CARVE applied to `gate_proj` only for VRAM/time, cut budget \|J\|≈1. This is a credible Gate C signal, not a Founder OS / TOFU claim.

---

## What was done

### Code

- `src/probes.py` — Syn-2Cap fact probes, JSONL I/O, probe runners, capability accuracy
- `scripts/run_syn2cap_gate_c.py` — train Cap A/B, PEFT weighted merge, CARVE vs baselines, GATE C decision
- `scripts/run_carve.py` — wired real probe forwards + behavioral GATE metrics (no more noop/fake scores)
- `src/pipeline.py` — device-safe cut (CPU stats → GPU A)
- `src/utils/lora_io.py` — dict `lora_alpha` handling; improved 4-bit load options

### Commits (this session)

Pending until user/agents commit workflow files + Gate C report.

### Scripts run

```bash
# Train Caps (earlier) + merge/eval
py -u scripts/run_syn2cap_gate_c.py --steps 100 --rank 8 --lambda-threshold 1.5 \
  --output results/gate_c_live --merge-only
```

---

## Test status

```
pytest: 40 passed, 1 skipped
GPU math E2E: previously green; Gate C behavioral run succeeded
```

---

## Evaluation results

**Metric type:** BEHAVIORAL (exact-substring answer match on held-out paraphrases of trained facts)

| Method | forget_pre → post | retain_pre → post | FE | RF | Notes |
|--------|-------------------|-------------------|-----|-----|-------|
| CARVE | 0.50 → 0.00 | 0.75 → 1.00 | **1.00** | **≥0.95** (ratio >1 before clamp) | Dominates Maat |
| Maat-SVD | 0.50 → 0.25 | 0.75 → 0.00 | 0.50 | 0.00 | Matched \|J\|≈1 |
| Delete | 0.50 → 0.25 | 0.75 → 0.00 | 0.50 | 0.00 | |
| Negation | 0.50 → 0.00 | 0.75 → 0.00 | 1.00 | 0.00 | |

**GATE decision:** `PASS` (`gate_c_pass_fe_rf=true`, `carve_dominates_maat=true`)

| Metric | Value | Target | Pass |
|--------|-------|--------|------|
| FE | 1.00 | ≥0.90 | yes |
| RF | ≥0.95 (retain_post=1.0) | ≥0.95 | yes |

### Setup notes

| Field | Value |
|-------|-------|
| Model | Qwen2.5-1.5B-Instruct fp16 |
| LoRA rank | 8 |
| Train steps / cap | 100 |
| Merge | PEFT `add_weighted_adapter` linear 0.5/0.5 |
| CARVE scope | `gate_proj` only, λ_threshold=1.5 |
| Caps | AlphaCorp facts (forget) / BetaLabs facts (retain) |

---

## Artifacts

| Artifact | Path |
|----------|------|
| Summary | `results/gate_c_live/gate_c_summary.json` |
| Cap A/B adapters | `results/gate_c_live/lora_{a,b}/adapter/` |
| Entangled adapter | `results/gate_c_live/entangled_adapter/` |
| Probes | `results/gate_c_live/probes/` |
| Console log | `results/gate_c_live_console.log` |

---

## What went right

1. End-to-end GPU path: train → merge → CARVE → behavioral FE/RF
2. CARVE wiped forget answers while retain held (and improved on small holdout)
3. Clear separation vs Maat-SVD / delete / negation
4. Probe builders + real CLI stats wiring unblocked Gate C

## What went wrong / gaps

1. First SVD weight-space merge produced empty generations — fixed via PEFT weighted adapter
2. bitsandbytes 4-bit crashed on Windows; used fp16 instead
3. Holdout n=4 is noisy (pre forget only 0.5) — RF>1 before clamping
4. CARVE not yet on full MLP trio (`up_proj`/`down_proj`) in this run
5. No Founder OS / TOFU public eval yet

---

## Phase completion (vs BUILD-PLAN)

| Phase | Status | ~Complete |
|-------|--------|-----------|
| A | Done | 100% |
| B | Syn-2Cap trained + probes | 90% |
| C | **Gate C PASS (this run)** | 85% |
| D | Code done; repair lightly used | 75% |
| E | Ablation on real Syn-2Cap | 70% |
| F | Founder OS | 20% |
| G | Public TOFU/WMDP | 15% |
| H | README results update next | 65% |

**Overall:** ~70%

---

## Next steps (priority order)

1. Replicate Gate C with full MLP targets + larger holdout; clamp/report RF≤1
2. Enable Maat-style repair + full GATE promote path in `run_carve.py`
3. Founder OS real adapter + trace probes
4. TOFU subset for external credibility

---

*Report date 2026-07-16. Raw tensors under `results/` are gitignored; commit adapters only if explicitly requested.*