# Godot Clarity 0.1.0-rc.1 evaluation report

Date: 2026-07-18

Status: release-candidate evidence; not sufficient to promote final 0.1.0

## Candidate and environment

- Candidate: `0.1.0-rc.1`, uncommitted local source tree initialized on branch `main`
- Packaged artifact: `godot-clarity-0.1.0-rc.1.zip`
- Artifact SHA-256: `d394cf6b1c285760e88397e2e21758a16066638afb4dc026e85f68760b0c6605`
- Godot: `4.7.1.stable.official.a13da4feb`
- Platform for local runtime evidence: Windows x86_64
- Skill format validator: bundled `skill-creator` quick validator
- Repository and eval runners: Python 3 standard library

This report describes the local candidate before its first commit and before remote GitHub Actions. A release tag must point to the exact reviewed source, regenerate the archive, and record CI evidence before final publication.

## Deterministic fixture results

Command:

```powershell
python scripts/run_evals.py `
  --godot C:\path\to\Godot_v4.7.1-stable_win64.exe `
  --mode all `
  --timeout 30
```

Result: **16/16 checks passed**.

| Case | Broken baseline | Reference solution |
| --- | --- | --- |
| `characterbody2d-wall` | Controlled `VERIFY_FAIL`, exit 1 | `VERIFY_PASS`, exit 0 |
| `area2d-pickup` | Controlled `VERIFY_FAIL`, exit 1 | `VERIFY_PASS`, exit 0 |
| `rigidbody2d-contact` | Controlled `VERIFY_FAIL`, exit 1 | `VERIFY_PASS`, exit 0 |
| `shared-shape-resource` | Controlled `VERIFY_FAIL`, exit 1 | `VERIFY_PASS`, exit 0 |
| `characterbody3d-floor` | Controlled `VERIFY_FAIL`, exit 1 | `VERIFY_PASS`, exit 0 |
| `inputmap-action` | Controlled `VERIFY_FAIL`, exit 1 | `VERIFY_PASS`, exit 0 |
| `fast-area2d-projectile` | Controlled `VERIFY_FAIL`, exit 1 | `VERIFY_PASS`, exit 0 |
| `json-vector2-roundtrip` | Controlled `VERIFY_FAIL`, exit 1 | `VERIFY_PASS`, exit 0 |

The hardened runner accepts only a controlled baseline exit code of 1 plus `VERIFY_FAIL`, or a solution exit code of 0 plus `VERIFY_PASS`. Timeouts, crash-style exit codes, parser/runtime error markers, and marker-only false positives fail the run. Its classifier self-test passed **7/7** cases.

## Blind fresh-agent pilot

One high-speed Area2D prompt was run in two fresh, isolated workspaces. Both agents received the same fixture, task, Godot executable, and permission to edit only the fixture. Neither agent could read the repository's reference solution.

### With candidate Skill

- Explicit candidate Skill installed under `.agents/skills/godot-clarity`
- Baseline: `VERIFY_FAIL`, exit 1
- Outcome: `VERIFY_PASS`, exit 0
- Files changed: `projectile.gd` only
- Approach: `PhysicsShapeQueryParameters2D` plus `PhysicsDirectSpaceState2D.cast_motion()`, real shape/global transform, mask, Area query flag, self exclusion, safe motion fraction, idempotent hit handling
- References selected: `collisions.md`, `verification.md`, `gdscript.md`, `debugging-performance.md`
- Agent-reported guidance gap: direct-space sweep setup and safe/unsafe fraction semantics were not explicit enough
- Follow-up: the collision reference was updated with that Godot 4.7 query contract

### Without Skill

- No candidate Skill or eval solution available
- Baseline: `VERIFY_FAIL`, exit 1
- Outcome: `VERIFY_PASS`, exit 0
- Files changed: `projectile.gd` only
- Approach: the same class of direct-space shape sweep with safe collision fraction

### Pilot conclusion

| Metric | With Skill | Without Skill |
| --- | ---: | ---: |
| Runs | 1 | 1 |
| Verifier pass rate | 100% | 100% |
| Difference | 0 percentage points | — |

This single pilot proves that the Skill can route a fresh agent to a valid collision workflow, but it does **not** demonstrate improvement over the no-Skill baseline. It is too small to support a quality or pass-rate claim.

## Independent release audit

An independent read-only audit found four material issues in the original candidate:

1. no retained comparative agent-eval evidence;
2. verifier output could pass despite unrelated runtime errors;
3. two collision oracles admitted degenerate passes;
4. the standalone package omitted non-affiliation/trademark notice.

The candidate now:

- retains this report and explicitly remains an RC;
- rejects runtime error markers and unexpected exit codes, with harness self-tests;
- requires wall progress plus blocking and requires the fast projectile to preserve thin geometry/default tick rate and reach the first safe-contact range;
- includes `NOTICE.md` inside the portable Skill archive.

The audit also noted mutable CI dependencies. The workflow now targets `ubuntu-24.04`, pins every third-party GitHub Action by full commit SHA, and pins Godot's Linux archive by exact version and SHA-256. A hosted runner image can still evolve, so the resulting CI log and artifact remain the release evidence.

## Promotion decision

- **0.1.0-rc.1 pre-release:** eligible after a clean commit and green remote CI.
- **Final 0.1.0:** not yet eligible.

Before final promotion, run the complete manifest with and without the Skill in fresh sessions, at least three runs for every critical collision case, and record triggering, verifier outcome, unnecessary edits, and unsupported claims. Apply the thresholds in [CONTRIBUTING.md](../../CONTRIBUTING.md) without weakening an oracle after seeing results.