# Proofmark benchmark report

- Run date: 2026-08-13
- Production model: `Proofmark/proofmark-webwild-v3`
- Browser weights: Q8 ONNX, SHA-256 `ed17ceb332bef84d0adcc2fa537eef85ed3ac6fb32c30393c326321fbbe54683`
- Decision threshold: **65%**, fixed before the final evaluation
- Bounty gate: **75% balanced accuracy**
- Proofmark release gate: **80% balanced accuracy**

## Binary result

**YES. The untouched generalized benchmark exceeds 80%.**

Proofmark scored **94.6% balanced accuracy** on 1,749 clean images and **91.4% balanced accuracy** on 1,749 deterministic web-stress variants. The final data came from a fourth Tiny-GenImage validation shard that was not used for training, calibration, model selection, or development.

| Final sealed test | AI recall | Real recall | Balanced accuracy | Correct |
| --- | ---: | ---: | ---: | ---: |
| Clean originals | **96.0%** (839/874) | **93.1%** (815/875) | **94.6%** | 1,654/1,749 |
| Web-stress variants | **93.6%** (818/874) | **89.3%** (781/875) | **91.4%** | 1,599/1,749 |

The ordinary-accuracy 95% Wilson interval is **93.4%–95.5%** clean and **90.0%–92.6%** under web stress. Both lower bounds remain above 80% on this sample.

## Generator breakdown

| Source | Clean recall | Web-stress recall |
| --- | ---: | ---: |
| ADM | 82.4% (103/125) | 82.4% (103/125) |
| BigGAN | 99.2% (124/125) | 100.0% (125/125) |
| GLIDE | 99.2% (123/124) | 92.7% (115/124) |
| Midjourney | 91.2% (114/125) | 86.4% (108/125) |
| Stable Diffusion 1.5 | 100.0% (125/125) | 98.4% (123/125) |
| VQDM | 100.0% (125/125) | 96.0% (120/125) |
| Wukong | 100.0% (125/125) | 99.2% (124/125) |
| Real | 93.1% (815/875) | 89.3% (781/875) |

The lowest final generator recall is ADM at 82.4%, so the aggregate result is not hiding a below-random generator. The degraded-real recall is 89.3%; that misses Proofmark's aspirational 90% sub-target by 0.7 points, but it does not miss either the bounty's 75% balanced-accuracy gate or the requested 80% gate.

## Data isolation and split

There is no misleading “80/20” claim. This was a multi-stage experiment with stricter role separation:

| Data role | Records | Used to change weights or threshold? |
| --- | ---: | --- |
| Head training | 4,614 image records; 9,228 frozen-backbone feature rows after augmentation | Weights only |
| Calibration | 1,104 clean/web records from four licensed domains | Threshold only |
| Development | 2,499 records, including the rejected v2 web set | Architecture and release decisions |
| Final sealed v3 | 1,749 originals plus 1,749 derived web variants | **No** |

The final set is balanced (874 AI / 875 real) and covers seven generators. Before scoring, its manifest was compared with 8,115 prior training, calibration, development, and sealed files. The audit found **zero SHA-256 duplicates and zero perceptual near-duplicates** at dHash distance ≤2 with matching aspect ratio.

The model hash, score fusion, raw cutoff (`0.020887045509173325`), displayed 65% cutoff, acceptance gates, and final manifest were frozen before labels from the final shard were scored. See:

- `benchmark-results/webwild-v3-release-lock.json`
- `benchmark-results/webwild-v3-locked-manifest.json`
- `benchmark-results/webwild-v3-sealed-original.json`
- `benchmark-results/webwild-v3-sealed-web.json`
- `benchmark-results/webwild-v3-release-summary.json`

## What changed and why it generalized

The original single-logit Community Forensics head generalized unevenly to ADM, BigGAN, and Midjourney after web transformations. Proofmark retained the MIT-licensed Community Forensics ViT backbone and froze it. It replaced only the classifier with a small 384→32→1 nonlinear head, trained with deterministic crops, JPEG recompression, blur, color, rotation, and erasing augmentation. Calibration and development were source-disjoint from the final shard.

This is intentionally a small local adaptation: the browser model remains about 23 MB and averaged 143 ms per clean image / 136 ms per web variant on the benchmark Mac CPU path. Quantized JavaScript outputs were used for the final numbers; Python training metrics were not substituted.

## Rejected experiments

| Version | Clean balanced | Web balanced | Why rejected |
| --- | ---: | ---: | --- |
| WebWild v1 linear head | 92.8% | 78.8% | Failed the 80% web gate |
| WebWild v2 nonlinear head | 94.9% | 86.6% | ADM web recall was only 66.4% |
| WebWild v3 production | **94.6%** | **91.4%** | Accepted; every generator ≥82.4% |

These failures remain checked in. They demonstrate that the protocol did not simply stop at the first attractive aggregate score.

## Web-stress protocol

Each sealed original receives exactly one deterministic transform selected from its precomputed SHA-256:

- JPEG quality 68 with 4:2:0 chroma subsampling;
- 62% downscale and WebP quality 72;
- centered 88% crop and JPEG quality 80; or
- light blur and JPEG quality 74.

The variant is linked to its original hash in the locked manifest. Variants are not independent observations, so confidence intervals are reported per pass rather than pretending there are 3,498 independent image families.

## Reproduce without model or API tokens

```bash
npm ci
npm run build:fresh
npm run check
npm run test:e2e

PROOFMARK_RESULTS_PATH=/private/tmp/results.json \
  npm run benchmark -- /path/to/labeled-set --quiet
```

The benchmark accepts `ai/` and `real/` folder trees and uses the same Q8 ONNX model, forensic fusion, and 65% displayed threshold as Chrome. No LLM, generator, cloud inference service, API key, or paid token is used.

- Final dataset: [TheKernel01/Tiny-GenImage](https://huggingface.co/datasets/TheKernel01/Tiny-GenImage), revision `6fe1304`
- Final source shard SHA-256: `e8efe734ebe1757d4995a763688ca91c5a2752edfea7f381708220d4c28b7ae2`
- Final manifest SHA-256: `c82cbda08af9f074e62357fc3a60b8bfbb90ad7313258414ccd36922dbc0e320`

## What this does—and does not—prove

The result is strong evidence that the frozen local detector exceeds 80% on a balanced, generator-diverse, browser-degraded sample it never used for fitting or selection. It is not mathematical proof of 80% on every future website. New generators, screenshots, severe edits, adversarial images, illustrations, and distribution shifts can still fail. Proofmark therefore presents a confidence and evidence panel rather than claiming forensic certainty.