# dsh-defend red-team benchmark results

> Deterministic scanner benchmark. Regenerate with
> `node --experimental-strip-types benchmark/run.mjs` (zero new dependencies; the scanner
> is the shipped `src/detect` source).

## Method

- **Detector**: `buildScanner()` — NFKC normalization, the Aho-Corasick automaton over all
  needles, per-rule tolerant regexes, and the Shannon-entropy gate on secret matches
  (default `minSecretEntropy: 3.0`). Each category is scanned with only that category's
  family enabled, so a match is a per-category detection.
- **Detection** = at least one match in the category's family.
- **Labels**: `label: true` = an attack/secret of that category; `label: false` = benign
  or near-miss text (including low-entropy secret lookalikes for the entropy gate).

## Per-category metrics

| Category | TP | FP | FN | TN | Precision | Recall | F1 | FPR |
|---|---|---|---|---|---|---|---|---|
| injection | 25 | 0 | 0 | 10 | 1.000 | 1.000 | 1.000 | 0.000 |
| jailbreak | 25 | 0 | 0 | 10 | 1.000 | 1.000 | 1.000 | 0.000 |
| secret | 22 | 1 | 3 | 9 | 0.957 | 0.880 | 0.917 | 0.100 |

## Overall

| Aggregate | Precision | Recall | F1 |
|---|---|---|---|
| Macro (mean of the three categories) | 0.986 | 0.960 | 0.972 |
| Micro (pooled TP/FP/FN across categories) | 0.986 | 0.960 | 0.973 |

- Samples: 105 total
  (75 positive,
   30 negative).
- Micro false-positive rate: 0.033.

## Per-sample notes

- Missed positives (false negatives): secret/sec-pos-23, secret/sec-pos-24, secret/sec-pos-25.
- Flagged negatives (false positives): secret/sec-neg-04.

## Comparison with the documented floor (27/28)

The repository pins a detection-rate floor on the upstream Prompt-Attack-Dataset fixture
(`fixtures/attacks.json`): **27 of 28** attacks detected (see README `Known limitations`
and `tests/detect.spec.ts`). Re-running that fixture with this same scanner reproduces:

- Detected: **27/28** (0.964).
- Missed: en_004.

The one documented miss (`en_004`'s lookalike Unicode) normalizes to "how to hack a system",
which the rule library still has no needle/regex for. This benchmark's category split is
stricter than the fixture: it adds per-category negatives (benign + low-entropy secret
lookalikes) and evaluates F1, not just recall, so a high detection rate here is a stronger
claim than the flat 27/28 recall.

## Known limitations (honest)

- **Vocabulary-bound.** Injection/jailbreak detection is only as broad as the ported rule
  library; novel phrasing and multi-step attacks can evade it.
- **Secret matching is structural.** Regex + entropy gate; no checksum/liveness validation
  and no issuer round-trip, so a well-formed synthetic credential is flagged by design.
- **GitHub token subtypes.** The secret vocabulary covers `ghp_` and `gho_` (plus
  `github_pat_`), not the `ghs_`/`ghr_`/`ghu_` GitHub App/refresh/user tokens, so
  those positives are reported as false negatives.
- **`aws-secret-key` grammar not in the committed dataset.** Its 40-character value is
  AWS-secret-shaped and GitHub push protection rejects committing such a literal, so that
  single grammar is exercised by a runtime test (`tests/benchmark.spec.ts`) instead of a
  dataset row. Every other secret grammar is measured here.
- **Entropy gate scores the whole regex match.** `found[0]` includes the key name and
  separator, so `password: <low-diversity-value>` can clear the gate when the key name
  itself adds enough entropy (see `sec-neg-04`).
- **No model verdicts.** The scanner is deterministic; it never judges intent.