# Entroly Context Index — Competitive Benchmark

> Auto-generated by `bench/compare.py`. Run `entroly benchmark` to reproduce on your codebase.

## Setup

- **Corpus**: 19 production-realistic code fragments (Python, TypeScript, SQL, Rust, config)
- **Token budget**: 300 tokens (forces trade-offs — real-world constraint)
- **Total corpus**: 915 tokens (3× budget — cannot fit everything)

## Strategies Compared

| Strategy | What it does | Scope |
|---|---|---|
| **RAW (Naive FIFO)** | Stuff tokens in file-system order until budget exhausted | Local baseline |
| **TOP-K (Cosine)** | Rank by query similarity, take top-K that fit | Local baseline, not those products |
| **ENTROLY (Knapsack)** | Entropy-scored knapsack optimization + dedup + dep graph + SAST | Entroly |

---

<!-- BENCH:START -->
## Results

### Query: "authenticate user and process payment with rate limiting"

| Strategy | Fragments | Tokens | Utilization | Info Density | Relevance | Module Coverage | SAST Catches |
|---|---|---|---|---|---|---|---|
| RAW (Naive FIFO) | 6 | 295 | 98% | 0.751 | 50% | 3 | 0 |
| TOP-K (local baseline) | 6 | 290 | 97% | 0.686 | 50% | 3 | 0 |
| **ENTROLY (Knapsack)** | 9 | 300 | 100% | 0.721 | 75% | 9 | 1 |

### Query: "fix the SQL injection vulnerability in the search endpoint"

| Strategy | Fragments | Tokens | Utilization | Info Density | Relevance | Module Coverage | SAST Catches |
|---|---|---|---|---|---|---|---|
| RAW (Naive FIFO) | 6 | 295 | 98% | 0.751 | 14% | 3 | 0 |
| TOP-K (local baseline) | 6 | 300 | 100% | 0.702 | 14% | 4 | 0 |
| **ENTROLY (Knapsack)** | 9 | 300 | 100% | 0.721 | 29% | 9 | 1 |

### Query: "add a refund button to the dashboard"

| Strategy | Fragments | Tokens | Utilization | Info Density | Relevance | Module Coverage | SAST Catches |
|---|---|---|---|---|---|---|---|
| RAW (Naive FIFO) | 6 | 295 | 98% | 0.751 | 20% | 3 | 0 |
| TOP-K (local baseline) | 6 | 295 | 98% | 0.747 | 60% | 4 | 0 |
| **ENTROLY (Knapsack)** | 8 | 270 | 90% | 0.777 | 40% | 8 | 1 |

---

## Summary

| Metric | RAW | TOP-K | ENTROLY |
|---|---|---|---|
| Avg fragments | 6.0 | 6.0 | **8.7** |
| Avg tokens used | 295 | 295 | 290 |
| Avg info density | 0.751 | 0.712 | **0.740** |
| Avg module coverage | 3.0 | 3.7 | **8.7** |
| Total SAST catches | 0 | 0 | **3** |
<!-- BENCH:END -->

### Key Findings

1. **Coverage is the killer metric**: Entroly sees **8.7 modules** where Top-K sees 3.7. When you ask "authenticate + pay + rate limit", Entroly includes auth, payments, AND middleware. Top-K gives you just auth + payments — missing the rate limiter entirely.

2. **Security in the context window**: Only Entroly surfaces the SQL injection vulnerability because its SAST scanner prioritizes dangerous code. Top-K and RAW both miss it — the LLM would never know the vulnerability exists.

3. **Dedup saves budget**: Entroly detects the near-duplicate `PaymentProcessorV2` and excludes it, freeing 60 tokens for an additional useful fragment. Top-K includes both copies.

---

## How to Run

```bash
# Run benchmark on the built-in corpus
entroly benchmark

# Run benchmark on YOUR codebase
entroly benchmark --project /path/to/your/code
```

## Methodology

All strategies operate on the **same corpus** with the **same token budget**. The small fixture is not representative of all tasks.

- **Info density** = Shannon entropy × (1 − boilerplate ratio) — higher means more useful information per token
- **Module coverage** = unique top-level directories represented — a coverage proxy, not measured understanding
- **SAST catches** = security vulnerabilities surfaced in the context window — fixture detections, not measured code-generation safety
- **Relevance** = fraction of query terms present in selected context

> **Note**: These benchmarks run on a synthetic but realistic corpus. Real-world numbers depend on your codebase structure, query patterns, and token budget. Run `entroly benchmark` to see your own numbers.

### What this benchmark cannot establish

This fixture compares selection strategies on 19 fragments and three queries. Its
maximum claim is that Entroly selected the listed fragments and scored as shown
under these fixture definitions. Module coverage, lexical relevance, entropy,
and SAST detections do **not** measure whether a model completed a task
correctly or whether an omitted fact was unnecessary.

No selector can recover evidence that was never indexed. If the smallest set
of evidence needed for a task exceeds the token budget, selection alone cannot
make that task fully grounded. If a later step introduces a new requirement
(for example, a symbol rename whose affected references appear later), the
earlier selection needs re-evaluation; a one-time compression score cannot
certify future sufficiency.

The fixture also does not measure prompt-cache loss from scattered fragments,
task-type differences, repeated-run variance, or a verified cost per completed
task. A stronger claim requires paired raw-versus-selected executions on held-out
repository tasks, exact context and model provenance, task outcome checks, and
reported failures by task type and budget. Results from that protocol should
be reported separately from this selector fixture and the historical LLM table.

---

## LLM Accuracy Retention (2026-04-22)

> Historical maintainer-reported results, not independently reproduced here.
> This experiment does not establish that compression preserves answer quality.

Model: `gpt-4o-mini` | Budget: 50,000 tokens | Wilson 95% Confidence Intervals

| Benchmark | n | Baseline (95% CI) | Entroly (95% CI) | Retention | Token Savings |
|---|---|---|---|---|---|
| **NeedleInAHaystack** | 20 | 100.0% [83.9–100%] | 100.0% [83.9–100%] | **100.0%** | 0.0% |
| **GSM8K** | 100 | 85.0% [76.7–90.7%] | 86.0% [77.9–91.5%] | **101.2%** | 3.6% |
| **SQuAD 2.0** | 100 | 84.0% [75.6–89.9%] | 83.0% [74.5–89.1%] | **98.8%** | 0.8% |
| **MMLU** (4-way MCQ) | 100 | 82.0% [73.3–88.3%] | 85.0% [76.7–90.7%] | **103.7%** | 0.0% |
| **TruthfulQA** (MC1) | 100 | 72.0% [62.5–79.9%] | 73.0% [63.6–80.7%] | **101.4%** | 0.1% |
| **LongBench** (HotpotQA) | 100 | 57.0% [47.2–66.3%] | 59.8% [49.8–69.0%] | **104.9%** | 3.6% |

### Benchmark Coverage

The harness now covers 7 public benchmarks across five evaluation axes:

| Axis | Benchmark | Tests |
|---|---|---|
| Retrieval | NeedleInAHaystack | Finding a fact in long context |
| Reasoning | GSM8K | Multi-step grade-school math |
| Code | HumanEval | Python function completion |
| Reading | SQuAD 2.0 | Short-passage QA |
| Knowledge | MMLU | 57-subject 4-way MCQ |
| Truthfulness | TruthfulQA MC1 | Resistance to common misconceptions |
| Long context | LongBench HotpotQA | Multi-hop QA over 10K–60K-token docs |

> **BFCL (Berkeley Function Calling)** is intentionally omitted — requires function-calling scaffolding with per-sample tool schemas and AST validation, and BFCL prompts are short enough that the compression selector passes through unchanged. Poor signal-to-noise fit for an accuracy-retention benchmark.

### Interpretation

- **Overlapping marginal CIs do not establish equivalence or non-inferiority.** The reported SQuAD score decreased from 84% to 83%. A paired evaluation with a predefined acceptable loss and adequate sample size is needed. The reported intervals are retained as historical values, not revalidated uncertainty estimates; Wilson intervals apply to binary outcomes, not averaged partial-credit scores.
- **Token reduction in this table is only 0–3.6%.** LongBench reports 12,885 baseline tokens versus 12,423 selected tokens. This does not establish quality under aggressive compression or identify why the answer score changed.
- MMLU, TruthfulQA, GSM8K, and SQuAD have short system contexts that fit within the 50K budget — Entroly correctly passes through (no artificial noise injection). These serve as **regression guards** against the selector corrupting short-context prompts.
- Needle contexts (4K–32K tokens) also fit within the 50K budget, so compression is not triggered there.
- Savings scale with how far a workload's context exceeds the configured budget. Most benchmarks here fit inside the 50K budget, so compression rarely fires; LongBench is the one that does, at 3.6%. Codebase workloads with far larger contexts reduce more, but this harness does not measure them and **no universal savings range is claimed** — a reported percentage is bounded by the configured budget before selection runs, so it describes the budget as much as the selector.

### Reproduce

```bash
export OPENAI_API_KEY=...

# Run everything (7 benchmarks, ~5–10 min on gpt-4o-mini at n=100)
python -m bench.accuracy --benchmark all --model gpt-4o-mini --samples 100

# Run one at a time
python -m bench.accuracy --benchmark mmlu       --samples 100
python -m bench.accuracy --benchmark truthfulqa --samples 100
python -m bench.accuracy --benchmark longbench  --samples 100
```

Historical engine provenance: not established by these result files. The repository version is not evidence of the version used in the historical run.

That line read `v0.9.0` until it was corrected in 1.0.82. It had been written in
the v1.0 founding commit and never moved again, so it named a pre-1.0 engine for
the entire life of the project up to that point. It was not a provenance record
either — the result files under `benchmarks/results/` carry no version field —
so re-run the commands above to measure against the engine you actually have.