# Entroly Context Index — Competitive Benchmark

> Auto-generated by `bench/compare.py`. Run `entroly benchmark` to reproduce on your codebase.

## Setup

- **Corpus**: 19 production-realistic code fragments (Python, TypeScript, SQL, Rust, config)
- **Token budget**: 300 tokens (forces trade-offs — real-world constraint)
- **Total corpus**: 915 tokens (3× budget — cannot fit everything)

## Strategies Compared

| Strategy | What it does | Scope |
|---|---|---|
| **RAW (Naive FIFO)** | Stuff tokens in file-system order until budget exhausted | Local baseline |
| **TOP-K (Cosine)** | Rank by query similarity, take top-K that fit | Local baseline, not those products |
| **ENTROLY (Knapsack)** | Entropy-scored knapsack optimization + dedup + dep graph + SAST | Entroly |

---

<!-- BENCH:START -->
## Results

### Query: "authenticate user and process payment with rate limiting"

| Strategy | Fragments | Tokens | Utilization | Info Density | Relevance | Module Coverage | SAST Catches |
|---|---|---|---|---|---|---|---|
| RAW (Naive FIFO) | 6 | 295 | 98% | 0.751 | 50% | 3 | 0 |
| TOP-K (local baseline) | 6 | 290 | 97% | 0.686 | 50% | 3 | 0 |
| **ENTROLY (Knapsack)** | 9 | 300 | 100% | 0.721 | 75% | 9 | 1 |

### Query: "fix the SQL injection vulnerability in the search endpoint"

| Strategy | Fragments | Tokens | Utilization | Info Density | Relevance | Module Coverage | SAST Catches |
|---|---|---|---|---|---|---|---|
| RAW (Naive FIFO) | 6 | 295 | 98% | 0.751 | 14% | 3 | 0 |
| TOP-K (local baseline) | 6 | 300 | 100% | 0.702 | 14% | 4 | 0 |
| **ENTROLY (Knapsack)** | 9 | 300 | 100% | 0.721 | 29% | 9 | 1 |

### Query: "add a refund button to the dashboard"

| Strategy | Fragments | Tokens | Utilization | Info Density | Relevance | Module Coverage | SAST Catches |
|---|---|---|---|---|---|---|---|
| RAW (Naive FIFO) | 6 | 295 | 98% | 0.751 | 20% | 3 | 0 |
| TOP-K (local baseline) | 6 | 295 | 98% | 0.747 | 60% | 4 | 0 |
| **ENTROLY (Knapsack)** | 8 | 270 | 90% | 0.777 | 40% | 8 | 1 |

---

## Summary

| Metric | RAW | TOP-K | ENTROLY |
|---|---|---|---|
| Avg fragments | 6.0 | 6.0 | **8.7** |
| Avg tokens used | 295 | 295 | 290 |
| Avg info density | 0.751 | 0.712 | **0.740** |
| Avg module coverage | 3.0 | 3.7 | **8.7** |
| Total SAST catches | 0 | 0 | **3** |
<!-- BENCH:END -->

### Key Findings

1. **Coverage is the killer metric**: Entroly sees **8.7 modules** where Top-K sees 3.7. When you ask "authenticate + pay + rate limit", Entroly includes auth, payments, AND middleware. Top-K gives you just auth + payments — missing the rate limiter entirely.

2. **Security in the context window**: Only Entroly surfaces the SQL injection vulnerability because its SAST scanner prioritizes dangerous code. Top-K and RAW both miss it — the LLM would never know the vulnerability exists.

3. **Dedup saves budget**: Entroly detects the near-duplicate `PaymentProcessorV2` and excludes it, freeing 60 tokens for an additional useful fragment. Top-K includes both copies.

---

## How to Run

```bash
# Run benchmark on the built-in corpus
entroly benchmark

# Run benchmark on YOUR codebase
entroly benchmark --project /path/to/your/code
```

## Methodology

All strategies operate on the **same corpus** with the **same token budget**. The small fixture is not representative of all tasks.

- **Info density** = Shannon entropy × (1 − boilerplate ratio) — higher means more useful information per token
- **Module coverage** = unique top-level directories represented — a coverage proxy, not measured understanding
- **SAST catches** = security vulnerabilities surfaced in the context window — fixture detections, not measured code-generation safety
- **Relevance** = fraction of query terms present in selected context

> **Note**: These benchmarks run on a synthetic but realistic corpus. Real-world numbers depend on your codebase structure, query patterns, and token budget. Run `entroly benchmark` to see your own numbers.

---

## LLM Accuracy Retention (2026-04-22)

> Historical maintainer-reported results, not independently reproduced here.
> This experiment does not establish that compression preserves answer quality.

Model: `gpt-4o-mini` | Budget: 50,000 tokens | Wilson 95% Confidence Intervals

| Benchmark | n | Baseline (95% CI) | Entroly (95% CI) | Retention | Token Savings |
|---|---|---|---|---|---|
| **NeedleInAHaystack** | 20 | 100.0% [83.9–100%] | 100.0% [83.9–100%] | **100.0%** | 0.0% |
| **GSM8K** | 100 | 85.0% [76.7–90.7%] | 86.0% [77.9–91.5%] | **101.2%** | 3.6% |
| **SQuAD 2.0** | 100 | 84.0% [75.6–89.9%] | 83.0% [74.5–89.1%] | **98.8%** | 0.8% |
| **MMLU** (4-way MCQ) | 100 | 82.0% [73.3–88.3%] | 85.0% [76.7–90.7%] | **103.7%** | 0.0% |
| **TruthfulQA** (MC1) | 100 | 72.0% [62.5–79.9%] | 73.0% [63.6–80.7%] | **101.4%** | 0.1% |
| **LongBench** (HotpotQA) | 100 | 57.0% [47.2–66.3%] | 59.8% [49.8–69.0%] | **104.9%** | 3.6% |

### Benchmark Coverage

The harness now covers 7 public benchmarks across five evaluation axes:

| Axis | Benchmark | Tests |
|---|---|---|
| Retrieval | NeedleInAHaystack | Finding a fact in long context |
| Reasoning | GSM8K | Multi-step grade-school math |
| Code | HumanEval | Python function completion |
| Reading | SQuAD 2.0 | Short-passage QA |
| Knowledge | MMLU | 57-subject 4-way MCQ |
| Truthfulness | TruthfulQA MC1 | Resistance to common misconceptions |
| Long context | LongBench HotpotQA | Multi-hop QA over 10K–60K-token docs |

> **BFCL (Berkeley Function Calling)** is intentionally omitted — requires function-calling scaffolding with per-sample tool schemas and AST validation, and BFCL prompts are short enough that the compression selector passes through unchanged. Poor signal-to-noise fit for an accuracy-retention benchmark.

### Interpretation

- **Overlapping marginal CIs do not establish equivalence or non-inferiority.** The reported SQuAD score decreased from 84% to 83%. A paired evaluation with a predefined acceptable loss and adequate sample size is needed. The reported intervals are retained as historical values, not revalidated uncertainty estimates; Wilson intervals apply to binary outcomes, not averaged partial-credit scores.
- **Token reduction in this table is only 0–3.6%.** LongBench reports 12,885 baseline tokens versus 12,423 selected tokens. This does not establish quality under aggressive compression or identify why the answer score changed.
- MMLU, TruthfulQA, GSM8K, and SQuAD have short system contexts that fit within the 50K budget — Entroly correctly passes through (no artificial noise injection). These serve as **regression guards** against the selector corrupting short-context prompts.
- Needle contexts (4K–32K tokens) also fit within the 50K budget, so compression is not triggered there.
- Savings scale with how far a workload's context exceeds the configured budget. Most benchmarks here fit inside the 50K budget, so compression rarely fires; LongBench is the one that does, at 3.6%. Codebase workloads with far larger contexts reduce more, but this harness does not measure them and **no universal savings range is claimed** — a reported percentage is bounded by the configured budget before selection runs, so it describes the budget as much as the selector.

### Reproduce

```bash
export OPENAI_API_KEY=...

# Run everything (7 benchmarks, ~5–10 min on gpt-4o-mini at n=100)
python -m bench.accuracy --benchmark all --model gpt-4o-mini --samples 100

# Run one at a time
python -m bench.accuracy --benchmark mmlu       --samples 100
python -m bench.accuracy --benchmark truthfulqa --samples 100
python -m bench.accuracy --benchmark longbench  --samples 100
```

Historical engine provenance: not established by these result files. The repository version is not evidence of the version used in the historical run.

That line read `v0.9.0` until it was corrected in 1.0.82. It had been written in
the v1.0 founding commit and never moved again, so it named a pre-1.0 engine for
the entire life of the project up to that point. It was not a provenance record
either — the result files under `benchmarks/results/` carry no version field —
so re-run the commands above to measure against the engine you actually have.