# muteval

[![CI](https://github.com/AshwinUgale/muteval/actions/workflows/ci.yml/badge.svg)](https://github.com/AshwinUgale/muteval/actions/workflows/ci.yml)
[![PyPI version](https://img.shields.io/pypi/v/muteval.svg)](https://pypi.org/project/muteval/)
[![Python versions](https://img.shields.io/pypi/pyversions/muteval.svg)](https://pypi.org/project/muteval/)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)

**Mutation testing for your LLM evals — find out if they'd actually catch a regression.**

![muteval run — mutation score, survivors, and the eval that would catch each](https://raw.githubusercontent.com/AshwinUgale/muteval/main/docs/assets/muteval-run.png)

Your evals are passing. That doesn't mean they work.

muteval answers the question every eval suite quietly dodges: *would my evals
actually fail if my system silently got worse?* It deliberately degrades the
thing under test, reruns **your existing eval suite** against each degraded
version (a "mutant"), and reports a **mutation score** — the percentage of
injected regressions your evals caught. The ones they miss are **survivors** —
candidate coverage gaps to triage: muteval surfaces them, you decide which ones
actually matter (see [docs/LIMITATIONS.md](docs/LIMITATIONS.md)).

It's `mutmut` / Stryker, but for evals.

```text
Mutation score: 33%  [████████░░░░░░░░░░░░░░░░]  (7/21 mutants killed, 95% CI 17-55%)
Effective score: 70%  (7/10 — excludes 11 inert mutant(s) whose output didn't change; 95% CI 40-89%)

3 SURVIVED  (output changed but evals didn't notice — real coverage gaps; 3 HIGH-severity):

  #1 [HIGH] SURVIVED  [flip_negation]  accept: 7870b8babd66
            inverted "don't" -> "do" (near: in the context, say you don't know.)
            fix: add an eval for the rule near "in the context, say you don't know." (e.g. a checks.llm_judge for that behavior)
  #2 [HIGH] SURVIVED  [drop_instruction_lines]  accept: 2f4e149fd0ec
            dropped line: "Answer using ONLY the provided context. If the answer is not in the c…"
            fix: add checks.llm_judge("the reply still follows: Answer using ONLY the provided context. If the answ…")
  …
```

(Real output, trimmed, of the keyless RAG quickstart: `muteval init --template rag`
then `muteval run` — see Quickstart. The suite checks the answers, but nothing
checks the "say you don't know" rule, so deleting or inverting it goes unnoticed.)

---

## Install

```bash
pip install muteval        # pure Python, zero required dependencies
pip install -U muteval     # already installed? upgrade (bare install is a no-op)
```

The core drags in **no** heavy LLM SDKs. Optional extras, only if you use them:

| Extra | Adds | For |
| --- | --- | --- |
| `muteval[promptfoo]` | PyYAML | `--promptfoo` ingestion |
| `muteval[deepeval]` | deepeval *(large tree)* | grading via deepeval metrics |
| `muteval[ragas]` | ragas *(large tree)* | grading via RAGAS metrics |

**Any provider for the system under test:** point it at any OpenAI-compatible
endpoint with `--base-url` (or `OPENAI_BASE_URL`) — Groq, Gemini-compat, GitHub
Models, Ollama, a local server. Discover what's available anytime with
`muteval list` (operators, checks, probes).

## 60-second quickstart (no API key)

```bash
muteval init --template rag        # scaffold a config (or --template basic)
muteval check --config muteval_config.py   # validate wiring + baseline first
muteval run   --config muteval_config.py
```

`muteval init` writes a runnable config with the four things you supply clearly
marked. `muteval check` is the doctor — it validates your pipeline, evals, and
baseline *before* a full run and tells you exactly which layer is broken. Then
`run` gives you a mutation score and a ranked list of survivors, each with a
suggested eval to close the gap.

Prefer zero-config? Point muteval at a prompt + cases and let it call the model:

```bash
export OPENAI_API_KEY=sk-...
muteval run --prompt-file system.txt --cases cases.jsonl --model gpt-4o-mini \
  --judge "the answer is grounded in the provided context" --fail-under 75
```

Already have a **promptfoo** suite? Point muteval straight at it — no muteval
config file, it reuses your prompt + tests + assertions:

```bash
muteval run --promptfoo promptfooconfig.yaml            # add --dry-run to preview
```

It reads the model from your `providers:` block, so it mutates your prompt against
the model your suite actually uses. Want it in CI? There's a GitHub Action —
see **[Run in CI](docs/ci.md)**.

Want to see it find a coverage gap first, with **no API key**? Run the offline
demo — it degrades a support-bot prompt and finds the rule the promptfoo suite
forgot to assert (~1s, mock model). It runs from a checkout of the repo (the
`examples/` aren't shipped in the wheel):

```bash
git clone https://github.com/AshwinUgale/muteval && cd muteval
muteval run --config examples/promptfoo_offline/muteval_config.py --no-color
```

Already have your own pipeline? Use it as the system under test — a function or a
deployed endpoint — no `run()` wrapper:

```bash
# your own function, called as fn(prompt, case) -> str
muteval run --target mypkg.app:answer --prompt-file system.txt --cases cases.jsonl --check contains:8080

# a deployed service: POSTs {prompt, case} JSON, reads the text output
muteval run --endpoint https://my-app/answer --prompt-file system.txt --cases cases.jsonl --judge "grounded in context"
```

Re-running is cheap: `--cache runs.sqlite` memoizes run outputs + eval outcomes,
so an identical re-run makes **zero** model/judge calls (skipped for noisy suites
with `--runs-per-mutant > 1`). Entries are keyed on a fingerprint of your `run`
and eval *code* (closures, thresholds, defaults), so editing an eval never serves
a stale verdict; set `cache_version` on an eval that reads a file or remote rubric:

```bash
muteval run --config muteval_config.py --cache .muteval-cache.sqlite
```

Slow because it's API-bound? Evaluate mutants in parallel (results are identical
to a serial run — order preserved; each case is copied per call, and the deepeval
adapter copies its metric per call. Your own *stateful* evals must be thread-safe):

```bash
muteval run --config muteval_config.py --concurrency 8 --cache .muteval-cache.sqlite
```

Worried about spend? Cap the model + judge calls; muteval fails closed (exit 2)
before overspending (cache hits and skipped judges don't count):

```bash
muteval run --config muteval_config.py --max-calls 500
```

Triage the survivors without re-running (the last run is saved to
`.muteval/last_run.json`):

```bash
muteval results        # ranked survivors (HIGH first) with ids
muteval show 1         # one survivor: operator, suggested fix, baseline→mutant diff
muteval report --html coverage.html   # a shareable standalone report
```

Beyond mutation coverage, `muteval probe` audits the eval suite along other
lenses. The load-bearing ones catch real, common defects: **judge reliability**
(does your LLM judge flip on identical re-runs?) and **discrimination** (can the
eval tell good outputs from bad, given good/bad exemplars?). The rest are hygiene
checks — statistical adequacy and redundancy — plus, only if you have labels,
**human agreement** (Cohen's κ via `muteval label`), the one true validity check.
A report card, no composite score. (A separate **judge-bias panel** —
position / verbosity / self-preference — is available as a library function for
*pairwise A/B judges*; it isn't part of the default card because it needs a
pairwise-judge harness.)

```bash
muteval probe --config muteval_config.py --html quality.html
```

## Why this exists

Regression tools (promptfoo, deepeval, OpenAI Evals, LangSmith) catch
regressions in your *system*. None tell you whether your *evals* are good enough
to catch those regressions in the first place. That meta-layer is the gap
muteval fills — mutation testing is the established answer to "is my test suite
any good?" in software engineering, brought to LLM eval suites.

| Tool | Mutates the… | Measures… |
| --- | --- | --- |
| promptfoo red team | input (jailbreaks) | your system's safety |
| Giskard | input (typos, swaps) | your model's robustness |
| deepeval synth data | output / ground truth | a metric's calibration |
| **muteval** | **the system (prompt → context → tools → model)** | **your eval suite's coverage** |

## How it works

Describe your system + evals in a small config, then muteval:

1. **Baseline** — confirms your suite passes on the *original* system. If it
   doesn't, muteval **refuses to score** (a red baseline makes every number
   meaningless) rather than hand you a misleading 100%.
2. **Mutate** — generates mutants by degrading the prompt / retrieved context /
   tool outputs / model (22 operators).
3. **Grade** — reruns your suite against each mutant. **Killed** = your evals
   caught it (good); **survived** = they missed it (a gap).
4. **Score** — `killed / evaluated`, with a 95% confidence interval, severity
   ranking, near-miss margins, and a suggested fix per survivor.

```python
from muteval import MutEvalConfig, checks

config = MutEvalConfig(
    prompt=SYSTEM_PROMPT,                 # the thing under test
    cases=[{"input": "...", "order_id": "A123"}],
    run=my_run_fn,                        # call your LLM/app -> output text
    evals=[                               # your existing checks, graded by muteval
        checks.contains_case("order_id"),
        checks.grounded("context"),       # LLM-judge preset (any OpenAI-compatible endpoint)
    ],
)
```

## Trustworthy by design

A coverage number you can't trust is worse than none. muteval **fails closed**:

- **Red or errored baseline → no score** (status `baseline_failed`/`errored`),
  CLI exits non-zero, no badge.
- **Partial mutant errors** above a budget → `partial_errors`, not a score over a
  shrunken denominator. `--max-error-rate` / `--allow-mutant-errors` to accept.
- **Non-determinism** → strict-majority verdicts over `runs_per_mutant`, Wilson
  confidence intervals on the score, flaky-mutant flagging. The original system
  must pass the same majority rule as a mutant, and a kill on output the
  original itself produces is a *noise kill*, not detection. Tied verdicts are
  gated like errors (`--max-unresolved-rate`; an odd `runs_per_mutant` can't tie).
- **A bad judge reply is an error, not a verdict** — an empty/unparseable judge
  reply, a NaN score, or an eval returning `0.2` / `"fail"` fails closed.
- **Cosmetic changes** → output-diffing separates real coverage gaps from
  "observationally unchanged" mutants. For free-text output, set `output_key=`
  (the part that *is* the behavior, e.g. the label) and `baseline_runs=N` so
  wording drift isn't mistaken for a behavior change.
- **Meaning-preserving edits** (paraphrase, reorder) are reported, never scored.
  Surviving them is healthy; a flip means an eval keys on wording.
- **No input-deleting mutants** → a mutant that drops a `{{placeholder}}` is a
  guaranteed kill that measures nothing, so it's never generated.

In a controlled, CI-enforced experiment the mutation score rises monotonically
with eval-suite coverage — **0% with no evals → 100% with complete coverage** —
across four domains (support bot, code review, RAG, HR policy). See
[FINDINGS.md](FINDINGS.md), and [docs/LIMITATIONS.md](docs/LIMITATIONS.md) for
when to distrust the number.

## What it can mutate (22 operators)

**Prompt:** `weaken_modals`, `flip_negation`, `drop_instruction_lines`,
`delete_sentences`, `truncate_prompt`, `drop_few_shot_example`,
`remove_emphasis`, `weaken_numeric_threshold`. Robustness (reported, not
scored): `swap_adjacent_instructions`, `paraphrase_instruction`.
**Retrieved context (RAG):** `drop_context_doc`, `clear_context`,
`corrupt_context_doc`, `swap_context_doc`, `shuffle_context`,
`duplicate_context_doc`, `truncate_context_doc`.
**Model:** `downgrade_model`. **Tools (agents):** `drop_tool_output`,
`corrupt_tool_output`, `swap_tool_output`, `deny_tool_output`.

Pass a `System(prompt=..., context=[...], tools=[...], model=...)` to make
context / tools / model mutable for RAG and agent suites. Bring your own operator
with `register_operator`, and scope which parts of the prompt mutate with
`[[mutate]]…[[/mutate]]` markers or `--scope-include/-exclude`.

## Gate CI + coverage badge

```bash
muteval run --config muteval_config.py --fail-under 75 --badge badge.json
muteval run --config muteval_config.py --junit junit.xml
```

Exits non-zero if coverage drops below 75%, so a PR that weakens your evals fails
the build. `--fail-on-severity high` gates on any unguarded high-severity gap.
Copy `examples/ci/github-actions.yml` to run it on every PR and publish a
[shields.io](https://shields.io) eval-coverage badge.

## Bring your existing metrics

Already have a suite? Reuse its metrics instead of rewriting them:

```python
from deepeval.metrics import FaithfulnessMetric
from muteval.adapters.deepeval import metrics_to_evals

evals = metrics_to_evals([FaithfulnessMetric()], input_key="question",
                         retrieval_context_key="context")
```

Adapters for **deepeval**, **RAGAS**, and **promptfoo** (`pip install
"muteval[deepeval|ragas|promptfoo]"`). Or use the built-in framework-free
`checks` — including `llm_judge` / `grounded` that hit **any OpenAI-compatible
endpoint** (OpenAI, Groq, Gemini, GitHub Models, Ollama…) via `base_url=`, using
only the standard library.

For a keyless integration using `scorer_to_eval`, see the
[Autoevals JSON-profile example](examples/autoevals_profile_json/README.md). It
compares JSON-only checks with `ExactMatch` on two controlled prompt mutations.

## Adopting it on your own suite

Pointing muteval at a real system is a ~1-hour integration, not plug-and-play —
[docs/ADOPTION.md](docs/ADOPTION.md) has the honest checklist, a
"where-it-breaks → what-to-change" table, judge-selection guidance, and the four
pieces you supply. Start with `muteval check` and fix a green baseline first.

## Roadmap

Shipped: prompt/context/tool/model mutation · deepeval/RAGAS/promptfoo adapters ·
scored evals + near-miss reporting · severity ranking + `--fail-on-severity` ·
confidence intervals + majority-vote stability · output-diffing · fail-closed
validity gate · `muteval check` doctor · RAG scaffold + adoption guide ·
zero-config ingestion (promptfoo/deepeval/callable/endpoint) · caching +
concurrency + budget caps · `results`/`show`/`report --html` triage · the
`muteval probe` eval-quality report card (adequacy, judge reliability + ICC,
discrimination, redundancy, judge bias, threshold calibration, human agreement) ·
an autofix verify loop that proposes an eval for a survivor and confirms it kills
the mutant while the baseline stays green.

Possible later (not the current focus): LLM-driven semantic mutations ·
agent/trace mutation · A/B suite comparison. The `muteval probe` layer stays a
documented part of muteval — an honest eval-quality/judge-audit layer, not a
separate product. Current focus is quality and honesty over new scope.

## Contributing & support

Early, open project — contributions are genuinely welcome, especially new
mutation operators, probes, and adapters (each is a small, self-contained add).

```bash
git clone https://github.com/AshwinUgale/muteval && cd muteval
pip install -e ".[dev]" && pytest -q          # dev setup + the suite
```

- **Start here:** [CONTRIBUTING.md](CONTRIBUTING.md) (dev setup, PR checklist,
  "add an operator" walkthrough) and [ARCHITECTURE.md](ARCHITECTURE.md) (a
  one-page map of the codebase).
- **Extend it:** [docs/PLUGINS.md](docs/PLUGINS.md) — the operator / probe /
  adapter / reporter API. `muteval list` shows what already exists.
- **Bugs & ideas:** open an [issue](https://github.com/AshwinUgale/muteval/issues)
  (there are templates); look for `good first issue`.
- **Docs:** <https://ashwinugale.github.io/muteval/>

Licensed [Apache-2.0](LICENSE). By participating you agree to the
[Code of Conduct](CODE_OF_CONDUCT.md).