# AVO — Agentic Variation Operators An open reproduction of **[AVO: Agentic Variation Operators for Autonomous Evolutionary Search](https://arxiv.org/abs/2603.24517)** (Chen, Ye, Xu et al., NVIDIA, 2026), runnable on a laptop. Classical evolutionary search, and the LLM-augmented systems that followed it, decompose the variation operator into a fixed pipeline: ``` Vary(P_t) = Generate(Sample(P_t)) ``` The framework samples parents; the model produces one candidate from them. AVO replaces that whole decomposition with a single autonomous agent run: ``` Vary(P_t) = Agent(P_t, K, f) ``` The agent sees the full lineage `P_t`, a domain knowledge base `K`, and the scoring function `f` — and decides for itself what to read, what to change, and when to measure. It stops being a candidate generator and becomes the variation operator. This repo implements that framework, plus the surrounding machinery the paper describes: a git-backed lineage, a correctness-gated score vector, the matches-or-improves commit policy, a supervisor that intervenes on stagnation, and trajectory plots. Two optimisation targets ship with it. --- ## The part that matters: it runs on the session you already have The default driver does **not** spawn an agent and does **not** call an API. It hands the variation prompt to the Claude Code session you are already talking to, and that session does the work. Nothing extra is billed, no `ANTHROPIC_API_KEY` is needed, and the agent doing the optimising is a real general-purpose coding agent — which is exactly what the paper used. Unattended mode (spawn an agent per step and let it run for days, like the paper's 7-day experiment) is available too, and is opt-in precisely because it spends quota. --- ## Install ```bash git clone https://github.com/gatordevin/avo cd avo pip install -e ".[all]" # or: pip install -e . for the core only avo doctor ``` On a system with an externally-managed Python (Homebrew, most Linux distros), use a virtualenv — the `--system-site-packages` flag reuses a NumPy and Matplotlib you already have: ```bash python3 -m venv --system-site-packages .venv .venv/bin/pip install -e ".[all]" .venv/bin/avo doctor ``` Requirements: Python 3.10+, `git`, and a C compiler if you want the `attention_c` target. `numpy` is needed by the bundled targets, `matplotlib` for plots. The core framework depends only on PyYAML. --- ## Quickstart — drive it from the agent you already have Full protocol, including Codex and plain-CLI use, in **[docs/DRIVING.md](docs/DRIVING.md)**. ### Claude Code Register the MCP server once, at user scope so it is available in every folder: ```bash claude mcp add avo -s user -- python3 -m avo.mcp_server # from a virtualenv, point at its interpreter: claude mcp add avo -s user -- /path/to/avo/.venv/bin/python -m avo.mcp_server ``` `claude mcp list` should show `avo — ✔ Connected`. Optionally install the bundled skill so `/avo` works anywhere: ```bash cp -r .claude/skills/avo ~/.claude/skills/avo ``` Then, in a Claude Code session in any directory: > Use the avo tools to evolve the game2048 target for 10 steps. Call > `avo_start_run`, then loop: `avo_next_step`, do the work it asks for, > `avo_evaluate` until you're happy, then `avo_submit`. If it reports a stall, > call `avo_supervisor_brief`, answer it, and file it with > `avo_record_supervisor`. The eleven tools are the whole loop: | tool | what it does | |---|---| | `avo_start_run` | seed `x_0`, score it, measure baselines, open the lineage | | `avo_next_step` | the variation prompt: `P_t`, the index of `K`, the contract for `f` | | `avo_evaluate` | run `f` on the work tree — free, call it as often as you like | | `avo_submit` | end the step: score, then commit or revert per the policy | | `avo_revert` | abandon an experiment without spending the step | | `avo_status` / `avo_lineage` | where the run is | | `avo_supervisor_brief` / `avo_record_supervisor` | the stagnation intervention | | `avo_plot` | render the trajectory | | `avo_list_targets` | what can be evolved | ### Codex Codex CLI speaks MCP and reads `AGENTS.md`, so both halves work: ```bash codex mcp add avo -- python3 -m avo.mcp_server ``` [`AGENTS.md`](AGENTS.md) at the repo root documents the loop and the rules that keep a run honest; Codex picks it up automatically when working in this directory. ### Without MCP Every tool has a CLI twin, so a plain shell works just as well — this is the most portable option and works with any agent, or by hand: ```bash avo start --target game2048 # seeds x0 and prints the first prompt # ... edit runs//work/, run runs//avo-eval as often as you like ... avo submit -m "expectimax depth 2 with a positional weight matrix" avo prompt # the next step's prompt avo status avo plot -o trajectory.png ``` --- ## Worked runs Two complete runs ship with the repo, both driven in session mode by a Claude Code session, both including their dead ends. ### `attention_decode` — beating the vendor kernel **[`examples/attention-decode-run/`](examples/attention-decode-run/)** evolves the decode step of attention: one query token against a long KV cache, the computation an LLM runs for **every generated token**. Scored against `mx.fast.scaled_dot_product_attention` — Apple's own fused Metal kernel. **0.05 → 1.14× MLX in three steps.** This is the one where the evolved kernel actually beats the vendor implementation, and the interesting part is *how*: - **Step 1 was implementation** — split-K flash-decoding took the kernel from 1.6 GB/s to 106 GB/s, about 95% of the machine's streaming limit. That reached 0.95× MLX and exhausted the lever: you cannot read bytes faster than the memory controller delivers them. - **Step 2 was mathematics.** The target's gate is an output-error budget rather than exact equality, so the search could change the computation. Measurement showed 99.9% of the softmax mass sits in ~11% of keys, so the kernel now scores every key but reads V only above a threshold derived so the discarded mass is provably under 0.3%. That crossed 1.0, spending 2% of the error budget. The lesson generalises: once a bandwidth-bound kernel is at the roofline, the only remaining lever is to read fewer bytes, and that is an algorithmic change. [Read the full write-up →](examples/attention-decode-run/RESULTS.md) ### `attention_c` — the paper's own domain **[`examples/attention-c-run/`](examples/attention-c-run/)** evolves a forward attention kernel in C, reaching **2.2× a straightforward NumPy/BLAS implementation** and close to the NEON roofline. Note the honest framing: that baseline is not a tuned attention library, and this kernel is *slower* than torch's CPU SDPA and MLX — Apple's AMX matrix units are unreachable from portable C. The write-up gives the full comparison. Three findings from it are worth the click: - **The paper's own algorithm was the wrong answer here.** A FlashAttention-style tiled kernel with a streaming online softmax measured *worse*, twice. At these sizes a whole head fits in L2, so blocking for locality buys nothing while the per-block rescale is pure added work. The cost is arithmetic, not memory. - **`-ffast-math` silently breaks the standard fast-`exp`**, by algebraically cancelling the add-magic-constant rounding trick it depends on. The correctness gate caught it on an N=3 shape; the throughput number never would have. - **The run forced a target fix.** Scoring raw GFLOP/s on a laptop doing other work is not a measurement — identical code ranged 44–76 GFLOP/s in twenty minutes. `eval.py` now times a NumPy/BLAS reference *in the same process, interleaved with the candidate*, and scores the ratio. [Read the full write-up →](examples/attention-c-run/RESULTS.md) ### `game2048` — evolving a game-playing policy **[`examples/game2048-run/`](examples/game2048-run/)** is a complete 8-step run of the `game2048` target, driven in session mode by a Claude Code session. The directory holds the unedited output: the evolved policy, the operator's working notes, the full trajectory, the screening tools it built, and its dead ends. **876 → 43 826 — 50× the seed, 14× the strongest baseline.** Games reaching 2048: 0% → 77%. Best tile: 512 → 8192. Apple M5, single-threaded, standard library only. ![evolution trajectory](examples/game2048-run/trajectory.png) Improvement arrives in discrete jumps separated by plateaus, matching the paper's Figure 5. The two flat versions are pure throughput work that bought the budget the next step spent — the same role the paper's v19→v20 branchless-rescale change plays. The largest single gain (+50.5%) was not an optimisation. The benchmark scores accumulated game points; the heuristic only measured how survivable a board looked, so nothing in the search knew that merging two 256s banks 512 points. Four steps of throughput work were worth +27% combined; one step of checking what was actually being optimised was worth +50%. [Read the full write-up →](examples/game2048-run/RESULTS.md) ## What ships with it ### `game2048` — evolve a game-playing policy Evolve `agent.py` into the strongest 2048 player you can, under a hard thinking-time budget. Scored as the geometric mean of mean game score across four banks of twelve deterministic seeds. Blowing the 120 s budget scores zero, not "slightly less" — so search depth, evaluation-function cost, and pruning all trade against each other, and that trade-off *is* the problem. Measured on an Apple M5: | policy | score | |---|---| | seed `x_0` (first legal move) | 876 | | random baseline | 1 076 | | corner heuristic baseline | 2 565 | | greedy one-ply baseline | 3 132 | Strong expectimax players score in the tens of thousands. The worked run above reached 43 826. ### `attention_c` — evolve a kernel, the paper's own domain Evolve a single-precision forward attention kernel in C: `O = softmax(QKᵀ/√D)V`, causal and non-causal, `D = 64`. Gated on agreement with a float64 reference over eighteen shapes — including prime and off-by-one sequence lengths, so a kernel that mishandles its tail fails rather than quietly scoring well. Scored as **speedup over a NumPy/BLAS reference timed in the same process**, geometric mean across four sequence lengths × two masking modes. 1.0 is parity with the library. Scoring a ratio rather than raw GFLOP/s makes the benchmark immune to whatever else the machine is doing — absolute throughput on a shared laptop moves by more than most optimisations are worth. | kernel | score | |---|---| | seed `x_0` (naive, materialises the full `N×N` score matrix) | 0.19× | | NumPy/BLAS baseline — the "cuDNN" of this setup | 1.00× | | evolved in 3 steps ([write-up](examples/attention-c-run/RESULTS.md)) | **2.11×** | The knowledge base covers the online-softmax formulation, tiling and block-size selection, CPU vectorisation, threading, and how to interrogate the host machine rather than assuming an ISA. Beating BLAS needs most of them. --- ## How it works ### The run directory ``` runs// work/ the candidate x_t — a standalone git repo whose history IS the lineage .avo/scores.jsonl every committed version's full score vector kb/ the knowledge base K, copied in so paths are stable avo-eval f, as a zero-argument shim the agent can call at will NOTES.md scratch space that survives across steps trajectory.jsonl every step, accepted or rejected rejected/ the diff of each rejected candidate, kept for the record logs/ evaluator and agent logs ``` Making the lineage a git repo means the agent inspects `P_t` with tools it already knows — `git log`, `git show v7:attention.c`, `git diff v6 v7` — instead of a bespoke API. Each accepted version is a commit tagged `vN` whose message carries the score vector. ### The commit policy Paper §3.2: a candidate is committed **only if** it passes the correctness gate *and* matches or improves the best committed score so far. Anything else is reverted and its diff archived — it stays part of the agent's internal search trajectory, but never enters the lineage. Correctness is a gate, not a dimension. A candidate that fails it scores zero regardless of what it measured (§3.1). In `attention_c` that means a kernel that is 10× faster and numerically wrong is worth exactly as much as one that does not compile. ### The score vector `f(x) = (f_1(x), …, f_n(x))` — one number per benchmark configuration, with the geometric mean as the scalar being maximised. This is what makes per-config movement diagnostic: a change that helps `n1024` and hurts `n128` is a blocking problem, not a win, and the aggregate alone would hide it. ### The supervisor Paper §3.3: long autonomous runs fail in two ways — the agent *stalls* when it exhausts its current line of attack, or enters *unproductive cycles* of edits that keep failing. After N steps without a new best (default 3), AVO stops and asks for a redirect: a review of the whole trajectory that proposes several concrete, different optimisation directions. The redirect is injected into the next variation prompt as a strong prior, and consumed by exactly one step. In session mode the supervisor is the same session wearing a different hat, which is cheap enough to actually use. In unattended mode it is a separate agent run with read-only intent. ### The trajectory `avo plot` renders the paper's Figure 5/6: running-best geometric mean as a step function, filled circles at each new best, dotted per-configuration curves, and the baselines as horizontal lines. Same caveat as the paper — it shows the *committed* sequence, not the internal search tree explored between commits. --- ## Unattended mode To reproduce the paper's setup, where the operator is a spawned agent and nobody is watching: ```bash avo run --target attention_c --backend claude_cli --max-steps 40 --time 12h avo run --resume runs/attention_c-20260321-091500 --time 24h ``` Backends: `claude_cli` (Claude Code headless — the closest analogue to the paper's agent), `api` (a self-contained agent loop on the Messages API, for people with only an API key), `agent_sdk` (in-process via `claude-agent-sdk`), and `mock` (a shell command, for testing the machinery without a model). This spends quota or credits on every step. Session mode does not. --- ## Adding your own target A target is a directory with a `target.yaml`, a seed program, a knowledge base, and an evaluator. The evaluator is any executable in any language; the whole contract is one JSON object on stdout: ```json {"correct": true, "metrics": {"config_a": 1520.3, "config_b": 1477.0}, "error": null, "notes": "shown to the agent"} ``` `correct` is the gate. `metrics` is the score vector. The scalar being optimised is their geometric mean unless you supply an explicit `primary`. ```yaml name: my_target description: One line, shown in `avo targets`. seed: seed # copied to work/ as x_0 knowledge_base: kb # copied to the run dir as K entrypoint: kernel.c # informational, used in prompts evaluate: command: ["python3", "{target}/eval.py", "--workdir", "{workdir}"] timeout: 30m baselines: # optional, measured once before evolution starts command: ["python3", "{target}/eval.py", "--baselines"] score: direction: maximize agent: goal: | What the agent is actually trying to do, and what the trade-offs are. ``` See `docs/TARGETS.md` for the full contract and `tests/fixtures/toy/` for a minimal working example. The knowledge base is worth real effort. It is the `K` in `Agent(P_t, K, f)`, and the difference between an agent that rediscovers tiling from first principles over ten steps and one that gets there in two. --- ## What is faithful, and what is not Faithful: - the operator formulation `Vary(P_t) = Agent(P_t, K, f)` — a real coding agent with file editing, shell access and persistent memory, given no task-specific modifications - single-lineage continuous evolution with git-backed state (§3.3) - the correctness gate and the n-dimensional score vector (§3.1) - the matches-or-improves commit policy, with failed attempts excluded from the lineage (§3.2) - supervisor intervention on stagnation and unproductive cycles (§3.3) - geometric-mean aggregation across benchmark configurations, and Figure 5/6 trajectory plots Not faithful, and deliberately so: - **The hardware.** The paper evolves attention kernels on B200 GPUs against cuDNN and FlashAttention-4. `attention_c` is the same problem on a CPU against NumPy/BLAS. The optimisations transfer in kind (tiling, online softmax, vectorisation, scheduling), not in magnitude. - **The scale.** The paper ran 7 days, 40 committed versions, 500+ explored directions. A session-mode run of 10–20 steps is a demonstration, not a replication. - **Population structure.** Like the paper, this implements the single-lineage case to isolate the operator. Archive- and island-based regimes are compatible with the formulation but not implemented. --- ## Repo layout ``` src/avo/ types.py Score, LineageEntry, the correctness gate, geomean config.py target specs and run configuration lineage.py P_t as git history scoring.py f as an external process knowledge.py K prompts.py the variation and supervisor prompts — the whole framework/agent interface run.py run state: seed, evaluate, commit policy, trajectory session.py driver: the session you already have is the operator loop.py driver: unattended, spawns an agent per step mcp_server.py the same operations as MCP tools (no dependencies) cli.py the same operations as subcommands plot.py Figure 5/6 agents/ backends for unattended mode targets/ game2048/ policy evolution under a time budget attention_c/ kernel evolution — the paper's domain, on a CPU examples/ attention-decode-run/ beats Apple's own fused kernel by changing the maths attention-c-run/ CPU kernel evolution, with an honest baseline caveat attention-metal-run/ GPU prefill — every CUDA instinct measured worse game2048-run/ policy evolution — 50x the seed docs/ PAPER_MAP.md every section of the paper, and where it lives in the code TARGETS.md the evaluator contract DRIVING.md how to drive a run from Claude Code, Codex, or a shell AGENTS.md cross-agent instructions (read automatically by Codex) .claude/skills/ the `/avo` skill for Claude Code ``` --- ## Citing This is an independent reproduction. Cite the original work: ```bibtex @article{chen2026avo, title = {AVO: Agentic Variation Operators for Autonomous Evolutionary Search}, author = {Chen, Terry and Ye, Zhifan and Xu, Bing and Ye, Zihao and Liu, Timmy and Hassani, Ali and Chen, Tianqi and Kerr, Andrew and Wu, Haicheng and Xu, Yang and Chen, Yu-Jung and Chen, Hanfeng and Kane, Aditya and Krashinsky, Ronny and Liu, Ming-Yu and Grover, Vinod and Ceze, Luis and Bringmann, Roger and Tran, John and Liu, Wei and Xie, Fung and Lightstone, Michael and Shi, Humphrey}, journal = {arXiv preprint arXiv:2603.24517}, year = {2026} } ``` Licensed under Apache-2.0. Not affiliated with or endorsed by NVIDIA.