# SkillClaw reproduction on harness evolution This example implements the SkillClaw method from [Evolving Skills for Autonomous Agents](https://arxiv.org/abs/2604.08377) using Reef's harness evolution engine (`harness_evolve`). Each day, the agent runs a fixed task list with its current skills. Each night, SkillClaw reviews the day's sessions and proposes skill changes. Every proposed change is accepted unless the method chooses to skip it; the next day measures the updated skills. Reef stores the skill pool as a tree and publishes each night's changes together as a new version that clients can download. The method supplies `propose`, `evaluate`, and a `CandidateSelector`. The engine renders the harness, runs evaluation tasks, records changes, and rolls back rejected changes. ## Directory layout ```text skillclaw/ skillclaw.yaml the recipe the driver boots: explicit implementation, selection always, batch_size 60, probe tasks, seed composition recipe.py the Reef recipe subclass: build_surface returns create_skill_surface([SkillCatalogModule(...)]) so every /v1 request carries the pool catalog; seed_skills seeds the benchmark's shipped skills library harbor/ one Harbor-format task (the standard layout the sibling examples follow): the benchmark's meeting-negotiation task vendored at the pin - task.toml, instruction.md, environment/ (the two mock services and their fixtures in a standalone image), tests/ (the benchmark's own grader writing the verifier reward); `run.py solve` runs it harness/ the method package skillclaw.yaml's dotted refs name tasks.json the frozen 60 task list at the WildClawBench pin harbor_agent.py the minimal Harbor agent `run.py solve` drives: one recorded model call through the embedded service day.py the pinned WildClawBench checkout (ensure_benchmark), the docker task lifecycle, and the benchmark's grading loop catalog.py SkillCatalogModule, ported from the sealed campaign: the OpenClaw catalog format, eligibility rules, and the catalog_names inverse skillclaw.py the method: propose runs the sealed night flow and maps its decisions to one composite mutation sequence; evaluate grades the probe episodes night.py the sealed night step: one decision per skill group plus the no-skill bucket, merge and registry semantics evolver.py the evolve server's LLM stages (summarize, judge, decide, create, merge) with the sealed bounds sessions.py recorded traffic to session digests: parse, annotate, aggregate, skill reference extraction prompts.py the five sealed system prompts and the agent preamble (all four files ported from benchmarks/skill_claw at commit 0519eefb) stats.py the preregistered gain criterion, carried verbatim from the sealed campaign (PR #175, commit 0519eefb) config.py paths, models, and campaign constants run.py the campaign loop, the sealed __main__ adapted: it embeds the Reef service, runs the docker day, reports every grade, and seals rounds; `run.py report` prints the gain table; `run.py solve` runs the harbor/ task once through reef-eval (the smoke) run.sh materializes the benchmark checkout (ensure_benchmark) before day one, then runs the driver pyproject.toml makes harness/ an installable package ``` ## The method, mapped onto the mechanism `propose` is the sealed night, unchanged in shape: rebuild each task's session digest from the recorded traffic, summarize every session, judge the unscored ones, group by referenced skill, then one decision per skill group plus the no-skill bucket - `improve_skill`, `optimize_description`, `create_skill`, or `skip`. The decisions land on a scratch pool (merge and registry semantics included) and the pool diff becomes one composite mutation sequence: one `create` or `update` per changed skill, applied under one snapshot and settled under one gate verdict. The sealed night never removes a skill, so no `remove` is ever proposed. The night's LLM is the served model itself, through the same upstream binding the day uses. The day is the sealed day: each task runs in its WildClawBench container, the agent's model calls go through the embedded Reef service (which injects the served pool's catalog and records the exchange), and the benchmark's own checks and judge grade the outcome. The report references the task's recorded traffic, so the night learns from exactly what the proxy saw. The night trigger is the batch: `data.batch_size` in skillclaw.yaml is the frozen task count (60 at the paper setting), no score window, so the whole day batches - failures and passes - and the day's last report schedules the background night step. The driver waits for that step before sealing the round and pulling the next day's pool. The selection policy is `selection: always`, so every applied night publishes in the paper's regime. The probe episodes (three exact-answer coding tasks) still run and their scores land in the version metrics, so every published pool version carries a measured before/after record without deciding on those scores. ## The two runs and the claim `REEF_SC_RUN` selects the run. `skillclaw` is the method run. `frozen` is the control run: it replays and seals the same rounds but reports nothing, so nothing batches, no night runs, and the pool never changes. The gain criterion is fixed before any data is read and carried verbatim in `stats.py` from the sealed campaign (PR #175, commit 0519eefb): a category's gain counts as real only when the method run's final day beats the control mean by more than two control standard deviations (one sided, calibrated to a 5.6 percent false positive rate per category); best day excesses stay descriptive. ## The 2026-08-29 results (GLM-5.3-Flash, preliminary) On GLM-5.3-Flash, served locally on 4 GPUs (sglang, no provider API), six nights of evolution applied 13 skill improvements and 8 creations. The pool grew from 9 to 17 skills, with same day create, next day improve loops. In the Productivity category the method run's final day beats the control mean by +12.05 points (2.29 sd). On the subset of tasks scored on every day in both runs, the margin grows to +15.72 (2.50 sd). Numbers recompute from the stored run data with `python run.py report`. The serving layer that lets the official code run against a locally served reasoning model is documented in `serving/`. ## Quick start What a run actually needs, beyond `pip install reef-client` and reef itself: - **docker**, and the WildClawBench container image (a tarball under the checkout's `Images/` directory, loaded automatically) - the day runs every task in its own container. - **the WildClawBench dataset**: `run.sh` clones the repo at the pin, but the multi gigabyte task workspaces come from the `internlm/WildClawBench` HuggingFace dataset and must be downloaded into `work/wildclawbench` first; `run.sh` fails fast with that instruction if they are missing. - **a provider key** (`REEF_UPSTREAM_API_KEY`, required): the agent sessions, the benchmark judge, and the night's evolve calls all bill against it. A full campaign is two runs (control first) of 7 days x 60 multi-turn agent sessions each, plus judge and night calls - the dominant cost is the agent sessions on the executor model (the paper setting is qwen3-max through OpenRouter; a locally served model through the `serving/` proxy replaces the bill with GPU time). Budget accordingly before starting; there is no dry mode in this script. - **a Brave search key** (`BRAVE_API_KEY`, optional): the benchmark's search tasks call the Brave API from inside their containers; without a key those tasks degrade and the rest of the day still runs. - **the pi binary** (`REEF_PI_BINARY`): the mechanism's probe episodes run it headless; their scores are recorded, never gating. ```bash pip install reef-client export REEF_UPSTREAM_API_KEY=sk-... # required export REEF_UPSTREAM_URL=https://openrouter.ai/api # no /v1 suffix export REEF_MODEL=qwen/qwen3-max REEF_SC_RUN=frozen ./run.sh # the control run first REEF_SC_RUN=skillclaw ./run.sh # then the method run python3 run.py report # the gain table over the sealed rounds ``` Each run resumes after its last sealed round, so a crashed or interrupted campaign is rerun with the same command: completed tasks replay from their stored verdicts, already landed reports are skipped, and a night whose trigger report landed before the crash is recovered at boot. ## The Harbor smoke task `harbor/` is one task in the Harbor format every sibling example uses, and `PYTHONPATH=../.. python3 run.py solve` is the standard one-episode smoke over it (`pip install -e .` first; docker required, the campaign env vars apply). The repository root on `PYTHONPATH` exposes the cookbook's canonical `recipes.skillclaw.recipe` entry point. The task is the benchmark's own `03_Social_Interaction/task_1_meeting_negotiation`, vendored in full at the campaign pin: the prompt is `instruction.md`, its Warmup block is the container startup (mock Gmail and Calendar services on localhost, ground-truth fixtures deleted before the agent starts), and its Automated Checks block is `tests/grade.py`, writing `overall_score` to the verifier reward. This task was chosen because it grades fully programmatically (fixture-driven audit endpoints, no LLM judge, no external hosts), so unlike the campaign day it needs neither the multi gigabyte dataset download nor the benchmark's release image: the environment builds standalone and `lab.run` works anywhere docker does. WildClawBench is MIT licensed; the vendored content carries its notice (`harbor/LICENSE`, `task.toml`'s `license_note`). The solve agent (`harness/harbor_agent.py`) is deliberately minimal, one recorded model call through the same embedded service the campaign runs - the smoke proves the task contract end to end, not agent quality. Expect a low reward: a one-shot completion cannot work the mock APIs. ## Task list and grading `harness/tasks.json` is the frozen 60 task list at the WildClawBench pin, carried from the sealed campaign: it fixes the day size (the night trigger) and the category tables the criterion reads. Every entry names its benchmark task file; the prompt, timeout, warmup, environment, and automated checks come from the checkout, and grading is the benchmark's own (checks first, its judge where the checks call one). A task error voids grading: the score is the unscored sentinel report (-1.0, which still batches), never a fake zero, and the night's judge backfills it from the session.