# Run or upgrade to ALE v1.1 ALE v1.1 selects **122 active tasks** after 30 retirements. Use matching task code, environment profiles, images and data together. The website has the [configuration reference](https://agents-last-exam.org/docs?p=pages/configure.html) and [run tutorial](https://agents-last-exam.org/docs?p=pages/run.html). Task designs and source are frozen. Use the matching images and data pinned in the [asset manifest](../../releases/v1.1/assets.json). The [task review](v1.1-task-review.md) records validation scope and limits; it does not certify complete agent validation of the release. To run the original requirements instead, follow the [v1 guide](v1.md). Use v1 code, legacy images and fresh logs, not v1.1 assets or results. ## What changes from v1 **57 active tasks require reruns; 65 remain eligible for retention.** The six open-source conversions below are included. | Change | Tasks | What counts | | --- | ---: | --- | | Instruction | 47 | Agent-facing prompts and instructions, including input documents | | Verification | 43 | Scoring logic, acceptance rules and grader execution | | Environment | 32 | Task-specific software, dependencies, setup, staging, paths, resources or platform | | Data | 23 | Input materials, starter/helper assets, models, media or reference answers | Counts overlap: a task can appear in several rows. Classification follows the content of a change, not which repository stores it; a prompt edit inside a data archive counts as Instruction. Retired tasks, unselected variants, inactive metadata, packaging-only cleanup and shared framework changes are excluded from these counts. The [task review](v1.1-task-review.md) records each modified task's decision, source/data disposition and validation type. Instruction-only schemas/examples, grader-only changes and runtime-only scripts do not separately count as Data. The changes follow three principles: - Preserve the original business goal and intended difficulty; changing software must not remove required work. - Make grading match the public requirements, accept valid equivalent answers, and correct unsupported thresholds or incorrect reference answers. - Supply necessary inputs and working runtimes without giving away the solution; keep hidden references separate from the agent's workspace. Butterfly animation accepts any software. Both butterfly and skeletal animation now run on CPU QEMU, bringing CPU coverage to **119 tasks**, with **3 GPU tasks** listed in `selected_tasks/gpu.txt`. Skeletal evaluation prevents duplicate render launches after transport retries and keeps the original rendering and scoring requirements. Task setup replaces the old butterfly input specification, so these updates work with the published images and data without rebuilding them. Six formerly licensed tasks keep their IDs and business goals while moving to open-source Linux workflows: | Task | v1.1 workflow | Supported environment | | --- | --- | --- | | `engineering/2d_drawings_to_3d_building_model` | Blender building reconstruction | Docker or Linux VM | | `engineering/cailian_road_highway_alignment_2` | FreeCAD Road alignment | Linux VM | | `engineering/gcode` | FreeCAD CAM, LinuxCNC and CAMotics machining | Linux VM | | `engineering/inner_support_elevation_optimization` | Kratos excavation and support analysis | Linux VM | | `visual_media/music_transcription` | MuseScore notation, PDF and MIDI | Docker or Linux VM | | `visual_media/project_migration` | Ardour project, mix and stems | Docker or Linux VM | All six use the Linux VM image with QEMU or Google Cloud. The default Docker profile supports the three workflows listed above. Runtime compatibility checks cover native save/reopen, export and task artifacts; see the task review for their scope and limits. G-code's replay requires process/network isolation unavailable in that profile. Road requires systemd process supervision; inner-support also requires writable delegated cgroups for its worker resource limits. Use a Linux VM for these three; installing the applications alone does not provide those OS facilities. `engineering/mold-flow` is removed: its full four-cavity fill/pack/cool/demold workflow has no validated open-source replacement at the original scope. `education_info/homework_grading_numerical_pdes_instance_02` is retired because feedback verification was overly tied to fixed wording. A further 28 tasks are retired from v1.1; the [task review](v1.1-task-review.md#retired-tasks) lists all 30. Retirement is a release-scope decision, not a claim that every retired task is defective. Hugging Face data packages cover the **122 selected tasks**. ## Framework fixes and improvements These changes apply across tasks and are separate from the task-change counts above. | Area | Fix or improvement | What it means for a run | | --- | --- | --- | | Agent completion before evaluation | After the main session exits, wait for supervised background tools within the remaining agent budget. At timeout or cancellation, stop the whole supervised process tree. Confirm it has exited before staging or decrypting references. | Background work can finish normally without gaining extra time. Reference access stays blocked until process exit is confirmed. | | Solver failure isolation | On Linux with systemd, run the solver and its supervisor in a separate service from CUA. Protect only the supervisor from OOM selection, retain normal worker priority, and record the agent result before cleanup. | A worker OOM does not trigger CUA's service-wide shutdown. Exit codes and original errors survive cleanup failures; unconfirmed termination still blocks reference access. | | Resume | Check the task-folder hash and a successful, error-free evaluation with the same numeric score before reusing a result. | Missing or changed task hashes and unscored evaluations trigger reruns. | | Image identity | Record resolved image identities for QEMU, Docker and Google Cloud. Verify previously untracked QEMU cache files by checksum before adopting them. Rename QEMU's `hf_revision` setting to `image_revision`. | A run can be traced to its actual image; equal file sizes alone do not establish identity. Task hashes do not cover external data, images or shared framework code, so keep the corresponding asset pins too. | | Judge configuration and preflight | Give shared text/image judges separate `JUDGE_BASE_URL`, `JUDGE_API_KEY` and `JUDGE_MODEL` settings; check the required text/image interface before starting sandboxes. | Agent and judge endpoints can be configured independently. Invalid judge configuration fails early; later judge failures remain evaluation failures requiring retry. | | Evaluation error reporting | Preserve evaluation failures, including invalid Judge replies, instead of substituting zero or machine scores or allowing an agent timeout to mask them. | Failed evaluations have `score: null`; valid low scores retain their meaning. Odoo uses database checks without a non-scoring screenshot Judge. | | Input and reference staging | Add neutral workspace aliases, correct Docker input ownership, restore validated GCS symlinks and test encrypted reference archives before extraction. Distinguish missing GCS objects from failed access. | Inputs retain their required paths and permissions. Failed GCS probes preserve existing inputs and stop the run; corrupt reference archives are detected before grading. | | Data download and Docker builds | Require a pinned data archive and SHA256, prefer the current `hf` CLI, reject nonempty extraction targets and memory-backed staging, and add disk/memory checks to the local Docker build workflow. | Downloads cannot silently mix with an old data directory; image builds check resource headroom before allocating large files or VMs. | | Agent adapters | Add Pi, Qwen Code and ZCode with verified CLI versions, private configuration, accurate usage and process cleanup. Support ZCode on Windows and preserve its CLI's output-token defaults unless explicitly overridden. Fix Kimi Code's Windows npm launch; add Claude Code output-token configuration; preserve OpenClaw streaming and tool schemas, and reject incomplete result envelopes. | Adapters preserve credentials and report unsuccessful runs accurately. New CLI adapters upgrade their private Node runtime when required. | | Node bootstrap | Allow 60 seconds for Linux npm prefix configuration and retry once only on timeout. Report nonzero exits immediately. | Slow startup gets a bounded retry; configuration errors remain visible. | | Desktop tools | Configure pixel or normalized coordinates; refresh the bundled tool bridge; acknowledge Unicode clipboard transfers on Linux and use native Unicode input on Windows. | Coordinates match the advertised tool schema, and successive text inputs do not overwrite pending pastes. | | Configuration and helper models | Ignore environment-variable examples in YAML comments. Route ALE-Claw compaction and memory helpers through the selected endpoint and credentials. | Comments do not require secrets; helper calls respect explicit routing without leaking credentials across providers. | | Provider and recording reliability | Enable Docker init and UTF-8; clean up containers after acquisition failures; verify GCloud partition and filesystem expansion; distinguish unavailable machine types from invalid configuration; bound Octavus recording settlement. | Failed startup releases container resources, disk expansion failures remain visible, placement preserves configured priorities, and incomplete recordings do not cause unbounded waits. | | Task discovery | List tasks without importing their scoring code. | Broken metadata does not stop task listing. | | Documentation and rollback | Align README, Markdown guides and website docs with the release profiles and asset manifest; publish a `v1` tag and a linked v1 run guide. | Users can select a consistent v1.1 setup or reproduce v1 with its original assets and separate logs. | HF Sandbox is excluded from v1.1: the current integration does not provide reliable fast startup for the release image or compatible reusable desktop environments. Hugging Face data/image downloads and inference remain supported. On Linux with systemd, the sandbox executor requires root or noninteractive `sudo` access to `systemd-run` and `systemctl`. It preserves the agent's user, groups and environment. Systems without systemd retain detached process supervision; Windows uses Job Objects. These changes deploy with the framework and do not require rebuilding the published VM images. See the [judge setup](../quickstart.md#7-configure-secrets), [image and data pins](#keep-versions-together), [log upgrade procedure](#upgrade-existing-logs), and [v1 guide](v1.md) for the corresponding configuration and commands. Bulk output collection remains off by default (`output_path: null`); this is not a newly introduced default. Run metadata, trajectories, evaluations and telemetry are still collected. With `cleanup_mode: delete`, deliverables left inside the sandbox are deleted with it. ## Run v1.1 From the v1.1 checkout: ```bash bash env/install_host_deps.sh uv sync --all-packages --locked bash env/verify_host_env.sh cp -n secret/.env.example secret/.env cp example_exp.yaml my_experiment.yaml ``` The [host setup](../../env/README.md#host-evaluation-environment) installs system tools and validates all 122 active task entry points plus deferred evaluator imports against the checked-in lockfile. It does not provision VMs or invoke judges. Runs repeat dependency checks before Judge preflight and sandbox creation; dry runs only inspect configuration. Fill in the agent and provider credentials in `secret/.env`. Follow the [Google Cloud setup](../quickstart.md), [QEMU setup](../local-qemu.md), or [Docker setup](https://agents-last-exam.org/docs?p=pages/local-docker.html), then choose the environment and task list in `my_experiment.yaml`: | Task list | Tasks | Environment | | --- | ---: | --- | | `selected_tasks/full.txt` | 122 | `configs/environments/environment_gcloud.yaml` | | `selected_tasks/cpu.txt` | 119 | `configs/environments/qemu.yaml` | | `selected_tasks/gpu.txt` | 3 | `configs/environments/environment_gcloud.yaml` | | `selected_tasks/ale_cli.txt` | 86 | Linux VM profile; includes desktop workflows | | `selected_tasks/docker_support.txt` | 78 | `configs/environments/docker.yaml` | Difficulty lists under `selected_tasks/full/` contain 45 near-term, 46 full-spectrum, and 36 last-exam tasks; they overlap. Docker excludes eight Linux tasks requiring nested runtimes, namespaces, or systemd/cgroups; use QEMU for those tasks. AWS and Alibaba Cloud CPU profiles require matching v1.1 images; their existing images are not verified for the six converted workflows. Start with `tasks: selected_tasks/hello_both.txt` on a VM provider, then switch to the desired benchmark list. Set `wall_time_s: 18000` for a five-hour agent budget, and size `concurrency` for your available resources. ```bash uv run python -m ale_run run my_experiment.yaml --dry-run uv run python -m ale_run run my_experiment.yaml ``` Environment profiles default to `output_path: null`: bulk task output stays in the sandbox; run metadata, trajectories, evaluations, and telemetry are still collected under `output.root`. Use `cleanup_mode: keep` to inspect deliverables there. Retained VMs consume memory; stop or delete them after preserving needed evidence. With `cleanup_mode: delete`, sandbox files are removed at cleanup. ## Keep versions together The [asset manifest](../../releases/v1.1/assets.json) records dataset revisions, disk checksums, cloud image identities, and the Docker digest. | Component | Pin to preserve | | --- | --- | | Host Python environment | Checked-in `uv.lock`; install with `uv sync --all-packages --locked` | | QEMU guest disks | `qemu.image_revision` in each snapshot of `configs/environments/qemu.yaml` | | Google Cloud images | Copy the named `images.*.gcloud` images; optional `gcloud.image_revision` verifies the numeric `sourceImageId` of your project's copy, which gets a new ID | | Docker runtime | The profile selects `docker.pinned_image` from the manifest; optional `docker.image_revision` verifies a SHA256 image ID or repository digest | | Task data | `huggingface.*.revision` and `task_data.archive_sha256` in the manifest | In custom QEMU profiles, rename `hf_revision` to `image_revision`, keeping the same pin. Image revision settings are nested in each snapshot's provider block. `run.json`'s `environment.image_revision` records the acquired identity even without a configured pin. For QEMU remote disks, it records the resolved HF commit or GCS generation; the requested HF pin remains in source metadata. QEMU and Google Cloud images contain task data. Keep QEMU disks and overlays on persistent storage, with capacity for downloads and retained guests. Check the actual filesystem and available RAM before allocating. Remove `image_revision` when changing a QEMU disk source from Hugging Face to GCS or a local file. Docker needs the separate data archive. Obtain approved access to the [archive dataset](https://huggingface.co/datasets/agents-last-exam/agents-last-exam-data-archive), authenticate, then run from the repository root on persistent storage: ```bash IMAGE=$(jq -r '.docker.pinned_image' releases/v1.1/assets.json) docker pull "$IMAGE" REVISION=$(jq -r '.huggingface.archive.revision' releases/v1.1/assets.json) SHA256=$(jq -r '.task_data.archive_sha256' releases/v1.1/assets.json) scripts/fetch_task_data.sh --revision "$REVISION" --sha256 "$SHA256" ``` The fetcher verifies the archive and installs into an absent or empty `task-data-v1.1`; do not merge it with an old data directory. The Docker profile uses `task_data_source: local:task-data-v1.1`. Only the current task's inputs and software are staged before the agent; references are staged afterward. Direct `hf://` task-data staging is not implemented. Guest-side GCS staging uses `task_data_source: gs://ale-data-public/v1.1` and evaluator credentials. ## Upgrade existing logs `run.json` records `task.revision` before setup: a deterministic SHA256 of task folder file names, contents, executable bits, and link targets, ignoring runtime caches. Resume skips completed or timeout units only with a matching hash and a successful, error-free `eval_result.json` containing the same numeric score as `run.json`. Missing or mismatched hashes and unscored evaluations rerun. `environment.image_revision` records provider image provenance. Keep image and dataset pins alongside the task code. Excavation-support and UV-reproduction evaluation use a separate sandbox directory for each invocation. Model snapshots, rendered images and diagnostic receipts stay in the sandbox; host-side scoring uses memory without writing to task folders. Evaluation therefore does not change the task hash. Use `cleanup_mode: keep` to inspect sandbox evidence before releasing it. To complete a v1 experiment, use `tasks: selected_tasks/v1.1_modified.txt` and a fresh `output.root` to run the 57 tasks modified at release. The other 65 active tasks were unchanged at release. Reuse valid v1 results only after checking that the task, data, environment and agent configuration match, and exclude the 30 retired tasks. Later changes to a task may require another run. Keep the original records intact; resume will rerun records without a matching task hash rather than automatically importing them. To reproduce v1 instead, follow the [v1 run guide](v1.md) with original code, legacy images and fresh logs. Do not reuse v1.1 logs for v1.