# LeanBuild Build a Lean project from GitHub on lean-cloud. Give it a repository, revision, and Lake targets. Workers compile ready modules using their local CPUs and share artifacts through blob storage. The final report links to the completed workspace, executables, and diagnostics. Lean's normal elaboration and kernel checking run as part of compilation. For a full example, follow [Build Mathlib on AWS](MathlibAWS.md): create four workers, build a pinned Mathlib revision, watch its progress, and clean up. ```json { "repository": "https://github.com/leanprover-community/batteries", "revision": "66e469188fe07963e5cf666c7d4433f45b8131c5", "targets": ["+Batteries.Data.Nat.Bisect", "+Batteries.Data.Nat.Bitwise.Lemmas"] } ``` Omit `targets` to build the project's default Lake targets. Use the same names as `lake build`: for example `Mathlib`, `my_executable`, `+My.Module:olean`, or `@dependency/Library`. Custom targets are supported too. A branch or tag is resolved once; the actual commit is recorded in the result. ## How it runs The Cloud workflow has five stages: 1. Fetch the revision, resolve Lake dependencies and generated-source prerequisites, discover the module DAG, and publish the prepared workspace. 2. Enqueue modules with no prerequisites. Record the number of builder sessions from `env.workers.size`. 3. Run the builder sessions in one `Cloud.parallel`. Each is a long-lived `exec` that continuously feeds ready modules to one persistent Lake session. 4. Collect the immutable result for every planned module. Failed modules block their dependents; independent modules can still finish. 5. Restore the artifacts on one worker and ask Lake to complete the original targets, including native compilation, linking, custom targets and dynamic dependencies. Publish the complete workspace manifest. Inside these `exec` boundaries, LeanBuild uses **Eff programs**. The [builder model](LeanBuild/Builders/Model.lean) makes pure scheduling decisions; the [program](LeanBuild/Builders/Program.lean) describes preparation, publication, successor notification and queue settlement. The [Lake session](LeanBuild/LakeSession.lean) has explicit start, submit, poll, finish and cancel operations. [IO handlers](LeanBuild/Builders/IO.lean) own tasks, leases and bounded transfer pools, and join outstanding work on exit. [Project preparation and finalization](LeanBuild/Project.lean) and [prefetch ordering](LeanBuild/Prefetch/Program.lean) are also Eff programs. These are local effects; they do not add Cloud replay records or scheduler round trips for individual modules. Eff programs use [BuildError](LeanBuild/BuildError.lean) to distinguish cancellation, network, tool and protocol failures. Errors stay typed across parallel jobs. Cleanup preserves the original cause and attaches later failures; it still joins tasks and attempts to release every lease. An operation error is an exec failure, not a worker crash or an automatic retry. Each builder restores the prepared sources once and loads Lake once. A bounded pipeline receives ready work, restores missing prerequisites, compiles modules and publishes their artifacts concurrently. Each worker admits up to twice its compiler-slot count, including preparation and jobs already submitted to Lake. The persistent Lake session queues prepared modules: seven compiler slots can have seven replacements waiting. The session starts the next module locally when a Lake job finishes, without waiting for the outer coordinator to poll it and submit more work. An explicit job limit enforces the compilation budget; Lean's task-pool size alone is not a hard process limit. The bounded window leaves further work available to other workers. Receives do not define execution batches. Shared prerequisite files are transferred once, and concurrent preparations await the same transfer. Ready modules are ordered by the longest remaining dependency chain, computed once from the DAG. The shared queue offers higher priorities first; equal priorities keep submission order. Parallel preparation can change when they reach Lake. Both initial modules and newly ready dependents use the same lookup. Every worker continuously looks ahead through this ordered queue while Lake is busy, prefetching the upcoming modules' prerequisites. The scan does not claim work, and continues beyond the first page. It revisits the front every 200 ms so new high-priority modules overtake unstarted downloads. Free download slots refill without waiting for a whole page. It does not download unrelated completed modules. Dequeuing a module waits only for prerequisites still missing or in flight; preparation and prefetch share transfers and the local cache. Dependency lookup follows the requested ancestry and stops at subtrees already prepared on that worker. It does not scan the full DAG or pass every historical prerequisite through the download pool again. Concurrent lookups share result reads and module installations; cache hits and waiters consume no transfer slots. Artifact bytes travel directly from blob storage. Read and upload pools each allow up to `min 32 (max 4 (4 × CPUs))` concurrent transfers, shared across nested operations. Compilation still uses `max 1 (CPUs − 1)` slots. As each module finishes, a bounded IO pool packs and uploads its artifacts, publishes its immutable result, notifies ready dependents, and acknowledges its delivery. Each module stays in one pack; parallel transfers do not split its small files into separate objects. Large packs upload chunks concurrently. Uploads overlap both other uploads and ongoing compilation. All claims are renewed until settled. The workspace, manifest cache and downloaded packs stay on the worker's local filesystem; already installed imports are not copied again. For `Base → [Left, Right] → Final`, publishing `Base` makes `Left` and `Right` available immediately. `Final` becomes available when both are committed. Any worker, including the one compiling `Left`, can start newly ready work while its other modules are still compiling or uploading. A compilation completion immediately frees a slot. There are no batch barriers, global dependency rounds or repeated Cloud forks for module scheduling. The queue lives in the scheduler's existing HTTP/SQLite service. Its short transactions run independently of the Cloud scheduler actor. All deployments use the same implementation; no SQS, Service Bus, or separate queue service is required. Sources, artifacts and module results move directly between workers and blob storage, not through the scheduler. Module names are idempotent queue keys: several parents can announce the same ready module without creating more work, even after its acknowledgement. Each ready successor is enqueued immediately. Other successors check uncached parents with bounded parallel IO and stop when a missing parent is found. Queue deliveries have renewable leases and may repeat. Artifacts are uploaded before publishing an immutable, run-scoped result; notifications precede ACK. After interruption, redelivery reuses a committed result and repeats any needed notifications. An empty receive is not completion: builders wait until the queue has no pending **or leased** messages, and the collection stage still requires one result for every module. This relies on seeding all roots before starting builders and sending every successor before acknowledging its predecessor. Pause or kill signals `env.cancellation`. Builders stop accepting work, terminate active compiler process groups, join local tasks, and release outstanding work. The runtime checks assignment ownership before journaling the `exec` outcome. Ordinary IO exceptions remain application failures; they are not crash retries. HTTP clients retry transient connection failures and temporary service responses with bounded backoff and jitter before reporting an exhausted operation as an error. `watch` shows one shared module completion counter and each worker's submitted Lake jobs (running plus queued), compiler-slot budget, restoring, ready, uploading, awaiting-upload and unlocking counts. Unlocking checks dependent work after the result is durable; it does not occupy an upload slot. Queued jobs do not imply extra active compilers. Failed/blocked modules are included in completed work and displayed separately. The service updates the durable counter atomically with settlement; repeated acknowledgements cannot inflate it. Workers report snapshots approximately once per second independently of compilation. Watch selects the newest counter revision and marks stalled activity updates stale. Finalization reports its current phase: gathering manifests, restoring artifacts, completing Lake targets, verifying the workspace, or publishing the result. Displayed progress is a best-effort last view, separate from replay records. ## Local deployment and first run From the lean-cloud repository: ```sh bash Applications/LeanBuild/build.sh lean-cloud-build:dev bash deploy/local.sh configure .lean-cloud/build lean-cloud-build:dev build 3 8080 bash deploy/local.sh up .lean-cloud/build lake exe lean_cloud ``` In the console, enter each command separately: ```text connect build http://127.0.0.1:8080 --token-file .lean-cloud/build/client.token programs run lean-build --input Applications/LeanBuild/example.json --id batteries ps watch batteries result batteries ``` The example builds two modules from a pinned Batteries revision. The same image works with the [Azure](../../deploy/Azure.md), [AWS](../../deploy/AWS.md), and [existing Kubernetes](../../deploy/README.md) deployment helpers; the console uses the same HTTP connection in each environment. ## Build results The report contains: - `commit`, `toolchain`, `platform`, and `source`: exact input provenance. - `modules`: per-module status, diagnostics, elapsed time and artifact manifest, in dependency order. Module durations run from their submission to Lake through compilation completion; they include waiting for Lake jobs and are not CPU time. Failed prerequisites produce `blocked` results. - `build`: final Lake target status, diagnostics, duration and an `artifacts` reference to the complete workspace manifest. Its duration includes final artifact publication and the local cache checkpoint. - `success`: true only when the modules and final target build succeed. Save a diagnostics or manifest reference from the report as JSON, then read it: ```sh bash Applications/LeanBuild/local-files.sh .lean-cloud/build ref.json ``` Each manifest entry records a relative path, a packed-file reference (pack, offset, length, SHA-256), and an executable flag. `files download CONFIG FILE_REF OUTPUT` downloads a top-level file reference; `files restore CONFIG MANIFEST_REF DIRECTORY` restores a workspace into a new directory. These application commands run inside the deployed image, where storage is configured. Transfers verify sizes and hashes. Native executables require a compatible platform and libraries. Remove the test deployment and its stored data: ```sh bash deploy/local.sh clean .lean-cloud/build --yes ``` ## Configuration and resource use | Request field | Meaning | Default | | --- | --- | --- | | `repository`, `revision` | Public GitHub HTTPS URL and Git revision | Required | | `targets` | Arguments to `lake build` | Project's default targets | | `subdirectory` | Lake project directory inside the repository | `"."` | | `options` | Lake configuration values, e.g. `{"feature":"enabled"}` passes `-Kfeature=enabled` | `{}` | | `cache` | `"rebuild"` clears package build directories and disables external Lake caches; `"reuse"` allows valid prepared cache artifacts | `"rebuild"` | | `limits` | Command and artifact limits | Below | Builder count is recorded once. Each builder uses its executing worker's current CPU budget, so a restarted session may have a different local width. These are concurrency budgets, not CPU affinity or placement guarantees. An N-CPU worker uses `max 1 (N - 1)` compilation slots, leaving headroom for coordination and artifact processing. Individual module compilers default to one thread so they do not each create a hardware-sized pool. Preparation and final targets also use the full budget. Native tools and project options may create additional threads; container limits remain authoritative. Configure `cpuMillis` and `memoryMiB` in `deployment.json` (see [worker resource budgets](../../deploy/README.md#worker-resource-budgets)). An unconfigured worker advertises one CPU. The local pool does not dynamically estimate module memory use: choose enough memory for the concurrent compilers. Artifact hashing, restoration and blob transfers also use bounded local pools. `limits` defaults to `seconds: 600` per subprocess, `helperMemoryMiB: 4096` per Lake helper, `bytes: 2147483648` per workspace, and `modules: 10000`. Preparation and the final Lake invocation must each finish within `seconds`. A persistent Lake session lasts as long as its worker's share of the build, so there `seconds` bounds inactivity instead: while modules wait in Lake, the next one must finish within `seconds` of the previous completion (or of submission when Lake was idle), and the session must exit within `seconds` after its last module. Fewer workers therefore take longer rather than failing, while a hung compiler still stops the build. Diagnostics are capped at 16 MiB per stream. Workspace size is checked at preparation, restoration/publication boundaries and after all local compilers stop. Container memory and disk limits bound peak usage between these checks. Workers cache immutable packs in `/data/build-cache` and one reusable workspace in `/data/build-cache.workspace`. The workspace key includes prepared sources, request, toolchain and helper. Switching builds replaces it. Cache hits are verified; an interrupted session leaves no clean marker and is reconstructed from committed blob artifacts. Cache directories must be private to each worker process. `LEAN_BUILD_CACHE` changes the pack directory and `LEAN_BUILD_CACHE_BYTES` bounds packs (default 4 GiB). Without a cache, builders use temporary workspaces. Provision extra disk for logs and temporary files. Concurrent readers share a pack download; independent packs download in parallel. Packs stay pinned while being copied. When active packs fill the cache, additional downloads use temporary space rather than evicting a pack still in use. Prepared inputs and installed artifacts remain immutable during module builds. Conflicting artifacts are rejected. Build hooks must declare dependencies to Lake and work in relocated workspaces. Custom targets and unusual facets remain the final Lake build's responsibility. Dependencies must be inside the snapshot; internal file symlinks are materialized, while external links, directory links, Git submodules and external dependency directories are rejected. ## Toolchains and compatibility Source projects need a stable Lean **4.35.0 or later** release. The project's `lean-toolchain` must match the deployed toolchain exactly; it is never rewritten. To select an exact release: ```sh bash Applications/LeanBuild/build.sh lean-cloud-build:dev 4.35.0 ``` The image pins the release checksum. Each worker downloads Lean directly from GitHub at first startup, verifies it and checks the Lake helper. Restarts reuse `/data/toolchains`; the scheduler does not download a toolchain. Lean 4.35.0 needs roughly 3.2 GB of persistent space plus installation scratch space. Kubernetes can use `startupSeconds: 900` to allow installation time. Alternatively, `LEAN_BUILD_TOOLCHAIN` selects an existing installation. `LEAN_BUILD_RELEASE` and `LEAN_BUILD_TOOLCHAINS` select the pinned release JSON and download cache. 4.35.0 is tested; future Lake API changes may need helper updates. The workflow is `lean-build/v6`. Use the same application image for a run's entire lifetime, including recovery. Run trusted repositories in dedicated deployments: Lake configuration, build hooks, plugins and metaprograms execute code inside the worker container. ## Tests ```sh cd Applications/LeanBuild elan toolchain install "$(cat source-toolchain)" export LEAN_BUILD_TOOLCHAIN="$(elan run "$(cat source-toolchain)" lean --print-prefix)" export LEAN_BUILD_HELPER="$PWD/Helper.lean" lake build cloud_app build_local build_tests .lake/build/bin/build_tests --native ``` Pure handlers test the same Eff builder over 680 seeded DAG schedules on 2–4 builders. Each turn advances one effect, with separate Lake sessions, bounded parallel jobs and local artifact caches. Schedules inject lease expiry/redelivery and cancellation between installation, submission, publication, commit, successor notification and acknowledgement. They check results, stale receipt fencing and cleanup, and assert that the intended fault interleavings actually occurred. 320 schedules inject one-off failures in result reads, queue sends/receives and artifact publication; half also fail compiler cleanup. They verify typed error propagation, preservation of the original cause and recovery of released work by the remaining builders. Tests also compare direct/replay execution on 40 generated DAGs. They cover concurrent builder sessions, bounded lookahead that leaves work available to idle workers, progress without round barriers, interruption between result publication and notification, cancellation, blocked dependencies, durable errors, cache reuse and integrity. A native rendezvous verifies concurrent publications while another compiler in the same session is still running. A single-worker rendezvous checks that newly ready dependencies advance before an unrelated slow compiler completes. Native compiler processes also verify that Lake queues replacements while all slots are occupied, starts them without another request, and keeps the compiler limit, including on a single-CPU worker. Incremental prerequisite lookup is compared with full DAG traversal, and shared installation tests cover concurrent reads, failures and retry from a cold workspace. Native fixtures compare output with ordinary Lake and cover libraries, executables, C FFI, generated/custom targets and restored artifacts. A native time-limit test checks that a chain of slow modules may outlast `seconds` while a hung compiler still stops the build. Runtime tests separately exercise the same HTTP/SQLite queue's leases, reopening, stale receipts, concurrent consumers and run removal. The container test uses the ordinary HTTP client and removes its temporary deployment on exit: ```sh bash Applications/LeanBuild/test-deployment.sh lean-cloud-build:dev ``` The [AWS Mathlib walkthrough](MathlibAWS.md) uses a full-build configuration tested on 31 October 2026: 9,031 modules completed on four workers in 25m27s. Shared queue throughput, artifact IO, module memory use and the final target build can still limit utilization. Finalization merges manifests in one pass, verifies files concurrently, retains existing entries from that scan, and uploads the final manifest in parallel chunks. Unchanged artifacts keep their original blob references.