# Testing This page is for contributors. It says what CI runs, how the end-to-end test works, and how to change its data. The stub runs in `nextflow.yml` check that the Nextflow wiring holds together, and `container-test.yml` runs each changed image's own tool on the fixture. Neither runs a pipeline step as the scripts and modules call it, so a step can be broken in real use while both stay green. The **E2E** workflow ([`.github/workflows/e2e.yml`](https://github.com/GeiserX/OwnGenome/blob/main/.github/workflows/e2e.yml)) closes that gap: it runs the real tools, in the pinned containers, on a small slice of a public genome, and checks what they write. Every `nextflow run` step in `nextflow.yml` has a time limit of its own, well under the job's 45 minutes. It is 10 minutes for the runs without containers, which take under a minute, and 20 for the stub run that pulls images, which took about four minutes in the run we timed. When a step of that job fails or is cut by its limit, the job prints which tasks were running and which were waiting to start, and uploads `.nextflow.log` as the artifact `nextflow-logs-` (`pinned` or `26.10`). Without that log the first two hangs of the retry step could not be explained ([Lessons learned](lessons-learned.md#the-local-executor-can-lose-track-of-free-memory-and-a-task-that-needs-all-of-it-then-never-starts)). The stub runs never execute a `bin/` helper the way a task does, by its bare name from the PATH, so a helper committed without its executable bit passes them and fails only in a real task (exit 126). The `helper-binaries` job of `guard.yml` runs [`scripts/ci/check-exec-bits.sh`](https://github.com/GeiserX/OwnGenome/blob/main/scripts/ci/check-exec-bits.sh): every `bin/` file that `main.nf`, `workflows/` or `modules/` calls by name must be committed as mode 100755 and start with a `#!` line. `tests/test_exec_bits.sh` plants a helper without the bit, which must fail. ## The fixture The test data is a slice of **HG002**, the Genome in a Bottle (GIAB) son of the Ashkenazi trio. HG002 is a public, consented reference sample, so nothing in the fixture is personal data. [`scripts/ci/build-fixture.sh`](https://github.com/GeiserX/OwnGenome/blob/main/scripts/ci/build-fixture.sh) builds it. It reads only the regions below from GIAB's 60x GRCh38 BAM on GIAB's S3 mirror (`https://giab.s3.amazonaws.com/`) over HTTPS (samtools fetches byte ranges through the `.bai`, so the 126 GB file is never downloaded) and samples them down to about 30x. | Region (GRCh38) | Why it is there | |---|---| | chr1:109,600,000-109,800,000 | GSTM1 (pypgx reads depth over it) | | chr2:233,600,000-233,800,000 | UGT1A1 and UGT1A4 | | chr4:68,500,000-68,700,000 | UGT2B15 and UGT2B17 | | chr16:28,580,000-28,630,000 | SULT1A1 | | chr19:40,800,000-41,050,000 | CYP2A6, CYP2A7 and CYP2B6 | | chr20:10,000,000-10,500,000 | small variants; the planted ClinVar record (in SNAP25); the GIAB truth slice | | chr22:42,000,000-42,300,000 | CYP2D6 and CYP2D7 with flanks | | chr12:47,800,000-47,950,000 | VDR, the control gene pypgx normalises depth with | | chr10:94,700,000-95,000,000 | CYP2C19 and CYP2C9 | | chr6:29,900,000-33,100,000 | HLA | | chr5:69,900,000-71,100,000 | SMN1 and SMN2 | | chrX:73,700,000-74,000,000 | non-PAR chrX | | chrX:1,000,000-1,200,000 | PAR1 | | chrY:2,700,000-3,000,000 | non-PAR chrY (HG002 is male) | | chrY:6,800,000-7,000,000, chrY:12,600,000-12,800,000, chrY:14,600,000-14,800,000, chrY:19,600,000-19,800,000, chrY:20,500,000-20,700,000 | five more single-copy stretches of chrY (TBL1Y, USP9Y, NLGN4Y, KDM5D, EIF1AY), each in a different piece of the single-copy part step 33 reads, so its mosaic loss-of-Y estimate rests on about 120 blocks of 10 kb over six pieces (since `fixture-v7`) | | chrM | the whole mitochondrial genome, sampled to about 500x (the source has thousands of x there) | The chr5, chrX and chrY slices are not used by every step yet. They are there so the paralog, ploidy, sample-QC and Y-haplogroup work needs no new fixture. What the release holds: | File | Content | |---|---| | `HG002_R1.fastq.gz`, `HG002_R2.fastq.gz` | the sliced reads as name-sorted pairs, input for step 02 | | `HG002_slice.bam` (+ `.bai`) | the same reads as GIAB aligned them | | `HG002_cyrius.bam` (+ `.bai`) | GIAB's alignment of the regions Cyrius (step 21) reads, sampled the same way: CYP2D6, CYP2D7 and its 3,000 depth-normalisation bins on chr1 to chr22, plus the two 50 kb flanks (chr22:42.05-42.10 and 42.20-42.25 Mb) step 21's depth check compares CYP2D6 with. Cyrius stops on the first bin whose contig the BAM lacks, so it cannot run on the BAM step 02 writes against the fixture reference | | `fixture_ref.fa.gz` (+ `.fai`, `.gzi`, `.dict`) | whole chr1, chr2, chr4, chr5, chr6, chr10, chr12, chr16, chr19, chr20, chr22, chrX, chrY and chrM from the NCBI GRCh38 no-ALT analysis set, the pipeline's default reference, so every coordinate and contig name is real | | `clinvar.vcf.gz`, `clinvar_chr.vcf.gz`, `clinvar_pathogenic_chr.vcf.gz` (+ `.tbi`) | ClinVar records inside the regions, built the way `setup.sh` builds the full files | | `planted.tsv` | one synthetic ClinVar record (CLNSIG Pathogenic, gene SNAP25, ID 900000001) at a SNV HG002 is homozygous for, so step 06 always has a hit with a known gene | | `HG002_vep.vcf` | up to 200 GIAB truth variants (HLA-A, -B, -C, CYP2C19, CYP2C9, SNAP25) annotated by VEP `--database --everything`; it stands in for step 13, whose offline cache does not fit a runner. `--database` gives no gnomAD frequencies, so steps 23 and 31 only run their no-gnomAD branch | | `revel_synthetic.tsv.gz` (+ `.tbi`) | a score file in REVEL's layout for the SNVs of the VEP subset. The scores are made up; they only show that a score track is applied | | `HG002_sv_manta_style.vcf.gz` (+ `.tbi`) | ten Manta-style SV records, input for AnnotSV and the SV readers | | `HG002_truth_chr20.vcf.gz` (+ `.tbi`), `HG002_truth_chr20.bed` | GIAB v4.2.1 truth for the chr20 slice | | `HG002_GRCh38_1_22_v4.2.1_benchmark.vcf.gz` (+ `.tbi`), `HG002_GRCh38_1_22_v4.2.1_benchmark_noinconsistent.bed`, `HG002_GRCh38_v5.0q_smvar.vcf.gz` (+ `.tbi`), `HG002_GRCh38_v5.0q_smvar.benchmark.bed` | GIAB's two HG002 truth sets, whole and byte for byte as GIAB publishes them, for `benchmark-variants.sh --giab` (its md5 check accepts nothing else). The GIAB e2e case reads them from here, so it needs no download. About 214 MB of the release. The build checks each file against the md5 the step accepts; MANIFEST.txt lists the md5s and sources | | `regions.bed`, `MANIFEST.txt`, `SHA256SUMS` | the slices, how this build was made (sources, depth, ClinVar date, build-script checksum), checksums | Why chr2, chr4 and chr16: pypgx (step 32) reads depth over the region of every gene it can call copy number for before it calls any of them, and samtools refuses a region on a contig the BAM does not have. One missing contig and step 32 calls no BAM-based gene at all, CYP2D6 included. Its GRCh38 region for GSTT1 is on an ALT contig, `chr22_KI270879v1_alt`, which the default reference does not have; step 32 leaves GSTT1 out on the fixture as it does on a real sample. Up to `fixture-v4` the fixture carried that contig; `fixture-v5` dropped it, so the e2e job runs on the contigs a default setup has. The whole release stays under 1.5 GB. The build checks that both BAMs pass `samtools quickcheck`, that every `.gz` file passes `gzip -t`, that `samtools idxstats` shows reads on every primary contig of the reference, that the reference has no ALT or HLA contig, that the Cyrius BAM has reads on chr1 to chr22 and reads with MAPQ >= 1 in both CYP2D6 flanks, and that each GIAB truth file has the md5 `benchmark-variants.sh` accepts. After download the e2e job repeats the BAM, `gzip -t`, contig and size checks (`tests/e2e/01-fixture.sh`); the GIAB case checks the truth files by md5 and `pgx-1-depth-check.sh` checks the flank depth. ### Where it lives and how to change it The fixture is published as assets of a GitHub release, a prerelease that is never marked latest. Its tag is the one line in [`tests/fixtures/VERSION`](https://github.com/GeiserX/OwnGenome/blob/main/tests/fixtures/VERSION) (now `fixture-v7`). The e2e job reads the same file, so the workflow never changes when the data does. To change the data: 1. Edit `scripts/ci/build-fixture.sh`. 2. Bump `tests/fixtures/VERSION` (for example from `fixture-v7` to `fixture-v8`). 3. Push the branch and open the pull request. The `build-fixture` job runs on every pull request and on every push run on main, and does nothing while the release exists and was built by the same script. When the new tag has no release yet, it builds the data on a GitHub runner (30 to 70 minutes, most of it VEP querying Ensembl's public database) and publishes the new release, `MANIFEST.txt` last. A newer push to the pull request cancels a build in progress; a release left without `MANIFEST.txt` is built again by the next run. The e2e job of your pull request waits up to 75 minutes for it, and that wait counts against the job's time limit (`timeout-minutes` of the `e2e` job in `e2e.yml`). A pull request from a fork gets a read-only token and cannot publish the release: `build-fixture` fails at once, and a maintainer builds the release from a branch of this repository with `job: build-fixture`. A pull request that changes the build script but keeps the old version fails `build-fixture` on purpose: the existing release was built by different code, and replacing its files would change the data under every open pull request. To rebuild a release in place anyway (for example after a failed upload), run the E2E workflow by hand with `job: build-fixture` and `rebuild: true`. It deletes the old `MANIFEST.txt` before it replaces the other files, so a cut rebuild is built again by the next run. You can build it yourself on Linux with Docker, `bgzip`, `tabix` and `python3` and about 10 GB of free disk: `scripts/ci/build-fixture.sh /path/to/out`. ## The e2e job [`scripts/ci/e2e-run.sh`](https://github.com/GeiserX/OwnGenome/blob/main/scripts/ci/e2e-run.sh) downloads the fixture, checks `SHA256SUMS`, lays out a `GENOME_DIR` the way `setup.sh` and step 13 would leave it, and runs the case files in [`tests/e2e/`](https://github.com/GeiserX/OwnGenome/tree/main/tests/e2e). It runs all of its cases even when one fails, so one run lists every broken step. In CI the cases run in parallel legs, each case at most once: a pull request runs only the cases it can affect ([Which cases a pull request runs](#which-cases-a-pull-request-runs)), main and the monthly run run them all. On 2026-11-02 the 130 cases took 230 minutes one after another on one runner (run 41390434697), against a 300-minute job limit. Now: - `e2e-plan` splits the cases with [`scripts/ci/e2e-shards.sh`](https://github.com/GeiserX/OwnGenome/blob/main/scripts/ci/e2e-shards.sh). The leg `e2e base` runs the cases of stage `base` in [`scripts/ci/e2e-case-times.tsv`](https://github.com/GeiserX/OwnGenome/blob/main/scripts/ci/e2e-case-times.tsv) (every numbered case: the fixture check, setup validation, alignment, DeepVariant, GATK, the single steps on HG002, the reports and the Nextflow run; and `bash-step-08-hla`), because most other cases read what those wrote. Every other picked case goes to one of the shard legs (`e2e shard 1`, `e2e shard 2`, ...). A full run uses all `E2E_SHARDS` (6) legs. For picked cases, `e2e-shards.sh count` sets how many: the cases' total seconds divided by the heaviest group's seconds (the critical path, since a group is never split) or by 900 seconds when that is larger, rounded down, plus one more leg when the rest is 900 seconds or more, and never more than `E2E_SHARDS` (6). A shard leg spends 18 to 20 minutes on top of its cases (pulling images, fetching and unpacking the base outputs, the parity checks and the uploads; run 41847167444), so a leg with less than 15 minutes of cases costs more runner time than it saves in wall time. With every case that rule gives 4 or 5 legs, and that is why a full run does not use it: the runner's disk, not time, is what a leg runs out of. With 5 legs, shards of 30 and 37 cases filled one runner's disk until the runner itself died (run 41995851301, shard 3), while 6 legs passed on the same main (run 41995719742). - The table holds each case's median time on the passing runs of nine e2e runs of 2026-11-04 (41766267038, 41767270253, 41773233871, 41774860458, 41779067613, 41790595358, 41799741941, 41807346887 and 41821518376), and its `needs`: the non-base cases whose outputs it reads. The needs were written from what each case says it reads (its header and its checks) and from the cases that failed in a split run without them (run 41455829120). A case and every case it needs, followed through, form a group that stays in one shard, in name order. Groups go to the least loaded shard, heaviest first. Every case needs a row: `e2e-shards.sh check`, which `tests/test_e2e_shards.sh` runs on the real table in the unit tests, fails and names each case without one. The split itself still runs a case with no row, as a 300-second case that needs nothing, so add the row with the case's measured seconds from its first passing run. - The base leg runs first. It packs every file its cases wrote or changed (at its absolute path, with owners, modes and times) and uploads it as the artifact `e2e-base`, kept for 3 days so that "Re-run failed jobs" the next day still finds it. The shard legs start only after it passed (`needs:`), so a shard never holds a runner while it waits: when many runs share the account's runners, shards that started early and waited took the runners the base legs needed (run 41506673662 had three shards running and its base leg still queued). Each shard fetches the artifact, tried up to five times, takes the base leg's work area, unpacks the archive there, deletes it (about 4 GB the cases need) and runs its cases. When the base leg fails, the shards are skipped and `e2e` is red. - The shard that holds `nextflow-from-fastq-2-nextflow` and the one that holds `nextflow-hardening-2-nf` run the parity checks; the plan fails when either case runs and the case before it in its pair is in a different shard. - The required check is the job `e2e`. It merges every leg's results into one table of case, result, time and job, followed by the failed checks of each failed case, and it is red unless the plan and every leg succeeded, every picked case reported exactly one pass and each parity check passed once when its cases ran (and not at all when they did not). It is red too when `e2e-scope` did not succeed. A leg that was skipped, cancelled or cut by its limit leaves cases with no result, and that is red too. The full logs are in the `e2e-logs-` artifacts. `tests/test_e2e_shards.sh` (unit tests) checks the split against planted faults: a case listed twice or missing, a stale or malformed table row, a need on a case that runs later, a pick that leaves out a need, and results missing, failed or reported twice; the leg count; and a case with no row, which `check` must name. `tests/test_e2e_scope.sh` checks the pick: a case with no paths row, a second row, a row for no case and a glob that matches no file must each fail `e2e-scope.sh check`, and a docs-only change, a change to `main.nf`, a change to one helper, to one module, to a case and to a table row must each pick the cases listed below. To measure the table again, run the E2E workflow by hand with `job: e2e` and `trace: true`. That run puts every non-base case in one shard, so it runs as long as one serial run. After each case `e2e-run.sh` records its time and, with the file systems remounted `strictatime`, every file under the work area and the checkout whose atime moved during the case, attributed to the case whose run last changed that file's ctime. The job `e2e` then prints a new `e2e-case-times.tsv` in its summary and uploads it as the artifact `e2e-case-times`; it keeps each row's stage and needs and adds the traced needs. It refuses a trace that saw no case read another case's file. The traced needs are an upper bound, not the table to commit: a file counts for the case that changed it last, so a case that rewrites or touches a shared file (a report, an index) shows up as a need of every later case that reads that file, and a case that scans a folder reads everything in it. On the trace of run 41454755531 they joined almost every case into one group, so check each traced need against the case before you add it. A read the trace cannot see, such as a test for whether a file exists, is not in the table: such a case fails in its shard, and its row then needs a hand-written `needs`. Every pull request starts the workflow, so main can require the e2e check. Its `e2e-scope` job picks the cases (next section); when the pull request changes no path in the e2e scope the plan and the legs are skipped and the job `e2e` passes without running anything, so a docs-only change is never blocked. The e2e job also runs once a month and by hand (`job: e2e`), with every case. It also runs, with every case, on main after every merge that touches the e2e scope, because main does not require a branch to be up to date before it merges: two pull requests that land close together each passed alone, and only the run on main tests them together. A docs-only merge starts no run. A run on main is never cancelled half way. A merge that lands while one runs waits, and GitHub keeps one waiting run, so a newer merge replaces the waiting one and the next run tests every merge since the last run. A red run on main names its commit, the newest merge it tested; the merges between the last green run on main and that commit are the suspects. The push paths in `e2e.yml` and the scope in `scripts/ci/e2e-scope.sh` (`e2e-scope.sh scope` prints it) are the same list written twice, and the first step of `e2e-plan` fails when they differ. A push to any other branch starts no run: its pull request runs e2e, and an `e2e` check from a push run would sit on the same commit as the pull request's required `e2e` check. The DeepVariant case passes the fixture slices as `INTERVALS`, so DeepVariant calls only those slices instead of walking all 1.8 Gb of the reference, which alone took 27 to 48 minutes. It uses a standard GitHub-hosted runner (4 CPUs, 16 GB of RAM). Every e2e leg and both `nextflow.yml` legs first run the composite action [`.github/actions/free-disk`](https://github.com/GeiserX/OwnGenome/blob/main/.github/actions/free-disk/action.yml): it deletes the preinstalled toolchains no job uses (Android, .NET, Haskell, Boost, Swift, PowerShell, Chromium, the Azure CLI, and CodeQL, Go, Python, PyPy, Ruby and Node from the tool cache; Java stays, and the scripts use the system `python3`) and the preinstalled Docker images, and prints the free space before and after to the log and the job summary. Legs ran out of disk on 2026-11-04 with only part of that removed. ### Which cases a pull request runs A pull request's e2e run used to run every case: 2 h 17 min from start to end and 339 job-minutes with free runners (run 41774860458, 2026-11-04), while most cases cannot be affected by a given change. [`scripts/ci/e2e-scope.sh`](https://github.com/GeiserX/OwnGenome/blob/main/scripts/ci/e2e-scope.sh) now picks the cases from the files the pull request changed since it left main (`git diff BASE...HEAD`, three dots): - **None** when no changed path is in the e2e scope: `scripts/`, `modules/`, `workflows/`, `bin/`, `conf/`, `assets/`, `tests/e2e/`, `tests/fixtures/VERSION`, `main.nf`, `nextflow.config`, `nextflow_schema.json`, `versions.env`, `.github/workflows/e2e.yml`, `.github/actions/`, and the committed files under `tests/fixtures/` that a case reads (each named in `SCOPE_FILES` and in that case's row; the other files there feed only the unit tests). - **Every case** when a changed path is shared wiring (`FULL` in the script): `main.nf`, `nextflow.config`, `nextflow_schema.json`, `versions.env`, `assets/steps.tsv`, `workflows/`, `conf/`, `scripts/lib/`, `scripts/run-all.sh`, steps 02 and 03 (the BAM and VCF every case starts from), the fixture (`tests/fixtures/VERSION`, `scripts/ci/build-fixture.sh`), and the e2e runner and planner (`e2e-run.sh`, `e2e-shards.sh`, `e2e-scope.sh`, `tests/e2e/lib.sh`, `tests/e2e/bin/`, `e2e.yml`) with the composite actions every leg runs (`.github/actions/`). Also when the pull request carries the label `e2e-full`: the scope job reads the labels when it runs, so add the label and re-run the workflow (or push). - **Otherwise** the smoke cases `nextflow-from-fastq-1-bash` and `nextflow-from-fastq-2-nextflow` (the bash steps and the pipeline from FASTQ, and their parity check), every case whose own file changed, every case whose row in `e2e-case-paths.tsv` or `e2e-case-times.tsv` changed, every case with a path glob that matches a changed file, and the cases those need (`needs`, followed through). The base leg runs every base case whenever any case runs, because every shard starts from what they wrote. A change to a base step other than 02 and 03 therefore runs the base cases, the smoke cases and the cases that list the step, but not a shard case that only reads that step's output: the full run on main after the merge is what tests those. The job summary of `e2e-scope` lists every picked case with the reason it was picked. [`scripts/ci/e2e-case-paths.tsv`](https://github.com/GeiserX/OwnGenome/blob/main/scripts/ci/e2e-case-paths.tsv) holds, for each case, the files it runs: the step scripts, bin helpers, modules (`modules/local//*`) and assets it calls, and what those call in turn. The globs are matched as in bash `[[ PATH == GLOB ]]`, so `*` matches slashes too. A base case's row is `-`. The table was first written from the text of each case, followed through the scripts, helpers and modules it names (comments left out). `e2e-scope.sh check` runs first in `e2e-scope` and fails when a case has no row or two, a row names no case, a base row lists paths, a glob matches no tracked file, or a glob matches a file outside the e2e scope (a change to that file would start no run, so the case would never see it); then `e2e-scope` is red and so is `e2e`. A glob too many costs runner time; a glob too few lets a break through to main, where the full run after the merge finds it. The images the cases use are cached in four GitHub Actions cache entries, one per group. `scripts/ci/e2e-run.sh --image-groups` names each image's group, taken from its repository name without the tag, so a version bump changes the key of one group only; a group whose key missed restores the newest older entry of that group and pulls only what changed. A group keeps its smallest images up to 1500 MB of zstd output, and the larger ones are pulled on every run. Four groups of 1500 MB and the six data entries below stay under the repository's 10 GB cache limit even while main replaces one group: 5.9 GB of images, 1.5 GB more for the group being replaced, and about 2.4 GB of data. Images pinned by digest are not cached, because `docker load` gives them back without the digest, and they are left out of the group's key, so a digest bump misses no entry. Only the base leg saves, and only on main (or in a run by hand with `save_images`, on any branch), after it pulled every image the cases use and got through its cases, and only the groups whose key missed: a cache saved by a pull request can be read by that pull request alone and would only push main's entries out. The job summary lists every image with its size and what the cache did with it: restored on an exact key hit, restored from an older entry, saved, or pulled. Every merge in the e2e scope starts a run on main, so the cache follows main's images. GitHub drops an entry no run has read for 7 days, so after a quiet week the next run on main fills the cache again. Until this layout the cache was one entry, and by October 2026 it had grown to about 11 GB, over the limit, so every run missed it. Six downloads are cached the same way. Two come from EBI's 1000 Genomes server, which is slow and at times stalls: MoChA's chr20 panel with its manifest (0.64 GB, case `mocha-1`), keyed on `MOCHA_PANEL_MANIFEST_SHA256` and the chromosome, and the 1000 Genomes phase 3 sites VCF (1.36 GB, case `archaic-ancestry-1`), keyed on `ARCHAIC_1KG_SITES_MD5`. On a miss, each of those two is tried first at `E2E_DATA_MIRROR`, the GitHub release `e2e-data-1` that holds copies of the pinned files, and then at EBI; the pin checks both. The release also holds the four PGS Catalog score files, the HAPNEST panel and the table builders' downloads below, so an EBI outage costs none of them on a cold cache. The one builder download it lacks is the TRExplorer catalog of `build-sv-rule-regions.sh`, which is not on EBI: on a miss it comes from its own GitHub release. Two are small and failed whole runs on 2026-11-04, because their case is in the base leg or a shard that cannot start without it: `hla.dat` of the two IPD-IMGT/HLA releases `bash-step-08-hla` installs, from raw.githubusercontent.com (which served it under 1 KB/s), with GENCODE's gene and exon files from EBI's FTP server (curl exit 7 and 28), and the four real PGS Catalog score files of `prs-scores-and-run-options-1-strand-sex`, from the same server. `scripts/ci/e2e-run.sh --data-key` names their keys: the releases and a hash of the code in `scripts/lib/common.sh` that installs them, and a hash of the score ids and their server. `scripts/ci/e2e-run.sh` hands each case its files just before it runs, at the paths `setup.sh` and `install_data_file` read them from, so they download nothing; the score files the case copies from the cache itself. On a miss it downloads them first, checked against their pins or the checksum files the servers publish beside them (`MD5SUMS`, `md5checksum.txt`, the `.md5` of each score), trying up to 6 times 30 s apart. The four score files are pinned by md5 in `PGS_MD5` of `e2e-run.sh` (the md5 the PGS Catalog served beside each on 2026-11-06), tried on the mirror first and then at EBI; a restored or downloaded score must match its pin as well as its `.md5`. A file gets at most `E2E_MIRROR_BUDGET` seconds on the mirror (900, or half the fill's budget when that is less), so a mirror that trickles leaves EBI its turn. One fill may take at most `E2E_DATA_BUDGET` seconds (2400 in `e2e.yml`): a source that trickles above `fetch`'s stall limit would otherwise run on to the leg's 240-minute limit, as EBI's 1000 Genomes server did twice on 2026-11-05. A fill still running then is stopped and its case fails as `FAIL (data fill over 2400s)` without running, since its own download would be as slow. A restored entry is checked again before use (`--data-check`): the IPD-IMGT/HLA and GENCODE entry against the sha256 list written when it was filled, the score files against their `.md5`; one that fails is downloaded again. `bash-step-08-hla` then runs with every HTTP and HTTPS request sent to a closed port, after a control shows that a request under that block fails at the closed port, so a warm cache cannot reach either host. Only the leg whose cases include the case restores and saves its entry, and it saves only on main or in a run by hand with `save_images`, as with the images. The job summary says whether each key hit, and the case's log says where its files came from. The fifth is the PGS Catalog's synthetic HAPNEST panel (0.27 GB, cases `prs-2-ancestry` and `prs-scores-and-run-options-2-panel-chrx`) from EBI's FTP server, which failed cases with curl exit 7 on 2026-11-04. It is keyed on the sha256 `tests/e2e/lib.sh` pins (`HAPNEST_SHA256`). The cases handle it themselves (`setup_ancestry_panel` in `lib.sh`): with the panel in the cache, or on a miss from `E2E_DATA_MIRROR`, they check the sha256 and run `setup.sh --ancestry-panel` with every HTTP and HTTPS request sent to a closed port, so an EBI outage cannot fail them; when neither has it, `setup.sh` downloads the panel from EBI as a user's run does. The first case puts a panel that did not come from the cache in the cache for the save step. A red control in `prs-2-ancestry` runs `setup.sh` with the network blocked and no panel, which must fail. The sixth, entry `catalogs`, holds the pinned downloads of `scripts/ci/build-callability.sh`, `scripts/ci/build-cn-intervals.py` and `scripts/ci/build-sv-rule-regions.sh`, whose cases `callability-1-catalog`, `cn-intervals-1-catalog` and `followups-ci-3-sv-rule-regions` rebuild `assets/callability_genes.tsv`, the interval rows of `bin/cn_loci.tsv` and `assets/sv_rule_regions.tsv` and compare them with the committed files. Each builder lists its downloads with `--list`; the key is a hash of those lists, so a new pin is a new key. GENCODE's GTF among them comes from EBI's FTP server, which refused every connection for minutes on 2026-11-06 and failed `callability-1-catalog` on main (run 42259432681). The runner fills the entry as it does the others, mirror first and every file checked against its sha256, and each case points its builder's cache at its folder there; a builder that still has to download tries for about five minutes, 30 s apart. Every download, there and in `setup.sh`, gives up on a transfer that stalls (`FETCH_STALL`, see `fetch` in `scripts/lib/common.sh`), and the next try resumes it. How long it took in one job, before the split: a passing e2e job took about 105 to 205 minutes with 84 cases, and 230 minutes of cases with 130 (run 41390434697, 2026-11-02). The runner decides much of it, not the cache: the same CPU model took 106 minutes in one run and 166 in another. In October 2026, with 84 cases, the same code took 106 and 166 minutes on an AMD EPYC 9V45 (runs 39197502832 and 39075176392, the second of which also saved the cache), 112 minutes on an Intel Xeon 6973P (run 39088086725), 156 and 191 minutes on an AMD EPYC 7763 (runs 39075764278 and 39075201465) and 205 minutes on an AMD EPYC 9V74 (run 39197512594, which saved three groups). The image cache saves about 7 minutes of that: pulling every image took 9 minutes, restoring and loading the four groups took 2.5 (run 39075201465 restored 39 images in 118 seconds), and saving them all took 12. Each leg has a limit of 240 minutes (355 for a traced shard leg); the base leg waits up to 75 minutes for a fixture a pull request bumps. Each leg's first steps print the runner's CPU model and CPU count to the log and the job summary, so a later timing can name the hardware. Before the first case of a full run, and of every leg, `e2e-run.sh` pulls every image the cases use that is not on the machine yet: the `NAME_IMAGE` lines of `versions.env` and of the case files, less a list of images no case runs. A pull that fails with a registry 5xx is tried again, up to four tries in all, 20, 40 and 80 seconds apart. Any other error, such as a missing tag or a rate limit, fails the job at once, and the tools' own `docker run` is never retried. A Docker Hub 500 on one image's manifest is what failed main's run 37980248545 the first time. `scripts/ci/e2e-run.sh --self-test-pull` checks the retry against a fake docker, and `e2e-plan` runs it. A run given a pattern skips the pre-pull, because most images reach a case through the steps it runs and cannot be picked per case; its cases pull their own images, with one try each. The five largest images (DeepVariant, PharmCAT, GATK, MEGAnE and VNtyper, 1.9 to 6.0 GB each to download, none of them in the image cache) are left out of the pre-pull: the docker shim `tests/e2e/bin/docker` pulls one, with the same retry, when a case first runs it, and removes it when that case ends. A case that runs several of them holds them all until it ends: `nextflow-from-fastq-2-nextflow` runs DeepVariant and PharmCAT in one Nextflow run, so both sit on disk together. The image cache keys start with `e2e-images-ovl-`: archives saved from containerd's store did not all load into overlay2. Each leg switches Docker to the classic overlay2 image store before it restores or pulls an image. Docker 29 on the ubuntu-26.04 runner image uses containerd's store, which keeps every layer twice (the compressed blob and the unpacked copy): DeepVariant's, GATK's and VNtyper's images took 24.8 GB there and 17.6 GB on overlay2, and the legs that landed on a 72 GB runner (48 GB free) ran out of disk in the pre-pull (run 41938436657). Before its first case `e2e-run.sh` fails the leg when less than `E2E_MIN_FREE_GB` is free under the work area (10 GB in `e2e.yml`), and after the cases it prints the least free space it saw while they ran. Some hosted runners have a 72 GB root disk (about 50 GB free after the clean-up) and a second, blank NVMe disk (220.7 GB on the one we measured); the 145 GB runners have none. Each leg formats such a disk and mounts it at `/mnt` before anything else, and then Docker's data root (`/mnt/docker`) and, on the base leg, the work area go there. A shard's work area follows the base leg's, so on a shard only the images move. A 72 GB runner without a spare disk still has only about 16 GB free before a shard's cases, which the heaviest legs do not fit (they used 26 to 33 GB); the `E2E_MIN_FREE_GB` gate stops only the legs below 10 GB. What it covers: - **Bash steps, in order:** `validate-setup.sh`, 02 (alignment from FASTQ), 03 (DeepVariant), 03a (GATK HaplotypeCaller on the chr20 slice), 06, 07 (PharmCAT), 11, 12, 16, 16b, 20 (Mutect2 on chrM), 21 (Cyrius, on `HG002_cyrius.bam`), 32 (pypgx with its bundle), 27, then 30, 23 and 31 on the VEP subset with the synthetic score file, then 24 and `generate-report.sh`. - **Nextflow:** `nextflow run main.nf -profile docker` with real containers on the VCF and BAM the bash steps produced, through today's VCF+BAM samplesheet, with `--tools clinvar,mosdepth,delly,vcfanno,roh,pharmcat,cpic`, `--max_cpus 4 --max_memory 14.GB`. - **Parity and the launcher:** `nextflow-from-fastq-*.sh` run the bash steps 01b, 02, 03, 06, 07, 11 and 25 and the pipeline on the same FASTQ pair, and [`scripts/ci/parity-diff.sh`](https://github.com/GeiserX/OwnGenome/blob/main/scripts/ci/parity-diff.sh) compares what they wrote ([Bash vs Nextflow parity](nextflow.md#bash-vs-nextflow-parity)). `run-all-launcher.sh` runs `scripts/run-all.sh` from the same reads in a `GENOME_DIR` of its own: it must publish the same files as the direct `nextflow run`, pass the same parity check against the bash steps, write both reports, start a second run from the BAM, VCF and gVCF the first one published with no FASTP, minimap2 or DeepVariant task, and on a third run take every task from the cache in under ten minutes. Each comparison has a negative control in the same run. - **Manta and the Nextflow hardening** (`nextflow-hardening-*.sh`): step 04 on a planted Manta-style VCF with two inversion breakend pairs (they must come out as two `SVTYPE=INV` records) and on the fixture BAM inside the fixture's regions (`MANTA_CALL_REGIONS`); then a Nextflow run with `--tools clinvar,pharmcat,mosdepth,delly,manta,vcfanno,pypgx,telomere_hunter,mito_haplogroup,survivor_merge,y_haplogroup`, which checks that every container ran with `--network none` and that no task built a `.fai` of the reference. The samplesheet declares HG002 male with `--sex_check warn` (indexcov reads the slices as female), so Yleaf runs; the SURVIVOR consensus of Manta and Delly must have records with `SUPP` 2, and Yleaf's table one row with markers read. The bash steps 22 and 37 then run on the same calls and BAM in a `GENOME_DIR` of hard links with a male indexcov `.ped`, and the workflow compares `sv_merged/` and `y_haplogroup/` with `PARITY_ITEMS="sv_merged y_haplogroup" scripts/ci/parity-diff.sh`, after its self-test on those two items. - **Fine-grained ancestry** (`ancestry-fine-grained-1.sh`): `setup.sh --ancestry-fine-panel` reads a reduced panel from gnomAD's bucket (the fixture's regions plus chr22:20-35 Mb, two people held out), then step 26b runs on HG002's gVCF, on HG002's GIAB genotypes at the panel's SNVs and on a planted half-Yoruba, half-Han-Chinese person, and the Nextflow module runs on the same genotypes. The painting checks have their negative control inside the case (the two sides of the planted junction swapped), and a red control refits the panel with every reference label moved to the next group and requires the fraction checks to fail on it. - **Raw-data export and its round trip** (`consumer-export-1.sh`): step 40 writes the 23andMe and AncestryDNA files from HG002's gVCF with the CI slice of the site table (`tests/fixtures/consumer_export/sites.tsv.gz`), and both pass the strict reader. Every site's genotype must match a second, independent reading of the gVCF, and a planted hole in the gVCF must turn that site into `--` and give no site another genotype. `scripts/chip-to-vcf.sh` then reads both files back and lifts them to GRCh38 with UCSC's hg19ToHg38 chain: the records must land on the table's GRCh38 positions with the alleles of case 21's calls. Red controls shift the table by one position on GRCh38 (the export stops) and on GRCh37 (the round trip fails). The Nextflow module must write the same files as step 40. - **Index times** (`run-all-sample-qc-and-stale-index-1-index-times.sh`): every index the direct `nextflow run` and the launcher published is not older than its data file, and samtools and bcftools print no stale-index warning on the BAM and the VCF. Then a short VCF-only run with three planted pairs checks `main.nf`'s completion handler: it sets the index times of the run's own files after the last copy, and leaves an index older than the run as it was. A copied pair with its index set an hour older is the negative control. - **The Nextflow report's tables** (`paralogs-report-section-1-nf-report-tables.sh`): `HTML_REPORT` runs for real on the CYP2D6 depth check, the cDNA read check and the phased compound-het pairs that earlier cases wrote, and its reports must show the depth-check line, the cDNA column with its flagged count and the pairs by phase; the same run without the three tables must show none of them. A stub run of every tool then checks that `main.nf` passes the cDNA table to each `HTML_REPORT` task that got ClinVar hits, against a copy of the pipeline with that join removed. - **Locityper, step 53** (`ci-e2e-2-locityper.sh`): `setup.sh --locityper-data` builds the database for the HLA-DQB1 row of `assets/locityper_loci.bed` from a chr6 and chr20 reference cut from the fixture's, and `scripts/53-locityper.sh` genotypes the fixture's reads aligned to it, with chr20 as the background region. The fixture has no reads over FCGR or KIR, and HLA-DRB1 is left out for time: with it the case took 59 minutes, 35 of them jellyfish's k-mer count and 23 `locityper augment`. With HLA-DQB1 alone `augment` takes about a minute. Jellyfish does not depend on the loci: with its hash fixed at 3G entries it took 36 minutes (2,289 s for the case, fork run 42325306077), and with the hash sized to the reference it takes 36 seconds (142 s for the case, fork run 42421585308). The case keeps a limit of 90 minutes (`# case-timeout: 5400`; every other case keeps 60) as room for a slow runner. The locus must come out called, with two panel haplotypes, a quality, reads and the authors' filter, in the columns `bin/locityper_report.py` writes. Red controls: the step stops without the database, without the BAM's index, and with a database whose `DONE` names another reference. - **The SV rule regions** (`followups-ci-3-sv-rule-regions.sh`): `scripts/ci/build-sv-rule-regions.sh --check` rebuilds `assets/sv_rule_regions.tsv` from its four pinned downloads and must find the committed file byte for byte, with its row counts (2,409 coding tandem repeats). Red control: `--check` on a copy with the MHC row's end moved by one must exit 1 and show that row. The downloads come from the `catalogs` data cache entry. - **The 1.0 output contract** (`output-contract-1-hg002.sh`): every stable file of [`assets/output_contract.tsv`](https://github.com/GeiserX/OwnGenome/blob/main/assets/output_contract.tsv) whose `e2e` column names a leg is in what the base cases' bash steps and case `60-nextflow` wrote, under its contracted name, with its required columns, VCF fields and JSON keys. Its red control drops a column and renames a file on a copy and must fail on both. `tests/test_output_contract.py` checks the same contract against the code with no run ([What 1.0 keeps stable](output-contract.md)). - **Report assets:** a last case lists every external `http(s)` address in a `src=` or `href=` attribute of the generated HTML reports, so the remote files a report loads when opened are known. The list goes to the job summary. A case that exits 0 must have run at least one check: `finish` in `lib.sh` fails a case that ran none, and `e2e-run.sh` fails one whose log has no `[PASS]` line (`FAIL (no check ran)`), so a case that loses its checks to an early `finish` cannot stay green. A case with nothing to do on a run sets `NO_CHECKS_REASON` before `finish`, which prints the reason and adds it to the job summary; `prs-3-panel-measure` does that on every run that does not measure the panel. `tests/test_e2e_finish.sh` checks `finish` on planted cases. Each check is a count or a column: a VCF with records, the sample name in the VCF header, a gene in the ClinVar hit, at least one called gene in PharmCAT's `report.json`, a CYP2D6 row from pypgx, the planted ClinVar row in the HTML report with its gene, its significance and no empty cell. An exit code alone never passes a case. Many steps ask Docker for `--cpus 8`, which Docker refuses on a 4-CPU machine. The job puts a small shim first on `PATH` ([`tests/e2e/bin/docker`](https://github.com/GeiserX/OwnGenome/blob/main/tests/e2e/bin/docker)) that lowers `--cpus` to the CPUs the machine has and passes everything else through. ### Adding a case A case is one executable file in `tests/e2e/`. It sources `lib.sh`, runs one step, checks its output and ends with `finish`: ```bash #!/usr/bin/env bash # What this case shows, in one or two lines. . "$(dirname "$0")/lib.sh" run_step 11-roh-analysis.sh "$SAMPLE" check_step_exit 11-roh-analysis.sh check_ge "per-site (ST) lines in the ROH output" \ "$(grep -c '^ST' "${GENOME_DIR}/${SAMPLE}/vcf/${SAMPLE}_roh.txt" || true)" 100 finish ``` Cases run in name order in the C locale: the numbered base cases first, then files named `-.sh`, which can read everything the base cases wrote. Add your own file rather than editing the runner, the workflow or another package's case. Every case needs a row in `scripts/ci/e2e-case-times.tsv` with its stage, its measured seconds and its needs, or `tests/test_e2e_shards.sh` is red. A case that reads what another case outside the base stage wrote must name it in the `needs` column of that row, or the split may put the two in different shards. Every case also needs a row in `scripts/ci/e2e-case-paths.tsv` with the globs of the files it runs (the step script, the module, the bin helpers and assets they call), or `e2e-scope` is red; a pull request runs the case when one of those files changes. `lib.sh` has the helpers: `check`, `check_eq`, `check_ge`, `has`, `lacks`, `bcf` and `sam` (bcftools and samtools from the pinned images, paths relative to `GENOME_DIR`), `vcf_count`, `planted`, `sample_side_hits` and `pharmcat_called`. A new check has to be seen failing once before it is trusted: run it against the code before your fix, or against a deliberately broken input, and link that red run in the pull request. ### What no e2e case runs This is every step script that no e2e case runs, plus the full-size index build of step 02a. The first eight rows need data, hardware or a binary an e2e job does not have. The three long-read rows need long reads, and the e2e fixture has short reads only. The others could run, but no case runs them yet. The last column says what checks each one instead. An image's smoke test is a row of `tests/smoke/commands.tsv` that `container-test.yml` runs on the fixture when the image changes and once a month; it runs the tool, not the step's script. | Step | Why no e2e case runs it | What checks it instead | |---|---|---| | 26b (fine-grained ancestry) on the full panel | the install reads about 19 GB from gnomAD's bucket | case `ancestry-fine-grained-1` runs it on a reduced panel of the fixture's regions plus chr22:20-35 Mb | | 43 (archaic ancestry) on the whole genome | `setup.sh --archaic` reads 150 GB of archaic genomes | case `archaic-ancestry-1` runs it on the whole of chr22; the `archaic-hg002` jobs of `public-evidence.yml` run setup and step 43 on HG002's 22 autosomes, by hand | | 44 (ancient populations) on the full AADR panel | the install reads the 7.1 GB AADR release and runs smartpca on it for an hour | case `ancient-similarity-1` runs it on a planted panel; the `ancient-footprint` job of `public-evidence.yml` runs setup and step 44 on HG002, by hand | | 13 (VEP, offline) | the cache is 26 GB; the fixture's VEP subset was annotated once with `--database` instead | the image's smoke test, which annotates 50 records with `--database` | | 17 (CPSR) | the PCGR bundle (about 8 GB) plus a second VEP cache. CI never runs real CPSR | the image's smoke test, and the fake-docker cases for the command line | | 05 (AnnotSV) | the annotation data is 5.3 GB | the image's smoke test; the row that annotates the fixture's SV records with the full data runs monthly and on dispatch only | | 04b (GRIDSS) | needs a 31 GB Java heap | the image's smoke test | | 02a on GRCh38 (BWA-MEM2 index build) | about 90 GB of RAM | case `alignment-markdup-gvcf-sex-4-bwamem2` runs step 02a on an 8.4 Mb reference cut from the fixture's regions | | 01 (ORA to FASTQ) | needs Illumina's `orad` binary, which runs outside Docker | nothing | | 02b (long-read alignment) | long reads; the e2e fixture has none | the minimap2 image's smoke test, which maps synthetic HiFi reads with `-x map-hifi`; nothing runs the script | | 03e (Clair3) | long reads; the e2e fixture has none | the image's smoke test, with a recall check on synthetic HiFi reads and a check that the two model directories the script names exist; the fake-docker suite runs the script | | 04c (Sniffles2) | long reads; the e2e fixture has none | the image's smoke test on the synthetic HiFi alignment; `tests/fake-docker/bash-step-partial-outputs.sh` runs the script | | 15 (duphold) | the bash step: no case yet | case `nf-wiring-2-sv-str-tools` runs the Nextflow `DUPHOLD` and `DUPHOLD_FILTER` with the real image on the fixture's Manta calls; the image's smoke test on a planted deletion; `tests/fake-docker/bash-step-sv.sh` runs the script | | 18 (CNVpytor) | the bash step: no case yet | case `nf-wiring-2-sv-str-tools` runs the Nextflow `CNVPYTOR` and `CNVPYTOR_VCF` with the real image and the pinned GC and mask files; the image's smoke test; `tests/fake-docker/followups-ci-2-steps-03c-18.sh` runs the script (resource checks, mounts, commands); the SV merge case reads a planted CNVpytor VCF, not one CNVpytor wrote | | 29 (somatic Mutect2) | no case yet | `tests/fake-docker/bash-step-somatic-mito.sh` checks the Mutect2 command line | | 03c (Strelka2) | no case yet | the image's smoke test, with a recall check against the truth slice; `tests/fake-docker/followups-ci-2-steps-03c-18.sh` runs the script (configure, run, resume, rerun) | | 03d (Octopus) | no case yet | the image's smoke test, with a recall check against the truth slice; case `bash-step-bash42` runs the script under bash 4.2 against the fake docker | | 04a (TIDDIT) | no case yet | the image's smoke test; settle-doubts question 4 runs the script with only a BWA-MEM2 index; case `bash-step-bash42` and `tests/fake-docker/bash-step-sv.sh` run it against the fake docker | | chip-to-vcf (array data to VCF) on a vendor download | case `consumer-export-1` runs it only on the 23andMe and AncestryDNA files step 40 wrote | `tests/test_chip_to_vcf.sh` in the unit-tests job runs the script with the real bcftools and Picard images on two ten-row vendor files | Case `nf-wiring-2-sv-str-tools` runs the Nextflow `EXPANSION_HUNTER` module with the real image on the fixture's BAM (the bundled catalog cut to the fixture's contigs); the bash step 09 runs in case `bash-step-09-str`. ## The settle-doubts job [`scripts/ci/settle-doubts.sh`](https://github.com/GeiserX/OwnGenome/blob/main/scripts/ci/settle-doubts.sh) answers questions a review could not settle by reading the code: whether GATK accepts a BAM without a read group, which Clair3 models the image ships, what TIDDIT does with only a BWA-MEM2 index, whether fastp trimming changes DeepVariant's accuracy against the GIAB truth, and so on. It prints one table row per question to the job summary, with the command and what was observed. It is not a gate. Run it by hand with `job: settle-doubts`; it also runs on a pull request, and on main after a merge, that changes the script. ## The public-evidence workflow `.github/workflows/public-evidence.yml` runs two opt-in steps whole, on public data only, and writes what they measured to the job summary. It is not a gate; it runs by hand only (`workflow_dispatch`). Two inputs pick what runs: `chromosomes` (default `all`) names the autosomes for the archaic jobs, for example `22` for one chromosome job (plus the `empty` red control) and that chromosome's three archaic files instead of 22 chromosome jobs and 150 GB, and `ancient` (default on) runs the ancient-footprint job: ```bash gh workflow run public-evidence.yml -f chromosomes=22 -f ancient=false ``` - `ancient-footprint`: `setup.sh --ancient-panel` on the full AADR release, then step 44 on a gVCF built from GIAB's HG002 v4.2.1 genotypes, with the peak memory of each step's containers (one maximum per step, not per container), the peak disk growth and the time. It fails if a command fails, if the sampler saw no container memory, or if step 44 used no site. What it measured is in [step 44, Runtime](44-ancient.md#runtime). - `archaic-hg002`: `setup.sh --archaic` and step 43 on HG002, one job per autosome; `archaic-share` joins them into one report per population. When all 22 autosomes ran, the summary says whether HG002's Neanderthal total lies inside the 49.9 to 82.9 Mb that Chen et al.'s released per-person calls give Europeans with the same method and cutoffs; it prints the comparison and does not gate on it. A job fails if a command fails or step 43 analysed no site, and the job `empty` is its red control: an empty region list must make setup fail. What it measured is in [step 43, Runtime](43-archaic.md#runtime). ## The prs-memory job It measures the memory of the two PRS tasks that read only the scoring files: `PRS_PREPARE` and pgsc_calc's `FORMAT_SCOREFILES` on the default score list (`tests/test_prs_memory.sh --measure`). One leg runs them under the memory `conf/base.config` gives them and must pass; the other runs them under what the labels gave before and must be killed. It runs only by hand (`job: prs-memory`) and is not a gate. What it measured is in [PRS: Memory](25-prs.md#memory). ## Running the e2e job yourself The e2e run needs Linux, Docker, Nextflow at `NEXTFLOW_VERSION` in `versions.env` with Java 17, the `gh` CLI and about 25 GB of disk, and all 130 cases took 230 minutes one after another on a 4-CPU GitHub runner (2026-11-02). Run it on a machine you do not need for anything else: ```bash E2E_WORK=/path/with/space scripts/ci/e2e-run.sh # every case E2E_WORK=/path/with/space scripts/ci/e2e-run.sh '3*' # only cases whose file name matches scripts/ci/e2e-shards.sh plan 6 # the split CI would run ``` A partial run is for debugging: later cases read what earlier cases wrote. ## Retaking the report pictures The pictures of the HTML report (step 24), the CPIC report (step 27) and the MultiQC report (step 28) in these docs show DEMO-001, an invented sample. [`tests/demo/retake-screenshots.sh`](https://github.com/GeiserX/OwnGenome/blob/main/tests/demo/retake-screenshots.sh) makes them again after a change to a report: ```bash DEMO_TERMS_FILE=~/private-terms.txt tests/demo/retake-screenshots.sh /path/to/work ``` It writes DEMO-001 from a fixed seed (`tests/demo/make_demo_sample.py`; no real sample is read), runs steps 36, 27, 24 and 28 on it with the pinned images, and stops if a section of the report has no value, so no card says "Not run". It then draws the three pictures into `/path/to/work/pictures` with a pinned Chromium and Pillow, and checks them: only image data in each PNG, and on macOS the OCR finds DEMO-001 and none of the terms in `DEMO_TERMS_FILE` (your names and host names, one regular expression per line; keep that file outside the repository). The check runs a self-test first, on planted bad pictures. Docker must see the work folder and the checkout: with Colima, keep both under the home folder or `/private/tmp`. The **Demo pictures** workflow ([`.github/workflows/demo-pictures.yml`](https://github.com/GeiserX/OwnGenome/blob/main/.github/workflows/demo-pictures.yml)) retakes them on `ubuntu-latest`, by hand or on a pull request that changes `tests/demo/` or the workflow. It sets `PLAYWRIGHT_WITH_DEPS=1`, so Playwright installs Chromium's system libraries itself, and returns the pictures as the `demo-pictures` artifact. On Linux the check reads the PNG chunks only, because the OCR half needs macOS: run `tests/demo/check_pictures.py --ocr` on a Mac before you commit pictures from the artifact. The workflow writes nothing to the repository. Look at the pictures, copy them into `docs/images/`, and update the alt text where the cards changed. A new report card that the generator does not feed yet stops the script with its name; add its input file to `make_demo_sample.py`. ## Telling a flaky test from a broken one The **Repeat a test** workflow (`.github/workflows/repeat-test.yml`) runs one `tests/test_*.sh` or `tests/test_*.py` many times, each run in a fresh job on its own `ubuntu-latest` runner, six at a time. Start it from the Actions tab with the test's path and the number of runs (1 to 60, default 30); it also runs, with its defaults (`tests/test_annotsv_benign_mane.sh`, 30 runs), on a pull request that changes the workflow. A flaky test fails a few runs out of many; a broken one fails them all. Each run records its exit code, and the summary job lists every run and fails when any failed. One more run, `control`, runs a test that always fails, and the summary fails when the control passed, so "0 of 30 failed" is never a count that could not see a failure.