# Testing managed Vitess The Vitess candidate is admitted to a release only when the same pinned runtime and controller images pass native acceptance and release verification. Tests run in `k3d-hakopod-dev` on the dedicated development VM. They do not use customer database credentials, storage or Kubernetes contexts. This page describes the test setup and what each check proves. Passing it on one VM does not establish availability across physical hosts, zones or cloud providers. ## Recorded native run All five native cases passed on October 5, 2026, using source `06a7b47baea53dcf920c7500c2f3c21eed4ae597` and the pinned runtime and operator images in [the qualification manifest](../release/managed-vitess/manifest.json). The [native acceptance record](../release/managed-vitess/native-acceptance.json) binds the test events, source files and runtime environment. | Case | Runner elapsed | | --- | ---: | | Standalone and cluster lifecycle | 999.946 seconds | | Maximum topology | 899.659 seconds | | Isolated recovery | 615.209 seconds | | Replica reseeding | 399.062 seconds | | Backup approval revocation | 288.702 seconds | The maximum topology contained eight primaries, 40 replicas, two gateways and three topology voters. Its three measured health observations took 14.967, 15.321 and 15.798 seconds. Each used 140 core, 123 grouped and 157 exec API requests, within the unchanged limits of 176 core requests, 150 grouped requests and 25 seconds. Every case preserved the 2,819-file source snapshot, restored the worker configuration and reclaimed its database namespaces and volumes. The three development workers each had 16 CPUs and 48 GiB memory and shared one physical VM. These results do not establish independent host or zone availability. Shared HTTP acceptance and package publication are separate release gates. ## Isolated native backup storage Run `scripts/setup-development-vitess-storage.py --provision-only` on the development VM. Set `HAKOPOD_DATABASE_FIXTURE_ROOT` to its acceptance scratch directory when it differs from the script's default. That directory must contain the named development kubeconfig and the reviewed `bin/kubectl` executable. The helper requires Python, boto3 and botocore. The helper creates one owned namespace with digest-pinned SeaweedFS. A ResourceQuota limits it to one CPU, 1 GiB memory, one pod and one 3 GiB volume. Connect that pod's port 8333 to a temporary HTTPS transport, then run the helper with `--https-endpoint https://`. The endpoint must have an ordinary publicly trusted certificate so native clients use their normal trust configuration. Record the provider, hostname, certificate issuer, process IDs and any disposable transport identity separately from database credentials. The transport must preserve every signed request header. Native AWS SDK requests sign `Accept-Encoding: identity`; a proxy that changes that value breaks S3 authentication. A direct Cloudflare quick tunnel changed this header during development. A temporary localhost.run tunnel preserved it but disconnected before a native test finished, so it was not retained for qualification. The current fixture uses a separate, bounded Caddy process behind Cloudflare. It restores `Accept-Encoding: identity` only for header-signed SigV4 requests whose signed-header list contains that exact field, whose method is HEAD, GET, PUT, POST or DELETE, and whose raw path belongs to one of the six fixture buckets. The matcher rejects duplicate or malformed Authorization headers and lookalike bucket paths. It leaves query-signed requests unchanged. SeaweedFS still checks the original signature and bucket permissions. Caddy listens only on loopback, disables upstream compression and returns `Cache-Control: no-store`. This fixture adjustment does not change the product's S3 client, TLS verification, public endpoint implementation or production proxy configuration. Check the public transport with the exact AWS SDK used by the candidate before starting native tests. The current probe covers binary and special-character object keys, complete multipart byte comparisons, range reads, explicit aborts, and refused path, query and checksum-header tampering. It also verifies that a warmed object URL refuses a foreign identity and an anonymous request, and that an overwrite is visible immediately. Ordinary PUT requests use `UNSIGNED-PAYLOAD`; the separate checksum test supplies a signed SHA-256 checksum and proves that changing the body is rejected. Do not describe an unsigned payload as cryptographically bound to its signature. Keep the probe source, proxy configuration, execution times, checksums and sanitized response metadata with the acceptance evidence. The observed `no-store`, uncached responses apply to that endpoint and run. Passing these checks establishes that the fixture can carry the native clients' requests; it does not qualify Vitess lifecycle or recovery. Bound the tunnel, proxy and localhost-only Kubernetes port forward, and verify that they remain active before each native case. Each test database receives a different S3 key and bucket. A key may administer only its own bucket. The helper verifies signed HTTPS head, put, get, list and delete requests, a multipart upload with a range read, and rejected reads and writes to another fixture's bucket. Every probe includes the signed identity encoding header. It saves credentials in a mode-0600 file and prints only structural results and the public fixture hostname. The native fixture config contains these separate entries: | Entry | Purpose | | --- | --- | | `standalone` | One MySQL tablet behind a gateway | | `cluster` | Two shards with one replica per shard | | `recovery-source` | Two-shard logical archive source | | `recovery-target` | Separate empty two-shard recovery target | | `reseed` | One shard with two replicas for native member recovery | | `revocation` | Storage approval removal, restoration and deletion | The scale case uses the `cluster` entry with an eight-shard test specification and five replicas per shard. Set `HAKOPOD_VITESS_NATIVE_FIXTURE_CONFIG` to the protected file path. Do not put its contents in a command, issue, log or repository. Each test run selects a fresh prefix inside its assigned bucket, so retained native backups cannot initialize a new fixture with old data. The test refuses an existing database namespace. A failed fixture can be retained for inspection with `HAKOPOD_KEEP_DATABASE_FIXTURES=1`; remove only that owned fixture before retrying. The tests accept one of two named development topologies: two or three unique nodes selected from `k3d-hakopod-dev-server-0`, `k3d-hakopod-database-worker-0` and `k3d-hakopod-database-worker-1`; or the exact dedicated triple `k3d-hakopod-vitess-worker-0`, `k3d-hakopod-vitess-worker-1` and `k3d-hakopod-vitess-worker-2`. They require `HAKOPOD_DATABASE_RECOVERY_TEST=1`, `HAKOPOD_DATABASE_VITESS_TEST=1` and `HAKOPOD_TEST_KUBECONFIG` pointing to the named development context. Coordinate their resource use with other acceptance work before starting them. ## Native checks `TestManagedVitessLive` creates standalone and two-shard databases through the same runtime path as normal provisioning. It checks binary and Unicode data, queries through every gateway, primary and replica routing, and the exact rows stored on each physical shard. It also checks wrong-host and wrong-issuer certificate failures, plaintext refusal and application SQL privileges. Cluster testing removes an owned primary process, waits for recovery, then verifies reads and writes. That is process replacement evidence; it does not test an unreachable physical host. Certificate renewal must produce a new observed leaf and issuer, preserve data and keep the previous CA working during its overlap period. The refusal and privilege checks run again after renewal. `TestManagedVitessRecoveryLive` captures a logical archive, changes the source, then restores the captured data into a separate empty target. It checks both shards, rejected corruption and truncation, existing-session failure, independent source and target writes, and nonempty-target refusal. Application ingress stays closed until both recovery and inspection are recorded. The archive records per-shard read-locked captures, not one transactionally consistent instant across all shards. `TestManagedVitessNativeReseedLive` disables automatic recovery on the exact owned vtorc process, stops one replica, and makes new writes through the healthy replica set. It creates a native backup, purges the primary's older binary logs and proves that the stopped replica is missing transactions from that purged range. It then pauses the database's namespace operator, deletes the stopped tablet and its owned disposable data claim, and verifies that Kubernetes removes the old PersistentVolume object before the operator resumes. The operator must recreate the same native tablet alias with new pod, claim and volume identities and restore it from the latest backup. Every original row and the later transaction must be present on that replacement before automatic recovery resumes, and on every member after the cluster is healthy again. `TestManagedVitessBackupRevocationLive` removes the trusted storage approval and checks that backup credentials, storage egress, native backup Jobs and their processes are gone. Remaining schedules must be suspended or deleting, and the namespace operator must be stopped. Database tablet identities and application data must remain intact. Restoring approval must restore the schedule and its owned credentials and egress. The fixture is finally deleted with approval removed, exercising ordinary garbage collection in that state. The namespace operator must remain paused after deletion is accepted so it cannot recreate children under the deleting database. Removing a Secret does not invalidate a key at its storage provider. Running tablets may retain native recovery credentials. The revocation test verifies the documented platform boundary and does not claim that all copies of a key have been destroyed. `TestManagedVitessScaleLive` creates the maximum admitted fixed topology: eight shards with five replicas per shard. It requires 48 tablets in total, comprising eight primaries and 40 replicas, behind two ready gateways and three ready topology voters. It verifies shard routing and data through the primary and replica target on both gateways. Three consecutive observations must preserve the topology fingerprint and every member UID, complete within 25 seconds, keep TLS verified with plaintext rejected, and remain within the test's bounded core and grouped Kubernetes API list requests. The recorded native run above passed this case and the other four cases against the same source and pinned images. ## Evidence and cleanup Run these tests against an isolated candidate source snapshot with the candidate image digests. Only that snapshot may lift the runtime gate for qualification. Keep the shipping-source gate closed until the release verifier has matched the successful native test events to the exact source, patches and image identities. Skipped tests and package compilation do not count as native acceptance. `scripts/run-development-vitess-acceptance.py` runs one case at a time against the `vitess-source-check` directory under the supplied scratch root. Pass the qualified runtime and operator image references, including their SHA-256 digests, with `--engine-image` and `--operator-image`. Choose `lifecycle`, `recovery`, `reseed`, `revocation` or `scale` with `--case`. Use a new `--attempt` number for each run; the helper refuses to overwrite earlier evidence. Run it inside the development VM's bounded systemd unit after coordinating the test lane. Pass the protected development cluster receipt and its exact checksum with `--cluster-receipt` and `--cluster-receipt-sha256`. Before starting Go, the runner binds the live cluster and every node UID in the selected topology's complete approved node set to that receipt, then checks the exact selected nodes used by the case. It also requires exactly the eight established, names-accepted, namespaced `planetscale.com` resource definitions. Warm the Go test cache against the exact candidate source before importing the images. Import the runtime, operator and etcd images into every selected node, including their canonical `repository@sha256:…` references. Then measure the space left. Image imports and compilation can consume several GiB even before a database starts. Do not lower the kubelet eviction or image collection thresholds to make a test fit. Pass a separate `--fixture-budget-gib` for the new data, native backups and container writes expected during the selected case. Use measured growth from a previous run, or a conservative allowance for a first run. Recovery keeps its source and target alive together, so its budget must cover both. Include any growth expected in the S3 fixture. A PVC request is not a measurement of actual space used, and local-path storage does not enforce it as a filesystem limit. The helper checks the executor and every selected node are Linux on amd64, schedulable and free of disk, memory and PID pressure. It reads nodefs and imagefs capacity, the kubelet image collection threshold and the exact cached image digests. Each filesystem must have the fixture budget available in addition to a reserve of at least 12 GiB. A larger reserve applies when required by the observed image collection threshold. The protected preflight report records these measurements even when the space or image check refuses the run. No Go test or database provisioning starts after a rejected preflight. The runner also reads allocatable CPU and the effective CPU requests of every active pod on the selected nodes. It reserves the fixed product envelope used by each test topology: 8,850m for lifecycle, 17,700m for simultaneous recovery source and target clusters, 6,550m for reseed and 5,100m for revocation. These figures include replacement and native backup surge from the production database capacity contract. The preflight records and rejects any shortfall. The scale case reserves 45,450m CPU and 97,050 MiB memory and requires its fixed 67 GiB fixture disk budget before it can start. It records the source inventory before and after the run, the image references, the log checksum and the actual Go test events. Raw output stays in a protected JSONL file. The evidence file contains structural test events without SQL or credential output. The runner limits its log to 64 MiB, gives Go 90 minutes, and stops the process group after 95 minutes. A skip, failure, incomplete event sequence, changed source inventory or exceeded bound cannot pass qualification. The source inventory includes the internal packages, their embedded scripts and data, and the template catalog used by those packages. It rejects symbolic paths and limits the inventory to 4,096 files and 128 MiB. The runner fixes the Go target, toolchain selection and module settings. User Go configuration, workspaces and inherited compiler flags cannot override the candidate. Verify the module cache with `go mod verify` before warming the candidate test cache. `release/verify-vitess-runtime.py` checks the combined native results against the source, upstream patches and binary checksums. It also pulls the published images without registry credentials and checks the binaries inside them. A local candidate run alone does not establish that a release image can be pulled or that a published image contains the tested binaries. Keep the storage pod running until the tests finish. Restarting its transport can change the public hostname; recreate the protected fixture config and its exact approved endpoint addresses before another run. Retained backups consume the fixture's bounded storage, so inspect and clean up only the owned fixture data between qualification rounds. After recording results and deleting all owned database fixtures, stop the recorded transport processes and remove their disposable identity. Then remove the recorded fixture namespace and its protected configuration and state files. Verify that the namespace and its owned volumes have been reclaimed. Do not delete a similarly named namespace without matching its recorded UID and `hakopod.io/development-fixture` label. See [the Vitess stack and limits](managed-vitess.md) and [managed database architecture](managed-database-architecture.md) for the product contract these tests are intended to qualify.