# Provider invoice load and recovery test Run from the repository root with Node 22 or newer. No Docker or local database is used. The scripts refuse the production URL and never load `.env.local`. They create clearly named synthetic companies in the `erp-base` staging branch (`metjnjrhvujscngnpzdv`). Fixtures remain there for inspection. ## Worker results recorded on 2026-09-19 Both full worker runs passed against staging over HTTP. The providers returned synthetic responses; the worker, provider clients, local throttling, mappers, encryption and database RPCs were real. The two volume runs ran concurrently. | Provider | Invoices | Invoice lines | Total duration | Longest returned worker run | Worker budget | | --- | ---: | ---: | ---: | ---: | ---: | | Visma | 25,000 | 75,000 | 14 min 3 sec | 50.183 sec | 60 sec | | Bokio | 25,000 | 75,000 | 14 min 14 sec | 50.178 sec | 60 sec | Both jobs completed with zero failed or pending records, correct totals, 250 customers and exactly 25,000 row-completion events. Each recovered from a real process kill after commit, a lost commit response and a simulated provider 429. A competing worker could not claim the active lease. The full import duration exceeds one invocation's budget because progress survives between invocations. Additional runs passed for each provider: 100 invoices with every detail request delayed 250 ms; 30 with two-second responses and a 15-second worker budget; and 30 with 1.7-second responses that interrupt a detail fetch at the invocation boundary. These exposed and verified a fix: exhausting only the remaining invocation time now leaves the invoice pending for automatic resumption instead of marking it for manual retry or applying provider-error backoff. The volume runs picked up this fix in subsequent invocations; the short-budget scenarios and final smoke tests ran after the fix. The full unit suite passed 24,239 tests, with 5 skipped. Changed files pass lint, and the type check reports no new errors. Exact metrics, synthetic fixture IDs and measurement limits are in [`results-2026-09-19.json`](./results-2026-09-19.json). No live provider account, hosted scheduler, deployed function runtime or large SIE ledger was exercised. The changes are in the feature worktree and are not deployed. Hosted cron recovery still needs verification after deployment. PR review follow-up: after requiring explicit encryption keys and removing name-only party adoption, another 300-invoice worker run passed for each provider with process termination, lost acknowledgement, rate limiting and exact-count checks. Each imported 900 lines and 250 customers without duplicates. Results are in [`results-2026-09-19-pr-review.json`](./results-2026-09-19-pr-review.json). The staging SQL test also verifies that same-named parties remain separate, source-ID retries remain stable and register writes cannot target journal tables. ## Results recorded on 2026-09-18 The database-only benchmark passed for both providers: | Provider | Invoices | Invoice lines | Failed / pending | Slowest commit plus replay | | --- | ---: | ---: | ---: | ---: | | Visma | 25,000 | 75,000 | 0 / 0 | 121.927 ms | | Bokio | 25,000 | 75,000 | 0 / 0 | 185.774 ms | Every ten-invoice batch was replayed. Both runs finished with 250 customers, 25,000 unique row-completion events, and exactly SEK 9,375,000 in invoice totals. Synthetic fixture IDs and assertions are recorded in [`results-2026-09-18.json`](./results-2026-09-18.json). Separate automated pagination tests passed 25,000 invoices per provider through the real provider HTTP clients and mappers, with simulated responses and throttling disabled. Worker deadline regressions passed with stalled database claims, reads, commits and releases, plus an oversized 2,001-line invoice. These earlier database-only numbers exclude worker and HTTP timings. See the 2026-09-19 results above for the subsequent worker runs. ## Worker test over HTTP Create an ignored `.env.provider-load.local` file with the staging values: ```dotenv NEXT_PUBLIC_SUPABASE_URL=https://metjnjrhvujscngnpzdv.supabase.co SUPABASE_SERVICE_ROLE_KEY= ``` A Supabase CLI `branches get staging --project-ref pwxtzglxptnnvjrpixpg -o env` file also works via `--env /path/to/file`, using its `SUPABASE_URL` and `SUPABASE_SERVICE_ROLE_KEY`/`SERVICE_ROLE_KEY` fields. Keep this file private. Start with a smoke run: ```bash node --import tsx --conditions react-server scripts/provider-migration/load-staging.ts --count 30 --budget 15000 ``` Then run 25,000 invoices per provider: ```bash node --import tsx --conditions react-server scripts/provider-migration/load-staging.ts ``` Use `--provider visma` or `--provider bokio` to run one provider. Increase volume with `--count 50000`. `--budget 60000` is the default invocation budget; use `--budget 210000` to exercise the production worker ceiling. The harness asserts that each invocation returns within its budget plus 1.5 seconds of scheduling allowance. A separate process watchdog fails a stuck run. The transport returns synthetic Visma/Bokio HTTP payloads. Provider clients, pagination, local rate limiters, mappers, encryption, consent resolution, worker code, PostgREST and database RPCs are real. Unexpected outbound hosts are rejected in worker processes. No real provider account is contacted. Each provider run tests: - 25,000 invoices with three lines each, three fiscal years, 250 shared customers, and paid/unpaid states. - Provider HTTP 429 followed by successful retry. - An actual process kill immediately after a database commit, then a new worker. - A lost commit acknowledgement and safe replay. - A competing claim while a worker holds the lease. - Invoice and line counts, exact header totals, completed receipts, and bounded request sizes and worker runtimes, with one row-completion event per invoice. The 429 is injected once per provider run, including when discovery spans several worker invocations. Repeating it on every restart could prevent a short-budget test from ever reaching its first commit. Only the synthetic job's lease/backoff timestamps are accelerated after injected failures, so recovery does not wait five minutes. The same claim/fencing RPCs still control ownership. The SIE prerequisite is a synthetic completed intake record, with no ledger entries. This test does not validate SIE import, matching to a large existing ledger, or real provider payload fidelity. The default list payloads contain complete invoice lines. To include detail requests and latency: ```bash node --import tsx --conditions react-server scripts/provider-migration/load-staging.ts --count 25000 --detail-every 10 --delay-ms 100 ``` For a Visma-style register requiring detail for every invoice, use `--provider visma --detail-every 1`. Real client throttling stays enabled, so 25,000 details at ten requests per second require at least about 42 minutes, before database work. Total job duration can exceed an invocation limit safely; each invocation must checkpoint and return. The simulated delay does not model all provider outages or distributed Redis rate-limit contention. To interrupt detail requests at an invocation boundary and verify automatic resumption with a small dataset: ```bash node --import tsx --conditions react-server scripts/provider-migration/load-staging.ts --count 30 --budget 15000 --detail-every 1 --delay-ms 1700 ``` An invoice that only exhausts the remaining invocation time stays pending. It does not need manual retry and does not incur provider-error backoff. An invoice that exhausts its full 15-second detail allowance still gets an explicit retry outcome, so one consistently failing invoice cannot stall all remaining invoices indefinitely. Results are written after every invocation to `.env.provider-load-report.json`. The report includes synthetic company/job IDs, elapsed time per invocation, request counts, maximum database HTTP time and process memory. No credentials are included. Use a separate `--report` path for each run you want to retain. ## Database-only benchmark through the Supabase app When HTTP credentials are unavailable: ```bash node --import tsx --conditions react-server scripts/provider-migration/write-sql-load.ts 25000 ``` This only generates `.env.provider-sql-load-plan.json`. Execute each provider's `seed`, then every `batches` statement in order, then `finish`, using the connected Supabase app's SQL tool with project ID `metjnjrhvujscngnpzdv`. Do not execute it on another project. These statements contain no schema changes. The SQL plan uses the production Visma/Bokio and invoice mappers. It persists 25,000 source receipts, calls the real commit RPC with ten invoices per batch, replays every batch, finishes follow-up receipts, and asserts invoices, lines, customers, totals and exactly one row-completion event per invoice. It reports maximum database time for a commit plus its replay. The database plan does not execute the worker, HTTP transport, encryption or provider pagination and must not be presented as an end-to-end timeout test. ## Targeted regressions and other data ```bash node node_modules/vitest/vitest.mjs run extensions/general/arcim-migration/lib/__tests__ lib/providers/__tests__ lib/providers/visma/__tests__ lib/providers/bokio/__tests__ ``` These cover stalled provider/database operations, oversized invoices, failed individual details, reconnect/account changes, credit notes, invoice numbers, VAT, payment states and fiscal-year scope. Deadline tests use a controlled clock; they are separate from measured staging invocation times. Before release, also run a scrubbed representative source payload from the original incident, a full 25,000-invoice register requiring every detail, a large existing SIE ledger with registration references, supplier invoices and settlement, foreign currencies, mixed VAT rates, missing fields, and reconnect after token expiry. These are additional integration datasets, not claims about the default 25,000-invoice fixture. Finally verify the deployed cron resumes a job with the browser closed; a CLI loop cannot prove the hosted scheduler is enabled.