# Outage: Production relay metrics-cardinality OOM

## Incident metadata

- **Date:** 2026-09-02
- **Severity:** Critical
- **Status:** Resolved operationally
- **Resolved:** 2026-09-05T19:53:19Z
- **Component:** production token.place relay and its application metrics endpoint
- **Incident ID:** `2026-09-02-production-relay-metrics-cardinality-oom`
- **Primary incident window:** approximately 2026-09-02 16:41 PDT / 23:41 UTC through
  2026-09-02 18:08:18 PDT / 2026-09-03 01:08:18 UTC

## Summary

On September 2, 2026, the sole production token.place relay entered an approximately 87-minute
crash loop. Default Flask request instrumentation retained raw or effectively unbounded HTTP path
values as Prometheus labels. A rapid increase in distinct unmatched paths grew the application
metric set from approximately 29,000 Flask series to approximately 71,000. Collection and
serialization of that multiprocess metric set increased `/metrics` response size and latency while
relay memory approached its 256Mi hard limit, after which Kubernetes OOM-killed the process.

Multiprocess metric state used pod-lifetime temporary storage without startup cleanup. An ordinary
application-container restart could therefore retain the unsafe exposition state and allow the OOM
cycle to repeat. Because production had one relay replica, a crash could
temporarily remove the only backend and produce Traefik's browser-visible `no available server`
response.

The source repository's `main` branch already contained bounded-registry hardening, including a
dedicated relay registry, disabled default exports, normalization of unmatched routes to `other`,
and a bounded-scrape regression test. The deployed `e46277d` image did not descend from that
hardening merge and still initialized default Flask metrics. This confirmed release-line/source-
provenance mismatch left the runtime mechanism present in production.

At 2026-09-03 01:07:50 UTC, an operator paused Prometheus discovery of only the token.place target
and deleted only the affected pod. Kubernetes created a healthy replacement with a fresh
pod-level `emptyDir`; the image and Deployment were not rolled back or changed. Production serves
traffic again, but application scraping remained intentionally paused until the recovery rollout
described below. The bounded-cardinality backport has since been deployed and authenticated
scraping restored. The operator classified the incident as **Resolved operationally** at `2026-09-05T19:53:19Z`:
service and observability were restored on the corrected deployment. That operational boundary
does not assert that every original qualification or issue-closeout requirement passed. The
operator reported the combined recovery image's staging qualification complete before deployment;
this public summary does not reproduce the complete staging evidence, and independent evidence
review and the 24-hour stability requirements remain undocumented.
As of 2026-09-06, maintainers independently closed the completed-work trackers
[token.place #1765](https://github.com/futuroptimist/token.place/issues/1765#issuecomment-5560895881),
[token.place #1766](https://github.com/futuroptimist/token.place/issues/1766#issuecomment-5560900290),
[sugarkube #2774](https://github.com/futuroptimist/sugarkube/issues/2774#issuecomment-5560902169),
and [sugarkube #2775](https://github.com/futuroptimist/sugarkube/issues/2775#issuecomment-5560903530).
Those closures do not establish every broader acceptance criterion or long-duration observation.

## Impact

- Production was intermittently unavailable during an approximately 87-minute crash loop, from
  the first Kubernetes-confirmed restart/OOM evidence at approximately 23:41 UTC until the
  replacement pod became healthy at 01:08:18 UTC. This was not measured as continuous downtime.
- While the sole relay pod was unavailable, browsers could receive Traefik's
  `no available server` response.
- The Deployment and Service reported an available endpoint during some healthy intervals between
  crashes, consistent with intermittent rather than continuous impact.
- Repeated process termination created an unquantified risk of losing active API-v1 correctness
  state because authoritative relay state remained memory-backed. No customer data loss or
  in-memory work loss was proven.
- Affected-user counts, failed-request counts, revenue impact, and exact request-loss duration were
  not measured and are not estimated here.
- Severity is **Critical** because a production PagerDuty alert accompanied user-visible
  unavailability of the only relay replica.

## Detection

The incident was detected through the production PagerDuty alert and user-visible unavailability.
The exact PagerDuty trigger and acknowledgement timestamps are absent from the sanitized evidence,
so this record does not assign a precise page-fire time.

Kubernetes was the authoritative source for the OOM determination: the previous container state
reported reason `OOMKilled`, exit code `137`, and finish time `2026-09-03T00:50:09Z`. The historical
restart counter and `last_termination_oom` signal first appeared at 23:41 UTC, and the Prometheus
target was down by 23:47 UTC. The `container_oom_events` query remained zero and is not treated as
evidence that no OOM occurred.

## Timeline

All incident times are shown in PDT and UTC. Prometheus observations came from the
`2026-09-02T23:20:00Z` through `2026-09-03T01:08:00Z` historical window at 30-second resolution;
sampled maxima may be lower than instantaneous peaks.

| Time (PDT) | Time (UTC) | Event |
| --- | --- | --- |
| 2026-08-27 11:10:41 | 2026-08-27 18:10:41 | The ServiceMonitor was created. It remained generation 1 before mitigation and was already scraping every 30 seconds; it was not activated at incident onset. |
| 2026-09-02 16:20:00 | 2026-09-02 23:20:00 | Historical evidence window began. Working set was 197,107,712 bytes. |
| 2026-09-02 16:30:00 | 2026-09-02 23:30:00 | Target was up. Working set was 197,111,808 bytes and RSS was 187,510,784 bytes. Metrics included 1,679 distinct path labels, 1,659 unknown paths, 28,964 Flask series, and 28,996 total samples; scrape duration was 3.102 seconds. |
| 2026-09-02 16:39:30 | 2026-09-02 23:39:30 | Maximum sampled working set was 247,996,416 bytes: `247,996,416 / 268,435,456 × 100 = 92.385%`, or approximately 92.4% of the 256Mi limit. |
| 2026-09-02 16:40:00 | 2026-09-02 23:40:00 | Target remained up. Maximum sampled RSS was 242,376,704 bytes. Cardinality reached 2,741 distinct paths, 2,719 unknown paths, 47,018 Flask series, and 47,050 total samples. `scrape_series_added` was 2,261 and scrape duration was 6.150 seconds. |
| Approximately 2026-09-02 16:41 | Approximately 2026-09-02 23:41 | The first historical restart and `last_termination_oom` signal appeared. `scrape_series_added` reached its observed maximum of 2,822. This is the first Kubernetes-confirmed OOM/restart evidence. |
| 2026-09-02 16:46:30 | 2026-09-02 23:46:30 | Metrics reached 4,124 distinct paths, 4,093 unknown paths, and 70,529 Flask series. |
| 2026-09-02 16:47:00 | 2026-09-02 23:47:00 | Prometheus target was down and the restart counter was 2. |
| 2026-09-02 16:48:00 | 2026-09-02 23:48:00 | Observed maxima reached 4,155 distinct paths, 4,121 unknown paths, 71,056 Flask series, and 71,088 scraped samples. |
| 2026-09-02 16:55:00 | 2026-09-02 23:55:00 | Restart counter was 5. |
| 2026-09-02 17:05:00 | 2026-09-03 00:05:00 | Restart counter was 7. |
| 2026-09-02 17:50:09 | 2026-09-03 00:50:09 | Kubernetes recorded the prior container termination as `OOMKilled`, exit code 137. |
| During initial triage | During initial triage | The affected pod had reached 15 restarts. Kubernetes events included 251 BackOff events over approximately 73 minutes. A later historical query showed a maximum restart count of 16. |
| 2026-09-02 18:07:50 | 2026-09-03 01:07:50 | After verifying the exact production target, the operator changed only its discovery selector so it no longer matched, then deleted only the OOM-looping pod. No image rollback or workload change occurred. |
| 2026-09-02 18:08:18 | 2026-09-03 01:08:18 | Replacement pod started, became ready, and ended the confirmed impact window with a fresh pod-level `emptyDir`. |
| 2026-09-05 12:10:05 | 2026-09-05 19:10:05 | Observed Helm revision 7 upgrade deployed the identity-verified combined recovery image. |
| 2026-09-05 12:32:18 | 2026-09-05 19:32:18 | Production cutover verification began; request and compute lifecycles succeeded through a bounded two-minute soak. |
| 2026-09-05 12:38:04 | 2026-09-05 19:38:04 | Root and metadata blackbox probe restoration began; both targets were uniquely discovered and healthy. |
| 2026-09-05 12:47:21 | 2026-09-05 19:47:21 | The application-metrics restoration procedure began. The authenticated baseline and bounded canary checks then preceded enabling periodic scraping. |
| 2026-09-05 12:53:19 | 2026-09-05 19:53:19 | A healthy application scrape completed the initial bounded six-minute restoration soak; required longer-duration exit criteria remained open. |

## Technical root cause

### Confirmed mechanism

The confirmed root cause was unbounded application metric label cardinality, not a Prometheus
memory leak.

1. Deployed commit `e46277d` called bare `PrometheusMetrics(app)` in `api/__init__.py` and pinned
   `prometheus_flask_exporter==0.23.2` and `prometheus-client==0.21.0` in
   `config/requirements_relay.txt`. The exporter defaults (`export_defaults=True`,
   `group_by="path"`) enabled its Flask request-latency histogram with `(method,path,status)`
   identity. Each unique tuple exposed 17 series: 15 buckets plus `_count` and `_sum`.
2. Default Flask request instrumentation therefore retained request-controlled raw paths as labels.
   Almost every observed path value during the buildup was an unmatched route: 1,659 of
   1,679 at 23:30 UTC, 2,719 of 2,741 at 23:40 UTC, and 4,093 of 4,124 at 23:46:30 UTC.
3. Distinct path values rapidly multiplied the number of application series. Flask series grew
   from 28,964 at 23:30 UTC to 70,529 at 23:46:30 UTC and peaked at 71,056 at 23:48 UTC.
4. Prometheus scraped 28,996 samples at 23:30 UTC, 47,050 at 23:40 UTC, and 71,088 at the
   23:48 UTC peak. During the crash loop, exposition responses were approximately 7.79 MB and took
   approximately 6–8 seconds. Historical Prometheus scrape duration rose from 3.102 seconds at
   23:30 UTC to 6.150 seconds at 23:40 UTC and reached an observed maximum of 8.238 seconds.
5. Relay memory approached the container's 268,435,456-byte limit. The largest 30-second working
   set sample was 247,996,416 bytes (approximately 92.4%); the largest RSS sample was 242,376,704
   bytes. The samples need not capture the instantaneous allocation that crossed the hard limit.
6. Kubernetes OOM-killed the process. Its previous-state termination evidence, not the zero-valued
   `container_oom_events` query, establishes the OOM.
7. Multiprocess metric state used pod-lifetime `/tmp` without startup cleanup. Application-container
   restarts within the same pod could retain the accumulated metric database. The old files were
   not directly inventoried, and scrape pause and pod replacement occurred together, so their
   individual effects were not production A/B tested.
8. Production had one desired and available replica. Each OOM interval could therefore leave no
   serving backend.

### Causal fingerprint

The tuple counts below are separate from the distinct-path figures in the timeline:

| UTC | `(method,path,status)` tuples | `17 × tuples` | Observed Flask series | Residual |
| --- | ---: | ---: | ---: | ---: |
| 23:30:00 | 1,703 | 28,951 | 28,964 | 13 |
| 23:40:00 | 2,765 | 47,005 | 47,018 | 13 |
| 23:46:30 | 4,148 | 70,516 | 70,529 | 13 |
| 23:48:00 | 4,179 | 71,043 | 71,056 | 13 |

Every checkpoint satisfies `flask_series = 17 × request_histogram_tuples + 13`. The increase was
exactly `2,476 × 17 = 42,092` series, equal to `71,056 − 28,964`. This is the causal fingerprint:
raw-path histogram cardinality accounts for all observed Flask-series growth, rather than a
generic Prometheus-server memory leak.

### Source hardening and deployed provenance

Source commit
[`a86fa1a`](https://github.com/futuroptimist/token.place/commit/a86fa1a47220023b7a236fefcb7f78c0c4487880),
merged to `main` by
[`e0e2685`](https://github.com/futuroptimist/token.place/commit/e0e2685062a1f4714dd6a10cda7586dd1534860a),
introduced `RELAY_METRICS_REGISTRY`, set `metrics_export_defaults=False`, normalized unmatched
routes through `_normalise_http_route()` to `other`, and added
`test_metrics_scrape_uses_bounded_relay_registry`.

PR [#1447](https://github.com/futuroptimist/token.place/pull/1447), merged to `main` as `e0e2685`
on July 13, implemented a dedicated `CollectorRegistry`, `metrics_export_defaults=False`,
`metrics_path=None`, a relay-owned bounded `/metrics`, and reviewed finite labels. Its negative
test proves unmatched paths expose neither raw paths nor default `flask_http_request*` families.

The deployed commit `e46277d` came from PR #1735's `release/relay-0.1.1` base
[`dc6ac09`](https://github.com/futuroptimist/token.place/commit/dc6ac09d7963d417ab6054b97c48e314a6494eef).
Repository ancestry confirms that `e46277d` is not a descendant of hardening merge `e0e2685`, and
its `api/__init__.py` still calls default `PrometheusMetrics(app)`. Thus the bounded implementation
existed on `main` but was omitted from the deployed release lineage. This is confirmed repository
evidence; this record does not speculate about why that release line was selected or imply a
rollout near incident onset.

On the divergent release line, PR [#1729](https://github.com/futuroptimist/token.place/pull/1729)
narrowly backported four gauges and bearer authentication while leaving the default exporter
enabled. PR [#1732](https://github.com/futuroptimist/token.place/pull/1732) fixed maintenance-gauge
PID labels while explicitly preserving the default registry and route. PR
[#1735](https://github.com/futuroptimist/token.place/pull/1735) changed build-info labels and
produced deployed `e46277d`. Release validation checked required metric presence and authentication
but lacked a negative/cardinality regression proving default Flask families and raw unmatched
paths were absent.

### Trigger classification and causal confidence

Aggregate evidence confirms **automated-style unmatched-path/404 probing** as the trigger class.
From 23:30 through 23:48 UTC, every one of the 2,476 new tuples was a 404: 2,469 were `GET 404` and
7 were `POST 404`. Of 2,998 incremental requests, 2,675 (89.23%) were 404s. The 2,476 new tuples
equaled 92.56% of incremental 404 requests, demonstrating extremely low path reuse. The singleton-
path ratio rose from 34.43% to 70.81%. Tuple growth accelerated from 106.2 per minute during
23:30–23:40 to 212.8 per minute during 23:40–23:46:30.

The actor, source network, external-versus-internal origin, and intent remain unknown. This record
does not characterize the traffic as a malicious attack or attribute it to a party. The traffic
exposed a latent defect; regardless of origin, request input must not create unbounded labels.

Confidence is separated as follows:

- **Confirmed root defect:** request-controlled raw paths became unbounded metric labels.
- **Confirmed trigger class:** rapid automated-style unmatched-path/404 probing.
- **Confirmed outcome:** Kubernetes OOM-killed the relay.
- **Strongly supported final pressure mechanism:** collection and serialization of approximately
  71,088 samples into a roughly 7.79 MB, 6–8 second exposition while only approximately 19.49 MiB
  of sampled cgroup headroom remained. No heap profile captured the exact final allocation.
- **Persistence factor:** pod-lifetime multiprocess metric state lacked startup cleanup. Old files
  were not directly inventoried, and scrape pause plus pod replacement occurred together, so the
  individual effects were not production A/B tested.
- **Impact amplifier:** one replica.
- **Systemic cause:** release-line safety parity and negative-test coverage failed.

### Deployment-change attribution and PR #1726

PR [#1726](https://github.com/futuroptimist/token.place/pull/1726) was unrelated to this outage. It
merged on 2026-09-02 at 03:53:30 UTC as an internal, non-runtime-wired Valkey
scheduler/reservation/enqueue slice. Although the PR contained 65 commits, production was running
`ghcr.io/futuroptimist/tokenplace-relay:sha-e46277d`. Commit
[`e46277d`](https://github.com/futuroptimist/token.place/commit/e46277daaeb76beeb9f2a2e9e265181287239b22)
was created on August 30 and predates the #1726 merge.

That deployed commit came from PR
[#1735](https://github.com/futuroptimist/token.place/pull/1735), whose relevant change was limited
to immutable build-info labels. No production rollout occurred near incident onset; the
ReplicaSet and pod were approximately three days old. Neither #1726's commits nor a same-time
deployment are attributed as the trigger.

## Contributing factors

- Implicit default per-path Flask instrumentation accepted effectively unbounded caller-controlled
  label values in the deployed release instead of the bounded registry already present on `main`.
- The deployed release lineage omitted `main`'s normalization of unmatched/404 paths to `other`.
- Release/artifact provenance checks did not prevent deployment of a lineage missing the source-
  level metrics hardening.
- Cardinality, scrape sample count, series additions, scrape duration, memory headroom, and restart
  acceleration were not correlated early enough to prevent the OOM.
- The 256Mi hard limit left little headroom as metric state and scrape serialization grew. Raising
  it alone would delay, not fix, an unbounded-cardinality failure.
- Multiprocess metric files shared general pod-level `/tmp` storage and were not safely cleared on
  every application-container startup.
- The `emptyDir` lifetime was the pod lifetime, not the application-container lifetime.
- A single replica amplified every process crash into a possible total loss of serving capacity.
  Adding replicas is not yet a safe standalone mitigation because authoritative relay correctness
  state remains memory-backed.
- The ServiceMonitor's established 30-second scrape schedule repeatedly exercised the expensive
  exporter. It had existed since August 27 and was not a newly activated incident-time change.

## Recovery and resolution

At 2026-09-03T01:07:50Z, the operator first verified the production context, workload, exact image,
replica count, memory limit, and scrape-target configuration. The operator then:

1. paused Prometheus discovery of only the application target by changing its selector to a
   nonmatching value;
2. deleted only the affected pod so Kubernetes created a replacement with fresh pod-level
   temporary storage; and
3. did not roll back the image or change the workload.

The replacement started at `2026-09-03T01:08:18Z`, reported Ready `True`, and had zero restarts
during mitigation verification.

This emergency action removed the accumulated pod-level metric files and stopped scheduled
scrapes from rebuilding or serializing the unsafe metric set. It restored application service but
was not the permanent fix. The later corrected-image deployment and deliberate scrape restoration
completed the operational recovery described below.

## Post-recovery verification

- The replacement pod was Ready `True` with zero restarts during verification.
- Public `/livez`, `/healthz`, and `/` requests returned HTTP 200.
- Application state was healthy.
- The deployed image and Deployment were unchanged by mitigation.
- token.place application scraping remained intentionally paused.
- The verification proves recovery of serving traffic, not that the cardinality defect has been
  permanently corrected or that scraping was safe to restore at that earlier point.

## Recovery deployment and initial verification

Production Helm revision 7 was upgraded at `2026-09-05T19:10:05.335904906Z` to
`ghcr.io/futuroptimist/tokenplace-relay:sha-6c39adc`, the combined recovery image supplied by the
bounded-metrics backport in PR [#1782](https://github.com/futuroptimist/token.place/pull/1782) and
the public-information exemption backport in PR
[#1789](https://github.com/futuroptimist/token.place/pull/1789). The observed artifact identity
was:

- OCI index digest
  `sha256:543fde33aff45253630090b52d16163e3586da12c973f5c4a658ddc8927d0a68`;
- source commit `6c39adc64e7bed4f85d07164aa2860e637919ca9`;
- immutable public build ref `sha-6c39adc`; and
- public semantic version `0.1.1`.

The new pod was created at `2026-09-05T19:10:09Z` and started at
`2026-09-05T19:10:13Z`. Production had one ready replica, zero restarts, and no termination reason.
Cutover verification began at `2026-09-05T19:32:18Z`: the digest and immutable ref matched, one
compute registration and continuing successful polling were observed, and two accepted requests,
two accepted responses, and two successful retrievals completed. All public health and identity
endpoints returned HTTP 200, no HTTP 5xx occurred, and a bounded two-minute soak held relay memory
between 63,971,328 and 64,786,432 bytes against the 268,435,456-byte limit.

Public-information monitoring restoration began at `2026-09-05T19:38:04Z`. The root and
`/api/v1/meta` blackbox probes were restored to `release=kube-prometheus-stack`, alongside the
continuously active `/livez` and `/healthz` probes. Prometheus uniquely discovered both restored
targets with no last error. It recorded successful root and metadata scrapes at
`2026-09-05T19:41:51.950530221Z` and `2026-09-05T19:42:08.689609225Z`, respectively. Each target
produced three samples in the final three-minute window with `min_over_time(probe_success[3m])=1`.
There were no HTTP 429 or 5xx responses; compute polling continued, the pod stayed ready with zero
restarts, and memory remained approximately 64–65 MiB.

Application-metrics restoration began at `2026-09-05T19:47:21Z`. Before periodic scraping was
enabled, an authenticated manual scrape returned HTTP 200, 35,733 bytes, and 227 samples. Twenty
deterministic unmatched-path canaries all returned HTTP 404; the following authenticated scrape
returned 35,749 bytes and 227 samples, with no canary path and no `pid` label. Each maintenance
metric (`tokenplace_build_info`, `tokenplace_compute_nodes_healthy`,
`tokenplace_compute_nodes_registered`, and `tokenplace_instrumentation_up`) had exactly one series.

The token.place ServiceMonitor was then restored to `release=kube-prometheus-stack`. Prometheus
discovered exactly one healthy application target, with no last error, in scrape pool
`serviceMonitor/tokenplace/tokenplace/0`. Its final successful scrape was recorded at
`2026-09-05T19:53:19.538689743Z`. The final three-minute window contained six successful `up`
samples with `min_over_time(up[3m])=1`, `scrape_samples_scraped=227`,
`scrape_duration_seconds=0.011584181`, `tokenplace_instrumentation_up=1`, and zero raw canary
series. A final authenticated payload contained 35,751 bytes and 227 samples, with zero canary and
zero `pid`-label occurrences.

During the bounded six-minute restoration soak, memory stayed between 65,101,824 and 65,425,408
bytes against the 268,435,456-byte limit, compute polls increased from 222 to 260, the pod remained
ready with zero restarts, and HTTP 429 and 5xx counts both remained zero. At
`2026-09-05T19:53:19Z`, the bounded exporter was deployed, authenticated application scraping was
restored, and **no intentionally disabled monitoring functionality remained**. This is the
operator-supplied operational-resolution boundary, but it is only bounded production observation:
20 canaries and six minutes do not independently document the required thousands-path regression,
same-pod restart-cleanup verification, operator-reported staging soak, or 24-hour stability windows
below.

PR [#1782](https://github.com/futuroptimist/token.place/pull/1782) provides repository regression
coverage for bounded metrics and restart cleanup. That automated evidence is distinct from the
operator-reported completed production-equivalent staging qualification and from the bounded
production observations above; this public summary does not reproduce the complete staging
evidence, and the 20 production canaries are not represented as a thousands-path staging test.

## What went well

- Triage verified the exact context, namespace, Deployment, image, replica count, memory limit, and
  scrape target before making a narrowly scoped mitigation.
- Kubernetes termination state provided authoritative OOM evidence even though a related
  Prometheus OOM query remained zero.
- Historical 30-second metrics and sanitized aggregate application-log measurements allowed
  cardinality, scrape cost, memory pressure, and restarts to be aligned without publishing raw
  paths or sensitive payloads.
- Pausing only the affected target and replacing only the affected pod restored service without an
  image rollback or Deployment change.
- Public health checks confirmed that the replacement served traffic.

## What went poorly

- Caller-controlled unmatched paths could produce new metric series without a fixed bound.
- Metrics collection competed for memory with the relay and could serialize an approximately
  7.79 MB response every 30 seconds under the incident state.
- Container restart did not clear multiprocess metric files, making automatic recovery ineffective.
- A one-replica deployment turned process instability into intermittent production unavailability.
- Detection did not warn on the combined cardinality, scrape-duration, memory-headroom, and restart
  trend before the hard OOM limit was crossed.
- Application scraping had to be paused to recover safely, leaving production telemetry degraded.
- Evidence retention could not identify the traffic source or quantify customer request failures
  and possible active in-memory work loss.

## Corrective actions

Corrective actions are tracked in the linked issues below. Creating or closing a tracker does not
by itself change an action's implementation, deployment, or restoration status. This documentation
change closes no issues and does not satisfy broader exit criteria that remain undocumented.

| Priority | Type | Action | Rationale | Owner | Status | Tracker | Verification or exit criterion |
| --- | --- | --- | --- | --- | --- | --- | --- |
| P0 | Prevent | Replace raw-path Flask grouping with bounded route-template or endpoint labels. | Removes caller control of label cardinality. | Unassigned | Deployed; trackers closed 2026-09-06; broader evidence limits retained | [token.place #1765](https://github.com/futuroptimist/token.place/issues/1765)<br>[sugarkube #2775](https://github.com/futuroptimist/sugarkube/issues/2775) | The corrected image identity is verified and a fixed allowlist of route labels is demonstrated under adversarial traffic. |
| P0 | Prevent | Collapse every unmatched/404 route to one fixed label; prohibit query strings, request IDs, model names, keys, tokens, and arbitrary path segments from labels. | One unknown route class must remain one series class and must not expose sensitive values. | Unassigned | Deployed; tracker closed 2026-09-06; broader evidence limits retained | [token.place #1765](https://github.com/futuroptimist/token.place/issues/1765) | Thousands of distinct unknown URLs yield the same bounded labels and no sensitive strings in exposition. |
| P0 | Prevent | Prefer a small explicitly registered metric set over implicit default per-path instrumentation. | Makes the exported contract reviewable and bounded. | Unassigned | Deployed; tracker closed 2026-09-06 | [token.place #1765](https://github.com/futuroptimist/token.place/issues/1765) | The deployed corrected image exports documented finite metric names and label domains. |
| P0 | Prevent | Add a regression/load test with thousands of unique unmatched paths and fixed budgets for series, samples, response size, scrape duration, and memory. | Reproduces the trigger class and prevents recurrence. | Unassigned | Repository regression and staging qualification completed; trackers closed 2026-09-06; complete public evidence not reproduced | [token.place #1765](https://github.com/futuroptimist/token.place/issues/1765)<br>[sugarkube #2774](https://github.com/futuroptimist/sugarkube/issues/2774) | The test passes explicit reviewed budgets and fails the prior unbounded behavior. |
| P0 | Prevent | Move multiprocess metrics to a dedicated directory and clear it safely on every application-container startup before Gunicorn launches. | A container restart must not inherit stale metric files; pod deletion must not be the cleanup mechanism. | Unassigned | Implemented and regression-covered by #1782; tracker closed 2026-09-06 | [token.place #1765](https://github.com/futuroptimist/token.place/issues/1765) | A container restart test proves the directory starts clean without deleting the pod. |
| P0 | Prevent | Validate Prometheus multiprocess worker cleanup. | Dead-worker files and series must not accumulate across worker lifecycles. | Unassigned | Repository regression covered by #1782; tracker closed 2026-09-06; broader qualification pending | [token.place #1765](https://github.com/futuroptimist/token.place/issues/1765) | Repeated worker start/exit testing leaves a bounded, correct exposition. |
| P1 | Prevent | Consider a bounded edge rate limit or scanner control for unmatched paths. | Reduces abusive load as defense in depth but cannot replace bounded labels. | Unassigned | Proposed | [sugarkube #2780](https://github.com/futuroptimist/sugarkube/issues/2780) | Legitimate routes remain usable and randomized-path traffic is bounded; exporter tests still pass without this control. |
| P1 | Prevent | Reassess the 256Mi memory limit only after measuring the corrected exporter. | Measured headroom is useful, but a temporary increase is not a root-cause fix. | Unassigned | Proposed | [sugarkube #2781](https://github.com/futuroptimist/sugarkube/issues/2781) | A sustained corrected-exporter test supports a documented limit and safety margin. |
| P0 | Detect | Alert on relay working-set-to-limit ratios at warning and critical thresholds. | Provides actionable headroom before kernel enforcement. | Unassigned | Proposed | [Threshold measurement: sugarkube #2781](https://github.com/futuroptimist/sugarkube/issues/2781)<br>[Alert implementation: sugarkube #2405](https://github.com/futuroptimist/sugarkube/issues/2405) | Controlled threshold crossing fires and resolves both alert levels. |
| P0 | Detect | Alert directly on `OOMKilled` state and restart acceleration. | Avoids waiting for only the scrape target to fail. | Unassigned | Proposed | [Alert implementation: sugarkube #2405](https://github.com/futuroptimist/sugarkube/issues/2405) | Synthetic OOM-state and restart-rate inputs exercise the alert path. |
| P0 | Detect | Define per-target budgets and alerts for `scrape_samples_scraped`, `scrape_series_added`, `scrape_duration_seconds`, and distinct bounded route-label counts. | Detects exporter growth before memory exhaustion. | Unassigned | Proposed | [Restoration budgets and thresholds: sugarkube #2777](https://github.com/futuroptimist/sugarkube/issues/2777)<br>[Alert implementation: sugarkube #2405](https://github.com/futuroptimist/sugarkube/issues/2405) | Each budget has a documented threshold and tested alert. |
| P1 | Detect | Add a dashboard combining application cardinality, scrape sample count/size, scrape latency, memory headroom, and restarts. | Correlated signals shorten diagnosis. | Unassigned | Proposed | [Existing observability work](https://github.com/futuroptimist/sugarkube/issues/2405) | Dashboard panels populate from a staging cardinality exercise. |
| P0 | Detect | Add a release/staging gate that rejects linear series growth from unique unknown paths. | Stops reintroduction before production. | Unassigned | Proposed | [sugarkube #2774](https://github.com/futuroptimist/sugarkube/issues/2774) | Gate fails an intentionally unbounded fixture and passes bounded instrumentation. |
| P0 | Detect | Add a release/artifact provenance gate that requires either verified `e0e2685` ancestry or an explicitly reviewed and tested behavior-equivalent backport for the focused release-line hotfix. | Prevents a divergent release lineage from omitting required hardening while retaining the documented manual-hotfix strategy. | Unassigned | Proposed | [token.place #1770](https://github.com/futuroptimist/token.place/issues/1770) | Promotion verifies `e0e2685` ancestry or an explicitly reviewed equivalent backport, passes the bounded-registry negative/cardinality tests, and records the image and source commit identities and qualification path. |
| P1 | Detect | Retain privacy-safe aggregate access evidence long enough to classify future triggers without raw credentials, query strings, encrypted payloads, or sensitive paths. | Improves attribution confidence without weakening privacy. | Unassigned | Proposed | [sugarkube #2782](https://github.com/futuroptimist/sugarkube/issues/2782) | Retention and redaction review proves only bounded aggregates are stored. |
| P0 | Mitigate | Write a metrics-induced OOM-loop runbook: validate context/target, pause only its ServiceMonitor, replace the pod, verify public health, and keep scraping paused until exit criteria pass. | Makes the safe, narrow recovery repeatable. | Unassigned | Proposed | [sugarkube #2779](https://github.com/futuroptimist/sugarkube/issues/2779) | A non-production exercise completes with no unrelated target mutation. |
| P1 | Mitigate | Add a bounded emergency setting that disables expensive application metrics while retaining liveness, readiness, and minimal operational metrics. | Preserves minimum observability without exercising unsafe exposition. | Unassigned | Proposed | [token.place #1773](https://github.com/futuroptimist/token.place/issues/1773) | Exercise shows minimal metrics and health remain available with bounded resource use. |
| P0 | Mitigate | Document that container restart does not necessarily clear a pod `emptyDir`; this incident required pod replacement. | Prevents ineffective restart loops. | Unassigned | Proposed | [sugarkube #2779](https://github.com/futuroptimist/sugarkube/issues/2779) | Runbook review and a pod-lifecycle test demonstrate the distinction. |
| P0 | Mitigate | Define a safe procedure to restore the ServiceMonitor label only after the corrected image is verified. | Prevents premature scrape restoration. | Unassigned | Proposed | [sugarkube #2777](https://github.com/futuroptimist/sugarkube/issues/2777) | Procedure includes all restoration gates and a rollback step. |
| P1 | Mitigate | Continue shared-state/HA work tracked by [#1569](https://github.com/futuroptimist/token.place/issues/1569), without treating replicas as safe standalone mitigation while authoritative state is memory-backed. | HA can reduce single-replica amplification only after correctness constraints are satisfied. | Unassigned | Proposed | [Existing non-incident HA work #1569](https://github.com/futuroptimist/token.place/issues/1569) | Shared-state correctness is proven before multi-replica availability is relied upon. |
| P0 | Mitigate | After the fix, run a staging soak with scraping enabled and adversarial unique-path traffic before production restoration. | Validates the whole scrape/traffic lifecycle. | Unassigned | Operator-reported complete before deployment; tracker closed 2026-09-06; complete public evidence not reproduced | [sugarkube #2774](https://github.com/futuroptimist/sugarkube/issues/2774) | Sustained soak passes every restoration exit criterion. |

The recovery sequence is tracked explicitly: the bounded backport in
[token.place #1765](https://github.com/futuroptimist/token.place/issues/1765), staging qualification
in [sugarkube #2774](https://github.com/futuroptimist/sugarkube/issues/2774), and exact-image
deployment in [sugarkube #2775](https://github.com/futuroptimist/sugarkube/issues/2775) led to
application-metrics restoration tracked by
[sugarkube #2777](https://github.com/futuroptimist/sugarkube/issues/2777), after
[sugarkube #2776](https://github.com/futuroptimist/sugarkube/issues/2776) restored both
public-information probes. The operator reported qualification of the combined recovery image in
staging complete before its production deployment. Service and observability restoration support
the operational classification, while this public summary does not reproduce the complete staging
qualification evidence; independent evidence review and the 24-hour stability criteria remain
undocumented even though the completed-work trackers closed. Repository restart-cleanup coverage in
#1782 is not a claim that every production or staging acceptance criterion passed.

The fifteen canonical action trackers are token.place #1765, #1766, and #1770–#1773, plus
sugarkube #2774–#2782. As of 2026-09-06, #1765, #1766, #2774, and #2775 are independently closed;
the remaining eleven (#1770–#1773 and #2776–#2782) are open. Links to token.place
[#1569](https://github.com/futuroptimist/token.place/issues/1569) and sugarkube
[#2405](https://github.com/futuroptimist/sugarkube/issues/2405) are contextual trackers outside
that canonical fifteen-item set; every tracker link and owner in the tables remains unchanged.

### Prometheus restoration path

The recommended near-term path is a focused hotfix based on the current `release/relay-0.1.1`
line, unless a newer main-derived release has independently completed every existing staging and
production-promotion gate. Urgent observability restoration must not be coupled to an otherwise
unqualified broad `main` rollout. The release hotfix must manually port the bounded behavior from
#1447 rather than blindly cherry-pick its historical diff. It must:

- disable prometheus-flask-exporter defaults and its automatic `/metrics` route;
- use a dedicated registry and relay-owned authenticated `/metrics`;
- preserve the required build, instrumentation, and node gauges;
- expose only reviewed finite `tokenplace_*` label domains;
- prove that no default `flask_http_request*` family, raw path, query string, request ID,
  credential, ciphertext, or arbitrary value appears;
- use a dedicated exact multiprocess directory and clear/recreate only that directory before
  Gunicorn starts, or formally remove multiprocess mode if the release is locked to one worker;
  and
- add a same-pod application-container restart test proving stale metric files are not inherited.

Before production scraping is restored, an immutable hotfix image must pass staging with the
production-equivalent one-worker, 256Mi, and 30-second scrape settings. The operator reported this
qualification complete for the combined recovery image before deployment, although this public
summary does not reproduce its complete evidence. The test must send thousands
of unique unknown paths while continuously scraping and verify bounded series/sample counts,
response size, scrape latency, RSS/working-set headroom, zero OOMs/restarts, clean restart behavior,
exact image identity, public health, and API-v1 compatibility.

Production must keep application scraping paused through corrected-image deployment. Because relay
authority remains memory-backed and the workload uses a single replica with Recreate semantics,
the existing controlled quiescence and rollout procedure is required. Operators must verify health
and build identity before deliberately restoring only the exact target discovery selector, then
observe a defined stability window with explicit rollback thresholds. If cardinality, scrape cost,
memory, or restarts regress, pause only that target again. Scraping must never be restored against
`e46277d`; a memory-limit increase, edge filtering, or extra replicas alone is not resolution.

These retained requirements continue to govern full qualification and issue closeout. The
identity-verified recovery deployment and bounded restoration soak ending at
`2026-09-05T19:53:19Z` support the operator's **Resolved operationally** classification. The staging
qualification is operator-reported complete, while independent review of its evidence and the
other unverified criteria below remain pending.

### Required restoration exit criteria

For full historical qualification and issue closeout, all of the following must be demonstrated:

- unmatched paths map to a bounded label set;
- thousands of unique paths do not grow Prometheus series linearly;
- `/metrics` sample count, response size, and latency remain within explicit reviewed budgets;
- relay working set remains comfortably below the configured memory limit during a sustained
  scrape-and-traffic test;
- an application-container restart cannot inherit stale multiprocess metric files;
- the corrected image is deployed and its identity is verified;
- public health and application functionality remain healthy; and
- the ServiceMonitor is restored deliberately and observed through a defined stability window.

## Post-incident closeout

The availability impact ended when the replacement pod became healthy at
2026-09-03T01:08:18Z. Current closeout state is:

- **Application:** healthy after mitigation.
- **Telemetry:** all intentionally paused monitoring restored; application scraping is healthy.
- **Incident:** Resolved operationally at `2026-09-05T19:53:19Z`; broader historical qualification
  evidence remains undocumented.
- **Runtime/deployment remediation:** Helm revision 7 runs the identity-verified combined recovery
  image with the bounded registry backport, and the restored production scrape remained bounded.
- **Closeout:** four completed-work trackers closed independently on 2026-09-06; eleven canonical
  corrective and preventive issues remain open, and this documentation change closes none.

Disabling scraping, replacing a pod, filtering traffic, raising memory, or adding replicas alone
do not constitute resolution; the corrected deployment plus initial monitored restoration are
recovery evidence but do not yet meet every closeout gate. Issue
#1569 concerns separate shared-state/HA resilience work and did not cause this incident.

## Evidence gaps and unknowns

- The historical evidence window begins at 23:20 UTC, so the exact first unmatched request that
  began the buildup is unknown.
- Raw historical request paths and exact source addresses are unavailable or intentionally
  excluded from the public record. The traffic source cannot be attributed.
- Exact customer request failures and any active in-memory work losses were not measured. The
  record identifies risk, not proven customer data loss.
- Exact PagerDuty trigger and acknowledgement timestamps are unavailable in the sanitized evidence.
- Direct pre-mitigation `kubectl top` and cgroup capture failed because the old container was
  already unavailable.
- `container_oom_events` remained zero. This is not evidence against the authoritative Kubernetes
  `lastState.terminated.reason=OOMKilled` record.
- `scrape_body_bytes` remained zero. It is not evidence of an empty response; response-size claims
  use sanitized application access-log measurements instead.
- Thirty-second memory samples may have missed the instantaneous allocation that crossed 256Mi.
- Raw paths, query strings, request IDs, credentials, tokens, ciphertext, prompts, responses, tool
  data, arbitrary diagnostic payloads, source addresses, private evidence paths, screenshots, and
  raw log archives are intentionally excluded.

## Related incident

Stable process uptime after this incident's mitigation exposed a distinct, pre-existing
[public-information rate-limit defect on September 3](2026-09-03-production-public-information-rate-limit-exhaustion.md).
Application-metrics scraping remained paused throughout the September 3 failure and initial
mitigation period. Scraping was subsequently restored before operational resolution. The later quota exhaustion
does not alter this incident's metrics-cardinality root cause; both records instead identify a
shared release-line parity and production-qualification weakness.

## Verification commands and public references

Repository commands used to validate this documentation change:

```bash
python -m json.tool outages/2026-09-02-production-relay-metrics-cardinality-oom.json >/dev/null
python -c 'import json; import jsonschema; jsonschema.validate(json.load(open("outages/2026-09-02-production-relay-metrics-cardinality-oom.json")), json.load(open("outages/schema.json")))'
pre-commit run --all-files
git diff --check
detect-secrets scan $(git diff --cached --name-only)
```

Public references:

- [Deployed commit `e46277d`](https://github.com/futuroptimist/token.place/commit/e46277daaeb76beeb9f2a2e9e265181287239b22)
- [Bounded-registry source commit `a86fa1a`](https://github.com/futuroptimist/token.place/commit/a86fa1a47220023b7a236fefcb7f78c0c4487880)
- [`main` hardening merge `e0e2685`](https://github.com/futuroptimist/token.place/commit/e0e2685062a1f4714dd6a10cda7586dd1534860a)
- [`release/relay-0.1.1` base `dc6ac09`](https://github.com/futuroptimist/token.place/commit/dc6ac09d7963d417ab6054b97c48e314a6494eef)
- [PR #1735: build-info label correction](https://github.com/futuroptimist/token.place/pull/1735)
- [PR #1732: maintenance-gauge PID-label correction](https://github.com/futuroptimist/token.place/pull/1732)
- [PR #1729: release-line metrics backport](https://github.com/futuroptimist/token.place/pull/1729)
- [PR #1447: bounded relay metrics registry](https://github.com/futuroptimist/token.place/pull/1447)
- [PR #1726: unrelated internal Valkey slice](https://github.com/futuroptimist/token.place/pull/1726)
- [Issue #1569: separate shared-state/HA resilience work](https://github.com/futuroptimist/token.place/issues/1569)
- [Deployed `api/__init__.py`](https://github.com/futuroptimist/token.place/blob/e46277daaeb76beeb9f2a2e9e265181287239b22/api/__init__.py)
- [Deployed relay requirement pins](https://github.com/futuroptimist/token.place/blob/e46277daaeb76beeb9f2a2e9e265181287239b22/config/requirements_relay.txt)
- [prometheus-flask-exporter 0.23.2 source and defaults](https://github.com/rycus86/prometheus_flask_exporter/blob/0.23.2/prometheus_flask_exporter/__init__.py)
- [prometheus-client 0.21.0 multiprocess requirements](https://github.com/prometheus/client_python/blob/v0.21.0/docs/content/multiprocess/_index.md)
- [Kubernetes `emptyDir` documentation](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir)

This public record uses sanitized aggregates and durable repository references. It does not publish
private evidence locations or raw diagnostic artifacts.