--- name: vp-pattern-dev description: >- Validated Patterns development guidelines for ia-computer-vision. Use when creating, extending, or modifying VP pattern files (values-*.yaml, Chart.yaml, subscriptions, applications, pattern-metadata.yaml, pattern.sh, Makefile). Also use when adding operators, charts, or namespaces to the pattern. --- # Validated Patterns Development -- ia-computer-vision ## Pattern identity - **Name**: ia-computer-vision - **Tier**: Sandbox - **Maintainer**: Maximiliano Pizarro, Specialist Solution Architect (mapizarr@redhat.com) - **Repo**: https://github.com/maximilianoPizarro/ia-computer-vision - **Base**: multicloud-gitops (VP standard) - **Source architecture**: hybrid-mesh-platform ## Core VP structure rules 1. **Root files**: `values-global.yaml`, `values-{clusterGroupName}.yaml`, `Chart.yaml` (dep on `clustergroup` from `charts.validatedpatterns.io`), `pattern.sh`, `Makefile` 2. **Cluster roles**: `hub`, `east`, `west` -- each maps to `values-hub.yaml`, `values-east.yaml`, `values-west.yaml` 3. **`global.singleArgoCD: true`** -- single ArgoCD instance managed by VP Operator in `vp-gitops` 4. **`multiSourceConfig.enabled: true`** with `clusterGroupChartVersion: "0.9.*"` 5. **VP Operator handles OpenShift GitOps** -- do NOT add `openshift-gitops-operator` subscription ## Adding an operator (4-step process) Per https://validatedpatterns.io/contribute/extending-a-pattern/: 1. **Add namespace** in `clusterGroup.namespaces` of the target `values-*.yaml` 2. **Add subscription** in `clusterGroup.subscriptions` (OLM installs the operator) 3. **Add ArgoCD application** in `clusterGroup.applications` (either `chart:` for VP published charts or `path:` for local charts) 4. **Add Helm chart** in `charts/all/{name}/` only if custom CRs are needed beyond the operator install Subscription format: ```yaml subscriptions: my-operator: name: my-operator namespace: my-namespace # default: openshift-operators channel: stable source: redhat-operators # default ``` Application format (VP chart): ```yaml applications: my-app: name: my-app namespace: my-namespace argoProject: hub chart: my-chart # from charts.validatedpatterns.io chartVersion: 0.1.* ``` Application format (local chart): ```yaml applications: my-app: name: my-app namespace: my-namespace argoProject: hub path: charts/all/my-app overrides: - name: clusterName value: east ``` ## VP published charts (use instead of local charts) Available at https://charts.validatedpatterns.io/ -- reference by `chart:` + `chartVersion:`: | Chart | Version | Purpose | |-------|---------|---------| | `acm` | 0.2.* | ACM MultiClusterHub config | | `hashicorp-vault` | 0.1.* | Vault server | | `openshift-external-secrets` | 0.0.* | ESO config | | `rhbk` | 0.1.* | Red Hat Build of Keycloak | | `servicemesh` | 0.1.* | OSSM 3.0 (but we use local for ambient control) | | `llm-inference-service` | 0.3.* | KServe inference | | `letsencrypt` | 0.1.* | TLS certificates | ## Local chart structure Each chart in `charts/all/{name}/` must have: ``` charts/all/{name}/ ├── Chart.yaml # apiVersion: v2, type: application ├── values.yaml # defaults └── templates/ └── *.yaml # Kubernetes CRs ``` Chart.yaml template: ```yaml apiVersion: v2 name: {name} description: {description} type: application version: 0.1.0 maintainers: - name: Maximiliano Pizarro email: mapizarr@redhat.com ``` Use `argocd.argoproj.io/sync-wave` annotations for ordering. Use `argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true` for CRDs that may not exist yet. ## Secrets (values-secret.yaml.template) - Schema version 2.0 - Default backend: vault - Use `onMissingValue: generate` with `vaultPolicy: validatedPatternDefaultPolicy` for auto-generated passwords - Never commit actual secret values to Git - Secrets loader searches: `~/.config/validated-patterns/values-secret-ia-computer-vision.yaml` first ## Sandbox tier requirements 1. VP-conformant structure (multiSourceConfig, clustergroup dep) 2. Deployable on fresh OCP without modification 3. README with business problem + solution 4. Architecture drawing 5. Technical review (post-implementation) 6. Architecture review (post-implementation) 7. SUPPORT.md documenting support policy ## Current chart inventory ### Hub charts (values-hub.yaml) | Chart | Type | Wave | ArgoProject | |-------|------|------|-------------| | `openshift-gitops` | local | 0 | platform (only sets `ArgoCD/vp-gitops` `spec.localUsers` via `argocdLocalUser.enabled=true` — deliberately does NOT set `spec.rbac`/`controller`/`server`/etc.; see the `patterns-operator` root-cause entry below for why) | | `openshift-lightspeed` | local | — | platform (OpenShift Lightspeed operator + `OLSConfig`; LLM provider = legacy `ai-gateway`/Kuadrant, MCP server = `argocd-mcp`; see the Lightspeed/Kuadrant WASM entry below for a known, unresolved LLM-connectivity issue) | | `platform-users` | local | 0 | platform | | `observability` | local | 1 | observability | | `acm` | VP chart | 1 | hub | | `vault` | VP chart | 2 | external-secrets | | `openshift-external-secrets` | VP chart | 2 | external-secrets | | `rhbk` | VP chart | 2 | workshop | | `gitlab-operator` | local | 3 | workshop | | `developer-hub` | local | 4 | workshop | | `servicemesh-config` | VP chart (`servicemesh:0.1.*`) | 5 | mesh (required for neuroface-gateway Istio) | | `openshift-ai-hub` | local | 5 | ai | | `models-as-a-service` | local | 6 | ai (native RHOAI 3.4 MaaS: Tenant, Gateway, Gen AI Studio) | | `argocd-mcp` | local | 3 | hub (mcp-for-argocd, hub-only) | | `mailpit` | local | 5 | workshop (Mailpit UI + PPE Kafka consumer) | | `rhbk-iam` | local | 2 | workshop (per-user realms: neuroface/maas/cv, `backstage-provisioner` client) | | `hub-interconnect` | local | 5 | mesh (Skupper VAN + Kafka listener) | | `skupper-network-observer` | local | 5 | mesh | | `mailpit` | local | 5 | workshop (Mailpit UI + PPE Kafka consumer) | | `neuroface-gateway` | local | 6 | mesh | | `workshop-kuadrant-apis` | local | 6 | workshop (AI Gateway, workshop API products; owns the cluster-wide `Kuadrant`/`Authorino`/`Limitador` singleton — see pitfall below) | | `acm-hub-spoke` | local | 6 | hub | | `acs-init-bundle-sync` | local | 7 | security (**disabled by default** — commented out to save resources) | | `devspaces` | local | 7 | workshop | | `workshop-registration` | local | 5 | workshop | | `console-links` | local | 10 | workshop | Verify the live app list with `oc get application -n vp-gitops` — it drifts from this table faster than the table gets updated; treat this as a starting point, not a source of truth. ### Spoke charts (values-east.yaml / values-west.yaml) | Chart | Type | Wave | ArgoProject | |-------|------|------|-------------| | `platform-users` | local | 0 | platform (creates `acm-import` SA with cluster-admin) | | `argocd-local-users` | local | 0 | platform (sole owner of `spec.rbac`/`localUsers` on spoke `vp-gitops` — **not deployed on hub**, see openshift-gitops singleton note; verify ACM mustonlyhave on spokes) | | `argocd-mcp-spoke-export` | local | 1 | platform (exports ai-agent token to ConfigMap for hub sync) | | `servicemesh-config` | VP chart (`servicemesh:0.1.*`) | 1 | mesh | | `openshift-external-secrets` | VP chart | 2 | external-secrets | | `observability` | local | 2 | observability (also deploys spoke DSC for KServe CRDs) | | `acs-secured-cluster` | local | 3 | security (**disabled by default** — commented out to save resources) | | `spoke-interconnect` | local | 4 | mesh | | `spoke-neuroface` | local | 5 | ai (includes ApplicationSet for user NeuroFace repos) | | `spoke-neuroface-cv` | local | 6 | ai | | `kafka-console` | local | — | ai (Kafka UI for PPE detection events) | | `skupper-network-observer` | local | — | mesh | | `devspaces` | local | 7 | workshop | | `console-links` | local | 10 | workshop | ## Hub-only + GPU + pre-installed-operators overlay stack Five OPT-IN overlay files at the repo root, composable via the Pattern CR's `extraValueFiles` on top of the default `values-hub.yaml`. None of them are loaded unless explicitly listed — the tagged/default install path is completely unaffected. **Authoritative Pattern CR decision table:** `docs/content/patterns/ia-computer-vision/pattern-cr-guide.adoc` and ready-to-apply YAML in `examples/pattern-cr/`. | Scenario | extraValueFiles | Example file | |----------|-----------------|--------------| | **A — Hub-only CPU (default)** | `values-hub-only.yaml` | `examples/pattern-cr/hub-only-cpu.yaml` | | B — Hub + spokes CPU | *(none on hub)* | `examples/pattern-cr/hub-spoke-cpu.yaml` | | C — Hub-only GPU multi-node | `values-hub-gpu.yaml`, `values-hub-only.yaml` | `examples/pattern-cr/hub-only-gpu-multi-node.yaml` | | D — Hub-only GPU single-node + pre-installed ops | gpu → single-node → rhpds → hub-only | `examples/pattern-cr/hub-only-gpu-single-node-preinstalled.yaml` | | E — Spoke | *(none)* | `examples/pattern-cr/spoke.yaml` | Public docs for pre-installed-operator sandboxes (no vendor name in prose): `ideas-for-customization.adoc`, "Configuring GPU inference on a cluster with pre-installed operators". | File | Purpose | Depends on | |------|---------|------------| | `values-hub-gpu.yaml` | GPU-backed vLLM model serving on the hub (`openshift-ai-hub` `gpu.enabled`, `charts/all/openshift-ai-hub/templates/gpu-vllm-models.yaml`); installs its own NFD + NVIDIA GPU Operator subscriptions | none — works standalone on a from-scratch multi-node hub | | `values-hub-single-node.yaml` | Scales `userCount`/replica counts to 5 across ~11 applications so the stack fits a single-node kubelet `maxPods` ceiling | composes with `values-hub-gpu.yaml` | | `values-hub-rhpds.yaml` | Adapts to a cluster where NFD, GPU Operator, Service Mesh, OpenShift AI, and Connectivity Link are **already installed by someone else**: skips creating a second `OperatorGroup` in those namespaces, pins ZTunnel's `spec.version` to match the pre-installed Istio, points `models-as-a-service` at the pre-installed Connectivity Link's `kuadrant-system` namespace, enables `autoApproveManualInstallPlans`, and sets a `ssoHostPrefix: rhdh-sso` override (six charts, see below) when a pre-existing SSO/Keycloak instance already claims the default `sso.` hostname | loads after `values-hub-single-node.yaml` — see the override-wholesale-replace warning below | | `values-hub-only.yaml` | Hub-only install with no east/west spokes: disables Skupper/`hub-interconnect`/`acm-hub-spoke`, deploys the upstream `neuroface` chart + local Kafka (Strimzi) on the hub, routes `neuroface(-cv).apps.*` via `neuroface-gateway` `hubLocal` mode | optional, loads last | | `values-hub-gpu-minimal.yaml` | Fallback for sandboxes where `maxPods` cannot be raised: disables the entire workshop/dev platform (GitLab, Developer Hub, DevSpaces, RHBK, Lightspeed) and keeps only Vault/ESO + OpenShift AI + vLLM `InferenceService`s | only when `values-hub-single-node.yaml` is not enough; loads after all four above | Documented Pattern CR order for **Scenario D only**: `values-hub-gpu.yaml` → `values-hub-single-node.yaml` → `values-hub-rhpds.yaml` → `values-hub-only.yaml` (→ `values-hub-gpu-minimal.yaml` if needed). **Scenario A (most common)** uses only `values-hub-only.yaml`. ### `ssoHostPrefix` — six charts must move together `developer-hub`, `rhbk-iam`, `neuroface-gateway`, `workshop-registration`, `devspaces`, `console-links` all read a chart-local `ssoHostPrefix` (default `"sso"`). If a pre-existing SSO instance already claims `sso.`, override all six to the same alternate value — a partial update leaves whichever chart(s) you missed pointing at the dead hostname. `values-hub-rhpds.yaml` sets all six to `rhdh-sso`. ### Single-node maxPods ceiling See `docs/content/patterns/ia-computer-vision/cluster-sizing.adoc`, "Single-node sandboxes" — a control-plane-and-worker-combined node running a pre-provisioned RHACM/Pipelines/RHOAI/GPU-Operator baseline consumes ~110-130 of the kubelet's default 250 `maxPods` before this pattern installs anything. Prefer raising `maxPods` to 500 via `KubeletConfig` (one-time reboot) over `values-hub-gpu-minimal.yaml`, which strips real functionality instead. ## Known limitations (sandbox) - **Image builds**: Use OpenShift internal registry for scaffolded NeuroFace Tekton pipelines (Quay removed from pattern). - **Per-user RHBK**: Software template deploys RHBK biometric + OIDC AuthPolicy per user on spokes via ApplicationSet. - **Hub sizing**: `m6a.2xlarge` workers saturate at 98-99% CPU requests. Use `m6a.4xlarge` for production or remove master taints for sandbox. - **Developer Hub plugins**: OCM, Tekton, Topology, Kafka, Quay community plugins do not ship in RHDH 1.10 image. Must be `disabled: true` in `dynamic-plugins-rhdh` ConfigMap. - **Vault secrets**: If installed via OCP console Pattern CR (not CLI), secrets are not auto-loaded. Populate Vault manually after Vault initializes (wave 2). See README "Option B — Console install". - **RHBK**: The `rhbk-credentials` secret must include both `admin-password` and `db-password` fields. - **Skupper console link**: Points to hub but Skupper only runs on spokes. The hub console link will always show 503. ## Lessons learned from installation and software template launch (Jun 2026) ### OperatorGroup install modes Operators like RHCL and Cluster Observability Operator do NOT support `OwnNamespace` install mode. Spoke namespaces for these operators MUST declare `operatorGroup: true` + `targetNamespaces: []` (AllNamespaces mode). Without this, CSVs fail with `OwnNamespace InstallModeType not supported`. ### ACM auto-import RBAC The `acm-spoke-auto-import` CronJob ServiceAccount requires these permissions beyond basic `managedclusters` CRUD: - `cluster.open-cluster-management.io/managedclustersets/join` (create) — ACM auto-adds clusters to the `default` clusterset - `cluster.open-cluster-management.io/managedclusters/accept` (update) — required to set `hubAcceptsClient: true` - `register.open-cluster-management.io/managedclusters/accept` (update) — alternative API group for the same - Namespace `create` verb — to wait for ACM to create the spoke namespace ### Spoke import tokens The default ServiceAccount `kube-system:default` does NOT have `cluster-admin` permissions. The pattern auto-creates `kube-system:acm-import` SA with `cluster-admin` via `platform-users` chart (wave 0). Users only need to generate the token: `oc create token -n kube-system acm-import --duration=87600h`. ### KServe CRDs on spokes Spokes subscribe to RHOAI but the `InferenceService` and `ServingRuntime` CRDs are only installed when a `DataScienceCluster` CR exists with `kserve.managementState: Managed`. The pattern places a minimal DSC (KServe-only, everything else Removed) in the `observability` chart (wave 2) — NOT in `spoke-neuroface-cv` (wave 6) because that creates a deadlock (chart needs CRDs that only exist after the chart syncs). ### Service Mesh on hub The hub MUST have `servicemeshoperator3` subscription and a `servicemesh-config` application (VP `servicemesh` chart with `profile: ambient`) for the NeuroFace Gateway (`gatewayClassName: istio`). Without it, the Gateway shows `Waiting for controller` / `PROGRAMMED: Unknown`. ### Developer Hub YAML config The `catalog.providers.gitlab` config must NOT have duplicate `schedule:` keys. RHDH YAML parser is strict and crashes on duplicate map keys (`YAMLParseError: Map keys must be unique`). ### Developer Hub missing namespaces RHDH templates reference namespaces `gitlab`, `keycloak`, `east`, `west` for RBAC/spoke-token-sync. These must exist before the developer-hub app syncs. If they don't exist, ArgoCD retries until they are created. ### Console install (Pattern CR) secrets When installing via the OCP console (Pattern CR), `make load-secrets` is NOT executed. Vault initializes empty. You must load secrets manually: ```bash oc exec vault-0 -n vault -- vault kv put secret/hub/gitlab-credentials root-password="$(openssl rand -base64 16)" runner-token="$(openssl rand -base64 16)" oc exec vault-0 -n vault -- vault kv put secret/hub/rhbk-credentials admin-password="$(openssl rand -base64 16)" db-password="$(openssl rand -base64 16)" oc exec vault-0 -n vault -- vault kv put secret/hub/developer-hub-secrets session-secret="$(openssl rand -base64 32)" gitlab-token="placeholder" oc exec vault-0 -n vault -- vault kv put secret/hub/maas-credentials api-key="placeholder" ``` After loading, force ESO refresh: `oc annotate externalsecret -n keycloak-system --all force-sync=$(date +%s) --overwrite` Note: `gitlab-token` can stay as `placeholder` — the `gitlab-token-setup` PostSync job (developer-hub chart) auto-creates a PAT and patches both the K8s Secret and Vault. ### GitLab external URL (`gitlab-gitlab.apps` vs `gitlab.apps`) The GitLab operator creates an Ingress with hostname `gitlab-gitlab.apps.` (from `hostSuffix: gitlab`). OpenShift translates this into a Route, but WITHOUT TLS termination by default. HTTPS git push to `gitlab-gitlab.apps` returns **503** (OpenShift "Application not available" page). **Fix**: Set `global.hosts.gitlab.name` in the GitLab CR to `gitlab.apps.` (via Helm helper `gitlab-operator.host`). This overrides the auto-generated hostname so clone URLs (http_url_to_repo) use the working `gitlab.apps` route. Also add `route.openshift.io/termination: edge` in ingress annotations. The custom `route-gitlab-apps.yaml` uses the same helper and has TLS edge termination. ### RHDH scaffolder `publish:gitlab` input schema The RHDH dynamic plugin `backstage-plugin-scaffolder-backend-module-gitlab-dynamic` does NOT accept `description` or `repoVisibility` as top-level inputs. These must go under `settings`: ```yaml action: publish:gitlab input: repoUrl: "gitlab.apps.{{ domain }}?owner=group&repo=name" defaultBranch: main settings: description: "Project description" visibility: public ``` Using top-level `description` causes `InputError: instance is not allowed to have the additional property "description"`. ### RHDH GitLab token lifecycle (stale pod token) ESO syncs the GitLab PAT from Vault into the `developer-hub-oidc-auth` Secret, but the RHDH pod only reads `GITLAB_TOKEN` at startup (`envFrom`). If the Secret updates after the pod starts, the pod keeps the old (possibly placeholder) value until restarted. The `gitlab-token-setup` PostSync job now: 1. Checks if the Secret token is valid (GitLab API 200) 2. Compares the Secret token with the running pod's `GITLAB_TOKEN` env var 3. If they differ, restarts the `backstage-developer-hub` deployment 4. If no valid token exists, creates a new PAT via `gitlab-rails runner` 5. Patches both the K8s Secret AND Vault (`secret/hub/developer-hub-secrets gitlab-token`) 6. Restarts RHDH to load the new token This eliminates the manual step of patching Vault after GitLab deploys. ### ApplicationSet SSH_AUTH_SOCK error When `user-neuroface-apps` ApplicationSet finds repos via scmProvider but they are marked for deletion (`deletion_scheduled`), ArgoCD tries to clone them and fails with `error creating SSH agent: SSH_AUTH_SOCK not-specified`. This is transient — it resolves when the repos finish deletion or new repos are created. ### GitOpsCluster placement namespace The `GitOpsCluster` CR MUST be in `openshift-gitops` (same namespace as the ACM `Placement`), NOT in `vp-gitops`. But `spec.argoServer.argoNamespace` MUST point to `vp-gitops` (where the VP-managed ArgoCD instance lives). Without this split, ArgoCD never discovers spoke clusters. ### ExternalSecret API version Use `external-secrets.io/v1` (not `v1beta1`). The ESO operator on OCP 4.20 does not serve `v1beta1` by default. Using the wrong version causes `no matches for kind "ExternalSecret"`. ### Gateway API HA replicas Istio Gateway API does not support `spec.replicas` directly. Use `spec.infrastructure.parametersRef` pointing to a ConfigMap: ```yaml spec: infrastructure: parametersRef: group: "" kind: ConfigMap name: neuroface-gateway-infrastructure ``` The ConfigMap contains `deployment: | spec: replicas: 2`. This works on both hub and spokes. ### OwnerPicker entity ref format The Backstage `OwnerPicker` field returns `user:default/user1`, not `user1`. Use the Nunjucks filter `parseEntityRef | pick('name')` throughout the template to extract just the username. Without this, repo names and namespaces contain `user-default-user1` instead of `user1`. ## Lessons learned from workshop deployment and spoke onboarding (Jun 29, 2026) ### GitLab 500 on session login (namespace_settings nil) In GitLab 18+, groups created programmatically via `gitlab-rails runner` do NOT auto-create `namespace_settings`. When a user logs in and accesses a group page, GitLab calls `namespace_settings` which returns nil and triggers a 500. **Fix**: Call `create_namespace_settings!` on every group in the `ensure_group()` function inside `charts/all/gitlab-operator/templates/job-gitlab-bootstrap.yaml`. Also add an `ensure_developer()` function to grant Developer role to all workshop users on the `ws-workshop` group. ### Keycloak CrashLoopBackOff (ambient mesh breaks DNS) Adding Istio ambient mesh labels (`istio.io/dataplane-mode: ambient`) to `keycloak-system` namespace causes DNS resolution failures inside Keycloak pods (`UnknownHostException: postgresql-db`). The ztunnel proxy intercepts DNS before CoreDNS responds. **Fix**: Do NOT add ambient mesh labels to `keycloak-system` namespace in `values-hub.yaml`. Keycloak must remain outside the mesh. ### SSO hostname mismatch (cookie domain) The Keycloak CR defaults to `hostname: keycloak.apps.` but the AuthPolicy `extAuth` redirects users to `sso.apps.`. This mismatch causes "Restart login cookie not found" because cookies are scoped to the wrong hostname. **Fix**: Override `keycloak.ingress.hostname` (not `keycloak.hostname` — ignored by upstream rhbk) to `sso.{{ .Values.global.localClusterDomain }}` in `values-hub.yaml` so Keycloak issues cookies/tokens on `sso.`. Empty ingress hostname falls back to `keycloak.`; a wrong key left Cluster Bot installs on `keycloak.apps.example.com`. Update console links to use `sso.` as well. ### OIDC blocking frontend API calls AuthPolicy on `neuroface-app-lb` HTTPRoute blocks XHR calls to `/api/health` (and other backend endpoints). The frontend badge logic needs `/api/health` to determine cluster identity (HUB/EAST/WEST). **Fix**: Add `kuadrant.oidc.enabled` flag (default `false`) in the neuroface-gateway chart. When false, exclude `neuroface-app-lb` from gateway OIDC realms list. Keep OIDC only on `neuroface-cv-lb` (the CV inference route). ### YOLO PPE S3 403 (Minio credential mismatch) GitLab's bundled Minio auto-generates access keys at install time. Hardcoded `minio/minio123` credentials in the YOLO PPE inference container don't match. **Fix**: Set `useVault: true` in `spoke-neuroface-cv` values for model storage. Add Minio credential extraction and Vault sync to the `gitlab-bootstrap` job so the real credentials propagate to spokes via ESO. ### East spoke deployment blockers Multiple issues deploying spokes: - `vault` namespace missing on spoke (ESO fails) - Duplicate Job resources: ArgoCD hook annotations (`argocd.argoproj.io/hook: PostSync`) cause ArgoCD to count them separately from the regular Job, creating conflicts - `istio-system` namespace not auto-created on spokes (Istio operator expects it) **Fix**: Remove ArgoCD hook annotations from the `download-model` Job (let it run as a normal sync resource). Pre-create missing namespaces (`vault`, `istio-system`) in `clusterGroup.namespaces`. For manual emergency deploys: `helm template charts/all/spoke-neuroface-cv | oc apply -f -`. ### Skupper east connection Spokes need a Skupper `Site` CR and an `AccessToken` to link back to the hub. The `hub-interconnect` chart has an `accesstoken-sync` CronJob that pushes tokens to spokes via ACM `ManagedClusterAction`. Verify the CronJob runs successfully after spoke import completes. ### Rate limiting Default Kuadrant `RateLimitPolicy` of 30 req/min is too low for the demo (multiple users hitting the CV inference endpoint simultaneously). Increased to 120 req/min in `neuroface-gateway/values.yaml`. ## Lessons learned from OIDC self-service, live-incident response, and systematic bug-hunting (Jul 1, 2026) ### Duplicate cluster-singleton CRs caused a full security outage `neuroface-gateway` and `workshop-kuadrant-apis` each declared their own `Kuadrant` CR (in different namespaces) plus their own `Authorino`/`Limitador` CRs. The kuadrant-operator always deploys the managed Authorino/Limitador into its own install namespace (`redhat-connectivity-link-operator`) regardless of which `Kuadrant` CR's namespace triggered it, so the two declarations raced for the same underlying objects. When one Application got recreated during a routine ArgoCD sync, it took down Authorino/Limitador **cluster-wide** — every `AuthPolicy`/`RateLimitPolicy` in the cluster (NeuroFace CV OIDC, AI Gateway auth, workshop API rate limits) started failing with 500s until manually restored. **Fix**: declare cluster-singleton CRs (Kuadrant, Authorino, Limitador, DataScienceCluster, Kiali, etc.) in exactly ONE chart. Never let two Applications both create "the" instance of something the underlying operator treats as global. **Prevention technique** — systematic duplicate-object scan: render every local chart with `helm template` (once with `clusterRole=hub`, once with `clusterRole=spoke`, supplying dummy `global.*` values) and cross-reference every `(kind, namespace, name)` tuple across all chart outputs. Any tuple produced by more than one chart is a latent version of this exact bug. ### Duplicate Subscription silently strips resource-limit overrides `values-hub.yaml`'s `clusterGroup.subscriptions.gitlab` (rendered by the shared `clustergroup` chart, which only supports `spec.config.env`) and `charts/all/gitlab-operator/templates/subscriptions.yaml` (which supports `spec.config.resources`, needed because gitlab-operator's OLM default 300Mi limit OOMs while reconciling the full GitLab chart) both targeted the same `Subscription` object. The clustergroup-rendered one always won and silently stripped the `config.resources` override on every sync, leaving the operator permanently CrashLoopBackOff. **Fix**: if a local chart needs `spec.config.resources` (or any field the shared `clustergroup` chart's Subscription template does not render — it only supports `name/namespace/channel/source/config.env`), do NOT also declare that Subscription under `clusterGroup.subscriptions` in the values file. Pick exactly one owner. ### KeycloakRealmImport does not reconcile existing realms Adding a new client to `spec.realm.clients` in a `KeycloakRealmImport` has **no effect** if the realm already exists in the cluster (confirmed: the operator only imports on first creation). For any realm/client change on a live cluster, create/update it directly via the Keycloak Admin REST API, then keep the Helm source updated for future fresh installs (which do get the client, since the realm doesn't exist yet there). ### KeycloakRealmImport placeholders require backing Secrets to exist BEFORE import — PostSync Jobs fix the race The `$(PLACEHOLDER_NAME)` mechanism in `KeycloakRealmImport` resolves secrets at import time only. If the backing K8s Secret (from ESO/Vault) is not yet synced when the operator processes the import, the literal placeholder string (e.g. `$(CLIENT_SECRET_USER1)`) is stored as the actual client secret in Keycloak — causing `unauthorized_client (Invalid client or Invalid client credentials)` on every OAuth/OIDC flow until manually fixed. This is a **race condition inherent to the Keycloak operator's design** (one-shot import, no reconcile), not a sync-wave ordering issue (ESO may take seconds-to-minutes after the Secret Store connects). **Fix**: PostSync Jobs that idempotently read the real secret from the K8s Secret and PUT it into Keycloak via Admin REST API after every ArgoCD sync: - `charts/all/rhbk-iam/templates/job-sync-client-secrets.yaml` — syncs all per-user clients (`client-{realm}-user{N}`) across cv/maas/neuroface realms plus the `backstage-provisioner` service account in cv - `charts/all/developer-hub/templates/job-sync-backstage-realm-secrets.yaml` — syncs `developer-hub`, `developer-hub-catalog`, `devspaces` clients in the backstage realm Both Jobs are idempotent (skip if secret already matches), graceful (exit 0 if backing Secret not ready yet), and share the `rhbk-iam-secret-sync` ServiceAccount in `keycloak-system`. ### RHDH login secret must share the exact same Vault path as other clients using it `keycloak-realm.yaml` had the `developer-hub` (login) client's secret rendered directly from a Helm value (`.Values.keycloakOidcClientSecret`), while RHDH's actual runtime `OIDC_CLIENT_SECRET` env var came from Vault via ExternalSecret. These diverged, causing `unauthorized_client (Invalid client or Invalid client credentials)` on every login attempt. **Fix**: any Keycloak client whose secret RHDH (or another workload) reads from a Vault-backed Secret must resolve that same secret via the KeycloakRealmImport `spec.placeholders` mechanism (`$(PLACEHOLDER_NAME)` in the client's `secret:` field, backed by `secret: {name, key}` pointing at the SAME K8s Secret), never a separate hardcoded Helm value. See `charts/all/developer-hub/templates/keycloak-realm.yaml`. ### Scaffolder template `spec.output.text` schema Backstage/RHDH scaffolder templates require `spec.output.text` to be an array of `{title, content}` objects, not plain strings. A template with `output.text: ["some string"]` is **silently rejected by the catalog processor** (`ScaffolderEntitiesProcessor` warning, `/spec/output/text/0 must be object`) — the template never appears in the catalog at all, with no error surfaced to a casual observer. ### RBAC gaps in hook Jobs that `oc exec`/`oc get pods -l` cross-namespace Two hook Jobs (`gitlab-token-setup`, `job-gitlab-bootstrap`) execute into pods or list pods by label in a namespace their `Role` never granted `pods`/`pods/exec` for. One failed hard (`BackoffLimitExceeded`, blocking the whole app sync) because the script used `set -e` without guarding the failing command; the other failed silently (stderr redirected, checked with `if [ -n ... ]`) and just printed a misleading "WARN: could not write to Vault" on every run even though nothing was wrong. **Prevention technique**: `grep -rl "oc exec\|oc get pods" charts/all/*/templates/*.yaml`, then for each hit, extract every `-n ` the script touches and confirm the Job's ServiceAccount has a `Role`/`RoleBinding` in that exact namespace (not just its own). ### Shared-namespace lists must not concat hub-only with spoke-only namespaces `charts/all/observability/templates/all.yaml` built its Istio mesh monitoring namespace list by starting with the hub-only namespace (`neuroface-gateway-system`) and then, when `clusterSuffix` was set (i.e., on spokes), **concatenating** the spoke-only namespaces (`neuroface`, `neuroface-cv`) on top instead of replacing. Every spoke install tried (harmlessly, since the namespace doesn't exist there, but uselessly) to create a `PodMonitor`/`RoleBinding` in `neuroface-gateway-system`. **Fix**: namespace sets that are mutually exclusive per cluster role must be selected with `if/else`, never built by concatenation. **Prevention technique**: render every chart actually declared in `values-east.yaml`/`values-west.yaml` `clusterGroup.applications` with `clusterRole=spoke`, collect every `metadata.namespace`, and diff against `values-east.yaml`/`values-west.yaml` `clusterGroup.namespaces`. Anything used-but-undeclared is a bug (this is exactly how the `imperative` namespace gap — see `vp-helm-values` skill — and this one were both found). ### values-secret.yaml.template must declare every Vault path an ExternalSecret reads A new feature's `ExternalSecret` (`keycloak/realms/cv/backstage-provisioner`) worked in the live cluster only because the secret was populated manually during development. It was never added to `values-secret.yaml.template`, so `./pattern.sh make load-secrets` on a fresh install would never create it. **Prevention technique**: for every new `ExternalSecret`, grep its `remoteRef.key` + `remoteRef.property` and confirm a matching `secrets[].name` + `fields[].name` entry exists in `values-secret.yaml.template`. Do this for the whole file periodically, not just new additions — see the systematic script in the `vp-helm-values` skill's pitfall list. ### CEL expressions in Kuadrant policies: single quotes for string literals `request.headers["x-forwarded-for"]` (double quotes) in a `RateLimitPolicy` counter expression crashes Limitador (`Invalid limit file: ... Syntax error`, CrashLoopBackOff) because Limitador's descriptor-file serialization mangles a double-quoted string literal nested inside an already double-quoted CEL expression. Use single quotes: `request.headers['x-forwarded-for']`. ### Windows/git-bash environment notes (when operating this repo from Cursor on Windows) - `oc exec ... -- sh -c '...'` (not a bare path) avoids git-bash mangling a leading `/` into a Windows drive path. - Redirect command output to a relative path (`./tmp-work/file.yaml`) with `working_directory` set explicitly, not `/tmp/...` — git-bash sometimes translates `/tmp` to `C:/Program Files/Git/tmp` depending on the tool invocation path. - Files extracted from a live ConfigMap via `oc get -o jsonpath` on Windows can pick up CRLF line endings; strip `\r\n` → `\n` before re-applying as a `data` key, or `oc apply`/`oc patch` may silently place the content under `binaryData` instead of `data`. ## Lessons learned from Backstage proxy auth + third duplicate-singleton-CR incident (Jul 1, 2026) ### Backstage's proxy plugin silently drops the Authorization header unless allowlisted The `oidc-credentials-self-service` scaffolder template authenticates against Keycloak (`/proxy/keycloak/.../token`, no auth needed) and then calls Keycloak's Admin REST API (`/proxy/keycloak/admin/realms/cv/clients`) with `headers.Authorization: Bearer `. Every one of those admin-API steps failed with a **real** (Keycloak-issued) `401 {"error":"HTTP 401 Unauthorized"}`, not a Backstage auth error — Keycloak was receiving the request with no `Authorization` header at all. **Root cause**: Backstage's proxy-backend plugin only forwards "CORS-safe" headers to the proxy target by default; `Authorization` is intentionally excluded unless the endpoint config adds `allowedHeaders: ['Authorization']` (and normally also `credentials: forward`/`dangerously-allow-unauthenticated`). This is *silent* — no error in RHDH logs, the caller just gets whatever the upstream target replies for "no credentials". **Diagnosis technique that cut through it fast**: fetch a token, then compare `curl -H "Authorization: Bearer $T" ` (200) vs `curl -H "Authorization: Bearer $T" https://developer-hub.../api/proxy/` (401) vs the SAME proxied call with no header at all (401, identical body). If the with-header and without-header proxied responses are byte-identical, the proxy is dropping the header — check `allowedHeaders` on that `proxy.endpoints` entry. **Fix**: any `proxy.endpoints.'/x'` block whose callers need to pass a dynamic, per-request Bearer token through to the target (as opposed to a static one baked in via `headers: {Authorization: ${ENV}}`) needs `allowedHeaders: ['Authorization']` explicitly. ### Third instance of the "two charts, one cluster-singleton CR" bug: Kiali `charts/all/observability` declared its own `Kiali` CR (`name: kiali`, `openshift-cluster-observability-operator` namespace, `cluster_wide_access: true`) *and* the external validatedpatterns `servicemesh` chart (the `servicemesh-config` Application, chart pulled straight from `https://charts.validatedpatterns.io`, **not** vendored locally) already provisions its own cluster-wide `Kiali/default` in `istio-system`. The Kiali operator only allows one `cluster_wide_access: true` instance cluster-wide, so ours always lost (`status.conditions[type=Failure]`: `"already installed with deployment.cluster_wide_access set to true"`) and never got a Route. Both charts *also* separately declared the identical cluster-scoped `ClusterRoleBinding/kiali-monitoring-rbac` (same name, different `subjects[].namespace`), so the two Applications fought over its `subjects` on every sync. Three other places had the dead route's hostname hardcoded and needed fixing in lockstep: `console-links` (the "Kiali Service Mesh" ConsoleLink), and `developer-hub.kialiEndpoint` helper (feeds the in-app Kiali plugin's `url`, the `dynamic-plugins-rhdh` Kiali backend provider `url`, and the `plugin-readiness` hook's Kiali reachability probe — the readiness Job kept failing `DeadlineExceeded` and blocking every future ArgoCD sync of `developer-hub` until this was found). **Fix**: deleted our `Kiali` CR + its `ClusterRoleBinding`; pointed the helper/console-link at `kiali-` i.e. `kiali-istio-system`. **Prevention technique**: when a chart is sourced from an *external* Helm repo (`repoURL` under `charts.validatedpatterns.io` or similar, not `path: charts/all/...`), you cannot `grep` it locally — `helm pull --repo --untar` it into `/tmp` (or `--version` if the exact version is pinned in `values-*.yaml`'s `chartVersion`) and diff its rendered `kind`s/names against every local chart, same as the local-only scan in the "Duplicate cluster-singleton CRs" lesson above. ### Gate demo-only `customResources` entries behind a real values flag, not a namespace collision `configmap-app-config-rhdh.yaml`'s `kubernetes.customResources` unconditionally listed Strimzi (`Kafka*`) and Camel K (`Integration`/`Kamelet`/`Pipe`) GVKs, neither of which this pattern installs by default. RHDH's kubernetes-plugin queries the *entire* configured list for every namespace tied to a Topology-visible entity, so every entity in `neuroface-gateway-system` produced 4 Strimzi + 3 Camel `NOT_FOUND` 404s in the Topology view forever. There was already a `plugins.kafka.enabled: false` values key... just nested one level too deep inside `iamCatalog:` (dead weight, nothing ever read `iamCatalog.kafka`), one screen-scroll below the real `plugins:` block, easy to miss. **Fix**: moved `kafka`/`camel` to genuine top-level values keys and wrapped both CR groups in `{{- if .Values.kafka.enabled }}` / `{{- if .Values.camel.enabled }}`. **Also caught in the same customResources block**: `tokenratelimitpolicies.kuadrant.io` was configured at `apiVersion: v1`, but `oc get crd tokenratelimitpolicies.kuadrant.io -o jsonpath='{.spec.versions[*].name}: served={.spec.versions[*].served}'` showed only `v1alpha1` is `served: true` (unlike `authpolicies`/`ratelimitpolicies`, already on `v1`) — a version bump landed in the CRD schema without yet flipping `served: true`, and nobody re-checked the sibling `customResources` entries when adding it. **Prevention technique**: for every `kubernetes.customResources` entry, `oc get crd . -o jsonpath='{range .spec.versions[*]}{.name}: served={.served}{"\n"}{end}'` and confirm the configured `apiVersion` is actually `served: true` — don't assume it matches siblings in the same group. ### After an argocd-application-controller pod restart, sync retries can look "stuck" for several minutes — this is normal Mid-session the `vp-gitops-application-controller-0` StatefulSet pod restarted (unrelated to anything we changed). For the next ~5-10 minutes every `Application` showed `operationState.phase: Running` with no progress and no new hook Jobs, because the controller was silently replaying "Resuming in-progress operation" for *every* Application in the cluster (visible in its logs) before it got to ours. `argocd app terminate-op ` (run via `oc exec deploy/-server -- sh -c 'export ARGOCD_OPTS=--core; argocd app terminate-op '`, since there's no local `argocd` CLI) followed by a plain re-sync got it unstuck once the controller had actually caught up — don't terminate-op immediately on the first sign of "stuck", it can make things worse mid-catch-up. **If you need the fix live *now* and don't want to wait on ArgoCD**: `helm template -f --set -o jsonpath='{.spec.source.helm.parameters}'`> --show-only