# MCP Gateway Official Chart Migration - READY_FOR_REVIEW

**Status:** READY_FOR_REVIEW
**Date:** 2026-11-09
**Priority:** P0 - Official chart migration required before MVP backend deployment
**Current:** app-template based deployment (commit a146c5ee9, healthy)
**Target:** Official agentgateway-standalone v1.7.0 chart

---

## Chart Provenance (VERIFIED)

**Official Chart:** `oci://cr.agentgateway.dev/charts/agentgateway-standalone:1.7.0`
**Digest:** `sha256:8c3fcfb7f35366b86ba9ddbac30df3f542dcca1cd089092227e6f24c33742947`
**Source:** https://github.com/agentgateway/agentgateway
**Docs:** https://agentgateway.dev/docs/standalone/latest/documentation/setup/install/helm/

**Verification:**
- Chart pulled and validated against standalone binary v1.7.0
- NOT the Gateway API control plane chart
- Matches agentgateway/agentgateway GitHub repository
- Helm template renders correctly with proper values structure

---

## Critical Migration Issues

### 1. Session Key Incompatibility (BLOCKER - RESOLVED)

**Problem:**
- Current ESO secret has key `SESSION_KEY`
- Official chart expects key `key` (hardcoded in deployment.yaml:86)
- Cannot override via values

**Solution:**
- Updated `templates/session-key-generator.yaml`: `secretKeys: [key]`
- Migration requires deleting existing ExternalSecret and Secret
- ESO will recreate with correct key name on next sync

**Impact:**
- In-flight MCP sessions invalidated (acceptable for foundation with no routes)
- Owner accepted short outage for migration

### 2. Deployment Selector Change (IMMUTABLE FIELD)

**Current selector:**
```yaml
app.kubernetes.io/controller: agentgateway
app.kubernetes.io/instance: mcp-gateway
app.kubernetes.io/name: mcp-gateway
```

**New selector:**
```yaml
app.kubernetes.io/name: agentgateway-standalone
app.kubernetes.io/instance: mcp-gateway
app.kubernetes.io/component: standalone
```

**Impact:**
- Deployment selector is immutable - cannot update in place
- ArgoCD will delete old Deployment, create new one
- Short outage during transition (acceptable)
- Service DNS name unchanged (fullnameOverride: mcp-gateway)

### 3. Service Ports (PRESERVED)

**Current:** 3000 (http), 15020 (metrics), 15021 (healthz)
**New:** 3000 (http), 15020 (metrics), 15021 (healthz)

No change - Service DNS and port numbers preserved. Ingress backend reference unchanged.

### 4. Monitoring: ServiceMonitor → PodMonitor

**Current:** ServiceMonitor (app-template)
**New:** PodMonitor (official chart)

**Impact:**
- Brief monitoring gap during transition (acceptable)
- Prometheus continues scraping same endpoint (15020/metrics)
- PodMonitor has `release: monitoring` label for Prometheus selector

### 5. ConfigMap Name (PRESERVED)

**Current:** `mcp-gateway-config`
**New:** `mcp-gateway-config` (fullnameOverride)

No change - ConfigMap name preserved.

### 6. NetworkPolicy

**Current:** Gated by `networkIsolation: false` (not rendered)
**New:** Chart has no NetworkPolicy templates

**Migration:** None needed - no NP rendered in either case.

---

## Resource Identity Changes

| Resource | Current | Official | Impact |
|----------|---------|----------|--------|
| Deployment name | mcp-gateway | mcp-gateway | None |
| Deployment selector | controller=agentgateway | component=standalone | **Recreate** |
| Service name | mcp-gateway | mcp-gateway | None |
| Service ports | 3000/15020/15021 | 3000/15020/15021 | None |
| ConfigMap name | mcp-gateway-config | mcp-gateway-config | None |
| Secret name | mcp-gateway-session-key | mcp-gateway-session-key | None |
| Secret key | SESSION_KEY | key | **Recreate** |
| ServiceMonitor | mcp-gateway | (deleted) | **Prune** |
| PodMonitor | (none) | mcp-gateway | **Create** |
| Ingress | mcp-gateway | (wrapper template) | None |

---

## Files to Commit

### apps/mcp-gateway/
- `Chart.yaml` - Switch dependency to agentgateway-standalone
- `values.yaml` - Official chart values (replace current, nested under `agentgateway-standalone:`)
- `templates/session-key-generator.yaml` - Change secretKeys to `[key]`
- `templates/session-key-externalsecret.yaml` - No change (ESO manages lifecycle)
- `templates/kanidm-group.yaml` - No change
- `templates/kanidm-oauth2-client.yaml` - No change
- `templates/ingress.yaml` - Wrapper template for Ingress (chart has none)
- `templates/configmap.yaml` - Delete (chart provides it)
- `templates/networkpolicy.yaml` - Delete (chart has no NP, networkIsolation=false)

### Not tracked (gitignored)
- `Chart.lock` - Generated by helm dependency build
- `charts/` - Vendored chart tarball (not committed)

---

## Validation Results

### helm template
```bash
helm template mcp-gateway apps/mcp-gateway/ --namespace mcp-gateway --include-crds
```
✓ Renders 11 resources:
- 1 ServiceAccount
- 1 ConfigMap (mcp-gateway-config)
- 1 Service (ClusterIP, ports 3000/15020/15021)
- 1 Deployment (image v1.7.0, selector component=standalone)
- 1 PodMonitor (release: monitoring)
- 1 Ingress (wrapper template)
- 1 ExternalSecret (session key)
- 1 Password generator (secretKeys: [key])
- 1 KanidmGroup
- 1 KanidmOAuth2Client

✓ SESSION_KEY env var references secret with key `key`
✓ nodeSelector: kubernetes.io/hostname: prusik
✓ resources: 10m/64Mi requests, 256Mi limits
✓ securityContext: runAsNonRoot, readOnlyRootFilesystem, drop ALL
✓ config.yaml: adminAddr: "off", gateways.default.port: 3000, mcp.targets: []
✓ No NetworkPolicy rendered

### helm lint
```bash
helm lint apps/mcp-gateway/
```
✓ 1 chart(s) linted, 0 chart(s) failed

### pre-commit
```bash
pre-commit run --files apps/mcp-gateway/Chart.yaml apps/mcp-gateway/values.yaml apps/mcp-gateway/templates/session-key-generator.yaml apps/mcp-gateway/templates/ingress.yaml
```
✓ All checks passed (yamllint, helmlint, etc.)

---

## Rollback Plan

**If migration fails:**
1. Revert to commit a146c5ee9 (app-template chart)
2. ArgoCD sync (prune new resources, recreate old)
3. ESO secret recreated with key `SESSION_KEY`
4. Deployment recreated with old selector
5. Verify pod starts, Service DNS resolves, Ingress routes

**Rollback impact:** Short outage during resource recreation (acceptable)

---

## Migration Execution Steps

**Pre-deployment:**
1. Commit whitelist (see above)
2. Push to origin/master
3. Wait for ArgoCD to detect changes

**Deployment:**
1. Delete existing ExternalSecret:
   ```bash
   kubectl --context=grigri delete externalsecret mcp-gateway-session-key -n mcp-gateway
   ```
2. Delete existing Secret:
   ```bash
   kubectl --context=grigri delete secret mcp-gateway-session-key -n mcp-gateway
   ```
3. ArgoCD sync with prune=true (already configured)
   - Old Deployment deleted (selector mismatch)
   - New Deployment created
   - Old ServiceMonitor pruned
   - New PodMonitor created
   - ESO recreates Secret with key `key`

**Post-deployment verification:**
1. Verify new Deployment created:
   ```bash
   kubectl --context=grigri get deployment mcp-gateway -n mcp-gateway
   ```
2. Verify pod starts, Ready 1/1:
   ```bash
   kubectl --context=grigri get pods -n mcp-gateway -l app.kubernetes.io/component=standalone
   ```
3. Verify ESO secret created with key `key`:
   ```bash
   kubectl --context=grigri get secret mcp-gateway-session-key -n mcp-gateway -o jsonpath='{.data}' | python3 -c "import sys, json; print(list(json.loads(sys.stdin.read()).keys()))"
   ```
4. Verify Service DNS resolves:
   ```bash
   kubectl --context=grigri run -it --rm debug --image=busybox -- nslookup mcp-gateway.mcp-gateway.svc.cluster.local
   ```
5. Verify Ingress routes (404 expected, no backends):
   ```bash
   curl -I https://mcp.grigri.cloud/
   ```
6. Verify TLS valid:
   ```bash
   curl -vI https://mcp.grigri.cloud/ 2>&1 | grep "SSL certificate verify ok"
   ```
7. Verify no NetworkPolicy:
   ```bash
   kubectl --context=grigri get networkpolicy -n mcp-gateway
   ```
8. Verify PodMonitor created:
   ```bash
   kubectl --context=grigri get podmonitor mcp-gateway -n mcp-gateway
   ```
9. Wait 180s stable window
10. Capture UTC timestamps for deployment completion

---

## Owner Decisions Required

1. **Session key migration:** Delete existing secret (invalidates sessions) - **ACCEPTED**
2. **Deployment recreation:** Short outage during selector change - **ACCEPTED**
3. **Monitoring gap:** Brief gap during ServiceMonitor→PodMonitor - **ACCEPTED**
4. **No rollback to app-template:** Official chart is mandatory going forward - **CONFIRMED**

---

## Next Steps After Migration

1. **MVP Backend Deployment (blocked until migration complete):**
   - Grafana route: `/grafana/mcp` with scope `mcp_grafana`, native per-user Kanidm ACCESS JWT forwarded
   - Context7 route: `/context7/mcp` with scope `mcp_context7`, strip incoming Authorization header
   - Both routes stateless sessions
   - Require actual security evidence (JWT validation, scope enforcement, header stripping)

2. **Client Integration (separate agent):**
   - Hermes: 2 routes (`/grafana/mcp`, `/context7/mcp`)
   - OpenCode: 2 routes (`/grafana/mcp`, `/context7/mcp`)
   - Single client mcp-gateway audience (already resolved)

3. **Security CNP (after MVP working):**
   - CiliumNetworkPolicy for gateway egress
   - DNS-based FQDN allowlist (JWKS ingress 443, Grafana backend pod 3000, Context7 FQDN 443)
   - No broad world/443 allow
   - Separate follow-up, not part of this migration

---

## Blockers

**None.** Official chart migration is ready for review and deployment.

**Future blockers (not blocking this migration):**
- Grafana identity linking (separate agent will resolve)
- Context7 auth stripping proof (requires synthetic test against local HTTP MCP capture)
- Client integration (separate agent preparing)

---

## Summary

Official agentgateway-standalone v1.7.0 chart migration is complete and validated. All resources render correctly, helm lint passes, pre-commit passes. Migration requires deleting existing session secret (invalidates in-flight sessions, acceptable for foundation) and recreating Deployment (immutable selector change, short outage acceptable).

**Ready for main review release. No commits/push/deploy until released.**