---
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
sidebar-title: Production Deployments
---

# Production Deployments

This guide covers patterns for running AIPerf benchmarks in production environments -- CI/CD pipelines, Kueue-managed clusters, private registries, GitOps workflows, and multi-tenant setups.

---

## CI/CD Integration

### Detach Mode

In non-interactive environments (CI pipelines, cron jobs), AIPerf automatically detaches after deploying. You can also force this explicitly:

```bash
aiperf kube profile \
  --config benchmark.yaml \
  --image nvcr.io/nvidia/aiperf:latest \
  --detach
```

After deploying, poll for completion:

```bash
# Wait for the job to complete
while true; do
  PHASE=$(aiperf kube list my-benchmark 2>/dev/null | awk 'NR==2{print $3}')
  [ "$PHASE" = "Completed" ] || [ "$PHASE" = "Failed" ] && break
  sleep 30
done

# Download results
aiperf kube results my-benchmark --output ./artifacts
```

`aiperf kube results` exits nonzero when the job cannot be resolved or the
requested download is incomplete, so CI artifact steps fail instead of
silently publishing an empty or partial directory.

### JSON-Based Monitoring

For automated pipelines, use JSON output:

```bash
# Preflight check with exit code
aiperf kube preflight -o json
echo "Exit code: $?"    # 0 = all checks passed, 1 = failures

# Validation with structured output (takes AIPerfJob/AIPerfSweep CRs, not --config files)
aiperf kube validate -o json aiperfjob.yaml

```

### Example GitHub Actions Workflow

```yaml
jobs:
  benchmark:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4

      - name: Preflight check
        run: |
          aiperf kube preflight \
            --endpoint-url http://dynamo-agg-frontend.dynamo-server.svc:8000/v1 \
            -o json

      - name: Run benchmark
        run: |
          aiperf kube profile \
            --config benchmark.yaml \
            --image nvcr.io/nvidia/aiperf:${{ github.sha }} \
            --name "ci-${{ github.run_number }}" \
            --detach

      - name: Wait for completion
        run: |
          kubectl wait --for=condition=Complete \
            aiperfjob/ci-${{ github.run_number }} --timeout=60m

      - name: Collect results
        if: always()
        run: aiperf kube results ci-${{ github.run_number }} --output ./artifacts

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: benchmark-results
          path: ./artifacts/
```

---

## GitOps Workflows

### Generate Manifests for Version Control

Instead of deploying from the CLI, generate Kubernetes manifests and commit them to your GitOps repository:

```bash
# Generate an AIPerfJob CR (operator mode)
aiperf kube generate --operator \
  --config benchmark.yaml \
  --image nvcr.io/nvidia/aiperf:latest \
  > deploy/aiperfjob.yaml

# Or generate raw manifests (no operator needed)
aiperf kube generate --no-operator \
  --config benchmark.yaml \
  --image nvcr.io/nvidia/aiperf:latest \
  > deploy/manifests.yaml
```

Commit the generated YAML and let ArgoCD, Flux, or your GitOps tool apply it.

### Operator Mode vs. Raw Manifests

| Feature | Operator Mode | Raw Manifests |
|---------|--------------|---------------|
| Output | Single AIPerfJob CR | Namespace + RBAC + ConfigMap + JobSet |
| Requires | Operator installed | Only JobSet CRD |
| Monitoring | Operator tracks phase/progress | Manual pod watching |
| Results | Stored on operator PVC | Must retrieve before TTL |
| Cancellation | `spec.cancel: true` | Delete the JobSet |
| Conditions | Status conditions populated (`ConfigValid`, `EndpointReachable`, `PreflightPassed`, `ResourcesCreated`, `WorkersReady`, `BenchmarkRunning`, `ResultsAvailable`, `IndexUpdated`, `PreflightHasWarnings`, `Complete`, `Failed`) | None |

Operator mode also persists the active pod startup blocker in
`status.startupIssue`. This keeps the first-observed timestamp across operator
restarts and lets CI distinguish a retryable capacity delay from a stable
image, configuration, or placement failure before the terminal `Failed`
condition is set.

For production use, the operator mode is recommended.

---

## Kueue Gang-Scheduling

If your cluster uses [Kueue](https://kueue.sigs.k8s.io/) for resource management, AIPerf integrates with it for quota-managed gang-scheduling.

### Submit to a Kueue Queue

```bash
aiperf kube profile \
  --config benchmark.yaml \
  --image nvcr.io/nvidia/aiperf:latest \
  --queue-name gpu-benchmarks \
  --priority-class high-priority
```

Or in YAML:

```yaml
spec:
  scheduling:
    queueName: gpu-benchmarks
    priorityClass: high-priority
```

When a queue is specified:

1. The JobSet is created in a suspended state
2. Kueue evaluates quota availability
3. Once resources are available, Kueue unsuspends the JobSet
4. The operator detects the transition and monitors normally

`aiperf kube list` shows Kueue suspension status:

```bash
aiperf kube list
# Shows phase "Queued" while the JobSet awaits Kueue admission
```

---

## Private Registries

### Image Pull Secrets

If your AIPerf image is in a private registry:

```bash
# Create the pull secret
kubectl create secret docker-registry my-registry \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password='YOUR_TOKEN' \
  -n my-benchmarks

# Reference it when deploying
aiperf kube profile ... --image-pull-secrets my-registry
```

Or in YAML:

```yaml
spec:
  podTemplate:
    imagePullSecrets:
      - {name: my-registry}
```

### API Keys and Secrets

Pass API keys to the benchmark pods without embedding them in the config:

```bash
# Create a secret with your API key
kubectl create secret generic llm-api-key \
  --from-literal=api-key='sk-...' \
  -n my-benchmarks

# Reference it as an environment variable. Name the pod variable, and pass
# `<secret>/<key>` as its value.
aiperf kube profile ... \
  --env-from-secrets.OPENAI_API_KEY llm-api-key/api-key
```

`--env-from-secrets` is a mapping flag and accepts three equivalent spellings.
`--annotations`, `--labels`, and `--env-vars` accept the same three:

```bash
--env-from-secrets.OPENAI_API_KEY llm-api-key/api-key   # dot-notation
--env-from-secrets OPENAI_API_KEY=llm-api-key/api-key   # KEY=VALUE
--env-from-secrets '{"OPENAI_API_KEY": "llm-api-key/api-key"}'  # JSON object
```

Repeat the flag to set more than one entry.

Endpoint credentials are transported out of band from the benchmark
ConfigMap. AIPerf writes only redacted placeholders to `run_config.json`, then
each service restores the real values from Secret-backed environment
variables at startup. A credentialed endpoint without the matching
`valueFrom.secretKeyRef` mapping is rejected before the JobSet is created.

| Endpoint credential | Secret-backed pod environment variable |
|---|---|
| `endpoint.apiKey` | `AIPERF_INJECTED_API_KEY` or `OPENAI_API_KEY` |
| Sensitive headers such as `Authorization` or `X-API-Key` | `AIPERF_INJECTED_HEADERS` containing a JSON object of header strings |
| URL userinfo such as `https://user:password@host` | `AIPERF_INJECTED_ENDPOINT_URLS` containing a JSON list of full URL strings |

Do not use `--env-vars` for these values: literal pod environment values are
not Secret-backed and fail the credential transport check.

Or in YAML:

```yaml
spec:
  podTemplate:
    env:
      - name: OPENAI_API_KEY
        valueFrom:
          secretKeyRef:
            name: llm-api-key
            key: api-key
```

### Mounting Secret Files

For secrets that need to be files (e.g., certificates, tokens):

```bash
aiperf kube profile ... \
  --secret-mounts '{"name": "tls-cert", "mount_path": "/certs/ca.pem", "sub_path": "ca.pem"}'
```

`--secret-mounts` accepts the same shapes `--tolerations` does: a single JSON
object, a JSON array covering several mounts in one token, or the flag repeated
once per mount.

---

## Node Placement

### Target Specific GPUs

```bash
aiperf kube profile ... \
  --node-selector '{"nvidia.com/gpu.product": "NVIDIA-A100-SXM4-80GB"}'
```

### Tolerate GPU Taints

Many clusters taint GPU nodes. Add tolerations so benchmark pods can schedule there:

```bash
aiperf kube profile ... \
  --tolerations '[{"key": "nvidia.com/gpu", "operator": "Exists", "effect": "NoSchedule"}]'
```

Or in YAML:

```yaml
spec:
  podTemplate:
    nodeSelector:
      nvidia.com/gpu.product: "NVIDIA-A100-SXM4-80GB"
    tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
```

---

## Scaling Workers

AIPerf distributes workers across pods automatically. The `--total-workers` flag sets the total number of workers. The system places 10 workers per pod by default:

| `--total-workers` | Pods Created | Workers Per Pod |
|-----------------|-------------|-----------------|
| 10 | 1 | 10 |
| 50 | 5 | 10 |
| 100 | 10 | 10 |
| 200 | 20 | 10 |

A `--total-workers` value above the per-pod count that is not a multiple of it
cannot be expressed as a JobSet of identical pods, so it is rejected with an
error naming the two nearest usable totals. Keep the total a multiple of 10, or
set `benchmark.runtime.workersPerPod` to change the packing factor -- there is
no CLI flag for it. A total at or below the per-pod count runs on a single pod.

Only an authored total is rejected. When you omit `--total-workers` and
`benchmark.runtime.workers`, the count derived from `ceil(concurrency /
connectionsPerWorker)` is rounded up to a whole number of pods instead, since
failing on it would report a worker count you never chose.

Each worker maintains up to `connectionsPerWorker` concurrent requests (default: 100). When you pass `--total-workers` explicitly, the CLI derives this from your concurrency and worker count: `connectionsPerWorker = ceil(concurrency / workers)`. Left unset, the relationship inverts and the worker count is derived instead: `workers = ceil(concurrency / connectionsPerWorker)`.

For high-concurrency benchmarks, scale workers rather than increasing connections per worker. This distributes the load across pods and nodes.

---

## Results Server

The operator includes a results server sidecar that provides HTTP access to stored results. This powers `aiperf kube results` (which retrieves from the operator's PVC by default) and provides analytics endpoints backed by the SQLite runs index (`.aiperf_index.sqlite` on the results volume).

### Available Endpoints

After the operator is running:

```bash
# Port-forward to the results server
kubectl port-forward -n aiperf-system svc/aiperf-operator 8081:8081

# List stored results
curl localhost:8081/api/v1/results

# Get summary for a specific job
curl localhost:8081/api/v1/analytics/summary/my-benchmarks/my-benchmark

# Leaderboard across all benchmarks
curl localhost:8081/api/v1/analytics/leaderboard

# Compare two runs (repeat the jobs parameter, one per job)
curl "localhost:8081/api/v1/analytics/compare?jobs=run-a&jobs=run-b"

# Job history
curl "localhost:8081/api/v1/analytics/history?model=Qwen/Qwen3-0.6B"
```

### Storage Configuration

Results are stored on the operator's PVC. Configure retention in the Helm values:

```yaml
operator:
  env:
    resultsTtlDays: "30"           # auto-cleanup after 30 days
    resultsCompressOnDisk: "true"  # zstd compression

storage:
  enabled: true                    # default; backs results in a PVC so they survive pod restarts
  size: 1Ti                        # only used when enabled: true
  storageClassName: ""             # cluster default; only used when enabled: true
```

`spec.resultsTtlDays` overrides this default per AIPerfJob or AIPerfSweep. A
sweep persists the selected value in each epoch's aggregate, so cleanup remains
effective after Kubernetes deletes the parent CR. Cleanup also removes the
corresponding sweep index rows and repairs the sweep's `latest.txt` pointer.

### Exposing Results Outside the Cluster (Ingress)

By default the results server is reachable only via `ClusterIP + kubectl port-forward`. To expose it through an Ingress (e.g. for a shared dashboard link), enable `ingress.*` in the Helm values:

```yaml
ingress:
  enabled: true
  className: "nginx"          # empty uses the cluster default IngressClass
  annotations: {}
  hosts:
    - host: aiperf.example.com
      paths:
        - path: /
          pathType: Prefix    # backend port defaults to resultsServer.port (8081)
  tls: []                     # optional: list of {hosts: [...], secretName: ...}
```

The template lives at `deploy/helm/aiperf-operator/templates/ingress.yaml` and routes to the operator Service on `resultsServer.port`. Each path may override the backend port via `portNumber`.

---

## Environment Variables

Fine-tune benchmark behavior with environment variables in the pod template:

```yaml
spec:
  podTemplate:
    env:
      - name: AIPERF_HTTP_CONNECTION_LIMIT
        value: "200"
      - name: AIPERF_HTTP_KEEPALIVE_TIMEOUT
        value: "120"
```

Probe tunables such as `AIPERF_K8S_HEALTH_STARTUP_PERIOD_SECONDS` and
`AIPERF_K8S_HEALTH_STARTUP_FAILURE_THRESHOLD` do not belong here. They are read
by whichever process renders the JobSet -- the operator pod, or the CLI in
direct mode -- and baked into the pod spec's probe fields before the benchmark
container exists. Setting them under `spec.podTemplate.env` has no effect; set
them on the operator Deployment (or in the CLI's own environment) instead.

There is no `AIPERF_HTTP_TIMEOUT` variable; set the per-request timeout with
the `--request-timeout-seconds` CLI flag (or `request_timeout_seconds` in the
benchmark config) instead. See the
[Environment Variables Reference](../environment-variables.md) for the full list.

---

## Multi-Tenant Clusters

### Operator Scope and Namespace Ownership

An operator install is one of two things, decided by `operator.id`:

| | `operator.id` | Namespaces watched | Writes a claim |
|---|---|---|---|
| **Global operator** | `""` (default) | all, or `operator.watchNamespaces` | no |
| **Scoped operator** | a unique string | `operator.watchNamespaces` | yes |

A scoped operator claims each namespace it watches by writing a
`coordination.k8s.io/v1` Lease named `aiperf-operator` into it, and renews that
Lease on a timer. Every operator — the global one included — skips any
namespace whose Lease is held by a different identity and still fresh. Nothing
is set on the AIPerfJob itself, so a job cannot be submitted to the wrong
operator by forgetting a label.

This is what lets a scoped build run beside the production operator:

```bash
helm install aiperf-test deploy/helm/aiperf-operator \
  --namespace aiperf-system-test --create-namespace \
  --set operator.id=my-branch \
  --set 'operator.watchNamespaces={aiperf-test}'
```

The global operator stops reconciling `aiperf-test` as soon as the claim lands,
and resumes when the claim lapses. `aiperf kube list` shows the holder in its
`OWNER` column; `-` means the global operator owns that namespace, and `?`
means the claim could not be read (usually missing RBAC on `leases`).

The global operator caches namespace ownership and refreshes it every
`AIPERF_OPERATOR_CLAIM_LEASE_SECONDS / 3` seconds (~100 s by default). During
that window after a scoped operator starts, the global operator may briefly act
on events in a newly-claimed namespace before its cache updates.

Two scoped operators cannot claim the same namespace: the second one fails at
startup naming the current holder, rather than silently double-reconciling.

The claim's duration (`AIPERF_OPERATOR_CLAIM_LEASE_SECONDS`, default 300s) is
deliberately long, so a crash-looping operator keeps its namespaces across
restarts. Uninstalling a scoped operator lets its claims expire and the
namespaces fall back to the global operator; nothing needs to be deleted by
hand for ownership to transfer.

The `aiperf-operator` Lease object itself does survive the uninstall in each
watched namespace. It is expired, has no effect on ownership, and is only
visible in `kubectl get leases`. Remove it manually if you want it gone:

```bash
kubectl delete lease aiperf-operator -n <namespace>
```

### Namespace Isolation

Benchmarks run in the namespace you name with `--namespace`. For multi-tenant setups, give each team its own:

```bash
aiperf kube profile ... --namespace team-a-benchmarks --operator
aiperf kube profile ... --namespace team-b-benchmarks --operator
```

By default the operator watches all namespaces for AIPerfJob CRs; set
`operator.watchNamespaces` in the Helm values to restrict it to a fixed list,
and `operator.id` to also claim those namespaces away from a global operator
(see [Operator Scope and Namespace Ownership](#operator-scope-and-namespace-ownership)).
Each job gets its own RBAC scoped to its namespace. Here `--operator` bypasses cluster-scoped CRD
discovery, and the namespaces must already exist -- the CLI never issues a
Namespace-create request. A tenant therefore needs permission
to create `AIPerfJob` resources in its namespace, not permission to read CRDs or
create namespaces.

The per-job Role grants no create, update, or delete verbs, so a compromised
benchmark pod cannot launch workloads of its own. What it retains inside its
namespace is read access to pod, ConfigMap, Service, Endpoint, Event, and Job
metadata, plus `patch` on its own AIPerfJob and JobSet — which is why a
namespace per team, rather than a shared benchmark namespace, is still the
tenancy boundary. See
[RBAC and Security](rbac-security.md#operator-created-per-job-role).

If cluster policy forbids chart-managed cluster-scoped RBAC, install with
`rbac.create=false`, `serviceAccount.create=false`, an explicit
`serviceAccount.name`, and `tests.enabled=false`; that combination emits zero
`ClusterRole`/`ClusterRoleBinding` objects. See
[Eliminating all cluster-scoped RBAC](rbac-security.md#eliminating-all-cluster-scoped-rbac).

### Resource Quotas

The preflight checker validates resource quotas before deploying:

```bash
aiperf kube preflight --namespace team-a-benchmarks --workers 20
```

If the namespace has a ResourceQuota, preflight projects the total CPU and memory requirements and warns if deployment would exceed the quota.

---

## Operator Management

### Upgrading

```bash
helm upgrade aiperf-operator oci://ghcr.io/ai-dynamo/charts/aiperf-operator \
  --version X.Y.Z \
  --namespace aiperf-system
```

From a source checkout, pass `deploy/helm/aiperf-operator` in place of the OCI reference.

> **WARNING:** Drain active benchmarks before upgrading from a chart older than
> 0.8.0. Those versions rendered the benchmark `Role` and `RoleBinding` into a
> chart-owned `aiperf-benchmarks` namespace; the upgrade deletes that RBAC and
> recreates it in the release namespace. A benchmark running in
> `aiperf-benchmarks` at that moment loses its RBAC mid-flight and starts
> failing its pod, ConfigMap, and JobSet reads. Wait for in-flight runs to
> reach a terminal phase (`aiperf kube list`) before running `helm upgrade`.

### Uninstalling

```bash
helm uninstall aiperf-operator --namespace aiperf-system
```

Both CRDs carry `helm.sh/resource-policy: keep`, so the `AIPerfJob` and `AIPerfSweep` CRDs -- and any existing custom resources -- survive the uninstall. To remove everything:

```bash
kubectl delete crd aiperfjobs.aiperf.nvidia.com
kubectl delete crd aiperfsweeps.aiperf.nvidia.com
kubectl delete namespace aiperf-system
```

### Monitoring the Operator

Check operator logs:

```bash
kubectl logs -n aiperf-system -l app.kubernetes.io/name=aiperf-operator -f
```

The operator emits structured logs with job events, phase transitions, and error details.

### Debugging Failed Jobs

By default the operator cleans up JobSet pods once a job terminates, which makes postmortems hard. Set `keepFailedPods: true` on the AIPerfJob spec to preserve failed pod attempts for inspection:

```yaml
spec:
  keepFailedPods: true   # default: false; retains failed JobSet pods for kubectl logs/exec
```

This changes JobSet lifecycle (completed pods still get reaped via `ttlSecondsAfterFinished`); enable it only while diagnosing, then remove for steady-state runs.

### Deploying Before the Endpoint Is Reachable

The operator runs a TCP/HTTP reachability probe against `spec.benchmark.endpoint` before spinning up workers. If the inference server isn't live yet at deploy time (e.g. GitOps applies the CR before the model pod is Ready), set `skipEndpointCheck: true` to bypass the probe:

```yaml
spec:
  skipEndpointCheck: true   # default: false; skips the operator-side endpoint reachability probe
```

Workers will still fail fast if the endpoint never comes up -- this only suppresses the upfront check.

---

## Related Documentation

- [Getting Started](getting-started.md) -- First benchmark walkthrough
- [Deploy from a Source Checkout](source-checkout-deploy.md) -- Build and push AIPerf, Helm install the operator, and run on a real cluster
- [Kubernetes Configuration](configuration.md) -- All CRD fields and deployment options
- [Monitoring and Troubleshooting](monitoring.md) -- Watch, debug, and diagnose issues
- [Environment Variables](../environment-variables.md) -- All AIPERF_* environment variables