--- title: kubectl ome instance description: List the Instances of an InferenceService's OMENative components, inspect one Instance, show a component's retry blocks, and release a held revision (alpha). since: v1.3 --- `kubectl ome instance` shows the [Instances](../../concepts/omenative/instances.md) of an [InferenceService](../../concepts/serving/inference-services.md)'s [OMENative](../../concepts/architecture/deployment-modes.md#omenative) components. An Instance is one replica of a component: one pod, or a leader pod and its workers, which OME updates, repairs and reports as one unit. Use `list` to find an Instance that isn't `Ready`, `status` to see its pods and their warnings, and `retry-blocks` to see which revisions OME is retrying or holding, and why. Once you've fixed the cause, the alpha `release-held` lets OME try a held revision again. OME reports the Instances in the status of the InferenceReplica it creates for each component, named `-`, such as `chat-engine`. Only components with such an InferenceReplica appear: [Opt in to OMENative](../../concepts/architecture/deployment-modes.md#opt-in-to-omenative) shows how a component gets the mode. A standalone InferenceReplica — one you create directly — appears in no report, even when an InferenceService [fronts it](../../concepts/serving/inference-services.md#referenced-replicas): see [Which replicas the command reads](#which-replicas-the-command-reads). ```text kubectl ome instance SUBCOMMAND INFERENCESERVICE [flags] ``` | Subcommand | What it does | | --- | --- | | [`list`](#list) | Lists the Instances of every OMENative component. | | [`status`](#status) | Shows one Instance, with its pods and their Warning events. | | [`retry-blocks`](#retry-blocks) | Shows the retry blocks of one component. | | [`release-held`](#release-held) | Alpha. Asks OME to release one held revision. | `status`, `retry-blocks` and `release-held` act on one component, which `--component` names: `engine`, `decoder` or `router`. `-o json` and `-o yaml` print the whole report, as an `InstanceListReport`, `InstanceStatusReport`, `InstanceRetryBlocksReport` or `ActionResult`. Only `status` has a wide view: `release-held` treats `-o wide` as `table`. The overview covers the other [output formats](overview.md#output-formats) and the [permissions](overview.md#required-rbac) each command needs. `list` and `status` say how current the status they show is, with an evidence level: | Evidence | Meaning | | --- | --- | | `Reported` | The status is current. | | `Stale` | OME hasn't caught up with the latest change, so the rows can be out of date. | | `Malformed` | The status contradicts itself or holds invalid values. | | `Unavailable` | The CLI has no status it can use, as when OME hasn't written it yet. | ## Which replicas the command reads Every subcommand takes an InferenceService, never an InferenceReplica. It reads the InferenceService first, so naming a standalone replica — an [InferenceReplica](../../reference/api/ome.v1beta1.md#ome-io-v1beta1-InferenceReplica) you create directly, with no parent InferenceService — fails: ```bash kubectl ome instance list pool-a -n prod ``` ```output error: get InferenceService "prod/pool-a": inferenceservices.ome.io "pool-a" not found ``` The subcommands then keep only the InferenceReplicas that record the InferenceService as their parent: `spec.parentRef` names it, the `ome.io/inferenceservice` label carries its name, and their controller owner reference points at it. OME writes all three on every InferenceReplica it creates, so those appear. A replica that claims none of them isn't shown and raises no issue, whatever its name. Only a replica that claims the InferenceService but fails the identity checks is reported as left out, as `REJECTED` or `ID_REJECT`. A standalone replica claims no InferenceService, so no subcommand shows its Instances. Read them from the replica itself: in `kubectl get inferencereplica pool-a -n prod -o yaml`, they're the rows of `status.instanceStatuses`, or of `status.instanceStatusColumns` when `status.instanceStatusEncoding` is `ColumnarV2`. [`kubectl ome get ir`](get.md#inferencereplicas-columns) finds standalone replicas: their PARENT column shows `-`. An InferenceService whose [`spec.replicaRefs`](../../concepts/serving/inference-services.md#referenced-replicas) fronts standalone replicas writes nothing on them, so they don't claim it, and it has no InferenceReplicas of its own. Its report is empty while the fronted pools serve: ```bash kubectl ome instance list front -n prod ``` ```output COMP IDX/INC PHASE PODS REVS AOF EVIDENCE - - - - - --- OK ``` The empty table isn't an error: the command read everything, found no InferenceReplica that claims `front`, and exits `0`. `status` and `retry-blocks` read the same collection, so they can't see the fronted replicas either. To inspect a fronted pool's Instances, read the replica itself, as above. ## `list` ```text kubectl ome instance list INFERENCESERVICE [flags] ``` `list` shows one row for each Instance of the InferenceService's OMENative components, engine first, then decoder and router, in index order. The rows come from the InferenceReplicas' status, so they show what OME last recorded. [`status`](#status) shows the live pods. ### Flags {#list-flags} | Flag | Default | Description | | --- | --- | --- | | `-o`, `--output` | `table` | Output format: table, json or yaml | Plus the standard kubeconfig flags, such as `-n` and `--context`. ### Output fields {#list-output-fields} | Column | What it shows | | --- | --- | | COMP | The component. | | IDX/INC | The Instance's [index](../../concepts/omenative/instances.md#index) and [incarnation](../../concepts/omenative/instances.md#incarnation), as `index/incarnation`. | | PHASE | The [Instance phase](../../concepts/omenative/instances.md#instance-phases): `Pending`, `Creating`, `Ready`, `Updating`, `Restarting`, `Migrating`, `Failed` or `Deleting`. | | PODS | The Instance's pods as OME last counted them: `serving/available/total`, as [Readiness and availability](../../concepts/omenative/instances.md#readiness-and-availability) defines them. | | REVS | The revision the Instance runs and the one it's moving to: `running>target`, `>target` before any revision runs, or the running revision alone. A long value keeps its first and last 4 characters, as in `chat...4d2b`. To spot an update, check PHASE and the `O` flag, or run `status`. | | AOF | Three flags. `A`: the Instance is admitted, with pods and no scheduling gate. `O`: an [operation](../../concepts/omenative/instances.md#operations) is under way. `F`: OME recorded a failure. `-` when a flag is off. | | EVIDENCE | How current the component's status is. | EVIDENCE shows one value for each component: | Value | Meaning | | --- | --- | | `OK` | The status is current. | | `SPARSE` | The status is current, and the indexes have gaps, as after a migration. | | `EMPTY` | The status is current, with no rows, as for a component scaled to zero. | | `STALE:/` | OME last observed an older generation of the InferenceReplica. The rows still show. | | `STALE:PARENT` | The InferenceReplica reflects an older generation of the InferenceService. The rows still show. | | `NOT-OBSERVED` | OME hasn't written status for the InferenceReplica yet. | | `NOT-REPORTED` | The status counts replicas but lists no Instances. | | `PARENT-UNAVL` | The InferenceReplica doesn't record which InferenceService generation it reflects. | | `ROWS-LIMIT` | The component's rows didn't fit in the table, so none of them show. | | `BAD:` | The status is invalid. `DUP-COMP`: a second InferenceReplica claims the component. `DUP-IDX`: two rows share an index. Any other problem names the invalid value: `GEN` (generation), `PARENT` (InferenceService generation), `REVISION`, `COUNTS` (pod or replica counts), `PHASE` or `INSTANCE` (index or incarnation). `COUNTS` can appear briefly while a pod stops serving, as during an update. | | `BAD`, `UNAVAILABLE` | The status is malformed or unavailable, with no more specific value. | A component with no rows to show gets one row of dashes with its EVIDENCE, such as `NOT-OBSERVED`. A problem with the whole collection adds a last row, `summary`. It shows `SOURCE-UNAVL` (the CLI couldn't list or decode an InferenceReplica), `REJECTED` (it left out an InferenceReplica that failed the identity checks) or `TRUNCATED` (there's more than the table shows). With no rows at all, the table shows one row of dashes with a `summary` value or the overall state: `OK`, `PARTIAL` or `UNAVAILABLE`. `-o json` and `-o yaml` print every value in full, and every issue behind the EVIDENCE values. ### Examples {#list-examples} List the Instances of `chat` in `prod`: ```bash kubectl ome instance list chat -n prod ``` ```output COMP IDX/INC PHASE PODS REVS AOF EVIDENCE engine 0/1 Ready 1/1/1 chat...ne-a A-- OK ``` The engine has one Instance, index 0 in its first incarnation. It's `Ready` on the revision `chat-engine-a`, cut to 11 characters, with its one pod serving and available. When the InferenceService also has a decoder that's updating, and OME hasn't caught up with its latest change, the table looks like this: ```output COMP IDX/INC PHASE PODS REVS AOF EVIDENCE engine 0/1 Ready 1/1/1 chat...ne-a A-- OK decoder 3/8 Updating 1/1/4 chat...-new AO- STALE:3/4 ``` The decoder's Instance 3 is in its eighth incarnation and moving from `chat-decoder-old` to `chat-decoder-new`, with an operation under way. One of its four pods is serving and available. `STALE:3/4` means OME last observed generation 3 of the decoder's InferenceReplica, which is now at generation 4. ## `status` ```text kubectl ome instance status INFERENCESERVICE INDEX --component COMPONENT [flags] ``` `status` shows one Instance: the row OME reports for it, the migrations it takes part in, and its pods with their Warning events. INDEX is the Instance's [index](../../concepts/omenative/instances.md#index), the first number under IDX/INC in `list`. The pod and event rows are live. The phase, revisions, pod counts, conditions, operation and last failure are what OME last recorded. `status` first works out the component's deployment mode, from the `ome.io/deploymentMode` annotation or else, as OME does, from the InferenceService and its runtime. A component in another mode gets the state `NotOMENative`. `status` finds the Instance's pods by their `ome.io/instance-index` label. When an Instance has more than 16 pods, it shows the 16 most likely to explain a problem, such as pods being deleted or not Ready. It shows the reason and count of each pod's Warning events, but not their messages: `kubectl describe pod` shows those. ### Flags {#status-flags} | Flag | Default | Description | | --- | --- | --- | | `--component` | None | Component: engine, decoder or router (required) | | `--ome-namespace` | `ome` | Namespace where the OME control plane is installed | | `-o`, `--output` | `table` | Output format: table, wide, json or yaml | Plus the standard kubeconfig flags, such as `-n` and `--context`. ### Output fields {#status-output-fields} The first row, `state`, gives the report's state, the Instance and the component's evidence level, as in `Reported engine[0] evidence=Reported`. The state is one of: | State | Meaning | | --- | --- | | `Reported` | OME reports the Instance on current evidence, and the CLI read everything it looked for. | | `Partial` | Part of the report is stale, malformed or missing, as when the CLI couldn't read the pods. The `issue` rows say what. | | `Missing` | The component's InferenceReplica has no row with that index. | | `NotProjected` | The component has no InferenceReplica that [claims the InferenceService](#which-replicas-the-command-reads). | | `NotOMENative` | The component doesn't run in OMENative. | | `Unavailable` | The CLI couldn't read the InferenceReplicas, or has nothing it can use for the component, as when OME hasn't written its status yet. | The `deployment` and `encoding` rows always show. The others appear when they have a value: | Field | What it shows | | --- | --- | | `deployment` | The deployment mode, how it was set (`source=`), where the CLI read it (`origin=`), and its evidence level. | | `encoding` | The status encoding, `DenseV1` or `ColumnarV2`, its evidence level, and why it's unavailable, if it is. | | `instance` | The InferenceReplica, the incarnation, the phase, and whether the Instance is admitted. | | `revisions` | The revision the Instance runs, and the one it's moving to. | | `persisted` | The pod counts OME wrote: total, serving and available. | | `lifecycle` | The Instance's active pod-name slot, 0 or 1 (`activeOrdinal=`), and when it last became Ready (`readySince=`). | | `condition` | One row for each condition: its type, status, observed generation, evidence level and reason. | | `announced` | One row for each warning OME has raised once for the Instance, as `reason@episode`, such as `GangSplitRisk@#1`. | | `operation` | The operation under way: its type, ID, step and retry count. | | `op target` | The revision the operation moves to, and why. | | `op timing` | When the operation started, when it last made progress, and its deadline. | | `op hold` | What holds the operation back and when quota admission last refused one of its pods, as `waiting= refused=