---
title: Ingress and external access
description: "How clients outside the cluster reach an InferenceService: the Service that OME creates by default, or the Kubernetes Ingress or Gateway API HTTPRoutes it creates when you turn on ingress."
---

With ingress creation on, OME exposes each [InferenceService](../serving/inference-services.md) outside the cluster through a [Kubernetes Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) or [Gateway API](https://gateway-api.sigs.k8s.io/) HTTPRoutes, and reports the address that clients use in `status.url`. Ingress creation is off by default. Until you turn it on, OME puts a Service in front of the model, and you decide how to expose it.

You set ingress with the `ome-resources` chart values under `ome.controller.ingressGateway`, which the chart writes to the `inferenceservice-config` ConfigMap in the `ome` namespace. [Configure ingress](../../guides/networking/configure-ingress.md) walks through the setup and lists every setting. The examples use an InferenceService named `llama-chat` in the namespace `llama-demo`.

## What OME creates for external access

Two of the values decide what OME creates for each InferenceService:

| `disableIngressCreation` | `enableGatewayAPI` | What OME creates | Use it when |
| --- | --- | --- | --- |
| `true`, the default | Either | [The external Service](#the-external-service) | You expose the model yourself, or keep it inside the cluster. |
| `false` | `false`, the default | [A Kubernetes Ingress](#through-a-kubernetes-ingress) | You run an ingress controller, such as ingress-nginx. |
| `false` | `true` | [HTTPRoutes](#through-the-gateway-api), and a default BackendTrafficPolicy | You run Envoy Gateway. You also get a shared hostname, several Gateways and route timeouts. |

The annotation `ome.io/ingress-disable-creation` overrides `disableIngressCreation` for one InferenceService: `"true"` turns ingress creation off, and any other value turns it on. `enableGatewayAPI` has no annotation, so it applies to every InferenceService. [Per-service overrides](../../guides/networking/configure-ingress.md#per-service-overrides) lists the annotations for the other settings.

OME copies the InferenceService's annotations to its Ingress or HTTPRoutes, so annotations for your ingress controller or Gateway go on the InferenceService. OME updates a route's annotations only when the route's spec changes too.

HTTPRoutes need v1.3 or later. On v1.2.2, `enableGatewayAPI: true` makes the ingress step fail for every InferenceService, so use an Ingress.

Turning ingress creation on deletes the external Service, so move clients in the cluster to a route, or to a component's Service such as `llama-chat-engine`. Turning it off, or switching between an Ingress and HTTPRoutes, leaves the old Ingress or HTTPRoutes serving traffic until you delete them, as [Turn off ingress creation](../../guides/networking/configure-ingress.md#turn-off-ingress-creation) shows.

The Ingress and HTTPRoutes send requests to the Services that OME creates for the InferenceService's [components](../serving/inference-services.md#components): `<name>-engine`, `<name>-decoder` and `<name>-router`. So they work the same for [OMENative](../omenative/overview.md) components as for `RawDeployment` ones. They also work the same when the InferenceService fronts standalone replicas with `spec.replicaRefs`; the Services then select the replicas' pods, as [Referenced replicas](#referenced-replicas) describes. Keep InferenceService names to 56 characters, or 55 with a decoder. With longer names, the routes point at Services that don't exist.

## The external Service

With ingress creation off, OME puts a `ClusterIP` Service named after the InferenceService in front of its router, or its engine when there's no router. [Cluster-local](#cluster-local-services) InferenceServices don't get one. The Service listens on the runtime's port, 8080 for `llama-chat`.

To reach the model from outside the cluster, set the annotation `ome.io/service-type` to `LoadBalancer` or `NodePort`, or put your own Ingress or Gateway in front of the Service. [Expose a service without ingress](../../guides/networking/expose-without-ingress.md) walks through it.

OME treats any Service with the InferenceService's name as its own, and rewrites or deletes it, so give your own Services other names.

## Through a Kubernetes Ingress

OME creates one Ingress, named after the InferenceService, with the IngressClass in `ome.controller.ingressGateway.ingressGateway.className`. The chart sets it to `istio`, so change it to your ingress controller's class, such as `nginx`. Each rule matches the path prefix `/` on a hostname of its own.

### Hostnames {#hostnames-and-urls}

With `domain: example.com`, the Ingress for `llama-chat` has these hostnames:

| Hostname | Sends requests to | When |
| --- | --- | --- |
| `llama-chat.llama-demo.example.com` | `llama-chat-router`, or `llama-chat-engine` when there's no router | Always |
| `llama-chat-router.llama-demo.example.com` | `llama-chat-router` | With a router |
| `llama-chat-engine.llama-demo.example.com` | `llama-chat-engine` | With a router or a decoder |
| `llama-chat-decoder.llama-demo.example.com` | `llama-chat-decoder` | With a decoder |

The hostnames come from `domainTemplate`, a Go template that defaults to `{{ .Name }}.{{ .Namespace }}.{{ .IngressDomain }}`. In the template, `.Name` is the InferenceService's name for the top-level hostname, and the component's Service name for the others.

## Through the Gateway API {since=v1.3}

To create HTTPRoutes, set these values in the `ome-resources` chart:

```yaml
ome:
  controller:
    ingressGateway:
      disableIngressCreation: false
      enableGatewayAPI: true
      domain: example.com
      omeIngressGateway: envoy-gateway-system/internal-gateway
```

- `domain` is the domain that the hostnames end in. The chart sets `svc.cluster.local`, so set a domain that your clients can resolve.
- `omeIngressGateway` is the Gateway that the HTTPRoutes attach to, as `<namespace>/<name>`. The chart leaves it empty, and the API server rejects HTTPRoutes without it.

Turning on `enableGatewayAPI` takes a controller restart. `helm upgrade` restarts the controller when the values change. If you edit the ConfigMap directly instead, restart it yourself:

```bash
kubectl rollout restart deployment ome-controller-manager -n ome
```

```output
deployment.apps/ome-controller-manager restarted
```

HTTPRoutes also need:

- a Gateway listener whose `allowedRoutes.namespaces.from` is `All`, or a `Selector` that matches the InferenceService's namespace. The default, `Same`, admits only routes in the Gateway's namespace;
- the Gateway API CRDs. If you install them after the controller starts, restart it, or `IngressReady` lags behind the Gateway;
- Envoy Gateway v1.7 or later, for the default BackendTrafficPolicy that OME writes for an InferenceService without [traffic intent](traffic-policy.md#defaults-when-you-declare-nothing).

If OME can't write a route or the policy, it still creates the workloads, but the InferenceService's status stops updating. [Troubleshooting](../../guides/networking/configure-ingress.md#troubleshooting) shows how to find the cause in the controller's log.

### Routes

For `llama-chat`, OME creates up to four HTTPRoutes, all with the hostname `llm.example.com` in the default scheme:

| HTTPRoute | Path prefix | Sends requests to |
| --- | --- | --- |
| `llama-chat` | `/llama-demo/llama-chat/` | `llama-chat-router`, or `llama-chat-engine` when there's no router |
| `llama-chat-engine` | `/llama-demo/llama-chat-engine/` | `llama-chat-engine` |
| `llama-chat-router` | `/llama-demo/llama-chat-router/` | `llama-chat-router`, with a router |
| `llama-chat-decoder` | `/llama-demo/llama-chat-decoder/` | `llama-chat-decoder`, with a decoder |

Clients use the top-level route, `llama-chat`. The other routes each reach one component, bypassing the router. OME creates a component's route once the component is ready. For an OMENative component, a [serving Instance](../omenative/instances.md#readiness-and-availability) is enough: OME creates and updates the route even while the component's `EngineReady`, `RouterReady` or `DecoderReady` condition is `False`, as long as the component's [lifecycle](../architecture/deployment-modes.md#lifecycle-status) reports `readyReplicas` and `servingReplicas` above `0`. So a component whose [condition](../architecture/deployment-modes.md#conditions) turns `False` with the reason `InsufficientAvailable` keeps its route, and `IngressReady` can stay `True`, while OME restores the Instances.

Each route rewrites its path prefix to `/`, so a request for `/llama-demo/llama-chat/v1/models` reaches the model server as `/v1/models`. It also sets the headers `OMe-Isvc-Name` and `OME-Isvc-Namespace` to the InferenceService's name and namespace.

A route's request timeout is its component's `timeoutSeconds`, such as `spec.engine.timeoutSeconds`, or else `defaultRouteTimeoutSeconds`. An InferenceService with `spec.replicaRefs` has no component blocks, so all of its routes take the default: see [Referenced replicas](#referenced-replicas). The chart sets `defaultRouteTimeoutSeconds` to `0`, which turns the timeout off. With `null`, the Gateway's default applies, and Envoy's 15 seconds cuts long generations short. `timeoutSeconds` applies only to HTTPRoutes. [Configure route timeouts](../../guides/networking/configure-route-timeouts.md) has examples.

### Hostnames

OME builds the hostnames in one of two schemes:

- In the shared scheme, the default, every HTTPRoute has the hostname `<sharedHostPrefix>.<domain>`, which is `llm.example.com` with the chart's `sharedHostPrefix: llm`, and a path prefix of its own.
- In the per-service scheme, which `perISVCSubdomain: true` turns on, an InferenceService's HTTPRoutes use a hostname from `domainTemplate`, such as `llama-chat.llama-demo.example.com`, and match the path prefix `/` with no rewrite.

!!! warning "Routes overlap in the per-service scheme"
    In the per-service scheme, all the routes of an InferenceService match the same hostname and path, and one of them takes every request. With a router or a decoder, requests can reach the wrong component. Keep those InferenceServices on the shared scheme, as [Choose a Gateway API host scheme](../../guides/networking/gateway-host-schemes.md) describes.

`additionalIngressGateways` attaches the routes to more Gateways, each with its own domain, and `namespaceIngressGateways` picks the Gateways for the InferenceServices in a namespace. See [Use multiple gateways](../../guides/networking/multiple-gateways.md) and [Use per-namespace gateways](../../guides/networking/namespace-gateways.md).

## Send requests

With the values in [Through the Gateway API](#through-the-gateway-api), create `llama-chat`. It serves the pre-configured model `llama-3-2-1b-instruct`, as [A minimal InferenceService](../serving/inference-services.md#a-minimal-inferenceservice) does:

```yaml title="llama-chat.yaml"
apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-chat
  namespace: llama-demo
spec:
  model:
    name: llama-3-2-1b-instruct
  engine: {}
```

Apply the file:

```bash
kubectl apply -f llama-chat.yaml
```

```output
Warning: Runtime vllm-llama-3-2-1b-instruct will be auto-selected for model llama-3-2-1b-instruct
inferenceservice.ome.io/llama-chat created
```

Once the engine is ready, OME creates two HTTPRoutes: `llama-chat`, which clients use, and `llama-chat-engine`. It also writes a default BackendTrafficPolicy named `llama-chat`. List the routes:

```bash
kubectl get httproutes -n llama-demo
```

```output
NAME                HOSTNAMES             AGE
llama-chat          ["llm.example.com"]   4m
llama-chat-engine   ["llm.example.com"]   4m
```

Read the address from the status:

```bash
kubectl get inferenceservice llama-chat -n llama-demo -o jsonpath='{.status.url}'
```

```output
http://llm.example.com/llama-demo/llama-chat/
```

`llm.example.com` must resolve to your Gateway's address, through DNS or an entry in `/etc/hosts`. List the models that the InferenceService serves:

```bash
curl -s http://llm.example.com/llama-demo/llama-chat/v1/models | jq -r '.data[].id'
```

```output
vllm-model
```

The runtime serves the model as `vllm-model`, which it sets with `--served-model-name`. Send chat completions to `http://llm.example.com/llama-demo/llama-chat/v1/chat/completions`, with `"model": "vllm-model"` in the body.

## Status

For `llama-chat`, with `domain: example.com` and the chart's other values, `status.url` is:

| What OME creates | `status.url` |
| --- | --- |
| The external Service | `http://llama-chat.llama-demo.svc.cluster.local:8080` |
| A Kubernetes Ingress | `http://llama-chat.llama-demo.example.com:8080` |
| HTTPRoutes | `http://llm.example.com/llama-demo/llama-chat/` |

- With an Ingress, the port is the router's or the engine's Service port. Send requests to your ingress controller's port instead.
- With HTTPRoutes, `status.url` is the address on the primary Gateway. With more Gateways, `status.addresses` lists the address on each, as [Use multiple gateways](../../guides/networking/multiple-gateways.md) shows.
- With HTTPRoutes, send requests to `status.url`. The routes don't serve the component URLs in `status.components`.

### When IngressReady is True

The InferenceService is `Ready` only when `IngressReady` is `True`, as [Status](../serving/inference-services.md#status) describes:

| What OME creates | `IngressReady` is `True` when |
| --- | --- |
| The external Service | Always, with the reason `IngressDisabled`. |
| A Kubernetes Ingress | OME has created the Ingress. It waits for the router, or the decoder when there's no router, or else the engine. It doesn't check your ingress controller. |
| HTTPRoutes | Every declared or [referenced](#referenced-replicas) component is ready, or is an OMENative component with a serving Instance. Every Gateway on every HTTPRoute has accepted the route's current generation, with no `False` condition at that generation: see [Route edits and gateway acceptance](#route-edits-and-gateway-acceptance). The Gateway can reject the default BackendTrafficPolicy without changing `IngressReady`. |

When `IngressReady` is `False`, its reason and message say why:

| Reason | Message | Meaning |
| --- | --- | --- |
| `ComponentNotReady` | `Target service not ready for ingress creation`, or `engine component not ready for HTTPRoute creation` | OME creates the Ingress once the component is ready, and the HTTPRoute once the component is ready or, for an OMENative component, has a serving Instance: see [Routes](#routes). |
| `ParentStatusNotAvailable` | `engine HTTPRoute awaiting gateway programming`, `engine HTTPRoute awaiting gateway reprogramming`, or `Engine HttpRouteNotReady` | A Gateway hasn't reported on the route's current generation: OME just created the route, just [updated it](#route-edits-and-gateway-acceptance), or a Gateway's report is missing or names an older generation. |
| The Gateway's reason, such as `NotAllowedByListeners` | `Engine`, then the Gateway's message | The Gateway reports a `False` condition at the route's current generation. |

The messages name the component, the engine in these examples. [Troubleshooting](../../guides/networking/configure-ingress.md#troubleshooting) in Configure ingress says what to do.

### Route edits and gateway acceptance

A Gateway's acceptance counts only for the route generation it observed. An HTTPRoute is ready when every Gateway in its `spec.parentRefs` reports `Accepted: True` at the route's current generation, with no `False` condition at that generation. A Gateway that's missing from the route's `status.parents`, or whose conditions describe an older generation, reads as not accepted, so `IngressReady` is `False` with the reason `ParentStatusNotAvailable`.

So a change that rewrites a route's spec — a component's `timeoutSeconds`, a switch of [host scheme](#hostnames), or a change to the Gateways the routes attach to — drops `IngressReady` to `False` the moment OME updates the route, with the reason `ParentStatusNotAvailable` and the message `engine HTTPRoute awaiting gateway reprogramming`, naming the component whose route changed. The InferenceService's `Ready` condition drops with it. This is the normal reprogramming window, not a failure: the update gave the route a new generation, so the acceptance the Gateways reported for the previous generation no longer counts. While the wait goes on, the message becomes `Engine HttpRouteNotReady`, with the same reason. Both conditions turn `True` again once every Gateway accepts the new generation. If `ParentStatusNotAvailable` persists, a Gateway isn't reporting on the route: see [Troubleshooting](../../guides/networking/configure-ingress.md#troubleshooting).

## Referenced replicas

An InferenceService that fronts standalone [InferenceReplicas](../../reference/api/ome.v1beta1.md#ome-io-v1beta1-InferenceReplica) gets the same external access as one that declares its components. A role that `spec.replicaRefs` names, as [Referenced replicas](../serving/inference-services.md#referenced-replicas) on the InferenceServices page describes, counts as its component in every choice on this page: whether requests go to the router or the engine, the "with a router" and "with a decoder" rows of the hostname and route tables, and `status.url`. For `llama-fronted`, which fronts the standalone engine replica `llama-pool`, `status.url` with ingress creation off is `http://llama-fronted.llama-demo.svc.cluster.local:8080`, and otherwise the address in the [Status](#status) table, with `llama-fronted` in place of `llama-chat`.

What changes is whose pods serve the requests. A standalone replica's pods are named and labeled for the replica, not for the service that fronts it:

- The Ingress and the HTTPRoutes send requests to the component Services, `llama-fronted-engine` here. Such a Service selects its replica's pods — `llama-pool-engine-0-default-0` and so on — by the label `ome.io/inferenceservice: llama-pool`. When the replica runs each Instance as a leader and workers, the Service selects only the leader pods.
- [The external Service](#the-external-service) `llama-fronted` also selects the replica's pods: the referenced router's, or the referenced engine's when there's no router.

A component Service takes its ports from the replica's runners — the leader's when it has one — or, when the replica carries no runners, from the replica's current revision. Until the replica reports a revision, the Service publishes the default port, 8080.

A referenced role backs its route by the same rules as a declared component. The Ingress waits for the role it targets to be ready, and a referenced role's ready condition follows the replica's counts. For HTTPRoutes, the [Routes](#routes) relaxation applies as written: a role whose ready condition turns `False` gets and keeps its route while the replica reports `readyReplicas` and `servingReplicas` above `0`. When the named replica is missing or invalid, the role's condition turns `False` with the reason `ReplicaRefMissing` or `ReplicaRefInvalid`, and the role stops backing its route: OME stops creating and updating the role's routes, and `IngressReady` turns `False` with the reason `ComponentNotReady`.

The routes' request timeout comes from `defaultRouteTimeoutSeconds` alone. The webhook rejects `spec.engine`, `spec.router` and `spec.decoder` next to `spec.replicaRefs`, and an InferenceReplica carries no `timeoutSeconds`, so no per-service setting exists: every route of a fronting service takes the default or, when it's unset, has no timeout of its own, and the gateway's default applies. [Referenced replicas](../../guides/networking/configure-route-timeouts.md#referenced-replicas) on Configure route timeouts shows how to set the default.

## Security

OME doesn't set up TLS. Terminate it at your Gateway's listeners or in your ingress controller. The `urlScheme` value and the annotation `ome.io/ingress-url-scheme` change only the scheme that OME reports. Set `https` when you terminate TLS, so that `status.url` matches.

Nothing authenticates requests by default. To control who can send them, use your Gateway's or ingress controller's authentication.

!!! warning "A LoadBalancer Service exposes the model to anyone who can reach it"
    With `ome.io/service-type: LoadBalancer`, the external Service gets an address outside the cluster, often a public one, and nothing authenticates the requests sent to it. Keep the load balancer internal with your cloud's annotations, which OME copies to the Service, or put an authenticating gateway in front of the model.

## Cluster-local services

The label `networking.knative.dev/visibility: cluster-local` stops OME from creating the external Service, and makes OME delete an existing one. The label doesn't stop an Ingress or HTTPRoutes. To keep an InferenceService inside the cluster whatever the config says, add the label and the annotation `ome.io/ingress-disable-creation: "true"`. OME then sets no `status.url`, and clients in the cluster use the router's or the engine's Service, such as `http://llama-chat-engine.llama-demo.svc.cluster.local:8080`.

On v1.2.2, while ingress creation is off, a cluster-local InferenceService never reports `Ready`. Every reconcile fails after OME creates or updates its serving workload, and OME records an `InternalError` warning event. This is a known bug. With ingress creation off, the external Service is a `ClusterIP` Service, so on v1.2.2, leave the label off: the model stays inside the cluster without it.

## Next steps

- [Configure ingress](../../guides/networking/configure-ingress.md): turn on an Ingress or HTTPRoutes, and set the domain, the Gateway and the other settings.
- [Expose a service without ingress](../../guides/networking/expose-without-ingress.md): make the external Service a `LoadBalancer` or `NodePort` Service.
- [Choose a Gateway API host scheme](../../guides/networking/gateway-host-schemes.md): pick the shared or the per-service scheme.
- [Configure route timeouts](../../guides/networking/configure-route-timeouts.md): keep long generations from timing out.
- [Traffic policy](traffic-policy.md): set load balancing and session affinity at the gateway.