# `dstack-proxy`

`dstack-proxy` is a set of `dstack` components responsible for exposing [services](https://dstack.ai/docs/concepts/services/).

- By default, services are published at `dstack` server URL subpaths. The component that handles traffic to such services is called **in-server proxy**. It runs as part of the `dstack` server. It is implemented in `dstack._internal.server.services.proxy`.
- Users can optionally deploy a **gateway** to handle traffic to their services. The gateway app runs on a dedicated instance. Although it requires additional configuration, it provides higher performance and supports more features than the in-server proxy. It is implemented in `dstack._internal.proxy.gateway`.
- The in-server proxy and the gateway share some business logic, such as the OpenAI-compatible API and the connection pool implementation. The common details are implemented in `dstack._internal.proxy.lib`.

## Proxy functions and modules

### Reverse proxy

`dstack-proxy` acts as a reverse proxy and load balancer for services.

The in-server proxy uses a custom reverse proxy implementation based on FastAPI and httpx. It routes requests based on the `/proxy/services/<project>/<service>` path. It performs load balancing by selecting a random service replica for each forwarded request.

The gateway uses Nginx. It automatically maintains Nginx configs for each service in `/etc/nginx/sites-enabled/*`. Each service is published at a subdomain. Nginx performs load balancing using the round-robin method. Nginx forwards requests directly to service replicas, so traffic does not go through the gateway app.

### HTTPS

The in-server proxy is part of the `dstack` server, so services are only available over HTTPS if the `dstack` server is deployed with HTTPS.

The gateway can enforce HTTPS in the Nginx config. It uses Certbot to obtain TLS certificates from Let's Encrypt or another configured CA when the service is registered, unless the service configuration specifies `https: false`.

### Auth

Unless the service configuration specifies `auth: false`, `dstack-proxy` checks the authorization of incoming requests based on `dstack` user tokens from the `Authorization` header.

The in-server proxy validates the tokens by querying the correct token from the database.

On gateways, Nginx makes a headers-only subrequest to the gateway app to check authorization for each incoming request. The gateway app then makes a request to the `dstack` server to validate the token. Responses from the `dstack` server are cached for 60 seconds. If the `dstack` server does not respond, the incoming request is denied.

### OpenAI-compatible API

The OpenAI interface emulates the real OpenAI API for chat completion models. It can list running models in the project, convert between OpenAI and TGI request formats, and forward requests to the correct service.

The in-server proxy forwards requests directly to service replicas.

```mermaid
sequenceDiagram
    User->>dstack server: http(s)://<server>/proxy/models/<project>/
    dstack server->>Service replica: http+unix:///path/to/ssh/socket
```

The gateway uses Nginx to forward requests so they are included in access logs and service stats.

```mermaid
sequenceDiagram
    User->>Nginx: http(s)://gateway.<gateway>/
    Nginx->>Gateway app: http://localhost:8000/api/models/<project>/
    Gateway app->>Nginx: http://<service>.<gateway>/
    Nginx->>Service replica: http+unix:///path/to/ssh/socket
```

### Stats collector

`dstack-proxy` collects service usage stats that are then used by the `dstack` server for autoscaling. Stats collection is only supported on gateways and is implemented by reading `/var/log/nginx/dstack.access.log`

### Service connection pool

`dstack-proxy` connects to service replicas via SSH and forwards their service port to a local Unix socket, which is then used by the reverse proxy. All SSH connections are added to a pool and reused between requests.

The in-server proxy opens an SSH connection when it needs to forward the first request to it, so there may be a delay when forwarding the first request.

The gateway opens SSH connections when each replica is registered.

### Communication with the `dstack` server

The in-server proxy is part of the `dstack` server, so no network communication takes place between them.

Gateway-to-server communication happens over an SSH connection that is established by the `dstack` server when it creates a new gateway or starts. The SSH connection includes bidirectional port forwarding, enabling the gateway and the `dstack` server to call each other's APIs:

- The server calls the gateway to register services, fetch stats, etc.
- The gateway calls the server to validate user tokens.

### Storage

`dstack-proxy` has a set of stored models and a common storage repo interface.

The in-server proxy repo implementation fetches the models from the database. It doesn't write to the database, as all the details about services and replicas are written by the `dstack` server.

The gateway maintains its own in-memory storage repo. A copy of the repo is also stored in `~/dstack/state-v2.json`. The copy is updated on every write operation and is used for data recovery after restarts. The repo is populated when new services and replicas are registered.

## Dependency injection

When a module has to be implemented differently for the in-server proxy and the gateway, `dstack-proxy` uses interfaces with multiple implementations. For example, there are common interfaces for the storage repo and for auth checks. The in-server proxy and the gateway provide their own implementations for both interfaces.

```mermaid
classDiagram
    class BaseProxyRepo {
        <<abstract>>
    }
    BaseProxyRepo <|-- ServerProxyRepo
    BaseProxyRepo <|-- GatewayProxyRepo

    class BaseProxyAuthProvider {
        <<abstract>>
    }
    BaseProxyAuthProvider <|-- ServerProxyAuthProvider
    BaseProxyAuthProvider <|-- GatewayProxyAuthProvider
```

`dstack-proxy` then uses dependency injection to select the relevant implementation. Both the in-server proxy and the gateway provide an "injector" class that is used to obtain concrete interface implementations.

```mermaid
classDiagram
    class ProxyDependencyInjector {
        <<abstract>>
        +get_repo() BaseProxyRepo
        +get_auth_provider() BaseProxyAuthProvider
    }
    class ServerProxyDependencyInjector {
        +get_repo() ServerProxyRepo
        +get_auth_provider() ServerProxyAuthProvider
    }
    class GatewayDependencyInjector {
        +get_repo() GatewayProxyRepo
        +get_auth_provider() GatewayProxyAuthProvider
    }
    ProxyDependencyInjector <|-- ServerProxyDependencyInjector
    ProxyDependencyInjector <|-- GatewayDependencyInjector
```

An instance of the relevant injector class is stored in the FastAPI global app state and can be accessed from FastAPI path operations. There are helper functions that can be used as FastAPI path operation dependencies to obtain the injector or a module implementation: `get_injector`, `get_proxy_repo`, `get_proxy_auth_provider`, etc.

There are also similar gateway-specific helper functions to obtain gateway-specific module implementations: `get_gateway_proxy_repo` (guaranteed to return the gateway repo, which has more methods than the base repo interface), `get_nginx`, `get_stats_collector`, etc.

## Gateway operations

Gateway instances are managed by the `dstack` server. A gateway is associated with a project and some backend in the project. Gateways can be shared between projects using Exports.

### Creation

Users can create a gateway using the `dstack apply` command. The gateway YAML configuration must specify the gateway's wildcard domain - all direct subdomains should resolve to the gateway IP address. Since the IP address is unknown during provisioning, `dstack` doesn't check DNS records.

Provisioning happens as follows:
1. Launch a non-GPU instance (usually the smallest) with all ports exposed.
2. Install Nginx, Certbot, and patch configs.
3. Run the gateway install script (`core/backends/base/resources/gateway_install.sh`), which installs uv, Python, and the `dstack[gateway]` package and runs the systemd service `dstack.gateway.service`. The same script is used for updates, see below.

### Update

The gateway has a "blue-green deployment"-like configuration: the app runs from one of two virtual environments, `blue` or `green`, as recorded in `~/dstack/version`. The systemd service uses the newly installed package after a restart.

On startup, the `dstack` server runs the install script on every gateway over SSH. The script:
1. Exits if the active venv already has the target `dstack` version on the script's `python_version`. Unversioned packages (master builds, `DSTACK_GATEWAY_PACKAGE_URL`) are always reinstalled.
2. Installs uv (`uv_version`) and Python (`python_version`) if missing. Both are pinned in the script.
3. Recreates the unused venv and installs the new package into it.
4. Updates scripts and systemd service config.
5. Swaps the active venv name in `~/dstack/version`.
6. Restarts the systemd service.

Changing `python_version` migrates existing gateways to the new Python on their next update. The venvs are created with pip (`uv venv --seed`) because update scripts of older servers call pip directly, e.g. after a server downgrade.

## Gateway development

The gateway app needs to interact with Nginx and certbot, so running it locally can be challenging. You can test your development version by deploying it to a live gateway.

For example, push your version to GitHub and run `dstack server` with the `DSTACK_GATEWAY_PACKAGE_URL` environment variable, which will instruct `dstack` to deploy that version to all gateways.

```shell
DSTACK_GATEWAY_PACKAGE_URL=https://github.com/dstackai/dstack/archive/refs/heads/my_development_branch.zip dstack server
```

To access logs or troubleshoot, extract the gateway's private key from the database and connect to it via SSH. Logs are available in `journalctl`.

```shell
sqlite3 ~/.dstack/server/data/sqlite.db "SELECT ssh_private_key FROM gateway_computes WHERE deleted = 0 AND ip_address = '$GATEWAY_IP_ADDR'" > /tmp/gateway.key
chmod 600 /tmp/gateway.key
ssh -i /tmp/gateway.key ubuntu@$GATEWAY_IP_ADDR sudo journalctl -u dstack.gateway.service
```