# TinyRouter

TinyRouter is a small, auditable, self-hosted LLM gateway. Bring your own provider keys, expose one OpenAI-compatible endpoint, and describe deterministic fallbacks in YAML.

It is intended to feel like **Caddy for LLM APIs**: one process, one configuration file, and very little magic.

> **Status:** alpha. The routing contract is unit-tested and smoke-checked against live providers on every release, but TinyRouter has not yet been exercised under production traffic.

## What it does

- Exposes `POST /v1/chat/completions` and `GET /v1/models`
- Works with clients that can target an OpenAI-compatible base URL
- Supports streaming and non-streaming responses
- Supports OpenAI, Anthropic, Gemini, and generic OpenAI-compatible providers
- Translates text, images, and function/tool calls for Anthropic and Gemini
- Resolves friendly model aliases to ordered provider/model targets
- Retries and falls back only before a response has started
- Emits structured logs without prompts or completions
- Exposes health, readiness, and dependency-free Prometheus metrics
- Runs from TypeScript with Bun or compiles into a standalone executable

TinyRouter has no accounts, billing, credits, database, Redis, semantic router, or background control plane. The gateway serves no dashboard either — the optional status page in [`ui/`](ui/README.md) is a static file that reads the same public endpoints anyone else could, compiled into nothing and imported by nothing.

## When to use TinyRouter

TinyRouter occupies a deliberate niche: deterministic model fallback behind one OpenAI-compatible endpoint, in a process small enough to read before you hand it your API keys.

Use it when:

- **You want to audit the thing that holds your keys.** A gateway sits between your provider credentials and every prompt you send. TinyRouter is roughly 3,000 lines of TypeScript, not counting comments and blank lines (plus about 4,500 of tests), with two runtime dependencies, `yaml` and `zod` — one person can read all of it in an evening.
- **You refuse to run infrastructure for a proxy.** One process, one YAML file. No database, no Redis, no admin UI. Prometheus metrics are built in, and the compiled binary runs with nothing else installed.
- **You run local models with cloud fallback.** Serve your own model first and fail over to a hosted one only when it is down or overloaded:

  ```yaml
  routes:
    assistant:
      - local/qwen3
      - anthropic/claude-sonnet-4-6
  ```

- **You need failover you can reason about.** The routing contract fits in a paragraph and never switches providers mid-stream. What triggers a retry is a configured status list, not a heuristic.
- **Clients you don't control need one stable endpoint.** Open WebUI, LibreChat, editors, agents — anything that accepts an OpenAI base URL gets model aliases and failover with no code changes.

Use something else when:

| You need | Better fit |
| --- | --- |
| Budgets, virtual keys, per-user spend tracking, an admin UI | [LiteLLM](https://github.com/BerriAI/litellm) |
| Guardrails, semantic caching, prompt management | [Portkey](https://github.com/Portkey-AI/gateway) |
| A hosted gateway with nothing to operate | [OpenRouter](https://openrouter.ai), [Cloudflare AI Gateway](https://developers.cloudflare.com/ai-gateway/) |
| A hundred providers out of the box | LiteLLM or a hosted gateway |
| Provider switching inside one app whose code you control | An SDK such as the [AI SDK](https://ai-sdk.dev) — you may not need a proxy at all |

Those are good projects, and TinyRouter is not trying to replace them. It exists for the case where the feature you want most is being able to understand the whole thing.

## Install

Every release attaches a standalone executable for Linux and macOS on x64 and arm64, alongside a `SHA256SUMS` file. They embed the Bun runtime, so nothing else needs to be installed. The install script picks the right one for your system, checks it against `SHA256SUMS`, and unpacks it into the current directory:

```bash
curl -fsSL https://tinyrouter.dev/install.sh | sh
```

Pin a release with `| sh -s -- 0.9.0`, or set `TINYROUTER_INSTALL_DIR` to unpack somewhere else. It needs no root and is [about a hundred lines](site/install.sh), so read it before you pipe it into a shell. To do the same by hand:

```bash
curl -fsSL https://github.com/honzabit/tinyrouter/releases/latest/download/tinyrouter-linux-x64.tar.gz | tar -xz
```

That leaves a `tinyrouter` executable and a `tinyrouter.example.yaml` in the working directory — copy the example to `tinyrouter.yaml` and edit it. The example travels inside the archive so it always matches the executable beside it, and is named `.example.` so extracting over an existing directory cannot overwrite a real configuration. Swap `linux-x64` for `linux-arm64`, `darwin-x64`, or `darwin-arm64` as needed, and check a download against `SHA256SUMS`.

On macOS, extract with `tar` as above rather than double-clicking the archive: Finder copies its quarantine flag onto the extracted executable, and Gatekeeper then reports the executable as damaged because these builds are ad-hoc signed rather than notarized.

On Alpine and other musl distributions, where the glibc-linked executables cannot start, use `linux-x64-musl` or `linux-arm64-musl` instead; the install script picks them for you. They are the one case where something else needs installing, the C++ runtime: `apk add libstdc++`. Releases up to 0.9.0 have no musl build, so use the container image for those.

The x64 executables and the amd64 container image use Bun's baseline runtime, which does not need AVX2, so they also start on older or low-power CPUs and on virtual machines that do not expose it. Releases up to 0.9.1 use the default runtime and stop with "Illegal instruction" there.

### Container

Every release also publishes a multi-architecture image (amd64 and arm64) to the GitHub Container Registry:

```bash
docker run --rm -p 8080:8080 --env-file .env \
  -v "$PWD/tinyrouter.yaml:/etc/tinyrouter/tinyrouter.yaml:ro" \
  ghcr.io/honzabit/tinyrouter:latest
```

```yaml
services:
  tinyrouter:
    image: ghcr.io/honzabit/tinyrouter:latest
    ports: ["8080:8080"]
    env_file: .env
    volumes:
      - ./tinyrouter.yaml:/etc/tinyrouter/tinyrouter.yaml:ro
    restart: unless-stopped
```

The image sets `TINYROUTER_HOST=0.0.0.0` for itself, because a process listening on loopback inside a container never receives a published port. The shipped example config reads that variable, so a mounted copy of it works either way; a config that hardcodes `127.0.0.1` will not.

`0.1.1` pins one release, `0.1` follows its patches, and `latest` follows the newest release; prereleases are only ever published under their exact version. The image runs as a non-root user and holds nothing but the executable and CA certificates.

To run from source instead, follow the quick start below.

## Quick start

You need [Bun](https://bun.sh/) 1.3.14 or later.

```bash
bun install
cp tinyrouter.example.yaml tinyrouter.yaml

export TINYROUTER_API_KEY=local-secret
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GEMINI_API_KEY=...

bun run start
```

Call an alias exactly as you would call a model:

```bash
curl http://localhost:8080/v1/chat/completions \
  -H 'Authorization: Bearer local-secret' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "smart",
    "messages": [{"role": "user", "content": "Explain this error"}],
    "stream": true
  }'
```

Or use an OpenAI client with a custom base URL. TinyRouter implements the Chat Completions shape documented in the [official API reference](https://developers.openai.com/api/reference/chat-completions/overview/).

```ts
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: "local-secret",
  baseURL: "http://localhost:8080/v1",
});

const completion = await client.chat.completions.create({
  model: "smart",
  messages: [{ role: "user", content: "Hello" }],
});
```

## Documentation

Full reference lives in [`docs/`](docs/), versioned alongside the code, so a clone or a source archive describes the commit you actually have rather than whatever is newest. [tinyrouter.dev](https://tinyrouter.dev) is the same material for reading online.

| Page | What is in it |
| --- | --- |
| [Configuration](docs/configuration.md) | Every key, plus circuit breaking, request filters, and finding model IDs |
| [Routing contract](docs/routing.md) | What TinyRouter does per request, and what each provider adapter translates |
| [Operations](docs/operations.md) | Endpoints, metrics, the status page, shutdown, TLS |
| [Development](docs/development.md) | Build, test, smoke against live providers, releasing |
| [Changelog](CHANGELOG.md) | What changed in each release, and which changes broke something |

The status page in [`ui/`](ui/README.md) is a single static HTML file that reads those endpoints and shows provider health, circuit state, token counts and route order.

## Deliberate omissions

The first release does not implement the Responses API, embeddings, image generation, audio, persistent usage accounting, pricing, budgets, rate limits, caching, a web UI, or multi-user administration.

These are product layers, not prerequisites for a reliable tiny gateway. Additions should preserve the ability to understand the complete routing path without running external infrastructure.

## License

Apache-2.0