[](https://github.com/deltawi/deltallm/actions/workflows/ci.yml) [](https://docs.deltallm.io/) [](https://github.com/deltawi/deltallm/releases) [](https://www.python.org/downloads/) [](LICENSE) [](https://github.com/deltawi/deltallm/commits/main) [](https://github.com/deltawi/deltallm/stargazers) [](https://railway.com/deploy/deltallm?utm_medium=integration&utm_source=template&utm_campaign=deltallm) **Run an LLM service for your teams or customers.** Connect model providers, create plans with model access, pricing, and rate limits, and assign them to organizations. Apps use an OpenAI-compatible API. You manage access and usage through the Admin UI. Self-hosted. MIT licensed. [SSO](docs/features/authentication.md#single-sign-on) and [audit logs](docs/features/audit-log.md) included. ⭐ If DeltaLLM is useful to you, [star the repository](https://github.com/deltawi/deltallm) so other teams can find it. ## Customer Plans Define a Growth plan with two models, set their prices and limits, and assign it to five customer organizations. When the plan changes, review and activate a new version without configuring each customer separately. [Service tiers](docs/admin-ui/tiers.md) manage these plans. Assign organizations to follow the active version so they receive plan updates together. ## Admin UI Manage organizations, access, and usage in the browser.  ## One Endpoint For Your Apps ```python from openai import OpenAI # Before: call OpenAI directly. client = OpenAI(api_key="sk-...") # After: send the same OpenAI-compatible requests through DeltaLLM. client = OpenAI( base_url="http://localhost:4002/v1", api_key="sk-deltallm-key", ) ``` Your application keeps its OpenAI request format. DeltaLLM handles provider credentials, routing, policy checks, caching, failover, and spend tracking. ## What It Handles - **Unified API** - Use one OpenAI-compatible endpoint across OpenAI, Anthropic, Azure OpenAI, Bedrock, Gemini, Groq, and other providers. - **Scoped API keys** - Issue virtual keys with model allowlists, rate limits, budgets, owners, and expiry. - **Customer plans** - Assign versioned model packages to organizations through service tiers, with model access, rate limits, and pricing managed together. - **Routing and failover** - Route by strategy, retry failed deployments, and separate provider credentials from application code. - **Batch API** - Run embeddings and non-streaming chat completions asynchronously through OpenAI-compatible files and batches, even when upstream providers are synchronous. - **MCP gateway** - Register external MCP servers and expose approved tools through controlled gateway flows. - **Guardrails** - Detect PII and prompt injection before provider calls. - **Spend and usage** - Attribute cost by key, team, organization, model, and provider. - **Admin UI** - Manage models, credentials, route groups, teams, users, usage, audit logs, and settings in the browser. - **Operations** - Export Prometheus metrics, request logs, audit events, cache behavior, and health checks. ## Performance Latest complete run: 11 October 2026. All four rates passed. | Rate | Duration | Successful / scheduled | p95 / p99 | Result | | --- | ---: | ---: | ---: | --- | | 50 RPS | 600 s | 30,000 / 30,000 | 22.86 / 26.15 ms | Passed | | 100 RPS | 600 s | 60,000 / 60,000 | 23.53 / 28.09 ms | Passed | | 200 RPS | 600 s | 120,000 / 120,000 | 28.16 / 41.67 ms | Passed | | 500 RPS | 600 s | 300,000 / 300,000 | 95.15 / 179.39 ms | Passed | Synthetic one-token provider on a twelve-CPU, 12 GiB VM. See the [RPS results](https://docs.deltallm.io/en/latest/project/issue-320-rps-report/) for setup and limits. ## Quick Start With Docker Use Docker Compose for the shortest local evaluation path. It starts DeltaLLM, PostgreSQL, and Redis. ```bash git clone https://github.com/deltawi/deltallm.git cd deltallm cp config.example.yaml config.yaml ``` For a first local request, enable one-time model bootstrap in `config.yaml`: ```yaml general_settings: model_deployment_source: db_only model_deployment_bootstrap_from_config: true ``` Generate the required secrets: ```bash python3 -c 'import secrets; print("DELTALLM_MASTER_KEY=sk-" + secrets.token_hex(20) + "A1")' python3 -c 'import secrets; print("DELTALLM_SALT_KEY=" + secrets.token_hex(32))' ``` Create `.env` with the generated values and a provider key: ```env DELTALLM_MASTER_KEY=sk-your-generated-master-key DELTALLM_SALT_KEY=your-generated-salt-key OPENAI_API_KEY=sk-your-openai-key PLATFORM_BOOTSTRAP_ADMIN_EMAIL=admin@example.com PLATFORM_BOOTSTRAP_ADMIN_PASSWORD=ChangeMe123! ``` Start the stack: ```bash docker compose --profile single up -d --build ``` Wait for the health endpoint to return `{"status":"ok"}`, then send a request. In this terminal, export the same master key you put in `.env`: Compose reads that file for its containers, but it does not export variables into your shell. ```bash curl http://localhost:4002/health/liveliness export DELTALLM_MASTER_KEY='paste-your-generated-master-key' curl http://localhost:4002/v1/chat/completions \ -H "Authorization: Bearer $DELTALLM_MASTER_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o-mini", "messages": [ {"role": "user", "content": "Hello from DeltaLLM"} ] }' ``` Open the Admin UI at `http://localhost:4002`. Got your first answer back? A [GitHub star](https://github.com/deltawi/deltallm) helps other teams find DeltaLLM. Full Docker setup, environment variables, the multi-instance evaluation profile, and Dockerfile usage are covered in the [Docker guide](docs/getting-started/docker.md). Compose is not the production high-availability reference. ## Deployment Paths | Path | Use it for | Start here | | --- | --- | --- | | Railway | Managed evaluation with hosted PostgreSQL and Redis | [Railway deployment](docs/deployment/railway.md) | | Docker Compose | Local evaluation, demos, small teams, simple self-hosting | [Docker guide](docs/getting-started/docker.md) | | Kubernetes | Multi-instance production, ingress, autoscaling, and managed infrastructure | [Kubernetes guide](docs/deployment/kubernetes.md) | | Local development | Backend, UI, and contributor workflow from the repository | [Installation guide](docs/getting-started/installation.md) | Moving from LiteLLM? The import tool converts a LiteLLM `config.yaml` and lists each setting that it could not import. See [Move from LiteLLM](docs/guides/migrate-from-litellm.md). ## Documentation - [Getting started](docs/getting-started/index.md) - [Gateway usage examples](docs/getting-started/quickstart.md) - [Configuration reference](docs/configuration/index.md) - [Model deployments](docs/configuration/models.md) - [Features](docs/features/index.md) - [Admin UI guide](docs/admin-ui/index.md) - [API reference](docs/api/index.md) - [Deployment guide](docs/deployment/index.md) - [Security model](docs/security/index.md) ## Contributing - [Report issues](https://github.com/deltawi/deltallm/issues) - [Request features](https://github.com/deltawi/deltallm/discussions) - [Report a vulnerability privately](SECURITY.md) - PRs are welcome. Start with the [local installation guide](docs/getting-started/installation.md). - Documentation changes follow the [documentation contribution guide](CONTRIBUTING_DOCS.md). If DeltaLLM solves a problem for you, consider starring the repository. Share your use case in [Discussions](https://github.com/deltawi/deltallm/discussions) so we can improve the workflows that matter to you. ## License See [LICENSE](LICENSE).