Eco-Guard architecture # Eco-Guard **A self-hosted inference gateway for teams who want their models behind one OpenAI-compatible API, with the operational furniture that usually gets left out: model management, cost accounting, live hardware telemetry, and a control plane that is actually pleasant to use.** [![CI](https://github.com/swadhinbiswas/Ecoguard/actions/workflows/ci.yml/badge.svg)](https://github.com/swadhinbiswas/Ecoguard/actions/workflows/ci.yml) [![Docs](https://github.com/swadhinbiswas/Ecoguard/actions/workflows/docs.yml/badge.svg)](https://swadhinbiswas.github.io/Ecoguard/) [![Python 3.12](https://img.shields.io/badge/python-3.12+-3776AB?logo=python&logoColor=white)](https://www.python.org/) [![FastAPI](https://img.shields.io/badge/FastAPI-0.115-009688?logo=fastapi&logoColor=white)](https://fastapi.tiangolo.com/) [![Vue 3](https://img.shields.io/badge/Vue-3.5-4FC08D?logo=vuedotjs&logoColor=white)](https://vuejs.org/) [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) [Website](https://swadhinbiswas.github.io/Ecoguard/) · [Documentation](https://swadhinbiswas.github.io/Ecoguard/docs/) · [API reference](https://swadhinbiswas.github.io/Ecoguard/docs/api-reference/inference.html) · [Architecture decisions](https://swadhinbiswas.github.io/Ecoguard/docs/adr/) · [Benchmarks](#benchmarks)
--- Eco-Guard overview dashboard ## Why this exists Running a model locally is easy for one person on one laptop. Running it for a team is where things get messy: who gets access, which model is loaded, what each request costs, why latency doubled last night, and what happens when the GPU box falls over. Eco-Guard is the layer around the model. It loads GGUF files (or talks to Ollama, vLLM, TGI, or OpenAI), serves them over `/v1/chat/completions`, records what happened, enforces limits, and puts a UI on top so nobody has to read a log file to answer "is it up?". It deploys as one container with SQLite by default and PostgreSQL + Redis when you need them. I built it because I wanted the operational side of local inference to be boring, and because most "LLM platform" repos are either a thin wrapper around a single endpoint or an unmaintainable sprawl of half-finished features. This one aims to be the middle: small enough to read, complete enough to run. ## What it does **Model management** - Scans a directory for GGUF files and reads their metadata (architecture, parameter count, quantization, context length, chat template) without loading them. - Downloads models from Hugging Face with live progress, cancel, and resume-safe `.part` files. Search and file listing are built into the UI. - Loads and unloads models on demand, sizing context and GPU offload from the GGUF header and live hardware telemetry (VRAM, RAM, CPU). - Supports a dedicated embedding model via `EMBEDDING_MODEL_PATH`; otherwise the loaded model serves embeddings when it can. **Serving** - OpenAI-compatible `/v1/chat/completions` (streaming and non-streaming, tools, JSON mode), `/v1/completions`, `/v1/embeddings`, `/v1/models`. - Native API with chat, completions, batch (up to 100 prompts with per-item error isolation), model comparison, and SSE streaming. - Real streaming: generation runs on a worker thread and is bridged to async with backpressure, so the event loop stays free and client disconnects stop the work. - Backends: local `llama.cpp`, or any OpenAI-compatible endpoint (Ollama, vLLM, TGI, OpenAI). Switching is one environment variable. **Operating it** - Prometheus metrics plus a WebSocket feed for the UI: latency, tokens/s, time-to-first-token, queue depth, cache hit rate, GPU utilization, VRAM, temperature, power. - Structured JSON logs with request correlation, OpenTelemetry tracing (OTLP). - Health and readiness probes that know the difference between "process is alive" and "can serve inference". - Latency drift detection with a z-score baseline and optional retraining triggers. **Controlling it** - API keys stored as SHA-256 hashes with roles (`admin`, `operator`, `member`, `viewer`) enforced per route. - scrypt password hashing, account lockout, HTTP-only JWT cookies, SSO (Google/GitHub) with CSRF state validation. - Sliding-window rate limiting that is exact across workers when Redis is present and correct in-process when it is not. - Content safety pipeline: prompt-injection detection, PII redaction, content classification, with per-stage actions and a dry-run endpoint. - Per-workspace budgets and token quotas, audit logging, and retention cleanup. - IP allowlisting by CIDR for the whole API surface, and optional trusted-host enforcement. **Chat and playground** - Persistent chat sessions with streaming responses, system prompts, and per-session sampling parameters. - Playground that runs one prompt across up to four models and shows output, latency, and tokens/s side by side. ## Screenshots | Chat | Models | |---|---| | ![Chat](docs/media/ui/chat.png) | ![Models](docs/media/ui/models.png) | | Observability | Playground | |---|---| | ![Observability](docs/media/ui/observability.png) | ![Playground](docs/media/ui/playground.png) | ## Quick start ### Docker Compose (recommended) ```bash git clone https://github.com/swadhinbiswas/Ecoguard.git cd Ecoguard export ECOGUARD_DOCKER_POSTGRES_PASSWORD=$(openssl rand -hex 16) export ECOGUARD_DOCKER_JWT_SECRET=$(openssl rand -hex 32) export ECOGUARD_DOCKER_ADMIN_PASSWORD=$(openssl rand -hex 16) docker compose up --build -d ``` Open . The landing page is at `/`; the console is at `/app`, and the first visit walks through a three-step setup (admin account → download a model → load it) before you have a working endpoint. The first build compiles `llama-cpp-python`, which takes a few minutes. Set `ECOGUARD_INSTALL_LOCAL=0` if you plan to point Eco-Guard at Ollama or vLLM instead. ### Local development ```bash uv sync --group dev --extra local # Python 3.12; the extra installs llama.cpp uv run uvicorn src.main:app --reload # http://localhost:8000 # Optional: the Vue dev server with hot reload cd frontend && npm install && npm run dev ``` Without `DATABASE_URL` the app falls back to SQLite in `./data` and applies migrations on start, so nothing else has to be running. Production refuses to start unless PostgreSQL, a JWT secret, admin credentials, and explicit CORS origins are configured. ### Then use it like OpenAI ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="eg_your_key") response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Explain GGUF in two sentences."}], ) print(response.choices[0].message.content) ``` ```bash curl http://localhost:8000/v1/chat/completions \ -H "Authorization: Bearer eg_your_key" \ -H "Content-Type: application/json" \ -d '{"model":"default","messages":[{"role":"user","content":"hi"}],"stream":true}' ``` ## Deploy Eco-Guard is one ASGI process plus a database for its metadata and models on disk. The [deployment guide](docs/operations/deployment.md) walks through each path, and every environment variable is listed in [configuration](docs/operations/configuration.md). ### Docker Compose `docker compose up --build -d` starts `api`, `redis` (7-alpine) and `postgres` (16-alpine). The API waits for the PostgreSQL health check, mounts `./models`, `./data` and `./backups`, health-checks `/api/v1/health` every 30 seconds with a 60-second start period, and runs with `ENVIRONMENT=production`. Three variables are mandatory; Compose fails at startup without them: ```bash export ECOGUARD_DOCKER_JWT_SECRET=$(openssl rand -hex 32) export ECOGUARD_DOCKER_ADMIN_PASSWORD=$(openssl rand -hex 16) export ECOGUARD_DOCKER_POSTGRES_PASSWORD=$(openssl rand -hex 16) ``` `ECOGUARD_DOCKER_ADMIN_USERNAME` defaults to `admin`, and `ECOGUARD_DOCKER_CORS_ORIGINS` defaults to `http://localhost:8000`. Postgres and Redis keep named volumes, so `docker compose down` leaves the data and `docker compose down -v` removes it. TLS and request-size limits belong to the proxy in front; the container listens on plain HTTP. The first build compiles `llama-cpp-python`, which takes a few minutes. Set `ECOGUARD_INSTALL_LOCAL=0` when you will point at Ollama or vLLM instead and do not want to pay for that compile. ### Docker image The multi-stage `Dockerfile` builds the frontend, copies the virtualenv into a slim runtime image, and runs as the `ecoguard` user. The default command is `uvicorn src.main:app --host 0.0.0.0 --port 8000`, and the image health-checks `/api/v1/health` every 30 seconds with a 45-second start period. `llama-cpp-python` is an optional extra, so `docker build` leaves it out unless you ask for it. Either build the slim image and point Eco-Guard at a remote backend, or compile llama.cpp into it: ```bash docker build -t eco-guard:latest . # remote backends docker build --build-arg INSTALL_LOCAL=1 -t eco-guard:local . # llama.cpp inside docker run -p 8000:8000 -v "$PWD/models:/app/models" eco-guard:local ``` ### Kubernetes `k8s/` holds the ConfigMap, Deployment, Service, HPA, NetworkPolicy and PodDisruptionBudget. The Deployment runs two replicas, takes `DATABASE_URL`, `JWT_SECRET`, `ADMIN_USERNAME`, `ADMIN_PASSWORD` and `API_KEYS` from a Secret named `eco-guard-secrets` that you create, reads everything else from `eco-guard-config`, and probes `/api/v1/health` for liveness and `/api/v1/ready` for readiness. Models come from a PVC called `eco-guard-models-pvc`, which no manifest ships, so create both before applying: ```bash kubectl create pvc eco-guard-models-pvc --size 20Gi kubectl create secret generic eco-guard-secrets \ --from-literal=JWT_SECRET="$(openssl rand -hex 32)" \ --from-literal=ADMIN_USERNAME=admin \ --from-literal=ADMIN_PASSWORD="$(openssl rand -hex 16)" \ --from-literal=DATABASE_URL="postgresql+asyncpg://user:pass@postgres:5432/ecoguard" \ --from-literal=API_KEYS="" kubectl apply -f k8s/ ``` The pod runs with a read-only root filesystem, all capabilities dropped, and `/app/models` mounted from the PVC so downloads still work from the UI. A commented nodeSelector and `nvidia.com/gpu` limit in the Deployment cover GPU scheduling when you run llama.cpp on a GPU node. `monitoring/servicemonitor.yaml` registers the service with the Prometheus Operator. ### Helm `helm/eco-guard/` renders ServiceAccount, ConfigMap, Secret, Deployment, Service, HPA, PersistentVolumeClaim, Ingress, ServiceMonitor, NetworkPolicy and PodDisruptionBudget: ```bash helm upgrade --install eco-guard ./helm/eco-guard \ --values my-values.yaml \ --set image.repository=ghcr.io/you/eco-guard \ --set ingress.host="ecoguard.example.com" ``` The template refuses to render without `secrets.jwtSecret`, `secrets.adminUsername` and `secrets.adminPassword`, so keep them in a values file; command lines end up in shell history and process listings. `replicaCount` starts at 2, `autoscaling` goes from 2 to 10 replicas at 70% CPU or 80% memory, `modelVolume` requests 20Gi, `redis.enabled` is off, and the pod runs as UID 1000 with a read-only root filesystem. One gap to close before you install: the Secret builds `DATABASE_URL` from `postgresql.auth` for a service named `-postgresql`, which is the Bitnami subchart's name, and `Chart.yaml` does not declare that dependency. Either declare it and run `helm dependency update ./helm/eco-guard`: ```yaml # helm/eco-guard/Chart.yaml dependencies: - name: postgresql version: ">=0" repository: https://charts.bitnami.com/bitnami condition: postgresql.enabled ``` or edit the Secret template to point at the PostgreSQL you already run. ### Manual In `production` the app refuses to start when `auth_enabled` (the default) is on and `JWT_SECRET` is shorter than 32 characters or `ADMIN_USERNAME` and `ADMIN_PASSWORD` are not both set, when `DATABASE_URL` is missing or is not a `postgresql+asyncpg://` URL, when `CORS_ORIGINS` contains `*`, or when `COOKIE_SECURE` is false. Passwords taken through the API need 8 characters or more. CORS otherwise allows the local dev origins, so point `CORS_ORIGINS` at the origin you actually serve the UI from: ```bash make migrate # when AUTO_MIGRATE=false uv run uvicorn src.main:app --host 0.0.0.0 --port 8000 --workers 4 ``` Set `ROOT_PATH` when the app is not mounted at the domain root. Build the frontend with `make ui` if you want the landing page and console served by the same process; without `frontend/dist/` the API answers and the root path does not. Set `REDIS_URL` before running more than one worker or replica; the next section explains why. ### Render `render.yaml` is a working blueprint: Python 3.12, the frontend built at deploy time, managed PostgreSQL, a generated `JWT_SECRET`, and a health check on `/api/v1/health`. Render has no GPU, so the blueprint sets `BACKEND=remote` and expects `BACKEND_URL`, `BACKEND_API_KEY` and `BACKEND_MODEL` to point at Ollama, vLLM, TGI or OpenAI. ### The website GitHub Pages serves the landing page at the repository root and the documentation under `/docs/`, both published by the `Docs` workflow. It builds the docs with `uv run python -m tools.docs.build`, builds the frontend with `VITE_BASE=//`, and uploads one artifact containing the landing page, `docs/`, a `.nojekyll` file, and a copy of `index.html` named `404.html`. That copy is what makes deep links work: GitHub serves `404.html` for `/app/overview`, the SPA boots with the original URL still in the address bar, and the router resolves it. Links from before the move, such as `/architecture/overview.html`, are sent to `/docs/architecture/overview.html`. A push to `main` that touches `frontend/`, `docs/` or `tools/docs/` republishes both. ## Scale ### Put Redis in front first Without `REDIS_URL` the rate limiter, the circuit state and cache coordination all live inside each process, so two replicas each enforce their own limit and each believe the circuit is closed. Add Redis, then confirm `GET /api/v1/system/status` reports `redis.connected: true`. With it, a limit set once holds across every uvicorn worker. ### Scale with replicas One model, one process. llama.cpp serialises on a lock because one context generates one sequence at a time, and `MAX_CONCURRENT_INFERENCE` (default 4) only decides how many requests are admitted before they queue. Requests beyond that show up as `ecoguard_inference_queue_depth` and `ecoguard_requests_in_flight`. Throughput per replica is therefore fixed by the model and the hardware, so growth comes from replicas behind a load balancer. The chart's HPA (2 to 10 pods at 70% CPU or 80% memory) and PodDisruptionBudget (`minAvailable: 1`) keep enough of them alive through deploys. `uvicorn --workers 4` earns its keep when the API sits in front of a remote backend, where this process handles rate limiting, budgets, caching and dashboards while token generation happens elsewhere. With local inference each worker loads its own copy of the model, so more workers means more memory and no extra tokens per second. ### Tuning that moves the numbers | Knob | Default | What it does | |---|---|---| | `MODEL_N_GPU_LAYERS` | `-1` | auto: all layers on the GPU when 90% of free VRAM clears the model size | | `MODEL_N_CTX` | `4096` | context per request; the KV cache is roughly 8K tokens per GB for a 7B-class model | | `MODEL_N_THREADS` | `0` | auto: physical cores, capped at 16; CPU decode is memory-bandwidth bound | | quantization | in the file | Q4_K_M is the usual size/quality point; bigger quants need more VRAM and more time per token | | `MAX_CONCURRENT_INFERENCE` | `4` | requests admitted before queueing; decode itself stays serial | | `STREAM_QUEUE_SIZE` | `256` | chunks buffered per slow streaming client before backpressure | `GET /api/v1/system/hardware` reports the CPU, RAM and GPU figures the sizing rules use, `GET /api/v1/models/recommendation?model_id=...` returns the GPU layer, context and thread counts it would pick for that model, and `GET /api/v1/models/loaded` reports what the backend actually received. Caching keeps repeat requests off the queue: exact-match caching is on by default for deterministic requests (temperature near 0) with a 300-second TTL and 1000 entries. Semantic caching stays off until you give it a model whose embeddings are worth trusting. ### Watching and backing it up `monitoring/grafana/dashboard.json` imports into Grafana as-is, and `monitoring/prometheus/alerts.yml` ships seven rules: 5xx error rate, p95 latency, drift above the threshold, model unloaded, rate limiting, low token throughput and a down instance. Everything else exported at `/metrics` is there for panels you add yourself: queue depth, time to first token, cache hits, circuit state, GPU utilisation, VRAM and temperature. `POST /api/v1/mlops/backups` shells out to `pg_dump` and writes a timestamped dump, with `pg_restore` on the other side; SQLite uses `sqlite3 .backup`. The models directory is plain files, so whatever backs up the disk covers them. [Backups](docs/operations/backups.md) has the commands and the verification steps. ## Benchmarks Measured with `make benchmark MODEL=...` on a desktop with an Intel i5-13600K, 32 GB RAM and an RTX 3060 Ti. The PyPI `llama-cpp-python` wheel is CPU-only, so these numbers are **CPU inference** for `Qwen2.5-0.5B-Instruct` (Q4_K_M, 0.46 GB, 32k context): | Metric | Value | |---|---| | Model load | 0.58 s | | Time to first token (avg) | 70.6 ms | | Time to first token (p95) | 71.1 ms | | Full response latency (avg, 128 tokens) | 1.94 s | | Generation speed | 57.7 tok/s | | Generation speed (best) | 59.1 tok/s | | 1 concurrent request | 1.59 req/s | | 4 concurrent requests | 1.45 req/s | | Embedding latency | 61.3 ms | Reproduce with `uv run python -m benchmarks.benchmark --model `; the raw JSON lands in `benchmarks/results.json`. For GPU inference, install a CUDA build (`CMAKE_ARGS="-DGGML_CUDA=on" uv pip install llama-cpp-python --no-binary llama-cpp-python`) or build the image with `--build-arg CMAKE_ARGS="-DGGML_CUDA=on"`. Local inference is serialised per model: one GPU context generates one sequence at a time, so a second request waits for the first. ## Architecture Requests pass through an explicit middleware stack, outermost first: ``` SecurityHeaders → CORS → RequestContext → BodySizeLimit → GZip → RateLimit → ShutdownGuard → Auth → DemoReadOnly → routes ``` CORS sits outside auth and rate limiting so error responses still carry CORS headers; request logging sits outside auth so rejected requests are logged with a correlation id. `src/main.py` builds the stack by wrapping the ASGI app explicitly, so the outermost-first order is visible in one place and does not depend on `add_middleware`'s ordering rules. ``` src/ ├── main.py # app factory, lifespan, middleware composition ├── api/ # curated routers (system, models, inference, chat, │ # openai, mlops, analytics, admin, workspaces, auth, ws) ├── core/ │ ├── backends/ # llama.cpp + OpenAI-compatible backends, thread bridge │ ├── auth/ # users, API keys, JWT, RBAC, SSO, middleware │ ├── model_manager.py # GGUF discovery, downloads, load/unload │ ├── gguf.py # dependency-free GGUF parser │ ├── hardware.py # CPU/RAM/GPU detection and model sizing │ ├── guardrails.py # injection, PII, content safety │ └── ... # config, errors, cache, rate limit, tracing, gitops ├── services/ # inference, chat store, cost, drift, notifications ├── mlops/ # registry, experiments, datasets, jobs, drift, │ # evaluation, benchmarks, prompts, RAG, backups ├── db/ # engine, session, model registry (30 tables) ├── schemas/ # pydantic request/response models └── monitoring/ # Prometheus metrics, GPU polling frontend/ # Vue 3 SPA (14 views, no chart/UI dependencies) tools/docs/ # Markdown → static docs site generator alembic/ # one clean linear migration history benchmarks/ # inference benchmark harness ``` The API surface is curated: 159 routes including the OpenAI surface. Three prompt-template systems, two audit systems and two backup systems were consolidated, and stubbed features were removed. See [ADR-0006](docs/adr/0006-curated-api-surface.md). ## Observability Prometheus metrics are exposed at `/metrics`: ``` ecoguard_http_requests_total{method,route,status} ecoguard_inference_duration_seconds{backend} ecoguard_time_to_first_token_seconds{backend} ecoguard_inference_tokens_total{direction,backend} ecoguard_inference_tokens_per_second{backend} ecoguard_model_loaded ecoguard_requests_in_flight ecoguard_cache_events_total{result} ecoguard_guardrail_events_total{guardrail,action,severity} ecoguard_backend_errors_total{backend,kind} ecoguard_circuit_state{name} ecoguard_gpu_utilization_percent{gpu,name} ecoguard_gpu_memory_used_mb{gpu,name} ecoguard_drift_score ``` The UI subscribes to `/ws/metrics` for a live feed, and the Grafana dashboard in `monitoring/grafana/` ships ready to import. ## Security - Passwords are hashed with scrypt (per-password salt, constant-time compare); failed logins lock the account for a configurable window. - API keys are 256-bit random tokens stored as SHA-256 hashes, which are fast to verify and impossible to brute-force. The token is shown exactly once, at creation. - Sessions are HTTP-only, SameSite cookies; SSO uses state validation and refuses unverified email addresses. - Security headers (CSP, HSTS in production, X-Frame-Options, nosniff, Referrer-Policy, Permissions-Policy) are applied to every response. - Model ids and file paths are resolved and confined to the models directory to prevent traversal; the API never accepts arbitrary filesystem paths for model operations. - Production startup fails loudly on weak configuration (missing JWT secret, wildcard CORS, non-PostgreSQL database, insecure cookies). - Data export and deletion endpoints exist for GDPR-style requests; audit logs record administrative actions. Report vulnerabilities per [SECURITY.md](SECURITY.md). ## What is not real (yet) These are the current limits: - Fine-tuning ships without a trainer. The endpoint validates your request and runs one only if you provide `ECOGUARD_TRAINER_COMMAND`; otherwise it returns a clear configuration error. There is no built-in LoRA trainer. - Retraining triggers only create a trigger and a dataset export. Wiring either of them to a trainer is your job. - Semantic caching is off by default because it depends on the quality of the loaded model's embeddings. A dedicated embedding model makes it useful; a 0.5B chat model makes it approximate. - The registry does not verify artifacts cryptographically beyond an optional SHA-256 checksum computed at registration. - No multi-GPU tensor parallelism. One model, one process, so scale with replicas behind a load balancer. - Concurrency is a queue. A single llama.cpp context generates one sequence at a time, so extra requests wait; parallel throughput needs replicas or a remote backend. ## Development ```bash make install-local # dependencies + llama.cpp make run # API with reload make ui # build the production UI make docs # build the documentation site make check # ruff + mypy + pytest (what CI runs) make benchmark MODEL=models/your-model.gguf ``` The test suite is hermetic: 155 tests run in under three seconds with a fake backend, no network, and a temporary SQLite database. Tests cover auth and RBAC, inference and streaming, the OpenAI wire format, model management and GGUF parsing, migrations (upgrade, parity, downgrade), security utilities, and the MLOps flows. CI additionally builds the frontend, builds and smoke-tests the Docker image, and builds the documentation site. ## Documentation Full documentation is generated from `docs/` into a static site: - [Getting started](docs/getting-started/installation.md): install, first run, model setup - [Architecture](docs/architecture/overview.md): request lifecycle, backends, database, security - [API reference](docs/api-reference/inference.md): every endpoint with examples - [Operations](docs/operations/deployment.md): deployment, observability, backups, troubleshooting - [ADRs](docs/adr/): why the system is built this way The same content is published at , rebuilt by the `Docs` workflow whenever `frontend/`, `docs/` or `tools/docs/` change. ```bash make docs-serve # build and serve at http://localhost:8000 ``` ## License MIT, see [LICENSE](LICENSE).