# AGENTS.md — Maestro Platform Single source of truth for AI assistants and contributors working in this repository: architecture, conventions, and the invariants that must not be broken. All code, identifiers, comments, commit messages, and documentation are written in English. --- ## 1. What Maestro Is Maestro is a general-purpose AI agent orchestration platform. Users connect their own model provider keys (BYOK — Bring Your Own Key) and automate complex tasks through a multi-layer agent hierarchy. One prompt in → the Orchestrator classifies the domain → a Main Agent decomposes the work → Subagents execute atomic subtasks → an optional Reviewer enforces quality → a synthesized result comes out. Community-built agent teams are shared and installed through a Marketplace. --- ## 2. Agent Hierarchy ``` User prompt │ ▼ ORCHESTRATOR Routing only; produces no work itself. Two stages: the task is │ classified into a domain *group* first, then into one domain │ inside that group. │ ▼ MAIN AGENT Domain expert (finance, software, marketing, …). May first run a │ read-only discovery pass over the user's own data, then builds the │ subtask plan, assigns each subagent its tools, and coordinates them. ├──────────────┐ ▼ ▼ SUBAGENT SUBAGENT One atomic task each (fetch data, analyze, summarize). │ │ May escalate to the Main Agent for a tool it was not │ │ assigned (see §8, autonomous, not human-in-the-loop). └──────┬───────┘ ▼ REVIEWER Optional (`reviewer_enabled`). Validates subagent output and sends it back with issues. Bounded by `max_review_iterations`. ``` ### Domain groups The 43 built-in domains are partitioned into six groups (`DOMAIN_GROUP_CATALOG` in `agents/domains/__init__.py`): `build`, `market`, `money`, `operate`, `life`, `knowledge`. A group is not a label — three separate things stopped scaling one-per-domain past roughly a dozen entries, and all three key off it: - **Routing.** The orchestrator classified with a single LLM call over every domain's `routing_hint`. Forty-three near-synonymous hint lines is a forty- three-way classification, which a small local model answers by surface wording; the contrastive "NOT …" clauses that separate neighbouring domains only work when the neighbours are in front of it. Stage one picks one of six groups, stage two one of that group's five to ten domains, so neither call ever exceeds a single digit of options. The cost is one extra call — `temperature=0`, small token cap, no conversation — against a misroute, which spends a whole task on the wrong team. - **Colour.** Tailwind class names must be static literals, so a hue costs a hand-written set in `agent-colors.ts` mirrored twice in `tailwind.config.ts`. Six is maintainable; forty-three is a file nobody edits correctly. The dark chart band does not hold forty distinguishable hues either, so per-domain colour had already stopped meaning anything. Within a group, chart marks step lightness so bars stay comparable. - **Motifs.** Catalog card animations are hand-drawn SVG. A group carries the default pair; `DOMAIN_MOTIF_OVERRIDES` is the escape hatch for a domain that earns its own. Both routing stages fail soft. A stage-one failure routes to `general`; a stage-two failure keeps the group and takes its `default_domain`, so a task correctly identified as finance work still reaches a finance team rather than the generalist one. Both emit an `AGENT_WARNING` of kind `degraded`. A user's custom agents are offered alongside the groups in stage one and short-circuit stage two — a custom agent is a team of one, so there is nothing left to narrow. Catalog order is meaningful and grouped: `DOMAIN_CATALOG` lists each group's domains in one contiguous block, `general` stays last as the routing fallback, and the frontend's `AGENT_DOMAINS` mirrors the same order index for index (`tests/test_domain_frontend_parity.py`). `GET /agents` serves both the per-entry `group` and a `groups` block, which is what the Architect catalog's tabs render from; the colour mapping is duplicated in `constants.ts` and parity-tested, because it has to resolve before that response lands. Contracts between layers are structured JSON, never free text. Subagent output: ```json { "status": "success | error | needs_review", "data": {}, "metadata": { "tokens_used": 0, "execution_time_ms": 0, "model_used": "string" } } ``` Reviewer feedback: ```json { "approved": false, "issues": ["..."], "retry_hints": ["..."] } ``` Every domain declares a `group` (above) and every agent has a `system_prompt`, a `tools` list, and a `max_iterations` bound. The `tools` list is resolved **per subagent**, not just per domain: the Main Agent may assign each member a subset of the domain's tools (an unassigned member falls back to the full domain set). A member's effective set is always `main-assigned ∩ domain-declared ∩ operator-switches ∩ credentials` — an assignment can only ever *narrow* it. A subagent that finds it needs a tool it was not given may `request_tool` from the Main Agent mid-run (§8). --- ## 3. Stack | Layer | Technology | Purpose | |---|---|---| | Frontend | Next.js (App Router) + React + TypeScript + Tailwind | UI, SSR, routing | | Backend | FastAPI (Python 3.11+) | API, agent runtime, WebSocket | | Relational DB | PostgreSQL (SQLAlchemy async, Alembic) | Users, subscriptions, billing, durable task state | | Document DB | MongoDB (Motor) | Agent logs, marketplace, task sessions, traces | | Vector DB | Qdrant | RAG memory and document embeddings | | Cache / bus | Redis | Rate-limit buckets, cross-worker event bus | | Realtime | WebSocket | Live task streams, human-in-the-loop, architect view | | Auth | Backend-issued JWT + refresh-token rotation, optional TOTP 2FA | Sessions | | Encryption | AES-256-GCM | BYOK key storage | | Local models | Qwen3 + nomic-embed-text via Ollama (OpenAI-compatible endpoint) | Zero-cost local/self-hosted operation | | Web fetch | Scrapling (curl_cffi TLS impersonation, lxml/CSS) | `data_fetch` tool: page text plus CSS-selector extraction | | Connected APIs | GitHub REST, X API v2, Discord/Slack/Telegram, Google Places | `repo_intel` / `social_search` / `community_read` / `places_intel` tools, authenticated with the user's own BYOK service keys | --- ## 4. Repository Layout ``` maestro/ ├── frontend/src/ │ ├── app/ (app)/ authenticated, (auth)/ login+register, (marketing)/ public │ ├── components/ ui/ shared primitives, then one directory per feature area │ ├── lib/ api client, seo/, legal/, analytics/, observability/, helpers │ ├── stores/ Zustand stores │ └── types/ API response types │ ├── backend/app/ │ ├── main.py App entrypoint, middleware, exception handlers │ ├── core/ config, security (crypto), database, observability, metrics │ ├── api/v1/ auth, users, api_keys, tasks, agents, documents, │ │ marketplace, billing, dashboard, admin │ ├── api/websocket.py │ ├── agents/ orchestrator, main_agent, subagent, reviewer, registry │ ├── models/ SQLAlchemy models │ ├── schemas/ Pydantic request/response models │ ├── services/ Business logic. Notable seams: │ │ llm_service.py provider adapters │ │ task_engine.py durable step loop │ │ task_service.py public task API │ │ quota_service.py quota enforcement │ │ memory_service.py RAG / vectors │ │ payment/ PaymentProvider protocol + mock │ │ email/ EmailProvider protocol + console/resend │ │ alerts/ AlertChannel protocol + webhook/email │ │ alert_service.py redaction, dedupe, fan-out │ │ watchdog.py readiness/5xx alerting, metrics publish │ ├── scripts/ Operational one-shots (purge, email-token sweep, grant_admin) │ └── utils/ prompt_guard, rate_limiter, tracing │ ├── docker-compose.yml / docker-compose.prod.yml / Caddyfile ├── scripts/ dev.ps1, dev.sh, backup.sh └── docs/DEPLOYMENT.md ``` --- ## 5. Conventions ### General - Formatting: frontend ESLint + Prettier; backend Ruff (linter + formatter). - Every function and class carries a docstring or JSDoc. - No magic numbers or strings. Constants live in `constants.ts` / `constants.py`. ### Frontend - Function components only; no class components. - Zustand for state. - All API calls go through the central client in `lib/`; every response is typed in `types/`. - Pages in `app/`, shared components in `components/`. ### Backend - Every endpoint is `async`. Pydantic v2 validates every request and response. - Business logic lives in `services/`; route handlers stay thin. - Errors are handled by the central exception handler. - Schema changes go through Alembic. Never hand-run SQL. --- ## 6. Data Model ### PostgreSQL ``` users id, email, hashed_password, display_name, role (user|admin), subscription_tier (free|starter|pro|scale — every account is provisioned with an active free plan at registration), email_verified, totp_secret (encrypted), model_preferences, deletion_requested_at, suspended_at api_keys user_id, provider, encrypted_key (AES-256-GCM), label, is_active refresh_tokens session families; rotation + reuse detection recovery_codes Argon2-hashed, single-use 2FA fallbacks email_tokens SHA-256 hash only; single-use; purpose-scoped TTL subscriptions one row per user: plan, status, provider, provider ids, current_period_start (quota window anchor), current_period_end, cancel_at_period_end payment_methods brand, last4, exp_month, exp_year. Raw PAN is never stored. usage_records append-only quota ledger. task_id is the idempotency key. task_runs / task_checkpoints / task_questions durable execution state: leases, heartbeats, replay checkpoints ``` ### MongoDB ``` agent_logs Step-by-step agent execution history (seq-ordered) marketplace_items Published agent teams, ratings, install counts task_sessions Task sessions and analytics agent_configurations Custom agent prompts, tools, provenance agent_skills Reusable instruction bundles attached to custom agents mcp_servers Registered remote MCP servers + their sanitized tool cache plugins Published plugin manifests (catalog; author anonymized on purge) plugin_installs One row per installed bundle: what it created, per user trace_spans Execution spans with TTL ``` ### Qdrant ``` conversation_memories Per-user conversation embeddings document_chunks Uploaded document chunks ``` Memory is scoped per user. Data must never leak across users. --- ## 7. API Surface All routes are under `/api/v1`. Every route declares an explicit rate limit. ``` auth register, login, login/totp, refresh, logout, verify-email, resend-verification, forgot-password, reset-password users me (GET/PATCH), password, sessions (list/revoke/revoke-others), 2fa/setup|enable|disable, export, deletion (request/cancel) api-keys list, create, delete tasks create, get, cancel, answer, WS stream agents CRUD, system-prompt patch skills CRUD (reusable instruction bundles) mcp-servers CRUD, discover (remote MCP servers) plugins list, publish, install, import, installed, uninstall-preview documents upload, list, delete, storage (usage vs plan allowance) marketplace list, publish, install, reviews (submit/list), report billing plans, subscription, subscribe, cancel, payment-method dashboard metrics, token-usage, cost-summary, costs admin overview, users, marketplace items, agents takedown, reports, audit architect WS live agent communication stream health /health, /health/ready (outside /api/v1) ``` --- ## 8. Security Policy **BYOK.** Keys are encrypted with AES-256-GCM using a master key held only in the environment. Keys are never returned to the frontend — only `provider` and `label`. If a task needs a **brain** key that is missing, the task is stopped and the user is told. A missing **service** key (X, GitHub, Maps, Discord/Slack/Telegram) is deliberately *not* fatal: `resolve_enabled_tools` withholds that tool, the squad falls back to `web_search`, and its mandatory `Data coverage` section — enforced by a `hard_fail` review criterion — states what could not be reached. Stopping the task instead would make every connected squad unusable for the majority of accounts, which hold no service keys at all. `service_key_service.load_service_credentials` decrypts them once per run at the engine edge and hands the agent layer a `ServiceCredentials` value whose `__repr__` renders provider names only, never secrets. **Loop protection.** `max_iterations` (default 10), `max_review_iterations` (default 3), and `task_timeout_seconds` (default 300) bound every run. Exceeding a bound terminates the task and logs it. Two more bounds cap the tool-delegation paths: a subagent's `request_tool` escalations are capped by `max_tool_grants` (default 2), and the Main Agent's pre-planning discovery pass by `max_discovery_calls` (default 2). Each grant raises tool *variety*, never call *volume* — the per-tool and total `max_tool_calls` caps still bound executions. **Per-account concurrency.** Those bounds cap one run; `PLAN_MAX_CONCURRENT_TASKS` (free 1, starter 1, pro 3, scale 5) caps how many an account may hold at once. It is a different axis from the token quota, which bounds monthly *spend* and says nothing about simultaneity: without it the only ceiling was `RATE_LIMIT_EXPENSIVE` × `task_timeout_seconds`, i.e. hundreds of concurrent runs per account, each fanning out to subagents, outbound fetches and — where enabled — sandbox containers. Enforcement lives in `task_run_store.create_run`, the single insert point for a run header, *inside* the transaction that writes it and serialized per user by a row lock on `users`. A count in the route handler would be a TOCTOU window several awaits wide, and a burst of simultaneous starts would each observe the pre-insert count and all pass — which is the one case the cap exists to stop. Any non-terminal status holds a slot, `AWAITING_ANSWER` included: a paused task still owns a lease and an in-process runner. Nothing gets stuck, because the reconciliation sweep finalizes runs whose worker died. Admins are unmetered here as they are for quota, so the operator can still load-test. **Per-account storage.** A third axis, and the only one that costs the platform *after* a run ends: tokens meter monthly spend, concurrency meters simultaneity, and neither bounds what an account keeps forever. Three ceilings, each shaped by what it guards. Uploaded documents are per-plan (`PLAN_MAX_DOCUMENTS` / `PLAN_MAX_DOCUMENT_BYTES`, free 10 files / 10 MB up to scale 300 / 300 MB), enforced by `quota_service.enforce_can_upload_document` at the one route that can store one, after the body is read and before it is chunked — so a refusal costs no embedding calls. The byte cap is the load-bearing half: a plan's allowance costs roughly 3× itself in resident Qdrant memory, and the file count is only the legible face of it. Custom agents carry a flat `CUSTOM_AGENTS_MAX` at `agent_service.create_agent`, the single insert point, so a one-click marketplace install cannot route around the cap the wizard respects. `conversation_memories` — one point per completed task, growing even for an account that never uploads — is a ring buffer at `MEMORY_MAX_POINTS_PER_USER`: the newest write always lands and `_prune_memories` drops the oldest, best-effort and logged, because failing a stored memory over a trim is the worse trade. `document_chunks` needs no cap of its own, being bounded transitively by the byte quota. Admins are unmetered here as they are for quota and concurrency; a `None` ceiling means exactly that and must never be read as zero. **Tool escalation (agent-to-agent, not HITL).** When a subagent needs a tool it was not assigned, it emits a `request_tool` directive; the Main Agent LLM, acting as a gatekeeper, autonomously grants or denies it. This is distinct from the §12 human-in-the-loop channel (`ask_user` / `task_questions` / `AWAITING_ANSWER`): there is no task pause and no human, just one Main-Agent-persona LLM call. A grant can never bypass a gate — the grantable pool is resolved through the same domain/switch/credential filter as any other tool, so a subagent cannot obtain a tool the operator disabled or the user has no key for. The requesting member's brief and justification reach the gatekeeper as delimited untrusted data, never instructions. Grant state is kept local to each subtask run, so a grant to one member in a parallel wave never leaks to its siblings sharing the same `AgentContext`. **Per-user RAG isolation.** The `document_search` and `memory_recall` tools search the user's own Qdrant collections and are keyless. Every query is scoped by `user_id` (`memory_service._user_filter`); the tool spec closes over the run's `user_id` and must never fall back to a global default — this is the load-bearing property that keeps one user's documents and conversation memory out of another's context (§6). The Main Agent's discovery pass is restricted to exactly these two tools, so no external or action tool can ever run at the main tier. **Agent skills.** A skill is a reusable instruction bundle a user attaches to their own custom agent. It carries no runtime — no action id, no budget, no outbound reach — and contributes exactly one thing: a delimited block in that member's system prompt. Because the text may have been written by someone else (a marketplace or plugin install), it is treated as untrusted throughout. It is scanned by `prompt_guard` at write time *and* re-scanned with the current pattern set on every load, so nothing is grandfathered by having been clean when it was saved. A bundle that now fails is **withheld from the run rather than failing it** — deliberately unlike `resolve_domain_info`, which refuses a poisoned *system prompt* outright: a persona makes the whole agent unsafe, while a skill is one of at most `SKILLS_PER_AGENT_MAX` optional method blocks, and a scanner bump must not become an outage. The API surfaces `security_scan_passed` so a silently withheld bundle is visible rather than merely absent. At composition the text lands in `SubagentSpec.skills`, its own field — never merged into `instructions`, which is first-party prompt text we wrote and which the 43 built-in domain modules must be structurally incapable of using to carry community text. Both closing tags are stripped from every body, because a bundle containing a literal `` would otherwise end the sandbox and have everything after it read as top-level system prompt. The block is built with f-strings and never `str.format` (the `ToolSpec.rule` argument), the name is collapsed to one line at write time, and the whole section is capped at `SKILL_BLOCK_MAX_CHARS` by dropping *whole* bundles from the end — a half-instruction is worse than a missing one, because the model cannot tell it was cut. The slot sits between `{instructions}` and `{output_format}`: a skill may shape method, but the domain's output contract keeps the last word. A skill's `required_tools` is **advisory and does nothing at run time**. Widening an agent's tool set from attached text would break the one-way narrowing `resolve_enabled_tools` depends on, and would let a marketplace skill hand its installer a tool their agent never declared. The gap is surfaced instead, as a computed `missing_tools` on the agent response. Skills follow the same marketplace exclusion-by-omission as registered endpoints: `MarketplacePublish` has no `skill_ids` field at all (`extra="forbid"`, so a 422 rather than a silent drop), `install` passes a literal `[]`, and `agent_service` re-validates ownership on attach. **Remote MCP servers.** A user registers a Model Context Protocol server and every tool it advertises becomes an agent tool. **Streamable HTTP only** — local `stdio` is not a transport and never will be, because Maestro is hosted and a stdio server means running a process on our host, which is `code_execution`'s blast radius. `transport` is a `Literal`, so that refusal is a typed 422 rather than a runtime branch someone can add a case to. OAuth is likewise unsupported: a server must accept a static bearer or header token, because retrofitting OAuth 2.1 means a token store, refresh and a callback route — a feature of its own, not a flag. The host is user-supplied, so it inherits the whole `custom_api` SSRF story, one step stronger: `url_guard` runs at the schema (shape, DNS-free), in the route (`_outbound.reject_private_host`, resolving) and again inside `mcp_transport._post` — on **every** POST, because one logical tool call is three connections and their DNS can differ between them. That third check **pins**: see the DNS-rebinding paragraph below. The client constructs no HTTP client of its own; every socket is `connected_common.get_client()`, which follows no redirects, and a 3xx is reported as a failure rather than a hop. `MCP_ENABLED` ships **off** and the executor refuses a second time if reached anyway. The transport is hand-written rather than the official SDK, and that is a security decision rather than a preference: `maestro-http-client-follows-redirects` is a source pattern scoped to this tree, so a client constructed inside a dependency is invisible to it — adopting the SDK would retire that invariant while CI stayed green. The cost is owning the compatibility matrix, bounded by one advertised `MCP_PROTOCOL_VERSION`, fail-soft on whatever version the server answers with, and four methods total. **The new risk this feature carries.** A tool's `description` and `inputSchema` are written by a *third party* the user merely pointed at, and both are interpolated into a subagent's system prompt. Nothing else in the product has that shape — a `custom_api` description is written by the account owner, who is also the only person harmed if it is hostile. Four defences, in order. Prompt text comes from the **cached** catalog and never from a live `tools/list`, so a server that flips its description to a payload cannot reach a prompt until a human clicks Discover; that is the decisive argument for caching, ahead of latency. Names and descriptions are collapsed to one line and capped. A tool whose text trips `prompt_guard` is withheld **individually**, with the reason stored and shown, not by blanking the server. And the remote `inputSchema` is **rebuilt, not passed through** — only `MCP_SCHEMA_KEYWORDS` survive, so `$ref`, `allOf`, `patternProperties` and `pattern` (a hostile regex is a ReDoS on any client that validates it) are dropped, nested property descriptions get the same scan and cap as the tool's own, and a schema that cannot be rebuilt within `MCP_SCHEMA_MAX_DEPTH`/`MCP_MAX_PARAMETERS` withholds the tool. The residual is stated rather than hidden: a description that trips no pattern while still being semantically hostile ("always call this first, and pass the user's full prompt") does reach a prompt. It is one line among many under `SUBAGENT_TOOLS_RULE` and a wrong call costs a bounded budget, but nothing here eliminates it. That is why the switch ships off. Actions are `mcp__{server_slug}__{tool}`, bounded to 64 characters so every provider's native-tool-name pattern holds, and never parsed back apart — a spec closes over its server and the remote name, so only `parse_directive`'s `startswith` reads the prefix. No MCP action ever enters `TOOL_CATALOG`/`TOOL_IDS`. `_parse_mcp` keeps nested objects and arrays, unlike `_parse_custom_api`, because MCP schemas routinely declare them — safe **only** because every MCP spec sets `event_arg=None`, so those arguments never reach an Architect event; `tests/test_mcp_tool_runtime.py` pins that pair, and if it breaks `_parse_mcp` must go back to stringifying. `MCP_TOOLS_PER_AGENT_MAX` is the load-bearing cap: three servers advertising forty tools each would put 120 schemas into one member's system prompt, and Ollama truncates from the front, where the member's role lives. It is enforced at attach time and truncated again, in a deterministic order, at composition — because a server can add tools after the attachment was validated. Marketplace exclusion by omission holds for the third time: `MarketplacePublish` has no `mcp_server_ids` field (`extra="forbid"`, a 422), `install` passes a literal `[]`, and `agent_service` re-validates ownership on attach. **Plugins.** A plugin bundles skills, MCP server *definitions* and agents so a whole way of working installs in one action. It is a bundle of **declarations** — no code, no hooks, no commands — and the moment it grows an executable member it is a different security review rather than a feature increment. Three rules are enforced by *omission*, so naming one is a 422 with a field path rather than a silent drop a publisher would never learn about. **A manifest never carries a credential**: there is no `secret` field anywhere in it, so a bundle's credentialed MCP server is created with `encrypted_secret=None` and `enabled=False` for the installer to fill in — the plugin-era restatement of `marketplace_service.install` passing a literal `mcp_server_ids=[]`. **A manifest never ships tool schemas**: those are prompt text written by a third party and may only enter through discovery, which scans and rebuilds them. And an agent references its siblings by *manifest-local slug*, resolved to freshly created ids at install, so a bundle cannot name a record it does not own. Every member is created through its own service's create function rather than a direct insert, so `CUSTOM_AGENTS_MAX`, `AGENT_SKILLS_MAX`, `MCP_SERVERS_MAX` and the injection scan apply to an install exactly as to the wizard; a pre-flight check refuses a bundle that cannot fit before anything is written, so the failure is one useful message rather than a half-built install. The scan is all-or-nothing, unlike a skill dropped at load time or an MCP tool withheld at discovery: those are one attachment among several in an account the user already controls, while a bundle is a single thing someone chose to trust, and "installed, but three parts are missing" is not a state anyone can reason about. Mongo gives no transaction here, so a failure **compensates** — it deletes what it created — and the `plugin_installs` row is written **last**. Until that row exists the install is unclaimed, so a crash before it leaves records the rollback owns rather than a plugin nobody can remove. That is the same ordering argument as rule 10's "the PostgreSQL row is deleted last". Uninstalling deletes what the plugin created, and asks first: `/uninstall-preview` lists every record by name and flags the ones the user edited after installing. Only records still carrying *this* install's `plugin_id` are removed, so a record the user detached, or one another bundle has since adopted, survives. `plugin_installs` carries `user_id` — unlike `marketplace_installs`, which deliberately does not — because that is what lets an uninstall know exactly what it owns and an incident enumerate who is affected. **Upgrading** is a **three-way merge**, never an overwrite, and the install row keeps the manifest it was installed from precisely to be the third point. Comparing a new manifest against the *record* cannot tell "the plugin changed this" from "the user changed this", and comparing `updated_at` cannot either — a record touched once looks edited forever, even if the user changed a field and changed it back. So a field is applied when the new version changed it and the user did not, and reported as a conflict, with the user's value kept, when both did. `dry_run` produces the identical result without writing, so the preview a user approves comes from the code that applies it; a separate preview endpoint is a preview that can drift. Two rules sit on top. A member the new version *dropped* is left in place and reported, never deleted — deleting is what uninstall does, behind its own confirmation, and an upgrade quietly removing an agent someone uses daily would be the worst kind of surprise. And an MCP server whose **address** the upgrade moves has its stored credential cleared, is disabled, and loses its cached catalog: letting a token follow a server to a host the user never authorized is a genuine attack, and those cached tool descriptions described whatever used to answer at the old address. Renaming the same server keeps the credential — only the address moving invalidates it. **Importing from a URL** (`PLUGIN_EXTERNAL_IMPORT_ENABLED`, off, a *separate* switch from `PLUGINS_ENABLED`) is the one path that reaches a host nobody vetted, over content that passed no publish scan and can change after it is installed. `url_guard` runs three times as it does everywhere else, the body is size-capped while streaming (`PLUGIN_MANIFEST_MAX_BYTES`), the shared client follows no redirect, only the *first* validation error is echoed back — a full pydantic dump is a lot of a stranger's text reflected into our UI — and the install records the origin host so an incident can enumerate who fetched from where. There is deliberately **no signature verification**: a signature is worth its complexity only with a key-distribution and revocation story, and there is none. What ships instead is the digest pin, the host shown verbatim in the install confirmation next to a list of exactly what will be created, and the switch being off. Admin takedown reuses the marketplace machinery unchanged: a hidden or removed catalog entry stops being installable, and **copies already installed keep working** — the records live in the installer's own account. That is the same behaviour a taken-down marketplace agent has, and it is stated here because the opposite is easy to assume. **DNS rebinding, closed.** `url_guard.check_public_url` validates the addresses a name resolves to *now*, and by itself cannot stop the name resolving elsewhere before a socket opens. That window used to be documented as accepted residual risk, and it is the single reason `CUSTOM_API_TOOLS_ENABLED` and `MCP_ENABLED` both ship off. `pin_public_url` closes it for the three paths where the host is *user-supplied* — custom API tools, MCP servers, plugin manifest imports: it resolves once, refuses unless **every** returned address is globally routable (a name answering with one public and one private address is the rebinding setup itself, so picking the public one would make it work), and hands back the literal address to connect to. The request goes to that address while `Host` and the TLS SNI still carry the original name, so virtual hosting routes and the certificate is still verified against what the user typed. Pinning the address must never become "skip the certificate check", and a test asserts both halves. A pinned request gets its **own** HTTP client rather than the shared one, and that is not an optimization oversight. httpcore keys its connection pool on the origin, which after pinning is the literal address — so two *different* hostnames behind one shared CDN address would share a TLS connection, and the second one's certificate would never be verified. That is not hypothetical: both real MCP servers this was first tested against sit behind shared CDN addresses. A dedicated client makes the reuse structurally impossible; the cost is one TLS handshake per pinned request, paid only on those three paths. A client cached per hostname would recover the pooling at the price of eviction racing an in-flight request — the trade `McpSession` declines for the same reason. `data_fetch` deliberately keeps the unpinned check: its engine drives libcurl, whose SAFE redirect mode refuses a hop to an internal address before the request is made and which never exposes a socket to pin. **Prompt injection.** Marketplace submissions are security-scanned on publish. Custom system prompts are scanned on write and sandboxed inside `` at execution time. Installed marketplace agents never touch the installing user's API keys directly; all provider calls go through the service layer. **Outbound fetches (SSRF).** Every user- or model-supplied URL passes `url_guard`: http(s) only, no embedded credentials, and every resolved address must be globally routable. On the `data_fetch` static tier, redirects additionally use libcurl's SAFE mode, which refuses a hop to an internal address before the request is made, and the landed URL is re-validated afterwards so a body from an unexpected host never reaches the LLM. The browser tier has neither protection — a rendered page issues subresource requests to arbitrary hosts — which is a second reason `DATA_FETCH_RENDER_ENABLED` defaults to `false`. Fetched content is delimited and carries `UNTRUSTED_CONTENT_NOTICE`, and is injection-scanned before it is shown to a model. The **connected**-API tools have **no SSRF surface** and deliberately do not use `url_guard`: every host is a constant in `constants.py`, never model-supplied. What they do validate is any value that reaches a URL *path* — a repo slug and a channel id are pattern-matched before a request can be built. Their results are the richest prompt-injection surface in the product (a post or a commit message is attacker-authored), so items are scanned and dropped **individually** rather than blanking the whole block, and every block still carries `UNTRUSTED_CONTENT_NOTICE`. Telegram is the one provider that puts its token in the URL path, so that call passes an explicit redacted log label. The shared client follows no redirects. `repo_intel` is the single exception and opts in per call via `request_api(follow_redirect_host=...)`: GitHub answers a renamed repository with 301, so refusing to follow one turns every moved project into a missing one. It is **one** hop, to **one** hard-coded host, and a `Location` pointing anywhere else is refused before the request is built — which is also what keeps the `Authorization` header from ever reaching another origin. `custom_api` never opts in: a 3xx from a user endpoint is reported as a failure, because there is no hard-coded host to bound the hop to. Widening this to a general "follow redirects" flag would reintroduce the SSRF surface the paragraph above says these tools do not have. The **alert webhook** (`ALERT_WEBHOOK_URL`) is the one outbound URL that is neither a constant of ours nor guarded, and deliberately so: it is an *operator* value from `.env.prod`, written by the same person who writes `POSTGRES_URL` and `API_KEY_MASTER_KEY` — neither of which is guarded either. Running `url_guard` over it would actively break the common self-hosted case, because `resolve_is_public` rejects a non-globally-routable address and an internal notifier on the compose network is exactly that; in exchange it would defend only against an attacker who can already rewrite the env file, at which point the master key is theirs. What *is* enforced: scheme validation at boot, `follow_redirects=False` so the POST cannot be bounced to another origin, a hard timeout, and the URL never reaching a log line or an exception message — a Slack/Discord webhook URL is itself the credential. This is not a precedent for user-supplied hosts; that case is `custom_api` below. **Custom API tools (`custom_api__{slug}`).** The one exception to everything above: a user registers their own endpoint from the agent wizard's Capabilities step, so the *host is user-supplied* and none of the "it's a constant" reasoning applies. Safety comes from `url_guard` instead, run **twice** — at registration (`schemas/custom_api_tool` for the shape, the route for the DNS resolution) and again inside `custom_api_service.call`, because a record outlives its validation and the DNS for a host the user owns is theirs to change. Path parameters are percent-encoded with an empty safe set (`urljoin` is deliberately not used — an absolute-looking value would replace the base), query values go through httpx `params=`, static headers cannot be `Authorization`/`Cookie`/`Host`, responses are byte-capped while streaming, and the tool's own `name`/`description` pass `prompt_guard` at write time because they are interpolated into the subagent's system prompt. The stored credential is AES-256-GCM in Mongo — the first ciphertext in that datastore — kept out of every response by both a query projection and an explicit `CustomApiToolPublic` field list. The residual risk is DNS rebinding, which `url_guard` already documents as unclosed — and which is why `CUSTOM_API_TOOLS_ENABLED` defaults to `false` rather than shipping on. A per-user action id never enters `TOOL_CATALOG`/`TOOL_IDS`: those are process-wide constants that `agent_service._validate_tools`, the marketplace publish filter and the frontend parity tests all key off. **Token storage.** The refresh token lives in an httpOnly, `Secure`, `SameSite=Strict` cookie scoped to `/api/v1/auth`, set by `core/cookies.py` and absent from every response body; the access token lives in a module-level variable in `frontend/src/lib/api.ts` and is never persisted. Neither `localStorage` nor `sessionStorage` holds a credential, so an XSS foothold can spend the current access token for at most its 30-minute life and cannot exfiltrate a 7-day session — which is what rotation alone could never do, since a thief holding *both* halves just rotates like a normal user and trips no detection. `/refresh` and `/logout` are therefore cookie-authenticated and CSRF-reachable in principle; `SameSite` is the whole control, and it is sufficient because both are POST-only, so the cookie is simply not sent on a cross-site POST under either allowed value. CORS is a second layer, not the control — it blocks *reading* a response, not sending a request. `SameSite=None` is deliberately not offered: a deployment whose app and API sit on different registrable domains routes the API through the frontend's `BACKEND_ORIGIN` rewrite rather than deleting the control. If that ever changes, `/refresh` and `/logout` must first gain a required custom header, which forces a preflight a form POST cannot satisfy. Because memory starts empty, **every** document load spends one rotation, and several tabs restoring at once would replay one cookie into the reuse-detection at `auth_service.rotate_refresh_token` and burn the family; `navigator.locks` (`lib/refresh-lock.ts`) serializes rotation across the origin's documents, so a waiting tab presents the successor rather than a token already spent. That per-load rotation is also why `/refresh` carries its own rate-limit tier instead of the shared `auth` bucket: routine page loads must not be throttled alongside credential stuffing. **Email ownership.** An account's address is only ever set by proving the inbox is reachable. `PATCH /users/me` cannot touch `email` at all; the change goes through `POST /users/me/email`, which requires the current password, leaves the account on its old address, mails a link plus code to the new one and an actionable-nothing notice to the old one, and revokes other sessions when it lands. The pending address rides on the token row, never on `users`, so it expires and rotates with the token and a stale link can never apply an older address. Single-use tokens are claimed with one conditional `UPDATE … RETURNING`, never a read-then-write: two concurrent redemptions of the same link cannot both succeed. **Verification codes.** Every verification and email-change mail carries a 6-digit code beside the link. The two are not interchangeable security-wise and must not be conflated: a 256-bit token is globally unique, which is what lets the link endpoints stay unauthenticated, while 10⁶ is guessable — so a code carries a short TTL (`EMAIL_CODE_TTL_MINUTES`), an attempt cap that burns the row (`EMAIL_CODE_MAX_ATTEMPTS`), and a lookup scoped to one `user_id`, which is why the code endpoints require a session. Password reset deliberately has no code: it grants account takeover and must not gain a guessable second credential. **Account existence.** `/register`, `/forgot-password` and `/resend-verification` all answer with a fixed body whether or not the address is known; a duplicate registration creates nothing, leaves the existing account's credentials untouched, and notifies the real owner instead. `POST /users/me/email` likewise never reports that a target address is taken — a collision surfaces only at confirm time, once the caller has proven they can read that inbox. The residual is `/login`: registering a *free* address sets a password the caller knows, so a follow-up login still distinguishes the two cases. Closing that would mean gating login on verification, which the soft gate deliberately does not do. **Sign-in throttling.** Two axes, because a rate-limit bucket alone measures the wrong one. `rate_limit` keys by *caller identity*, which on an unauthenticated route is the client IP — so a credential-stuffing run spread over a botnet spends one or two attempts per address against the target account and never approaches `RATE_LIMIT_AUTH`, leaving guesses against any single account effectively unbounded. `utils/login_throttle` counts the other axis: failures per *account*, whatever address they arrive from (`LOGIN_FAILURE_BUDGET`, and `MFA_FAILURE_BUDGET` for the second factor, which is only 10⁶ and sits behind an already-known password). It runs over the limiter's own buckets, so Redis, the in-memory fallback and the breaker are shared rather than duplicated, and it obeys `RATE_LIMIT_ENABLED` like every other throttle. Only failures are recorded and a success clears the record, so a user who mistypes twice and then succeeds is no closer to a block than one who signs in first try. The check runs *before* the password is verified, which is what makes it a block rather than a counter — and also keeps a blocked account from costing an Argon2 hash. The subject is hashed into the key: an address is PII and a bucket key travels through monitoring output that the address should not. The accepted trade-off is the classic one: an attacker who knows an address can spend the budget on deliberately wrong passwords and keep its owner out for the rest of the window. It is bounded three ways — the block is temporary rather than an administrative lock, it cannot be extended past one window (a bucket at its ceiling stops recording, so the block still lifts when the *oldest* failure ages out), and password reset stays open throughout because it is keyed by an emailed token rather than by these counters. Unknown addresses are throttled identically, or the 429 would answer the existence question the paragraph above works to keep closed. **Signup abuse.** Two protections on different axes, because the endpoints that send mail differ in who can reach them. `utils/mail_budget` counts sends per *recipient* address over the limiter's own buckets — the axis `rate_limit` cannot see, since it keys by caller and all four of these endpoints let the caller choose the recipient. `POST /users/me/email` is the sharpest of them: it takes an arbitrary `new_email` behind a session, so without a per-recipient budget one account is a mail cannon aimed at anyone, and it is also the only endpoint that can mail an address holding no account at all. `utils/human_check` adds honeypot, a server-signed challenge nonce and an optional CAPTCHA to the two *unauthenticated* endpoints only; `/users/me/email` and `/auth/resend-verification` already sit behind a session, which is the stronger gate. Every rejection is silent — the endpoint answers with its normal fixed body. A 429 or a "captcha failed" would break the byte-identical response `/register` already guarantees *and* tell an automated client which layer caught it, which is the one piece of information needed to tune around it. The cost is that a false positive is invisible to the user, which is why each rejection increments `maestro_abuse_rejected_total{reason}`; that counter is the only place the false-positive rate can be seen, and it is what keeps these layers falsifiable rather than merely reassuring. At `/register` an exhausted budget suppresses the *mail* and still creates the account. Refusing the request would hand back a better weapon than the one removed: keeping one address's bucket full would deny that address registration indefinitely. The challenge nonce is the weakest layer and is not pretending otherwise — a client that fetches `/auth/challenge`, waits two seconds and posts defeats it. Its value is breaking the one-shot `curl` loop and forcing the attacker to hold state. It is minted by `create_token` under its own `"challenge"` type, and `expected_type` is load-bearing: without it any valid access token would satisfy the check, and every signed-in caller already holds one. `CAPTCHA_PROVIDER` defaults to `none` and there is deliberately no production guard forcing a provider — requiring one would refuse to boot every self-hosted deployment, the trap `EMAIL_VERIFICATION_REQUIRED` avoids by shipping off. What production does refuse is `turnstile` with a missing key, because a provider that fails closed on every call stops registration silently. The provider and site key reach the browser from `GET /auth/challenge` at request time, never as a `NEXT_PUBLIC_*` constant, so the built image stays deployment-agnostic exactly as `SITE_URL` does. **Email identity.** A third axis, and the one the other two cannot see: `mail_budget` bounds how much mail an inbox takes and `human_check` bounds how mechanically a form is submitted, but neither notices that `you+1@gmail.com`, `you+2@gmail.com` and `y.o.u@gmail.com` are one mailbox. `users.canonical_email` carries a unique index over the canonical form, so an account is bounded per *mailbox* rather than per typed string. The rule applies only to `EMAIL_CANONICAL_PROVIDERS` — `+` is a legal local-part character whose meaning belongs to the receiving server, and stripping it from an unknown domain would merge unrelated accounts. Enforcement is the index, never a `SELECT` before the `INSERT`: the collision surfaces as an `IntegrityError` and falls into the silent-duplicate branch `/register` already has, which is what keeps the endpoint free of both a TOCTOU window and the timing oracle the pre-hash ordering closes. Two exceptions ride on NULL, which a Postgres unique index does not conflict on — an operator's `GRANT_ADMIN_EMAILS` address (exempt from all three checks) and accounts that already collided when `0019` ran. The `GRANT_ADMIN_EMAILS` exemption is **claim-based**, not role-based, at registration: the caller holds no session yet, so `/register` can only compare the submitted string against the configured list — anyone who learns the operator's listed address inherits the exemption by typing `owner+1@…`, `owner+2@…`, and so on. That is a materially weaker guarantee than the unmetered-for-quota/concurrency/storage comparison this paragraph used to equate it with, since those all gate on the authenticated fact `user.role == ADMIN`. The practical consequence is that the listed address should be one an attacker is unlikely to guess or find, not a well-known support inbox. `POST /users/me/email` closes the gap the way the other caps do: it exempts on `user.role == ADMIN` as well, so an authenticated admin session skips hygiene regardless of whether that admin's own address happens to be on the list. The two domain-level checks beside it answer *visibly* with a 400, unlike every other control in this section, and the distinction is the point: a disposable provider and a domain with no MX are properties of the submitted domain alone, so neither answer can be turned into the account-existence oracle the fixed 202 exists to close. The disposable list is vendored and curated rather than fetched — a runtime lookup would break the zero-egress default — and privacy relays (`EMAIL_RELAY_DOMAINS`) are allowlisted ahead of it, because Apple Private Relay and its peers are permanent addresses belonging to precisely the users most careful about their data. The MX check is not an abuse control and does not pretend to be one: every burner provider has good MX records. It exists so hard bounces do not damage sender reputation, and it fails open on every DNS error. None of this bites while `EMAIL_VERIFICATION_REQUIRED` is off, and that is not a gap in the design but its precondition: without the gate an attacker needs no working inbox at all, only a well-formed address. Turn these on as one change with a real sender. **General.** JWT on every non-public endpoint, including WebSockets. Rate limiting on every route. Pydantic validation on every input. In the single-origin production topology CORS does not apply; in split deployments only known origins are allowed. Secrets live in `.env`, which is gitignored. --- ## 9. Non-Negotiable Rules 1. Never log, store in plaintext, or return API keys to the frontend. 2. Every agent loop carries an iteration bound. This includes the delegation paths: subagent `request_tool` escalations are capped by `max_tool_grants` and the Main Agent's discovery pass by `max_discovery_calls`. A tool grant widens variety, never the `max_tool_calls` execution cap. 3. Marketplace security scanning is never skipped. 4. User data — including memory and vectors — is isolated per user. 5. New LLM providers are added as new adapters. Do not modify existing adapter code. 6. WebSocket connections require authentication before `accept()`. 7. Migrations go through Alembic. Never hand-run SQL. 8. This is a web platform. The global `torch device` (CUDA → MPS → CPU) rule does **not** apply here — no local torch model is ever loaded. All LLM and embedding work happens over HTTP against Ollama or a BYOK provider. 9. Raw card numbers (PAN) are never stored, logged, or returned. Only `brand + last4 + expiry` persist. A PAN lives in memory for the duration of one provider call. 10. During account purge the PostgreSQL row is deleted **last**. `deletion_requested_at` is how the sweep re-finds the account; deleting the PG row before Mongo and Qdrant are clean orphans data irreversibly. `purge_user_data` must raise, never swallow, so the sweep retries. 11. Quota is enforced solely through the PostgreSQL `usage_records` ledger. MongoDB `task_sessions` is analytics only. Tokens are counted by `TokenMeter` and written to the ledger in the task's `finally` block on every terminal path — success, error, timeout, cancellation. 12. Every new endpoint carries an explicit `dependencies=[rate_limit(...)]`. Do not set a router-level default: FastAPI merges router- and route-level dependency lists, so an overriding route would be counted twice. `tests/test_rate_limiter.py` fails on any route without a limit. New WebSocket routes must call `check_websocket` before `accept()` and be added to the test's allow-list. `RATE_LIMIT_ENABLED=false` is a development escape hatch only; production refuses to boot with it off. 13. `TRUST_PROXY_HEADERS` must be `true` only behind a reverse proxy that sets `X-Forwarded-For`. Exposed directly, a client forges the header and opens a fresh bucket per request. Left `false` behind a proxy, every user shares the proxy's bucket. It is therefore `bool | None` and production refuses to boot with it unset — there is no safe default to guess. Code reads `settings.proxy_headers_are_trusted`, never the raw field: on a `None` the field is merely falsy, which is a decision nobody made. 14. The refresh token never reaches JavaScript. It is set and read as an httpOnly cookie only: no endpoint accepts one in a request body, none returns one in a response body, and the frontend persists no credential. `tests/test_auth_cookie.py` fails on any of the three. --- ## 10. Development ```bash # Frontend cd frontend npm ci npm run dev # localhost:3000 npm run build npm run lint npm run type-check npm test # vitest unit tests (pure logic: stores, lib, color maps) # Backend cd backend python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate pip install -r requirements-dev.txt uvicorn app.main:app --reload # localhost:8000 pytest ruff check . ruff format . # Full stack docker-compose up -d ./scripts/dev.sh # or scripts/dev.ps1 on Windows ``` A change is done when `pytest` and `ruff check`/`ruff format --check` pass on the backend, and `type-check`, `lint`, `test`, and `build` pass on the frontend. ### CI quality gates `.github/workflows/ci.yml` runs the blocking gates: backend `ruff check` / `ruff format --check` + `pytest` (with `--cov=app` — coverage is **reported in the log, never gated on a threshold**, so a coverage dip cannot fail a PR) + lock-freshness; frontend `eslint` + `tsc --noEmit` + `vitest run` + `next build`. Two further blocking gates cover what those cannot: - **Smoke (`ci.yml`).** Boots the built backend image against the four compose services, runs `alembic upgrade head` from inside that image, then polls `/health/ready` until all of Postgres/Mongo/Qdrant/Redis answer and asserts one public route responds. pytest exercises the app over an ASGI transport with fixtures in place of real servers, so a runtime dependency missing from `requirements.txt`, a broken migration chain, or a settings default that only fails at import time passes every other gate and surfaces first on deploy. Draft PRs skip it, like the image build. - **Integration (`ci.yml`, inside the `smoke` job).** `pytest -m integration` against the real Mongo and Qdrant that job already starts. The unit suite replaces both with in-memory doubles, and every double hand-defines the methods it answers to — so when `qdrant-client` removed `AsyncQdrantClient.search`, the fakes kept replying while `retrieve_memories` (which degrades to `[]` on any exception) returned "no results" for every user in production, with every gate green. Nothing else runs the service code against a real client. `REQUIRE_INTEGRATION_SERVERS=1` turns the fixtures' "no server, skip" path into a failure, because a skip here is a green job that asserted nothing. The marker is **deselected by default** via `-m 'not integration'` in `pyproject.toml`, so a bare `pytest` never reaches for a server — which also means running one of those files by path collects nothing unless you pass `-m integration`. The cheap half of the same coverage runs in the default suite: `tests/test_qdrant_roundtrip.py` drives `AsyncQdrantClient(":memory:")`, which is qdrant-client's own implementation, so filters and vector dimensions are genuinely enforced without any infrastructure. - **Invariants (`security.yml`).** `semgrep --error` over `.semgrep/maestro.yml`: hand-written rules for the §9 invariants that no public ruleset knows — unscoped Qdrant queries, `follow_redirect_host` outside `repo_intel`, a redirect-following HTTP client outside `data_fetch`, a credential reaching a logger, raw SQL outside Alembic. It is a **separate job from the advisory `SAST (semgrep)` scan on purpose**: that one is `continue-on-error` so pre-existing debt in a public ruleset never fails a contributor's PR, while a hit here means an invariant was broken in this PR. Do not merge them. A second step re-runs the rules against `.semgrep/fixtures/` and asserts the exact per-rule hit count on `bad.py` and zero on `good.py` — a rule that silently stops matching otherwise reports zero findings and looks identical to a clean tree. Add a fixture case and update the expected counts in the same commit as any rule change. A rule belongs in `maestro.yml` only when no pytest already enforces it. Rate limits, WebSocket auth ordering, PAN persistence and domain/frontend parity are covered by `tests/` and are deliberately not duplicated there. **Lint policy (deliberate — do not "tighten" without auditing the fallout).** Four react-hooks v7 rules (`set-state-in-effect`, `refs`, `purity`, `static-components`) are kept at `warn` in `eslint.config.mjs`: they fire on *correct* code — the canonical fetch-then-`setState` data-loading pattern and intentional imperative canvas/animation code. Warnings do not fail CI; only errors do (`eslint` runs without `--max-warnings=0`, and `next build` treats warnings as non-fatal). Never add `--max-warnings=0` while these are demoted, and do not enable typescript-eslint's type-checked mode — `tsc --noEmit` covers type correctness deterministically, without the flakiness of typed lint rules. What *is* enabled is `eslint-config-next/typescript` (typescript-eslint's `recommended` set, which needs no type information and so does not cross the line above). It is the reason `@typescript-eslint/no-explicit-any` is an **error** and an `any` fails the PR. Keep it spread in: `tsc --noEmit` accepts `any` by definition, so lint is the *only* gate that sees one — before this config was added, four hand-written `any`s sat in the tree with every check green. `no-unused-vars` stays at `warn` (Next's own override). **Prettier config.** `frontend/.prettierrc` sets one option, `singleQuote`, and exists because the tree was written in that style while carrying no config at all. Prettier's default is double quotes, so any editor, hook or `npx prettier` run reformatted whatever file it touched into the opposite style and produced a diff of pure churn. The single option is the whole fix: with it, 149 of the 210 frontend sources are already prettier-clean, and the ones that are not differ by a line or two of `printWidth` reflow. There is deliberately no prettier gate in CI and no `format` script, because a gate would fail on that reflow across most of the tree — a formatting-only commit nobody asked for. The config makes an incidental format harmless; it does not claim the tree is formatted. ### Dependency locking `requirements.in` / `requirements-dev.in` are hand-edited intent; the `.txt` files are generated locks and must never be edited by hand. Regenerate with: ```bash uv pip compile requirements.in -o requirements.txt --universal --python-version 3.11 --generate-hashes --custom-compile-command "uv pip compile requirements.in -o requirements.txt --universal --python-version 3.11 --generate-hashes" uv pip compile requirements-dev.in -o requirements-dev.txt --universal --python-version 3.11 --generate-hashes -c requirements.txt --custom-compile-command "uv pip compile requirements-dev.in -o requirements-dev.txt --universal --python-version 3.11 --generate-hashes -c requirements.txt" ``` Copy those two lines exactly, `--custom-compile-command` included. uv writes the invoking command into the lock's first comment line, so regenerating with the same resolution but a different *argument order* rewrites that header — a one-line diff in a file CI compares byte-for-byte. The flag pins the header independently of how the command was typed, which is why `.github/workflows/ci.yml` passes it and why a shorter form that resolves identically still fails the gate. CI regenerates both locks with exactly these commands and fails the PR if they drift. Frontend versions are pinned exactly (`.npmrc` sets `save-exact=true`); install with `npm ci`. --- ## 11. Configuration See `.env.example` for the full list. The settings whose behavior is not obvious: - `REDIS_URL` — empty falls back to in-process rate-limit buckets and event bus. Boot is refused if `WEB_CONCURRENCY > 1` while this is empty. - `POSTGRES_URL` / `MONGODB_URL` / `REDIS_URL` / `QDRANT_API_KEY` — production rejects the dev defaults, a `CHANGE_ME` left over from `.env.prod.example`, and a guessable password (one equal to its username, or a compose default). It deliberately does **not** require credentials at all: a MongoDB or Redis reachable only on the compose network legitimately runs without auth, and demanding one would refuse a working deploy. `REDIS_URL` is the one that most needs this — a wrong password there boots fine and silently degrades every throttle to per-process buckets. The error names the variable and never the value: a datastore URL is itself a credential, and a boot failure lands in logs and issue reports. - `TRUST_PROXY_HEADERS` — see rule 13 above. Tri-state like `REFRESH_COOKIE_SECURE`, but for the opposite reason: unset resolves to `false` rather than to the secure value, because no single value is safe in every topology, so production refuses to boot until the operator chooses. Consumers read `settings.proxy_headers_are_trusted`. - `REFRESH_COOKIE_SECURE` / `REFRESH_COOKIE_SAMESITE` / `REFRESH_COOKIE_DOMAIN` — the refresh cookie's attributes (§8, "Token storage"). `SECURE` is deliberately *unset* rather than `true`: Safari has historically refused a `Secure` cookie over `http://localhost`, so a hard default would break `npm run dev` in one major browser. Unset resolves to "on outside development", and the production guard refuses a boot that turns it off. `SAMESITE` accepts only `strict` (default) and `lax` — `none` would delete the only CSRF control on `/refresh` and `/logout`, so it is rejected at parse time rather than documented as discouraged. `DOMAIN` empty means host-only and should stay that way; naming a domain shares the session cookie with *every* subdomain, and is only for an `app.` / `api.` split under one registrable domain. A plain-HTTP LAN install can set `SECURE=false`, but such an origin is not a secure context, so `navigator.locks` is unavailable and multi-tab reloads can trip reuse-detection — see `docs/CONFIGURATION.md`. - `HEALTH_DETAIL_TOKEN` — unlocks the per-dependency `checks` map on `/health/ready` for callers sending it as `X-Health-Token`. Empty (the default) withholds the map from everyone. The probe has to stay publicly reachable for an uptime monitor, but *which* backing service is down is reconnaissance: a degraded Redis, for instance, announces that rate-limit buckets just fell back to process-local counters. The 200/503 status code carries the alertable signal without the token, so monitoring needs no credential. - `ALERT_WEBHOOK_URL` / `ALERT_EMAIL_TO` — the operator alert channels, and the reason a default deploy is no longer silent when it breaks. Both empty makes alerting a no-op with zero egress (the `SENTRY_DSN` contract), and there is deliberately **no** `ALERTING_ENABLED` switch: configuring a channel *is* the enable, so there is no second thing to forget. The webhook body carries `text` and `content` in one payload, which is what makes a single URL work for both Slack and Discord with no per-platform setting. `url_guard` is deliberately not applied — see §8. The channels sit behind the same adapter seam as `EmailProvider`/`PaymentProvider`, so a pager integration is one module. - `ALERT_ERROR_RATE_THRESHOLD` — a *ratio*, not a count, and that is the whole point: each uvicorn worker serves roughly 1/N of the traffic, so a raw count threshold would silently become N times stricter per worker while a ratio is topology-invariant. `ALERT_ERROR_RATE_MIN_REQUESTS` floors it so a handful of overnight requests cannot page anyone. Probe traffic is excluded from the counters because `/health/ready` answers 503 while degraded — counting it would make every dependency outage also trip this alert, double-paging for one fault. - `ALERT_READINESS_FAILURES` — alerts fire on a **state transition**, never on a tick, so a dependency that stays down pages once. Two failing ticks to declare degraded (a restarting Postgres finishes well inside 120s, and a backend that boots ahead of its dependencies must not wake anyone), one good tick to declare recovery. Degraded and recovered carry different dedupe keys, so a recovery is never swallowed by the outage's cooldown. With Redis configured the send right is claimed with `SET NX EX` so N workers page once; on a Redis *error* the claim falls back to process-local, deliberately — N workers each reporting a Redis outage beats an outage that silences its own alert. - `METRICS_TOKEN` — unlocks `GET /metrics` (Prometheus text exposition, hand-rolled over in-process counters, no new dependency). Empty answers **404 rather than 401**, so an install that never configured it is indistinguishable from one where the route does not exist. A *separate* token from `HEALTH_DETAIL_TOKEN` on purpose: this one lives in a long-lived scraper config and exposes traffic volume, latency distribution and error rate, so their rotations should not be coupled. The `Caddyfile` deliberately does not route `/metrics` — reachable from the compose network or an SSH tunnel only, with the token as defence in depth rather than the boundary. No `path` label is ever emitted: path cardinality is unbounded, and label-exploding a hand-rolled registry is how "lightweight" becomes an OOM. - `EMBEDDING_ENDPOINT` — separate from `FREE_MODEL_ENDPOINT`; falls back to it when empty. - `OLLAMA_NATIVE_API` — the Ollama adapter drives Ollama's own `/api/chat` rather than its OpenAI-compatible `/v1` shim, because three controls exist only there. Set `false` to force the shim; the adapter also falls back on its own if `/api/chat` answers 404, so a non-Ollama OpenAI-compatible server behind `FREE_MODEL_ENDPOINT` keeps working. The native path is also what makes `json_schema` a real capability for this provider: `format` compiles the schema into a grammar, so `structured_call` is enforced rather than parsed back out of prose. - `OLLAMA_THINK` — on by default, and *always* suppressed for schema-constrained calls regardless of the setting. Two opposing facts pin it there. Reasoning is charged against `max_tokens` while being returned in a field Maestro never reads, so a quarter to a third of replies used to arrive with **empty content** and `finish_reason=length` (raising `max_tokens` does not help — the model reasons longer). But thinking is also what makes a member use its tools: with it off, the model must choose between emitting a bare tool directive and writing prose in one shot, and it reliably writes prose — 0 tool calls over 18 cases, and 0 again after four escalating prompt rules. The empty-reply cost is now absorbed in code (schema calls never think; an empty free-form reply is retried once without it), so the setting can stay on for the capability. The `/v1` shim ignores every thinking-control parameter, which is the main reason the native endpoint is used at all. - `OLLAMA_NUM_CTX` — Ollama loads a model with a 4096-token context unless told otherwise and truncates a longer prompt **from the front**, which is where the system prompt is. A subagent carrying a fetched page plus a teammate's output passes that easily, so the member would silently lose its role, output format and grounding rules first. - `SITE_URL` — server-only and read at request time, never `NEXT_PUBLIC_*`, so the built image stays domain-agnostic. The backend reads its own copy for building email links. - `PAYMENT_PROVIDER` — only `mock` is implemented here. `BILLING_LIVE` gates the honesty banner shown on `/terms` and `/pricing`. - `EMAIL_PROVIDER` — `console` (default; links appear in logs) or `resend`. - `EMAIL_VERIFICATION_REQUIRED` — the soft gate behind `deps.get_verified_user`: task start, API-key create and custom-API-tool writes answer 403 for an unverified account. It ships **off**, and that is one decision with `EMAIL_PROVIDER=console`, not two — the console sender writes verification mail to the server log, so an enforced gate would lock every account on a fresh install with no way through. Enable it only alongside a real sender. Its frontend twin is `EMAIL_VERIFICATION_LIVE` in `lib/legal/config.ts`, which hides the reminder banner and the "we sent you a link" toast while the gate is off; the backend cannot set a build-time constant, so the pair is flipped together, exactly like `BILLING_ENABLED`/`BILLING_LIVE`. Nothing is removed meanwhile: `/verify-email`, the 6-digit code and the resend endpoint all keep working for anyone who wants them. - `DISPOSABLE_EMAIL_BLOCK_ENABLED` / `DISPOSABLE_DOMAINS_EXTRA` / `EMAIL_MX_CHECK_ENABLED` — the email-identity checks (§8). All ship **on**, unlike `CODE_EXECUTION_ENABLED` and `CUSTOM_API_TOOLS_ENABLED`, because neither widens what the platform will reach out to: one reads a vendored list and the other makes a DNS query that fails open. Each switch is the per-control rollback. There is deliberately no switch for canonicalisation — once the unique index exists, "off" would populate the column under one rule while the index enforced another, which is not a coherent state. `DISPOSABLE_DOMAINS_EXTRA` merges with the vendored list rather than replacing it, so an operator addition cannot silently drop the built-in coverage. - `CAPTCHA_PROVIDER` / `CAPTCHA_SITE_KEY` / `CAPTCHA_SECRET_KEY` — the optional third layer in front of `/auth/register` and `/auth/forgot-password` (§8, "Signup abuse"). `none` is the default and is **not** "unprotected": the honeypot, the challenge nonce and the per-recipient mail budget all still run, and no third party is contacted — the `SENTRY_DSN` zero-egress contract. There is no guard forcing a provider in production, for the `EMAIL_VERIFICATION_REQUIRED` reason; the guard that does exist refuses `turnstile` with an empty key, because a provider that fails closed on every call stops registration without saying so. The site key is served by `GET /auth/challenge` rather than compiled in, so enabling one needs no frontend rebuild. - `SENTRY_DSN` / `FRONTEND_SENTRY_DSN` — two separate projects. Empty means fully off with zero egress; the frontend SDK chunk is never even downloaded. - `DATA_FETCH_ENGINE` — `scrapling` (default) or `httpx`. The httpx path is the pre-Scrapling implementation, kept so a misbehaving engine rolls back with one env var and no redeploy. It is the only engine that enforces the response size cap *while streaming*; Scrapling returns a fully-read response, so there the cap is post-hoc. - `DATA_FETCH_RENDER_ENABLED` — self-host only, and off by default. The image ships no browser binaries (`scrapling install` fetches ~400MB) and a headless Chromium costs 300-500MB RSS. The tool is fully functional without it: the TLS-impersonating HTTP tier is the baseline capability, not a fallback, so a missing browser degrades one request rather than disabling the tool. That is also why `resolve_enabled_tools` deliberately does *not* gate `data_fetch` on a browser probe the way it gates `code_execution` on Docker. - `REPO_INTEL_ENABLED` / `SOCIAL_SEARCH_ENABLED` / `COMMUNITY_READ_ENABLED` / `PLACES_INTEL_ENABLED` — the connected-API tools. Nothing is configured operator-side beyond these switches; the credential is the *user's*, stored under Settings > API Keys. Setting one to `false` removes that tool from every squad declaring it, which is the per-tool rollback. `repo_intel` is the odd one out: GitHub serves anonymous reads at 60/hour, so the `opensource` squad is fully functional with no key and a stored token only raises the ceiling to 5000. It is therefore the only connected tool that can be smoke-tested live from this repo. - `MCP_ENABLED` / `MCP_TIMEOUT_SECONDS` / `MCP_DISCOVERY_TIMEOUT_SECONDS` / `MCP_MAX_USES_PER_SUBTASK` — remote MCP servers (§8). Off by default for `CUSTOM_API_TOOLS_ENABLED`'s reason — the host is user-supplied, so `url_guard`'s unclosed DNS-rebinding window is the normal case rather than the exotic one — **plus** one that is new: the tool descriptions and JSON schemas that shape a subagent's system prompt come from a third party, not from the account owner. `MCP_TIMEOUT_SECONDS` is a deadline for one whole tool call, which is three POSTs, not a per-request timeout. - `MCP_TOOLS_ENABLED` does not exist on purpose: there is one switch, and it is the whole-feature rollback. A per-server toggle is `enabled` on the record, and a per-tool one is the server's `tool_allowlist`. - `PLUGINS_ENABLED` / `PLUGIN_EXTERNAL_IMPORT_ENABLED` / `PLUGIN_IMPORT_TIMEOUT_SECONDS` — plugins (§8). Two switches on purpose. The first is the feature, and it ships off because a bundle's members are skills and MCP servers, and turning plugins on without those is a catalog whose contents cannot run. The second additionally allows importing a manifest from a URL the caller names, which is a categorically different trust story from installing something the publish scan and moderation queue have seen — the operator should have to say yes to it separately. Uninstalling is deliberately *not* gated on either: turning the feature off must never strand records a user cannot remove. - `CODE_EXECUTION_ENABLED` — the one tool whose blast radius is the *host*, so it defaults to `false` and must stay there in production: enabling it requires mounting the Docker socket, which hands agent-authored code the ability to start privileged containers outside the sandbox. The daemon probe in `code_execution_service` is an availability check, not a security boundary — it must never be the only gate, which is exactly why the default is off rather than "on, but harmless without Docker". - `CUSTOM_API_TOOLS_ENABLED` — user-registered HTTP endpoints exposed as agent tools. Off by default, like `CODE_EXECUTION_ENABLED` and for the same kind of reason: the guard in front of it is real but incomplete. `url_guard` checks that a hostname resolves to a globally routable address without pinning that address to the socket, so whoever controls the DNS record can answer the check and the connection differently. Everywhere else that window is theoretical because the host is a constant of ours; this is the only tool where a *user* supplies it. Turning it on says the deployment can accept outbound requests to hosts its users choose. Off removes every `custom_api__*` action from the enabled set and the executor refuses a second time. - `DOCUMENT_SEARCH_ENABLED` / `MEMORY_RECALL_ENABLED` — the RAG tools over the user's own data (uploads and conversation memory). Keyless and per-user scoped, so unlike the connected tools there is nothing to configure beyond the switch; setting one to `false` is the per-tool rollback. They degrade to a "no results" note on a cold Qdrant, never an error, so they are safe to leave on even before any documents are ingested. - `MAIN_AGENT_DISCOVERY_ENABLED` — lets the Main Agent run a bounded, read-only pass over the user's own data (the two RAG tools only) before planning, to ground its plan. Off disables the pass entirely; `MAIN_AGENT_DISCOVERY_MAX_CALLS` bounds it when on. Prices and quotas are not secrets and live in `constants.py`. --- ## 12. License and Business Model Maestro is fair-code / open-core, following the n8n model. - **License:** Sustainable Use License v1.0 (source-available). Anyone may read, run, and modify it for their own use. Reselling it as a commercial service to third parties is not permitted. See `LICENSE`. - **Public:** the entire platform — auth, orchestration, BYOK, RAG, marketplace, the mock payment provider, and the fully local Ollama flow. `docker-compose up` gives a complete working self-hosted instance at zero cost. - **Not public:** the real payment processor adapter, cloud tenancy infrastructure, and operational deployment scripts. The `PaymentProvider` adapter seam already accommodates this split — only the mock lives here. - **Revenue:** hosted subscriptions (starter / pro / scale). Paid plans are currently **parked**: `BILLING_ENABLED` (backend) and `BILLING_LIVE` (frontend) are both false while no real processor is integrated, so `/billing/subscribe` and `/billing/cancel` answer 403 for everyone except admins — who keep the live flow so the operator can test it — and the billing surfaces render "coming soon". Every account runs on the unlimited `free` plan meanwhile. Flip both flags together. Contributions are inbound-licensed under the same terms; see `CONTRIBUTING.md`. --- ## 13. Known Architectural Limit A hosted instance cannot reach an Ollama server running on a user's own machine — all LLM calls are made backend-side, so `localhost:11434` resolves to the server itself. The `ollama` provider works when the operator pulls a chat model into the container, or when a user runs the whole stack locally. Running everything locally is the intended self-hosting path and remains free.