Free-AI Gateway Architecture # ⚑ Free-AI Gateway **Enterprise-grade capability-routed AI Gateway monorepo aggregating free-tier and local AI APIs into reusable libraries, Model Context Protocol (MCP) servers, and OpenAI-compatible HTTP proxies.** [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) [![TypeScript](https://img.shields.io/badge/TypeScript-5.x-blue.svg)](https://www.typescriptlang.org/) [![Node.js](https://img.shields.io/badge/Node.js-%3E%3D18.0.0-green.svg)](https://nodejs.org/) [![Fastify](https://img.shields.io/badge/Fastify-5.x-black.svg)](https://fastify.dev/) [![Docker](https://img.shields.io/badge/Docker-Ready-2496ED.svg)](Dockerfile) [![Learn Guide](https://img.shields.io/badge/Guide-LEARN.md-orange.svg)](LEARN.md) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)
> πŸ“š **New to Free-AI Gateway?** Check out the comprehensive **[Architecture & Developer Guide (LEARN.md)](LEARN.md)** for detailed deep dives, tutorials, and integration patterns. --- ## πŸ“– Architecture & Monorepo Overview **`free-ai-gateway`** is organized as an enterprise monorepo separating **pure AI orchestration infrastructure** from **protocol-specific delivery mechanisms** (HTTP Fastify Proxy, MCP Server, CLI, & Web Examples): ```mermaid flowchart TD subgraph CoreLayer ["@free-ai-gateway/core (Standalone npm package)"] Router["CapabilityRouter & Strategy Engine"] Providers["20 Provider Adapters & Dynamic Registry"] Resilience["QuotaTracker & CircuitBreaker"] Observability["EventBus & MetricsTracker"] Transport["HttpClient with Exponential Backoff"] end subgraph Consumers ["Consumer Applications & Examples"] GatewayApp["apps/gateway (@free-ai-gateway/gateway)
Fastify HTTP OpenAI Proxy & SSE Streamer"] McpApp["packages/mcp (@free-ai-gateway/mcp)
Model Context Protocol Server"] SkillsPkg["packages/skills (@free-ai-gateway/skills)
Agentic IDE Skills & CLI"] CliApp["packages/cli (@free-ai-gateway/cli)
Terminal Assistant & Diagnostics"] NextApp["examples/nextjs-chat
Next.js 14+ Fullstack Streaming UI"] Collections["examples/collections
Postman & Bruno API Test Suites"] end GatewayApp -->|consumes| CoreLayer McpApp -->|consumes| CoreLayer SkillsPkg -->|integrates with| CoreLayer CliApp -->|consumes| CoreLayer NextApp -->|proxies to| GatewayApp Collections -->|tests| GatewayApp ``` ### Monorepo Workspaces Matrix | Package / App | Location | Purpose | Dependencies | | :--- | :--- | :--- | :--- | | **`@free-ai-gateway/core`** | `packages/core` | Protocol-neutral capability router, resilience engine, and 20 provider adapters. | `ajv`, `dotenv` (Zero HTTP server) | | **`@free-ai-gateway/mcp`** | `packages/mcp` | Model Context Protocol server exposing capability tools to AI agents (Claude Desktop, Cursor). | `@free-ai-gateway/core` | | **`@free-ai-gateway/skills`** | `packages/skills` | Agentic IDE skills (`SKILL.md`) and installer CLI for Antigravity, Claude, Cursor, and Copilot. | Standalone CLI & API | | **`@free-ai-gateway/cli`** | `packages/cli` | Terminal AI assistant, interactive chat REPL, model catalog, and diagnostics tool. | `@free-ai-gateway/core`, `@free-ai-gateway/skills` | | **`@free-ai-gateway/gateway`** | `apps/gateway` | High-throughput Fastify HTTP proxy serving OpenAI-compatible endpoints with auto-discovery and SSE. | `@free-ai-gateway/core`, `fastify` | | **`examples/nextjs-chat`** | `examples/nextjs-chat` | Fullstack Next.js 14+ App Router demo with Vercel AI SDK & zero-dep native SSE streams. | `ai`, `openai`, `next`, `react` | | **`examples/collections`** | `examples/collections` | Complete Postman v2.1 and Bruno collections covering all 19 API endpoints. | Standard JSON / `.bru` suites | --- ## ✨ Key Capabilities - 🎯 **Capability-Based Routing**: Request what you need (`model: "auto:tool_calling+structured_output"`), and let the router choose the fastest healthy free provider. - πŸ“ **Strategy Pattern Engine**: Pluggable load balancing strategies (`AdaptiveHealthStrategy`, `LowestLatencyStrategy`, or custom `IRoutingStrategy`). - πŸ”„ **Autonomous Failover**: Transparently cycles through ranked candidate providers until success upon encountering upstream `429` (Rate Limit) or `5xx` errors. - πŸ›‘οΈ **Circuit Breaker**: Detects failing providers and enters exponential cooldown backoff to prevent cascade failures. - ⏱️ **Sliding-Window Quota Tracking**: In-memory accounting of RPM, TPM, and RPD with proactive limit protection. - πŸ”‘ **Multi-Key Pool Rotation**: Configure comma-separated API keys (`PROVIDER_KEY=k1,k2,k3`) with round-robin rotation and per-key rate-limit isolation. - πŸ”Œ **Dynamic Provider Autoloader**: Add new providers by dropping a class extending `BaseProvider` into `packages/core/src/providers/`. - πŸ“‘ **Typed Event Bus & Observability**: Lifecycle events for OpenTelemetry and zero-overhead Prometheus `/metrics` endpoint. - πŸ€– **Model Context Protocol (MCP) Ready**: Use directly in Claude Desktop, Cursor, or agent workflows. --- ## 🧩 Supported Providers Matrix (20 Adapters) | Provider | Modalities / Capabilities | Authentication | Limit Scope | | :--- | :--- | :--- | :--- | | **Google AI Studio** | `text`, `tool_calling`, `vision`, `structured_output`, `embedding`, `tts` | `GOOGLE_API_KEY` | Per Model | | **Groq** | `text`, `tool_calling`, `structured_output`, `reasoning`, `speech_to_text` | `GROQ_API_KEY` | Account | | **Ollama (Local)** | `text`, `code`, `reasoning` | None (Local Daemon) | Local Node | | **SambaNova Cloud** | `text`, `tool_calling`, `reasoning`, `vision` | `SAMBANOVA_API_KEY` | Account | | **NVIDIA NIM** | `text`, `tool_calling`, `reasoning`, `vision`, `embedding`, `rerank`, `moderation` | `NVIDIA_API_KEY` | Account | | **Cohere** | `text`, `tool_calling`, `structured_output`, `reasoning`, `embedding`, `rerank` | `COHERE_API_KEY` | Account | | **OpenRouter** | `text`, `tool_calling`, `vision`, `reasoning`, `embedding`, `tts`, `moderation` | `OPENROUTER_API_KEY` | Account | | **OpenCode Zen** | `code`, `tool_calling`, `reasoning`, `text` | `OPENCODE_API_KEY` | Account | | **Bazaarlink.ai** | `text`, `code` | `BAZAARLINK_API_KEY` | Account | | **aimlapi.com** | `text` | `AIMLAPI_API_KEY` | Account | | **OVHcloud AI** | `text` | `OVHCLOUD_API_KEY` | Per Model | | **Voyage AI** | `embedding` | `VOYAGE_API_KEY` | Account | | **Jina AI** | `embedding`, `rerank` | `JINA_API_KEY` | Account | | **Hugging Face** | `text`, `tool_calling`, `image_gen` | `HUGGINGFACE_API_KEY` | Shared Pool | | **Cloudflare Workers AI** | `image_gen`, `embedding` | `CLOUDFLARE_API_TOKEN` | Shared Pool | | **Google Cloud Platform** | `translation`, `speech_to_text`, `text_to_speech`, `vision` | `GCP_API_KEY` | Account | | **MyMemory** | `translation` | `MYMEMORY_API_KEY` | Account | | **Unstructured.io** | `document_processing` | `UNSTRUCTURED_API_KEY` | Account | | **Exa AI** | `web_search` | `EXA_API_KEY` | Account | | **Tavily** | `web_search` | `TAVILY_API_KEY` | Account | --- ## πŸš€ Quick Start ### 1. Installation ```bash # Clone the repository git clone https://github.com/zaber-dev/free-ai-gateway.git cd free-ai-gateway # Install dependencies across all monorepo workspaces npm install ``` ### 2. Configure Environment Copy `.env.example` to `.env` and provide keys for the providers you wish to enable: ```bash cp .env.example .env ``` ```env PORT=3000 GROQ_API_KEY=gsk_... GOOGLE_API_KEY=AIza... NVIDIA_API_KEY=nvapi-... COHERE_API_KEY=... ``` ### 3. Build & Run ```bash # Compile all workspace packages npm run build # Run all 55 automated tests across all packages npm test # Start the Fastify HTTP Gateway (Dev mode) npm run dev # Start the Gateway in Production npm start ``` --- ## πŸ’» Usage Modalities ### Option A: HTTP Gateway (OpenAI Compatible) Call the local proxy with any OpenAI SDK or `curl`: ```bash curl http://localhost:3000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "auto:tool_calling+structured_output", "messages": [ { "role": "user", "content": "Extract name and age from: Alice is 30 years old." } ] }' ``` ```typescript import OpenAI from "openai"; const client = new OpenAI({ baseURL: "http://localhost:3000/v1", apiKey: "not-needed", }); const completion = await client.chat.completions.create({ model: "auto:reasoning", messages: [{ role: "user", content: "Solve: How many r's in strawberry?" }], }); console.log(completion.choices[0].message.content); ``` For a runnable Python version using the official OpenAI SDK and LangChain, see [examples/python-langchain](examples/python-langchain/README.md). --- ### Option B: Embedding `@free-ai-gateway/core` as a TypeScript Library Embed the capability router directly into your application without launching an HTTP server: ```typescript import { CapabilityRouter, Registry, QuotaTracker, CircuitBreaker, EventBus, LowestLatencyStrategy, } from "@free-ai-gateway/core"; const registry = new Registry(); const quota = new QuotaTracker(); const breaker = new CircuitBreaker(); const eventBus = new EventBus(); // Listen to lifecycle telemetry eventBus.on("request:fallback", (evt) => { console.warn(`[Fallback] Failed on ${evt.attemptedProvider}: ${evt.error}`); }); const router = new CapabilityRouter( registry, quota, breaker, undefined, eventBus, new LowestLatencyStrategy() ); const response = await router.route({ capabilities: ["text", "tool_calling"], payload: { messages: [{ role: "user", content: "Hello AI!" }], }, }); console.log("Served by:", response.servedBy); console.log("Data:", response.data); ``` --- ### Option C: Model Context Protocol (MCP) Server Connect Free-AI Gateway to Claude Desktop or Cursor: ```json { "mcpServers": { "free-ai-gateway": { "command": "node", "args": ["/path/to/free-ai-gateway/packages/mcp/dist/index.js"], "env": { "GROQ_API_KEY": "gsk_...", "GOOGLE_API_KEY": "AIza..." } } } } ``` **Exposed MCP Tools:** * `freeai_generate`: Generate text, reasoning, or code with automatic failover. * `freeai_search`: Web search queries via Exa / Tavily. * `freeai_embed`: Generate vector embeddings via Voyage, Jina, Gemini. * `freeai_rerank`: Rerank documents for retrieval augmented generation (RAG). * `freeai_analyze_image`: Multimodal vision analysis. --- ### Option D: Agentic IDE Skills (`@free-ai-gateway/skills`) Install Free-AI Gateway agent skills directly into your IDE or autonomous coding assistant: ```bash # Install to Google Antigravity (.agents/skills) npx @free-ai-gateway/skills install --target=antigravity # Install to Cursor (.cursor/skills) npx @free-ai-gateway/skills install --target=cursor # Install to Claude Code (.claude/skills) npx @free-ai-gateway/skills install --target=claude # Install to all supported AI assistants npx @free-ai-gateway/skills install --target=all ``` --- ### Option E: Terminal CLI Tool (`@free-ai-gateway/cli`) Use Free-AI directly from your terminal or command-line scripts: ```bash # One-off prompt execution with auto-routing npx @free-ai-gateway/cli "Explain MapReduce in simple terms" # Interactive chat REPL in terminal npx @free-ai-gateway/cli chat --capability=reasoning # Check model catalog with formatting & capability filter npx @free-ai-gateway/cli models --format=markdown --capability=code # Run system diagnostics npx @free-ai-gateway/cli doctor ``` --- ## πŸ›οΈ Monorepo Structure ``` free-ai-gateway/ β”œβ”€β”€ packages/ β”‚ β”œβ”€β”€ core/ # @free-ai-gateway/core β”‚ β”‚ β”œβ”€β”€ AGENTS.md # Agentic guidelines for @free-ai-gateway/core β”‚ β”‚ β”œβ”€β”€ src/ β”‚ β”‚ β”‚ β”œβ”€β”€ capabilities/ # Capability definitions & parsing β”‚ β”‚ β”‚ β”œβ”€β”€ config/ # providers.json, schema, config sources β”‚ β”‚ β”‚ β”œβ”€β”€ errors/ # ProviderError, NoProviderAvailableError β”‚ β”‚ β”‚ β”œβ”€β”€ observability/ # EventBus, MetricsTracker β”‚ β”‚ β”‚ β”œβ”€β”€ providers/ # 20 Provider Adapters + Registry + Loader β”‚ β”‚ β”‚ β”œβ”€β”€ resilience/ # QuotaTracker, CircuitBreaker β”‚ β”‚ β”‚ β”œβ”€β”€ router/ # CapabilityRouter & Strategy Pattern β”‚ β”‚ β”‚ β”œβ”€β”€ transport/ # HttpClient with exponential backoff β”‚ β”‚ β”‚ β”œβ”€β”€ types/ # Unified contracts & response schemas β”‚ β”‚ β”‚ └── index.ts # Public Core API β”‚ β”‚ β”œβ”€β”€ tests/ # 28 Core unit tests β”‚ β”‚ └── package.json β”‚ β”‚ β”‚ β”œβ”€β”€ mcp/ # @free-ai-gateway/mcp β”‚ β”‚ β”œβ”€β”€ AGENTS.md # Agentic guidelines for @free-ai-gateway/mcp β”‚ β”‚ β”œβ”€β”€ src/ β”‚ β”‚ β”‚ β”œβ”€β”€ tools/ # generate, search, embed, rerank, analyze-image β”‚ β”‚ β”‚ β”œβ”€β”€ resources/ # capabilities, models catalog β”‚ β”‚ β”‚ β”œβ”€β”€ server.ts # FreeAiMcpServer handler β”‚ β”‚ β”‚ └── index.ts β”‚ β”‚ β”œβ”€β”€ tests/ # 3 MCP server tests β”‚ β”‚ └── package.json β”‚ β”‚ β”‚ β”œβ”€β”€ skills/ # @free-ai-gateway/skills β”‚ β”‚ β”œβ”€β”€ AGENTS.md # Agentic guidelines for @free-ai-gateway/skills β”‚ β”‚ β”œβ”€β”€ src/ β”‚ β”‚ β”‚ β”œβ”€β”€ skills/ # Built-in skills (free-ai-gateway, scaffolding, mcp) β”‚ β”‚ β”‚ β”œβ”€β”€ installer.ts # Multi-target installer β”‚ β”‚ β”‚ β”œβ”€β”€ cli.ts # CLI executable (free-ai-skills) β”‚ β”‚ β”‚ └── index.ts β”‚ β”‚ β”œβ”€β”€ tests/ # 4 Skills tests β”‚ β”‚ └── package.json β”‚ β”‚ β”‚ └── cli/ # @free-ai-gateway/cli β”‚ β”œβ”€β”€ AGENTS.md # Agentic guidelines for @free-ai-gateway/cli β”‚ β”œβ”€β”€ src/ β”‚ β”‚ β”œβ”€β”€ commands/ # prompt, chat, models, doctor, skills β”‚ β”‚ β”œβ”€β”€ cli.ts # Argument parsing & dispatcher β”‚ β”‚ β”œβ”€β”€ bin.ts # CLI executable (free-ai, freeai) β”‚ β”‚ └── index.ts β”‚ β”œβ”€β”€ tests/ # 8 CLI tests β”‚ └── package.json β”‚ β”œβ”€β”€ apps/ β”‚ └── gateway/ # @free-ai-gateway/gateway (HTTP App) β”‚ β”œβ”€β”€ AGENTS.md # Agentic guidelines for @free-ai-gateway/gateway β”‚ β”œβ”€β”€ src/ β”‚ β”‚ β”œβ”€β”€ adapters/ # OpenAI chat response normalizer & SSE chunker β”‚ β”‚ β”œβ”€β”€ api/ β”‚ β”‚ β”‚ β”œβ”€β”€ routes/ # Fastify route modules & RouteLoader β”‚ β”‚ β”‚ └── server.ts # Server factory, timing hooks, 404 handler β”‚ β”‚ β”œβ”€β”€ jobs/ # Background JobScheduler & reverify worker β”‚ β”‚ └── index.ts β”‚ β”œβ”€β”€ tests/ # 7 Gateway HTTP tests β”‚ β”œβ”€β”€ Dockerfile # Monorepo container builder β”‚ └── package.json β”‚ β”œβ”€β”€ examples/ β”‚ β”œβ”€β”€ nextjs-chat/ # Next.js 14+ App Router & Vercel AI SDK example β”‚ └── collections/ # Ready-to-import Postman & Bruno API suites β”‚ β”œβ”€β”€ tests/ β”‚ └── e2e/ # 5 Cross-package E2E integration tests β”‚ β”œβ”€β”€ AGENTS.md # Monorepo Root Agentic Guidelines β”œβ”€β”€ CLAUDE.md # Claude Code Instructions β”œβ”€β”€ .agents/ # Workspace Skills Directory β”œβ”€β”€ .github/workflows/ci.yml # Matrix CI workflow β”œβ”€β”€ docker-compose.yml β”œβ”€β”€ package.json # Root workspace definition β”œβ”€β”€ tsconfig.base.json # Shared TypeScript compiler settings └── README.md ``` --- ## 🀝 Community & Governance - πŸ“– **[Architecture Blueprint](ARCHITECTURE.md)**: Deep dive into the internal system design and data flow. - πŸŽ“ **[Developer & Learning Guide](LEARN.md)**: Tutorials, programmatic usage, and SDK patterns. - πŸ—ΊοΈ **[Product Roadmap](ROADMAP.md)**: Planned milestones, distributed state, and upcoming features. - πŸ’¬ **[Support Guide](SUPPORT.md)**: Troubleshooting, community discussions, and help channels. - πŸ›οΈ **[Project Governance](GOVERNANCE.md)**: Decision-making process, maintainer roles, and release policies. - ✍️ **[Contributing Guide](CONTRIBUTING.md)**: Step-by-step instructions for adding new provider adapters. - πŸ”’ **[Security Policy](SECURITY.md)**: Vulnerability disclosure guidelines. - πŸ“œ **[Code of Conduct](CODE_OF_CONDUCT.md)**: Community standards and expectations. --- ## πŸ‘€ Author Created and maintained with ❀️ by **[Md. Mahedi Zaman Zaber](https://github.com/zaber-dev)**. --- ## πŸ“„ License This project is open source and available under the [MIT License](LICENSE).