π± Quick Preview: Animated overview of the LLMRouter chat interface showing real-time routing and model selection.
Launch the chat interface (requires API keys - see [Setting Up API Keys](#-setting-up-api-keys) section):
```bash
# Basic chat interface
llmrouter chat --router knnrouter --config config.yaml
# Custom host and port
llmrouter chat --router knnrouter --config config.yaml --host 0.0.0.0 --port 7860
# With public sharing link
llmrouter chat --router knnrouter --config config.yaml --share
# Specify query mode
llmrouter chat --router knnrouter --config config.yaml --mode full_context --top_k 5
```
Query Modes:
- `current_only`: Routes based on current query only (default)
- `full_context`: Combines all chat history with current query
- `retrieval`: Retrieves top-k similar historical queries for context
### Direct Script Execution
You can also run the CLI scripts directly:
```bash
# Training
python -m llmrouter.cli.router_train --router knnrouter --config config.yaml
# Inference
python -m llmrouter.cli.router_inference --router knnrouter --config config.yaml --query "Hello"
# Chat
python -m llmrouter.cli.router_chat --router knnrouter --config config.yaml
```
## π¨ ComfyUI Interface
LLMRouter offers a powerful **Visual Interface** via [ComfyUI](https://github.com/Comfy-Org/ComfyUI), transforming how you interact with the routing pipeline. Instead of editing YAML files and running terminal scripts, you can drag, drop, and connect nodes to build your workflow.
### Key Highlights
- **Visual Configuration**: Forget complex YAML files and terminal scripts. Adjust parameters (e.g., sample size, model candidates) and select datasets directly on the canvas.
- **End-to-End Automation**: Seamlessly link nodes to build a complete pipeline: Data Generation $\to$ Router Training $\to$ Evaluation.
- **Real-Time Monitoring**: Track the status of query generation, embedding extraction, and model training with instant visual feedback.
- **Modular Design**: Custom construct your pipeline by dragging, dropping, and connecting nodes for Datasets, LLMs, and Routers.
### Installation & Setup
Prerequisites: You must have [ComfyUI](https://github.com/Comfy-Org/ComfyUI) installed.
To install the LLMRouter custom nodes, you need to create two symbolic links (soft links).
#### 1. Link the Custom Nodes
This allows ComfyUI to load the LLMRouter Python backend logic in the ComfyUI "Nodes" category.
```bash
ln -s /path/to/LLMRouter/ComfyUI /path/to/ComfyUI/custom_nodes/LLMRouter
```
#### 2. Link the Workflow Example (Optional)
This allows you to see the pre-configured workflow in the ComfyUI "Workflows" category.
```bash
ln -s /path/to/LLMRouter/ComfyUI/workflows/llm_router_example.json /path/to/ComfyUI/user/default/workflows/llm_router_example.json
```
#### 3. Running the Application
To start the ComfyUI server with the LLMRouter nodes:
```bash
python /path/to/ComfyUI/main.py
```
#### 4. Remote Access & Port Forwarding
If you are running ComfyUI on a remote server (e.g., a compute cluster) and wish to access the interface locally, you can use SSH tunneling. Once the tunnel is established, access the interface at `http://127.0.0.1:8188`.
### Using the ComfyUI Interface
#### Find the Nodes
To use the nodes:
1. Open the ComfyUI web interface.
2. Use the **Node Library** sidebar or **Right-click** on the canvas.
3. Navigate to the **`LLMRouter`** category.
4. You will find nodes organized by function:
- **Data**: `Select Datasets`, `Select LLMs`, `Generate Data`.
- **Single-Round**: `KNN Router`, `SVM Router`, `MLP Router`, etc.
- **Multi-Round / Agentic**: Specialized routers for complex tasks.
#### Load the Example
To use the ready-to-run example:
1. Click the **`Workflows`** tab.
2. Select **`llm_router_example.json`**.
3. This loads a complete pipeline.
## π§ Creating Your Own Routers
LLMRouter supports a **plugin system** that allows you to add custom router implementations without modifying the core codebase. This makes it easy to experiment with new routing strategies or domain-specific routers.
### Quick Start
**1. Create your router directory:**
```bash
mkdir -p custom_routers/my_router
```
**2. Implement your router** (`custom_routers/my_router/router.py`):
```python
from llmrouter.models.meta_router import MetaRouter
import torch.nn as nn
class MyRouter(MetaRouter):
"""Your custom router implementation."""
def __init__(self, yaml_path: str):
# Initialize with a model (can be nn.Identity() for simple routers)
model = nn.Identity()
super().__init__(model=model, yaml_path=yaml_path)
# Get available LLM names from config
self.llm_names = list(self.llm_data.keys())
def route_single(self, query_input: dict) -> dict:
"""Route a single query to the best LLM."""
query = query_input['query']
# Your custom routing logic here
# Example: route based on query length
selected_llm = (self.llm_names[0] if len(query) < 50
else self.llm_names[-1])
return {
"query": query,
"model_name": selected_llm,
"predicted_llm": selected_llm,
}
def route_batch(self, batch: list) -> list:
"""Route multiple queries."""
return [self.route_single(q) for q in batch]
```
**3. Create configuration** (`custom_routers/my_router/config.yaml`):
```yaml
data_path:
llm_data: 'data/example_data/llm_candidates/default_llm.json'
hparam:
# Your hyperparameters here
# Optional: Default API endpoint (used as fallback if models don't specify their own)
# Individual models can override this by specifying api_endpoint in the llm_data JSON file
api_endpoint: 'https://integrate.api.nvidia.com/v1'
```
**4. Use your custom router** (same as built-in routers!):
```bash
# Inference
llmrouter infer --router my_router \
--config custom_routers/my_router/config.yaml \
--query "What is machine learning?"
# List all routers (including custom ones)
llmrouter list-routers
```
### Plugin Discovery
Custom routers are automatically discovered from:
- `./custom_routers/` (recommended - project directory)
- `~/.llmrouter/plugins/` (user home directory)
- `$LLMROUTER_PLUGINS` environment variable (colon-separated paths)
### Example Routers
LLMRouter includes example custom routers you can learn from:
**RandomRouter** - Simple baseline that randomly selects an LLM
```bash
llmrouter infer --router randomrouter \
--config custom_routers/randomrouter/config.yaml \
--query "Hello world"
```
**ThresholdRouter** - Advanced trainable router with difficulty estimation
```bash
# Train the router
llmrouter train --router thresholdrouter \
--config custom_routers/thresholdrouter/config.yaml
# Use for inference
llmrouter infer --router thresholdrouter \
--config custom_routers/thresholdrouter/config.yaml \
--query "Explain quantum computing"
```
### Documentation
For detailed guides on creating custom routers:
- π **Quick Start**: [custom_routers/README.md](custom_routers/README.md)
- π **Implementation Summary**: [CUSTOM_ROUTER_SUMMARY.md](CUSTOM_ROUTER_SUMMARY.md)
### Common Routing Patterns
**Rule-based routing:**
```python
def route_single(self, query_input):
query = query_input['query'].lower()
if 'code' in query:
return {"model_name": "code-specialist"}
elif len(query) < 50:
return {"model_name": "small-fast-model"}
else:
return {"model_name": "large-capable-model"}
```
**Embedding-based routing:**
```python
from llmrouter.utils import get_longformer_embedding
def route_single(self, query_input):
embedding = get_longformer_embedding(query_input['query'])
# Use embedding similarity to select best model
selected = self._find_best_model(embedding)
return {"model_name": selected}
```
**Cost-optimized routing:**
```python
def route_single(self, query_input):
difficulty = self._estimate_difficulty(query_input)
# Select cheapest model that can handle the difficulty
for model_name, info in sorted(self.llm_data.items(),
key=lambda x: x[1]['cost']):
if info['capability'] >= difficulty:
return {"model_name": model_name}
```
## π Adding Your Own Tasks
LLMRouter supports **custom task definitions** that allow you to add new task types with custom prompt templates and evaluation metrics. Custom tasks are automatically discovered and integrated into the data generation and evaluation pipeline.
### Quick Start
**1. Create a task formatter** (`custom_tasks/my_tasks.py`):
```python
from llmrouter.utils.prompting import register_prompt
from llmrouter.prompts import load_prompt_template
@register_prompt('my_task', default_metric='my_metric')
def format_my_task_prompt(sample_data):
system_prompt = load_prompt_template("task_my_task")
user_query = f"Question: {sample_data.get('query', '')}"
return {"system": system_prompt, "user": user_query}
```
**2. Create a prompt template** (`custom_tasks/task_prompts/task_my_task.yaml`):
```yaml
template: |
You are an expert at [task description]. [Instructions].
```
**3. Register a custom metric** (optional):
```python
from llmrouter.evaluation import evaluation_metric
@evaluation_metric('my_metric')
def my_metric(prediction: str, ground_truth: str, **kwargs) -> float:
return 1.0 if prediction == ground_truth else 0.0
```
**4. Use your custom task:**
```python
import custom_tasks.my_tasks # Import triggers registration
from llmrouter.utils import generate_task_query
from llmrouter.utils.evaluation import calculate_task_performance
# Generate prompt
prompt = generate_task_query('my_task', {'query': '...'})
# Evaluate (metric automatically inferred from task)
score = calculate_task_performance(
prediction="...",
ground_truth="...",
task_name="my_task"
)
```
### Documentation
For detailed guides on creating custom tasks:
- π **Complete Guide**: [custom_tasks/README.md](custom_tasks/README.md)
### π₯ Hands-on: Multi-View Video Tasks
Follow our **step-by-step walkthrough** in the [Charades-Ego Integration Guide](data/charades_ego/README.md) to process paired egocentric videos, generate VLM-based features, and train routers for **Activity**, **Object**, and **Verb** recognition.
## π xRouteBench Benchmark Pipeline
Reproduce the full router benchmark with one command. The
[`benchmark_pipeline/`](benchmark_pipeline/) folder trains and evaluates
**17 routers on the 8 [xRouteBench](https://huggingface.co/datasets/ulab-ai/xRouteBench)
datasets** (classic NLP, memory, time-series, video, multimodal math,
personalized), including cost-aware Pareto training with a composite
`alpha * performance - beta * cost` reward.
```bash
cd benchmark_pipeline
python download_data.py # pull data from HF (ulab-ai/xRouteBench)
python generate_embeddings.py # Qwen3-Embedding-0.6B query embeddings
python run_pipeline.py --datasets all --routers local # 13 local routers, zero API cost
python aggregate_results.py --csv # per-dataset tables + overall ranking
```
Evaluation **replays pre-recorded model executions** β every query in
xRouteBench was pre-run against all 18 candidate LLMs β so the local-router
sweep costs nothing to run. API-calling routers (multi-round, Router-R1,
Automix) are available behind `--include-api-routers`. See
[`benchmark_pipeline/README.md`](benchmark_pipeline/README.md) for details.
## π OpenClaw Router (OpenClaw Integration)
**OpenClaw Router** is an OpenAI-compatible API server that brings LLMRouter's intelligent routing to production environments. It integrates seamlessly with [OpenClaw](https://github.com/openclaw/openclaw), enabling you to deploy LLM routing via Slack, Discord, and other messaging platforms.
### Why OpenClaw Router?
| Feature | Benefit |
|---------|---------|
| **OpenAI-Compatible API** | Drop-in replacement for any OpenAI client (`/v1/chat/completions`) |
| **All Routing Strategies** | Use any of the 16+ LLMRouter strategies (KNN, SVM, MLP, LLM-based, etc.) |
| **Multimodal Understanding** | Process images, audio, and video - convert to text for routing decisions |
| **Routing Memory** | Persist queryβmodel history; retrieve similar past routes for better decisions |
| **Streaming Support** | Full streaming responses with optional `[model_name]` prefix |
| **Multi-Provider** | Route to Together AI, NVIDIA, OpenAI, Anthropic, or local models |
### Architecture
```
βββββββββββββββββββ ββββββββββββββββββββββββ βββββββββββββββββββββββ
β Slack/Discord ββββββΆβ OpenClaw Gateway ββββββΆβ OpenClaw Router β
β (Mobile/Web) β β (Socket Mode) β β (Port 8000) β
βββββββββββββββββββ ββββββββββββββββββββββββ ββββββββββββ¬βββββββββββ
β
ββββββββββββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββ
β β β
βΌ βΌ βΌ
βββββββββββββββββββ βββββββββββββββββββ βββββββββββββββββββ
β Fast Model β β Balanced Model β β Powerful Model β
β (e.g. 8B) β β (e.g. 70B) β β (e.g. 405B) β
βββββββββββββββββββ βββββββββββββββββββ βββββββββββββββββββ
```
### Quick Start
**1. Configure OpenClaw Router** (`openclaw_router/config.yaml`):
```yaml
serve:
host: "0.0.0.0"
port: 8000
show_model_prefix: true
router:
strategy: llm # or: random, round_robin, rules, llmrouter
provider: together
base_url: https://api.together.xyz/v1
model: "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo"
api_keys:
together: ${TOGETHER_API_KEY}
llms:
llama-3.1-8b:
provider: together
model: "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo"
base_url: https://api.together.xyz/v1
description: "Fast responses"
llama-3.3-70b:
provider: together
model: "meta-llama/Llama-3.3-70B-Instruct-Turbo"
base_url: https://api.together.xyz/v1
description: "Complex reasoning"
```
**2. Start the server**:
```bash
# Using the startup script (recommended - also starts OpenClaw gateway)
./scripts/start-openclaw.sh
# Or directly via CLI
llmrouter serve --config openclaw_router/config.yaml
# With ML-based router
llmrouter serve --config openclaw_router/config.yaml --router knnrouter
```
**3. Test the API**:
```bash
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "auto",
"messages": [{"role": "user", "content": "Explain quantum computing"}]
}'
```
### Optional Features
**Routing Memory** (retrieval-augmented routing):
```yaml
memory:
enabled: true
path: "${HOME}/.llmrouter/openclaw_memory.jsonl"
top_k: 10
retriever_model: "facebook/contriever-msmarco"
```
**Media Understanding** (multimodal support):
```yaml
media:
enabled: true
vision_model: "Qwen/Qwen3-VL-8B-Instruct"
audio_model: "openai/whisper-large-v3"
```
### Documentation
For complete setup instructions including Slack/Discord integration:
- π **Full Guide**: [openclaw_router/README.md](openclaw_router/README.md)
## πΊοΈ TODO
- [ ] Improve personalized routers: stronger user profiling, cold-start strategies, and online feedback updates.
- [ ] Integrate a multimodal router: support image/audio inputs and route by modality + task type to the right multimodal model.
- [ ] Add continual/online learning to adapt routers to domain drift (e.g., periodic re-training + feedback loops).
## π Acknowledgments
LLMRouter builds upon the excellent research from the community. We gratefully acknowledge the following works that inspired our router implementations:
- [**RouteLLM**](https://arxiv.org/abs/2406.18665) - Learning to Route LLMs with Preference Data (ICLR 2025)
- [**RouterDC**](https://arxiv.org/abs/2409.19886) - Query-Based Router by Dual Contrastive Learning (NeurIPS 2024)
- [**AutoMix**](https://arxiv.org/abs/2310.12963) - Automatically Mixing Language Models (NeurIPS 2024)
- [**Hybrid LLM**](https://arxiv.org/abs/2404.14618) - Cost-Efficient and Quality-Aware Query Routing (ICLR 2024)
- [**GraphRouter**](https://arxiv.org/abs/2410.03834) - A Graph-based Router for LLM Selections (ICLR 2025)
- [**GMTRouter**](https://arxiv.org/abs/2511.08590) - Personalized LLM Router over Multi-turn User Interactions
- [**PersonalizedRouter**](https://arxiv.org/abs/2511.16883) - Personalized LLM Routing via Graph-based User Preference Modeling
- [**Router-R1**](https://arxiv.org/abs/2506.09033) - Teaching LLMs Multi-Round Routing and Aggregation via RL (NeurIPS 2025)
- [**FusionFactory**](https://arxiv.org/abs/2507.10540) - Fusing LLM Capabilities with Multi-LLM Log Data
We warmly welcome contributions from the community! A powerful open-source router framework requires the collective effort of everyone. If you have developed a new routing method, please consider submitting a PR to add it to LLMRouter. Together, we can build the most comprehensive LLM routing library!
## π€ Contribution
We warmly welcome contributions from the community. **LLMRouter is a living, extensible research framework**, and its impact grows through the creativity and expertise of its contributors.
If you have developed a **new routing strategy, learning objective, training paradigm, or evaluation protocol**, we strongly encourage you to submit a pull request to integrate it into LLMRouter. **All accepted contributions are explicitly credited**, documented, and made available to a broad research and practitioner audience.
Contributing to LLMRouter is more than adding code. It is an opportunity to **increase the visibility, adoption, and long-term impact of your work** within the LLM systems community. Together, we aim to build the **most comprehensive and extensible open-source library for LLM routing**.
> **Notable contributions** may be highlighted in documentation, examples, benchmarks, or future releases.
## Star History
## π Citation
If you find LLMRouter useful for your research or projects, please cite it as:
```bibtex
@article{feng2026llmrouter,
title={LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers},
author={Feng, Tao and Yu, Fangxu and Zhang, Haozhen and Dai, Zhongjie and Yuan, Liangqi and Lei, Zijie and Zhang, Weizhi and Zhu, Kunlun and Yue, Haodong and Xuan, Keyang and others},
journal={arXiv preprint arXiv:2608.06867},
year={2026}
}
```
