---
title: "Comfy Router realtime models"
sidebarTitle: "Realtime models"
description: "Open live sessions with Genie 4, the fal world models, and Grok Voice through Comfy Router: WebSocket sessions, WebRTC signalling, authentication, limits, and billing."
---
Most Router models take one request and return a finished result. Realtime models instead hold a live session: you send input while the model runs and receive its output as it is produced. Router serves them on two surfaces:
| Model | What it does | Surface |
| --- | --- | --- |
| `google/genie-4` | Genie 4 interactive world model, steered by keys and text | WebSocket session, `GET /v2/realtime/google/google/genie-4` |
| `xai/grok-voice/realtime` | Grok Voice speech-to-speech conversation | WebSocket session, `GET /v2/realtime/fal/xai/grok-voice/realtime` |
| `fal-ai/abot-world-0` | Interactive world model, video over WebRTC | WebRTC signalling, `POST /v2/realtime/wma/fal` |
| `fal-ai/lingbot-world-2` | Interactive world model, video over WebRTC | WebRTC signalling, `POST /v2/realtime/wma/fal` |
All four are billed **per second of session time**, drawn from the same workspace credit balance as other Router calls. See [Router pricing](/development/comfy-router/pricing) for rates. A session needs a funded workspace: with no credits, opening one is refused with `402`.
The WebSocket path is `/v2/realtime/{provider}/{model}`, where `{provider}` is the broker that serves the model and `{model}` is the full model id. That is why `google` appears twice in the Genie 4 path.
## Authentication
Realtime routes accept the same credentials as other Router calls.
- **From a server**, send your API key in `X-API-Key: comfyui-...` or `Authorization: Bearer comfyui-...` on the WebSocket upgrade request or the signalling request. Keep API keys on your server; never ship them to a browser.
- **From a browser**, a WebSocket cannot set request headers. A browser signed in to a Comfy account passes its Firebase ID token in the subprotocol offer instead, alongside `comfy-realtime.v1`:
```js
const ws = new WebSocket(url, ["comfy-realtime.v1", "comfy-bearer." + idToken]);
```
The token must be an unexpired Firebase ID token. API keys are never accepted this way, and credentials are never accepted in the URL query string. An offer that carries `comfy-bearer.` without `comfy-realtime.v1` is refused with `400` before any session opens. When you offer `comfy-realtime.v1`, the server selects it in the `101` response.
The subprotocol offer applies to the WebSocket sessions only. WebRTC signalling is ordinary HTTP, so a browser sends `Authorization: Bearer ` as a header there.
## WebSocket sessions
Open a session with a WebSocket upgrade to `wss://api.comfy.org/v2/realtime/{provider}/{model}`. Router authenticates the upgrade, checks your balance, and relays frames between your client and the model for the life of the session.
### Session rules
- **One live WebSocket session per workspace.** A second upgrade while one is live is refused with `429` and `code: router.realtime.session_already_live`. Close the first session before opening another.
- **Metering starts at the `101`.** Billing runs from the moment the upgrade succeeds until the session ends, in whole seconds, with a minimum of one second. Opening a session and closing it at once still bills that minimum.
- **Balance is re-checked while the session runs.** Router meters the session and re-checks your balance every 15 seconds. If the balance still covers some time, the session's cap is shortened to match (a Genie 4 session then ends at that cap with `1000`). When the balance can no longer cover even one more second, or the account is blocked, Router drops the connection without a close frame (most clients report close code `1006`). If you ran out of credits, top up before reconnecting. A session's maximum length is also shortened up front to what your balance covers.
### Genie 4
`google/genie-4` generates an interactive world from a text prompt that you then move through with key presses. It accepts **text frames only**, each a JSON object with an `event_type`. A binary frame, or a message over 64 KiB, ends the session.
| `event_type` | Fields | Notes |
| --- | --- | --- |
| `world_setup` | `world_prompt: { "text": "..." }` | Must be the first event. The prompt is required, non-empty and at most 8 KiB. Text only: a `reference_image` or any image or video field is refused. Repeating the same `world_setup` is ignored; sending one with a different prompt ends the session. |
| `user_action` | `action`, `input` | `action` is `key_down`, `key_up` or `key_click`. `input` is one of the ten keys below. |
| `input` | `content: [{ "type": "text", "text": "..." }]` | Steers the world with text. Text parts only, each at most 8 KiB. |
| `interaction.complete` | none | Ends the session gracefully. Send it with no other fields. |
The ten keys a `user_action` can name:
| Purpose | Keys |
| --- | --- |
| Movement | `key_w`, `key_s`, `key_a`, `key_d` |
| Camera | `arrow_up`, `arrow_down`, `arrow_left`, `arrow_right` |
| Actions | `space`, `key_e` |
Any other event type, field or key, a `user_action` or `input` before `world_setup`, or a sustained flood of events (beyond a burst of about 120, then about 60 per second) ends the session with close code `1008` (policy violation). Nothing in the refused message reaches the model.
The model's own events come back on the same socket. A session ends:
- after the model's `interaction.completed` event, with close code `1000`;
- after the model's `error` event, with the provider's close code or `1011`;
- at the **120-second** session cap (or sooner, if your balance covers less), with close code `1000`.
Error text in the model's events can be replaced with a generic message before it reaches you.
```python Python
# Python 3.10+
# Install: python -m pip install "websockets>=13"
# Run: COMFY_API_KEY=comfyui-... python genie.py
import asyncio
import json
import os
from websockets.asyncio.client import connect
from websockets.exceptions import InvalidStatus
URL = "wss://api.comfy.org/v2/realtime/google/google/genie-4"
async def receive(ws):
async for message in ws:
if isinstance(message, bytes):
print(f"<- {len(message)} bytes")
continue
event = json.loads(message)
print("<-", event.get("event_type"))
async def main():
try:
ws = await connect(
URL,
additional_headers={"X-API-Key": os.environ["COMFY_API_KEY"]},
max_size=None,
)
except InvalidStatus as err:
# A refused upgrade: the body is JSON with detail, error_type and code.
print(err.response.status_code, err.response.body.decode(errors="replace"))
return
async with ws:
reader = asyncio.create_task(receive(ws))
async def send(event):
await ws.send(json.dumps(event))
await send({
"event_type": "world_setup",
"world_prompt": {"text": "A quiet harbor town at sunset, seen from the docks"},
})
await send({"event_type": "user_action", "action": "key_down", "input": "key_w"})
await asyncio.sleep(2)
await send({"event_type": "user_action", "action": "key_up", "input": "key_w"})
await send({"event_type": "user_action", "action": "key_click", "input": "arrow_left"})
await send({"event_type": "input", "content": [{"type": "text", "text": "Make it start to rain"}]})
await asyncio.sleep(5)
await send({"event_type": "interaction.complete"})
await reader
print("closed:", ws.close_code)
asyncio.run(main())
```
```js Browser
// A page signed in with a Comfy account. getIdToken() is your Firebase auth
// call; the token travels in the subprotocol offer, never in the URL.
const idToken = await auth.currentUser.getIdToken();
const ws = new WebSocket(
"wss://api.comfy.org/v2/realtime/google/google/genie-4",
["comfy-realtime.v1", "comfy-bearer." + idToken],
);
ws.addEventListener("open", () => {
ws.send(JSON.stringify({
event_type: "world_setup",
world_prompt: { text: "A quiet harbor town at sunset, seen from the docks" },
}));
});
// Map the ten supported keys to Genie 4 key names.
const keys = {
KeyW: "key_w", KeyS: "key_s", KeyA: "key_a", KeyD: "key_d",
ArrowUp: "arrow_up", ArrowDown: "arrow_down",
ArrowLeft: "arrow_left", ArrowRight: "arrow_right",
Space: "space", KeyE: "key_e",
};
for (const [type, action] of [["keydown", "key_down"], ["keyup", "key_up"]]) {
window.addEventListener(type, (e) => {
const input = keys[e.code];
if (!input || e.repeat || ws.readyState !== WebSocket.OPEN) return;
ws.send(JSON.stringify({ event_type: "user_action", action, input }));
});
}
ws.addEventListener("message", (e) => {
if (typeof e.data === "string") console.log("<-", JSON.parse(e.data).event_type);
});
ws.addEventListener("close", (e) => console.log("closed", e.code));
```
The browser example sends one `key_down` and one `key_up` per physical key press and skips auto-repeat, which keeps it well inside the event rate.
### Grok Voice
`xai/grok-voice/realtime` is a speech-to-speech conversation. The session speaks fal's realtime wire format: send each event as a [MessagePack](https://msgpack.org)-encoded binary frame; responses arrive as MessagePack binary frames or JSON text frames. The events inside are xAI's realtime voice events.
You can send these client events: `session.update`, `x-fal-session.configure`, `input_audio_buffer.append`, `input_audio_buffer.commit`, `input_audio_buffer.clear`, `conversation.item.create`, `conversation.item.delete`, `conversation.item.truncate`, `response.create` and `response.cancel`. Each event may carry only its documented fields.
- Tools are limited to client-executed `function` tools. Server-side tools such as MCP servers and web or X search are refused.
- Image and video content is refused anywhere in an event.
- If `session.model` is set, it must be `grok-voice-latest`, `grok-voice-3.0` or `grok-voice-2.0`.
A message that breaks these rules is not forwarded and ends the session with close code `1008`. The session cap is **30 minutes**, shortened up front to what your balance covers; at the cap the connection is closed without a close frame.
A Grok Voice session costs **$0.0011869 per second** (about $0.0712 per minute; at 211 credits = $1, about 0.25 credits per second or 15 credits per minute). It bills for the whole time the socket is open, whether or not audio is flowing.
```js Node
// Node 20+
// Install: npm install ws @msgpack/msgpack
// Run: COMFY_API_KEY=comfyui-... node grok-voice.mjs
import { decode, encode } from "@msgpack/msgpack";
import WebSocket from "ws";
const ws = new WebSocket("wss://api.comfy.org/v2/realtime/fal/xai/grok-voice/realtime", {
headers: { "X-API-Key": process.env.COMFY_API_KEY },
});
const send = (event) => ws.send(Buffer.from(encode(event)));
ws.on("unexpected-response", (_req, res) => {
let body = "";
res.on("data", (chunk) => (body += chunk));
res.on("end", () => console.error("refused:", res.statusCode, body));
});
ws.on("open", () => {
send({
type: "session.update",
session: {
voice: "Ara",
instructions: "Answer in one short sentence.",
turn_detection: { type: null },
},
});
send({
type: "conversation.item.create",
item: { type: "message", role: "user", content: [{ type: "input_text", text: "Say hello." }] },
});
send({ type: "response.create" });
});
ws.on("message", (data, isBinary) => {
const event = isBinary ? decode(data) : JSON.parse(data.toString());
console.log("<-", event.type);
if (event.type === "response.done" || event.type === "error") ws.close(1000);
});
ws.on("close", (code) => console.log("closed", code));
```
To send speech instead of text, stream audio with `input_audio_buffer.append` (base64 `audio`), then `input_audio_buffer.commit` and `response.create`.
## WebRTC world models
`fal-ai/abot-world-0` and `fal-ai/lingbot-world-2` stream video over WebRTC. The media travels **directly between the browser and fal**, peer to peer; Router handles only the HTTP signalling that sets the connection up, adding fal's credentials so your client never holds them.
Every signalling call is a `POST` to `https://api.comfy.org/v2/realtime/wma/fal` with your Comfy credential, a JSON body, and the real destination in the `x-fal-target-url` header. This is the proxy contract fal's JavaScript client uses, so you can point that client's proxy setting at this route. Only these destinations are accepted; anything else is refused with `400` and `code: router.realtime.wma_target_rejected`:
| `x-fal-target-url` | Body | Purpose |
| --- | --- | --- |
| `https://wma.fal.run/ice` | `{"app_id": ""}` | Fetch ICE servers. Opens no session and is not billed. |
| `https://wma.fal.run/session` | includes `"app_id": ""` | Open a session. The response carries its `session_id`. |
| `https://wma.fal.run/session/heartbeat` | `{"session_id": ""}` | Keep the session alive. |
| `https://wma.fal.run/session/close` | `{"session_id": ""}` | End the session. `DELETE /v2/realtime/wma/fal` with the same body does the same. |
The body must be a JSON object of at most 64 KiB with no repeated keys. Spell `app_id` and `session_id` exactly so; a different capitalization is refused with `400`.
```bash
# Fetch ICE servers for a world model (no session, no charge).
curl -X POST https://api.comfy.org/v2/realtime/wma/fal \
-H "X-API-Key: $COMFY_API_KEY" \
-H "Content-Type: application/json" \
-H "x-fal-target-url: https://wma.fal.run/ice" \
-d '{"app_id": "fal-ai/lingbot-world-2"}'
```
```js
// Heartbeat and close from a browser signed in with a Comfy account.
const signal = (target, body) =>
fetch("https://api.comfy.org/v2/realtime/wma/fal", {
method: "POST",
headers: {
Authorization: `Bearer ${idToken}`,
"Content-Type": "application/json",
"x-fal-target-url": target,
},
body: JSON.stringify(body),
});
const beat = setInterval(
() => signal("https://wma.fal.run/session/heartbeat", { session_id: sessionId }),
5000,
);
// When the user stops: tear down the peer connection, then close the session.
clearInterval(beat);
peerConnection.close();
const res = await signal("https://wma.fal.run/session/close", { session_id: sessionId });
console.log(await res.json()); // { session_id, closed: true, settled: true | false }
```
### Session lifecycle and billing
- **Maximum length.** Each session runs for at most **480 seconds**.
- **Heartbeats.** Send one every 5 seconds, as fal's client does. Router forwards at most one per second per session; a faster heartbeat is refused with `429` and `Retry-After: 1`. Heartbeats and closes are accepted only for a `session_id` your account opened and has not closed; otherwise the answer is `404` with `code: router.realtime.wma_session_not_found`.
- **Billing.** A session is billed per second from when it opens until 5 seconds after its last heartbeat, up to the 480-second maximum. Closing the session settles the bill right away; a session whose heartbeats stop for more than 15 seconds is settled on its own. The close response's `settled: false` means the bill is still being finalized and lands shortly.
- **Balance check at open.** Router cannot stop a peer-to-peer session once it starts, so opening one requires a balance that covers the full 480 seconds of this session plus the full length of every WebRTC session your workspace already has open. If it does not, `/session` is refused with `402` and `code: router.realtime.insufficient_credits`. This is a check, not a reservation: other spend can draw on the same balance while a session runs.
- **Concurrent sessions.** A workspace can hold **2** WebRTC world-model sessions at a time. A closed session keeps its slot until about 480 seconds after it opened (plus a short settle window), so a quick close-and-reopen can be refused with `429` and `code: router.realtime.concurrency_limit`; wait for the `Retry-After` the response names.
When fal itself answers a signalling call with an error, Router passes fal's status and body through unchanged.
## Errors
A refused upgrade or signalling call answers with an HTTP status and a JSON body carrying `detail`, `error_type` and a `code` that names the reason. Branch on `code` or `error_type` rather than on `detail`. Once a WebSocket session is open, problems are reported with a close code instead.
| Status | `code` | Meaning |
| --- | --- | --- |
| `400` | `router.realtime.subprotocol_required` | A `comfy-bearer.` offer came without `comfy-realtime.v1`. |
| `401` | | No valid credential. |
| `402` | `router.realtime.insufficient_credits` | The workspace balance is too low to open a session. |
| `403` | `router.realtime.model_unavailable` | The model is not available to your account. |
| `403` | `router.realtime.account_blocked` | The account is blocked. |
| `404` | `router.realtime.model_not_allowed` | The model is not served on this route. Check the path and model id. |
| `426` | `router.realtime.upgrade_required` | The WebSocket route got a plain HTTP request. Check that no proxy strips the `Upgrade` and `Connection` headers. |
| `429` | `router.realtime.rate_limited` | Too many requests, or the provider is at capacity. Wait for `Retry-After`. |
| `429` | `router.realtime.session_already_live` | Your workspace already has a live WebSocket session. |
| `429` | `router.realtime.concurrency_limit` | Too many concurrent sessions. Retry later. |
| `503` | `router.realtime.shutting_down` | The server handling the request is restarting. Retry. |
The upgrade and the `/ice` and `/session` signalling calls spend the same per-caller [request allowance](/development/comfy-router/limitations#requests-are-rate-limited-per-caller) as other Router calls.
| Close code | Meaning |
| --- | --- |
| `1000` | Normal end: the model finished, or a Genie 4 session reached its cap. |
| `1006` | No close frame arrived. Router causes include a Grok Voice session reaching its time cap, running out of credits, or a blocked account; a network drop or proxy timeout looks the same. Not sent by Router; your client reports it when the connection drops. |
| `1008` | A client message broke the model's rules (see each model above). |
| `1011` | The model reported an error. |
| `1012` | The server is restarting. Open a new session. |
## Next
What Router supports, and its deadlines and rate limits.
Rates and billing options.