--- title: "Comfy Router realtime models" sidebarTitle: "Realtime models" description: "Open live sessions with Genie 4, the fal world models, and Grok Voice through Comfy Router: WebSocket sessions, WebRTC signalling, authentication, limits, and billing." --- Most Router models take one request and return a finished result. Realtime models instead hold a live session: you send input while the model runs and receive its output as it is produced. Router serves them on two surfaces: | Model | What it does | Surface | | --- | --- | --- | | `google/genie-4` | Genie 4 interactive world model, steered by keys and text | WebSocket session, `GET /v2/realtime/google/google/genie-4` | | `xai/grok-voice/realtime` | Grok Voice speech-to-speech conversation | WebSocket session, `GET /v2/realtime/fal/xai/grok-voice/realtime` | | `fal-ai/abot-world-0` | Interactive world model, video over WebRTC | WebRTC signalling, `POST /v2/realtime/wma/fal` | | `fal-ai/lingbot-world-2` | Interactive world model, video over WebRTC | WebRTC signalling, `POST /v2/realtime/wma/fal` | All four are billed **per second of session time**, drawn from the same workspace credit balance as other Router calls. See [Router pricing](/development/comfy-router/pricing) for rates. A session needs a funded workspace: with no credits, opening one is refused with `402`. The WebSocket path is `/v2/realtime/{provider}/{model}`, where `{provider}` is the broker that serves the model and `{model}` is the full model id. That is why `google` appears twice in the Genie 4 path. ## Authentication Realtime routes accept the same credentials as other Router calls. - **From a server**, send your API key in `X-API-Key: comfyui-...` or `Authorization: Bearer comfyui-...` on the WebSocket upgrade request or the signalling request. Keep API keys on your server; never ship them to a browser. - **From a browser**, a WebSocket cannot set request headers. A browser signed in to a Comfy account passes its Firebase ID token in the subprotocol offer instead, alongside `comfy-realtime.v1`: ```js const ws = new WebSocket(url, ["comfy-realtime.v1", "comfy-bearer." + idToken]); ``` The token must be an unexpired Firebase ID token. API keys are never accepted this way, and credentials are never accepted in the URL query string. An offer that carries `comfy-bearer.` without `comfy-realtime.v1` is refused with `400` before any session opens. When you offer `comfy-realtime.v1`, the server selects it in the `101` response. The subprotocol offer applies to the WebSocket sessions only. WebRTC signalling is ordinary HTTP, so a browser sends `Authorization: Bearer ` as a header there. ## WebSocket sessions Open a session with a WebSocket upgrade to `wss://api.comfy.org/v2/realtime/{provider}/{model}`. Router authenticates the upgrade, checks your balance, and relays frames between your client and the model for the life of the session. ### Session rules - **One live WebSocket session per workspace.** A second upgrade while one is live is refused with `429` and `code: router.realtime.session_already_live`. Close the first session before opening another. - **Metering starts at the `101`.** Billing runs from the moment the upgrade succeeds until the session ends, in whole seconds, with a minimum of one second. Opening a session and closing it at once still bills that minimum. - **Balance is re-checked while the session runs.** Router meters the session and re-checks your balance every 15 seconds. If the balance still covers some time, the session's cap is shortened to match (a Genie 4 session then ends at that cap with `1000`). When the balance can no longer cover even one more second, or the account is blocked, Router drops the connection without a close frame (most clients report close code `1006`). If you ran out of credits, top up before reconnecting. A session's maximum length is also shortened up front to what your balance covers. ### Genie 4 `google/genie-4` generates an interactive world from a text prompt that you then move through with key presses. It accepts **text frames only**, each a JSON object with an `event_type`. A binary frame, or a message over 64 KiB, ends the session. | `event_type` | Fields | Notes | | --- | --- | --- | | `world_setup` | `world_prompt: { "text": "..." }` | Must be the first event. The prompt is required, non-empty and at most 8 KiB. Text only: a `reference_image` or any image or video field is refused. Repeating the same `world_setup` is ignored; sending one with a different prompt ends the session. | | `user_action` | `action`, `input` | `action` is `key_down`, `key_up` or `key_click`. `input` is one of the ten keys below. | | `input` | `content: [{ "type": "text", "text": "..." }]` | Steers the world with text. Text parts only, each at most 8 KiB. | | `interaction.complete` | none | Ends the session gracefully. Send it with no other fields. | The ten keys a `user_action` can name: | Purpose | Keys | | --- | --- | | Movement | `key_w`, `key_s`, `key_a`, `key_d` | | Camera | `arrow_up`, `arrow_down`, `arrow_left`, `arrow_right` | | Actions | `space`, `key_e` | Any other event type, field or key, a `user_action` or `input` before `world_setup`, or a sustained flood of events (beyond a burst of about 120, then about 60 per second) ends the session with close code `1008` (policy violation). Nothing in the refused message reaches the model. The model's own events come back on the same socket. A session ends: - after the model's `interaction.completed` event, with close code `1000`; - after the model's `error` event, with the provider's close code or `1011`; - at the **120-second** session cap (or sooner, if your balance covers less), with close code `1000`. Error text in the model's events can be replaced with a generic message before it reaches you. ```python Python # Python 3.10+ # Install: python -m pip install "websockets>=13" # Run: COMFY_API_KEY=comfyui-... python genie.py import asyncio import json import os from websockets.asyncio.client import connect from websockets.exceptions import InvalidStatus URL = "wss://api.comfy.org/v2/realtime/google/google/genie-4" async def receive(ws): async for message in ws: if isinstance(message, bytes): print(f"<- {len(message)} bytes") continue event = json.loads(message) print("<-", event.get("event_type")) async def main(): try: ws = await connect( URL, additional_headers={"X-API-Key": os.environ["COMFY_API_KEY"]}, max_size=None, ) except InvalidStatus as err: # A refused upgrade: the body is JSON with detail, error_type and code. print(err.response.status_code, err.response.body.decode(errors="replace")) return async with ws: reader = asyncio.create_task(receive(ws)) async def send(event): await ws.send(json.dumps(event)) await send({ "event_type": "world_setup", "world_prompt": {"text": "A quiet harbor town at sunset, seen from the docks"}, }) await send({"event_type": "user_action", "action": "key_down", "input": "key_w"}) await asyncio.sleep(2) await send({"event_type": "user_action", "action": "key_up", "input": "key_w"}) await send({"event_type": "user_action", "action": "key_click", "input": "arrow_left"}) await send({"event_type": "input", "content": [{"type": "text", "text": "Make it start to rain"}]}) await asyncio.sleep(5) await send({"event_type": "interaction.complete"}) await reader print("closed:", ws.close_code) asyncio.run(main()) ``` ```js Browser // A page signed in with a Comfy account. getIdToken() is your Firebase auth // call; the token travels in the subprotocol offer, never in the URL. const idToken = await auth.currentUser.getIdToken(); const ws = new WebSocket( "wss://api.comfy.org/v2/realtime/google/google/genie-4", ["comfy-realtime.v1", "comfy-bearer." + idToken], ); ws.addEventListener("open", () => { ws.send(JSON.stringify({ event_type: "world_setup", world_prompt: { text: "A quiet harbor town at sunset, seen from the docks" }, })); }); // Map the ten supported keys to Genie 4 key names. const keys = { KeyW: "key_w", KeyS: "key_s", KeyA: "key_a", KeyD: "key_d", ArrowUp: "arrow_up", ArrowDown: "arrow_down", ArrowLeft: "arrow_left", ArrowRight: "arrow_right", Space: "space", KeyE: "key_e", }; for (const [type, action] of [["keydown", "key_down"], ["keyup", "key_up"]]) { window.addEventListener(type, (e) => { const input = keys[e.code]; if (!input || e.repeat || ws.readyState !== WebSocket.OPEN) return; ws.send(JSON.stringify({ event_type: "user_action", action, input })); }); } ws.addEventListener("message", (e) => { if (typeof e.data === "string") console.log("<-", JSON.parse(e.data).event_type); }); ws.addEventListener("close", (e) => console.log("closed", e.code)); ``` The browser example sends one `key_down` and one `key_up` per physical key press and skips auto-repeat, which keeps it well inside the event rate. ### Grok Voice `xai/grok-voice/realtime` is a speech-to-speech conversation. The session speaks fal's realtime wire format: send each event as a [MessagePack](https://msgpack.org)-encoded binary frame; responses arrive as MessagePack binary frames or JSON text frames. The events inside are xAI's realtime voice events. You can send these client events: `session.update`, `x-fal-session.configure`, `input_audio_buffer.append`, `input_audio_buffer.commit`, `input_audio_buffer.clear`, `conversation.item.create`, `conversation.item.delete`, `conversation.item.truncate`, `response.create` and `response.cancel`. Each event may carry only its documented fields. - Tools are limited to client-executed `function` tools. Server-side tools such as MCP servers and web or X search are refused. - Image and video content is refused anywhere in an event. - If `session.model` is set, it must be `grok-voice-latest`, `grok-voice-3.0` or `grok-voice-2.0`. A message that breaks these rules is not forwarded and ends the session with close code `1008`. The session cap is **30 minutes**, shortened up front to what your balance covers; at the cap the connection is closed without a close frame. A Grok Voice session costs **$0.0011869 per second** (about $0.0712 per minute; at 211 credits = $1, about 0.25 credits per second or 15 credits per minute). It bills for the whole time the socket is open, whether or not audio is flowing. ```js Node // Node 20+ // Install: npm install ws @msgpack/msgpack // Run: COMFY_API_KEY=comfyui-... node grok-voice.mjs import { decode, encode } from "@msgpack/msgpack"; import WebSocket from "ws"; const ws = new WebSocket("wss://api.comfy.org/v2/realtime/fal/xai/grok-voice/realtime", { headers: { "X-API-Key": process.env.COMFY_API_KEY }, }); const send = (event) => ws.send(Buffer.from(encode(event))); ws.on("unexpected-response", (_req, res) => { let body = ""; res.on("data", (chunk) => (body += chunk)); res.on("end", () => console.error("refused:", res.statusCode, body)); }); ws.on("open", () => { send({ type: "session.update", session: { voice: "Ara", instructions: "Answer in one short sentence.", turn_detection: { type: null }, }, }); send({ type: "conversation.item.create", item: { type: "message", role: "user", content: [{ type: "input_text", text: "Say hello." }] }, }); send({ type: "response.create" }); }); ws.on("message", (data, isBinary) => { const event = isBinary ? decode(data) : JSON.parse(data.toString()); console.log("<-", event.type); if (event.type === "response.done" || event.type === "error") ws.close(1000); }); ws.on("close", (code) => console.log("closed", code)); ``` To send speech instead of text, stream audio with `input_audio_buffer.append` (base64 `audio`), then `input_audio_buffer.commit` and `response.create`. ## WebRTC world models `fal-ai/abot-world-0` and `fal-ai/lingbot-world-2` stream video over WebRTC. The media travels **directly between the browser and fal**, peer to peer; Router handles only the HTTP signalling that sets the connection up, adding fal's credentials so your client never holds them. Every signalling call is a `POST` to `https://api.comfy.org/v2/realtime/wma/fal` with your Comfy credential, a JSON body, and the real destination in the `x-fal-target-url` header. This is the proxy contract fal's JavaScript client uses, so you can point that client's proxy setting at this route. Only these destinations are accepted; anything else is refused with `400` and `code: router.realtime.wma_target_rejected`: | `x-fal-target-url` | Body | Purpose | | --- | --- | --- | | `https://wma.fal.run/ice` | `{"app_id": ""}` | Fetch ICE servers. Opens no session and is not billed. | | `https://wma.fal.run/session` | includes `"app_id": ""` | Open a session. The response carries its `session_id`. | | `https://wma.fal.run/session/heartbeat` | `{"session_id": ""}` | Keep the session alive. | | `https://wma.fal.run/session/close` | `{"session_id": ""}` | End the session. `DELETE /v2/realtime/wma/fal` with the same body does the same. | The body must be a JSON object of at most 64 KiB with no repeated keys. Spell `app_id` and `session_id` exactly so; a different capitalization is refused with `400`. ```bash # Fetch ICE servers for a world model (no session, no charge). curl -X POST https://api.comfy.org/v2/realtime/wma/fal \ -H "X-API-Key: $COMFY_API_KEY" \ -H "Content-Type: application/json" \ -H "x-fal-target-url: https://wma.fal.run/ice" \ -d '{"app_id": "fal-ai/lingbot-world-2"}' ``` ```js // Heartbeat and close from a browser signed in with a Comfy account. const signal = (target, body) => fetch("https://api.comfy.org/v2/realtime/wma/fal", { method: "POST", headers: { Authorization: `Bearer ${idToken}`, "Content-Type": "application/json", "x-fal-target-url": target, }, body: JSON.stringify(body), }); const beat = setInterval( () => signal("https://wma.fal.run/session/heartbeat", { session_id: sessionId }), 5000, ); // When the user stops: tear down the peer connection, then close the session. clearInterval(beat); peerConnection.close(); const res = await signal("https://wma.fal.run/session/close", { session_id: sessionId }); console.log(await res.json()); // { session_id, closed: true, settled: true | false } ``` ### Session lifecycle and billing - **Maximum length.** Each session runs for at most **480 seconds**. - **Heartbeats.** Send one every 5 seconds, as fal's client does. Router forwards at most one per second per session; a faster heartbeat is refused with `429` and `Retry-After: 1`. Heartbeats and closes are accepted only for a `session_id` your account opened and has not closed; otherwise the answer is `404` with `code: router.realtime.wma_session_not_found`. - **Billing.** A session is billed per second from when it opens until 5 seconds after its last heartbeat, up to the 480-second maximum. Closing the session settles the bill right away; a session whose heartbeats stop for more than 15 seconds is settled on its own. The close response's `settled: false` means the bill is still being finalized and lands shortly. - **Balance check at open.** Router cannot stop a peer-to-peer session once it starts, so opening one requires a balance that covers the full 480 seconds of this session plus the full length of every WebRTC session your workspace already has open. If it does not, `/session` is refused with `402` and `code: router.realtime.insufficient_credits`. This is a check, not a reservation: other spend can draw on the same balance while a session runs. - **Concurrent sessions.** A workspace can hold **2** WebRTC world-model sessions at a time. A closed session keeps its slot until about 480 seconds after it opened (plus a short settle window), so a quick close-and-reopen can be refused with `429` and `code: router.realtime.concurrency_limit`; wait for the `Retry-After` the response names. When fal itself answers a signalling call with an error, Router passes fal's status and body through unchanged. ## Errors A refused upgrade or signalling call answers with an HTTP status and a JSON body carrying `detail`, `error_type` and a `code` that names the reason. Branch on `code` or `error_type` rather than on `detail`. Once a WebSocket session is open, problems are reported with a close code instead. | Status | `code` | Meaning | | --- | --- | --- | | `400` | `router.realtime.subprotocol_required` | A `comfy-bearer.` offer came without `comfy-realtime.v1`. | | `401` | | No valid credential. | | `402` | `router.realtime.insufficient_credits` | The workspace balance is too low to open a session. | | `403` | `router.realtime.model_unavailable` | The model is not available to your account. | | `403` | `router.realtime.account_blocked` | The account is blocked. | | `404` | `router.realtime.model_not_allowed` | The model is not served on this route. Check the path and model id. | | `426` | `router.realtime.upgrade_required` | The WebSocket route got a plain HTTP request. Check that no proxy strips the `Upgrade` and `Connection` headers. | | `429` | `router.realtime.rate_limited` | Too many requests, or the provider is at capacity. Wait for `Retry-After`. | | `429` | `router.realtime.session_already_live` | Your workspace already has a live WebSocket session. | | `429` | `router.realtime.concurrency_limit` | Too many concurrent sessions. Retry later. | | `503` | `router.realtime.shutting_down` | The server handling the request is restarting. Retry. | The upgrade and the `/ice` and `/session` signalling calls spend the same per-caller [request allowance](/development/comfy-router/limitations#requests-are-rate-limited-per-caller) as other Router calls. | Close code | Meaning | | --- | --- | | `1000` | Normal end: the model finished, or a Genie 4 session reached its cap. | | `1006` | No close frame arrived. Router causes include a Grok Voice session reaching its time cap, running out of credits, or a blocked account; a network drop or proxy timeout looks the same. Not sent by Router; your client reports it when the connection drops. | | `1008` | A client message broke the model's rules (see each model above). | | `1011` | The model reported an error. | | `1012` | The server is restarting. Open a new session. | ## Next What Router supports, and its deadlines and rate limits. Rates and billing options.