(hwinference-architecture)= # HWInference: the on-device hardware-accelerated inference process `HWInference` is a utility process that runs native, hardware-accelerated inference libraries (currently `parakeet.cpp`, backed by `libggml`) outside of any content process, and outside the main process. Unlike the [Firefox AI Runtime](inference-architecture) inference process, it does not run JavaScript: its job is purely computational, receiving some input, running it against a model file, and producing some output. This page describes the generic facility: the process itself, how a consumer connects to it, how models are provisioned, and the security properties that hold regardless of who the consumer is. It does not cover the specifics of any one consumer. The plumbing lives in {searchfox}`toolkit/components/ml/ipc`, and the process itself is managed by {searchfox}`UtilityProcessManager `. [`SpeechRecognition`](/media/SpeechRecognition), which implements the on-device recognition side of the Web Speech API, is the only consumer today, and is used as the worked example throughout. ## The `HWInference` process The process is a utility process with its own `SandboxingKind`, {searchfox}`HW_INFERENCE `. Its sandbox policy resembles that of the GPU process, but it doesn't have access to the display server, or to things like fonts, or other special system calls or capabilities related to rendering. It only does computations: it receives some input (e.g. text, image, audio data) and uses a model file and a library to perform inference, and produces some output (e.g. timed text fragments, summary). On macOS it gets a dedicated profile, {searchfox}`SandboxPolicyHWInference `, rather than the generic utility one; on Linux, Windows it shares the generic utility policy. It is generally only started when needed, and closed quickly when not needed anymore, but lifetime is in the hands of the *consumer* of the `HWInference` system, see [Process lifetime](#process-lifetime). It delegates all model management tasks to the {searchfox}`ModelHub `, which it calls via IPC to the parent. This includes model availability checks and download (`ModelHub` handles caching). It can also acquire a handle to a model file using a `FileDescriptor` passed via IPC, without copy, important because model files can be quite big. This also allows `mmap`ing some models (notably mixture-of-experts models), for significant memory footprint gains. Since it doesn't run JavaScript, it will eventually be possible to tighten the sandbox further on macOS by making it a different executable, relinquishing the capability to mark pages as executable for JITing code. `ONNX Runtime` (for non-LLM type inference) and `llama.cpp` (for LLM-type inference on text) are eventually expected to also run inside `HWInference`, to be able to use hardware acceleration for tasks unrelated to speech recognition. ## One process, many users There is a single `HWInference` process, keyed like every other utility process by its `SandboxingKind` alone (see `GetProcess`/`LaunchProcess` in {searchfox}`ipc/glue/UtilityProcessManager.cpp`), with one `HWInferenceParent` on the main-process side, `HWInferenceParent::GetSingleton()`. Content-driven inference reaches it through `UtilityProcessManager::StartContentHWInferenceManager`. A privileged, parent-process-triggered consumer — future "browser AI" features — launches the same process, with `UtilityProcessManager::LaunchProcessWithKeepAlive`. What such a consumer does need is a manager protocol of its own alongside {searchfox}`PHWInferenceManager `, which is content-specific: today the only way into the process from outside it is the content path described below. Isolating consumers from each other in separate processes — chrome-driven from content-driven, per origin, per feature — is a matter of keying `UtilityProcessManager` by more than the `SandboxingKind`, so that a single kind can have several live processes. The topology this produces: **every** content process shares the one `HWInference` process, getting its own `HWInferenceManagerParent` there, which gives its task actors their identity. Solid arrows are task traffic, going straight between content and the utility process; dotted ones are model provisioning, which always goes through the main process. ```{mermaid} %%{init: {"flowchart": {"htmlLabels": false}}}%% flowchart LR subgraph CP1[Content Process A] SR1[SpeechRecognition] end subgraph CP2[Content Process B] SR2[SpeechRecognition] end subgraph HWC["HWInference"] direction TB HMP1[Manager for A] --> SRP1[SpeechRecognitionParent] HMP2[Manager for B] --> SRP2[SpeechRecognitionParent] end subgraph MP[Main Process] HWP["HWInferenceParent"] end SR1 --> HMP1 SR2 --> HMP2 SRP1 -.-> HWP SRP2 -.-> HWP ``` ## Process lifetime Users of the `HWInference` process decide how long it lives. `UtilityProcessManager::LaunchProcessWithKeepAlive` hands out a `UtilityProcessKeepAlive` on the process (main thread only), a single one shared by every caller; when the last reference to it goes away the process is shut down with `CleanShutdown`, rather than lingering until browser shutdown like other Utility processes. - **Content-process consumers** go through {searchfox}`PContent `: `RequestHWInferenceConnection` acquires a keep-alive for the requesting content process -- whether or not the process then starts -- and `ReleaseHWInferenceConnection` drops it. `HWInferenceManagerChild` sends exactly one release per request, from `ActorDestroy`. `ContentParent` holds a single keep-alive for as long as its content process has a connection outstanding, and drops it in its own `ActorDestroy`, so a crashed content process cannot pin the utility process forever. - **Parent-process consumers** call `LaunchProcessWithKeepAlive` directly, with no IPC involved, and bind their actor to the process it hands back with `UtilityProcessKeepAlive::StartUtility`. A keep-alive holds the process it was acquired on rather than its `SandboxingKind`, so one that outlives that process — it crashed, or the browser is shutting down — cannot shut down the process that replaced it. `UtilityProcessManager` has no policy of its own: it shuts the process down the moment the last keep-alive on it goes away. Other policies can be implemented, they belong in the user of the process. An example is `SpeechRecognition`: the Web API has numerous async static methods, and it would be wasteful to shutdown the process every time one of those static methods finish, when another one is about to be called. ## Connecting from a content process A content process gets a direct channel to the utility process the first time one is needed, and reuses it: `PHWInferenceManager` is a process-wide singleton, shared by every `HWInference` consumer in that content process. - `HWInferenceManagerChild::AcquireConnection()` returns an `HWInferenceConnectionGuard`, and establishes the connection if it is not up yet. Establishing it creates a `PHWInferenceManager` endpoint pair, binds the child-side endpoint locally as `HWInferenceManagerChild` right away, and calls `ContentChild::SendRequestHWInferenceConnection` with the parent-side endpoint. The connection works right away, no need to wait. - The connection is owned by its guards: it is closed once the last one is dropped, and each consumer decides how long to hold one, so no consumer can tear the channel out from under another. - `ContentParent::RecvRequestHWInferenceConnection`, in the main process, acquires a keep-alive for that content process and brokers the endpoint via `UtilityProcessManager::StartContentHWInferenceManager`, which starts (or reuses) the `HWInference` process and hands the endpoint over via `PHWInference::NewContentHWInferenceManager`. - The utility process binds it as `HWInferenceManagerParent` (`HWInferenceManagerParent::CreateForContent`), the parent side of {searchfox}`PHWInferenceManager `. This detour through the main process happens once per content process (subsequent callers reuse the same `HWInferenceManagerChild`). It is also the riskier of the two `HWInference` IPC boundaries, since content, unlike a parent-process consumer, may be compromised. See [Security](#security) for what a task's actor under this manager can and cannot trust from content. Once established, task traffic, (e.g. audio and timed text for speech recognition) flows directly between the content and utility processes, without going through the main process on every message. Model install and consent still route through the main process, see [Model provisioning](#model-provisioning-task-resolvers-and-modelhub) below. ```{mermaid} sequenceDiagram autonumber box Content Process participant SRB as HWInferenceManagerChild participant CC as ContentChild end box Main Process participant CP as ContentParent participant UPM as UtilityProcessManager participant HWP as HWInferenceParent end box HWInference participant HWC as HWInferenceChild participant HMP as HWInferenceManagerParent end Note over SRB: AcquireConnection():
CreateEndpoints(parentEp, childEp)
for PHWInferenceManager Note over SRB: bind childEp locally SRB->>CC: SendRequestHWInferenceConnection(parentEp) CC->>CP: PContent::RequestHWInferenceConnection(parentEp) CP->>UPM: StartContentHWInferenceManager(parentEp, contentId) Note over UPM: LaunchProcessWithKeepAlive(HW_INFERENCE), then
keepAlive->StartUtility(HWInferenceParent):
launches the process if not already running Note over CP: ++mHWInferenceConnections
keeps the UtilityProcessKeepAlive it got back UPM->>HWP: SendNewContentHWInferenceManager(parentEp, contentId) HWP->>HWC: PHWInference::NewContentHWInferenceManager(parentEp, contentId) HWC->>HMP: CreateForContent(parentEp, contentId) (bind) Note over SRB,HMP: direct channel established: task actors
(e.g. PSpeechRecognition) are created
directly over it as soon as childEp is bound,
no further main-process hop ``` ### Task protocols `Endpoint::Bind()` binds an actor to the thread that calls it, and that is the thread every one of that actor's `Recv` methods then runs on. Each task protocol is a separate toplevel connection created through the manager, so the two sides choose their threads independently. `PHWInferenceManager` itself is bound on the main thread in both processes. The manager answers a request (e.g. `CreateSpeechRecognition()` for speech recognition) by creating the endpoint pair, binding the utility-process side on its own thread, passing it the trusted content id it carries (`HWInferenceManagerParent::ContentId()`), and resolving with the content-process endpoint. `HWInferenceManagerChild::CreateSpeechRecognitionSession()` takes the event target the caller wants its side bound on, dispatches the bind there, and resolves its promise there too. Speech recognition passes its `SpeechIPC` thread, keeping audio off the content main thread. The utility side is bound on the main thread and dispatches inference to a `Parakeet` thread of its own, so `RecvProcessAudioData` returns without blocking on it. ```{mermaid} sequenceDiagram autonumber box Content Process participant C as Consumer
(main thread) participant HMC as HWInferenceManagerChild
(main thread) participant SRC as SpeechRecognitionChild
(SpeechIPC thread) end box HWInference participant HMP as HWInferenceManagerParent
(main thread) participant SRP as SpeechRecognitionParent
(main thread) end C->>HMC: CreateSpeechRecognitionSession(SpeechIPC) HMC->>HMP: CreateSpeechRecognition() Note over HMP: CreateEndpoints(parentEp, childEp) HMP->>SRP: parentEp.Bind() here, so SRP is bound
to the HWInference main thread HMP-->>HMC: resolve(childEp) HMC->>SRC: dispatch to SpeechIPC, childEp.Bind() there,
so SRC is bound to SpeechIPC SRC-->>C: promise resolves on SpeechIPC SRC->>SRP: PSpeechRecognition, SpeechIPC to HWInference main ``` Creating one costs a round trip; teardown is `Close()`. ## Model provisioning: task resolvers and `ModelHub` Every {searchfox}`PHWInference ` request carries a `(task, id)` pair. Two things happen with it, both in the parent process, in `HWInferenceParent`. **Resolution of a model:** `task` selects an {searchfox}`nsIMLModelResolver `, looked up as the XPCOM component `@mozilla.org/ml/model-resolver;1?task=`. Its `resolve()` maps `id` to the `engine`/`model`/`revision`/`filename` of a `ModelHub` artifact, out of static in-tree data compiled into the binary. An unknown `task` or `id` fails the request before any `ModelHub` call. For example, {searchfox}`SpeechModelResolver ` resolves the ids declared in {searchfox}`models.yaml `. **Model download gating:** ML models can be pretty big, and so user consent (or arbitrary asynchronous code) can be inserted prior to a download with `authorizeDownload()`. It gets the resolved model (and e.g., its size, but other metadata can be added) and the `WindowGlobalParent` responsible for the request (0 denotes a parent-process user). If the model is already present locally, this is resolved immediately. For example, in `SpeechRecognition`, a doorhanger on that window's tab is displayed the first time a specific language is requested End to end, with speech recognition as example, originating from a Content process: ```{mermaid} sequenceDiagram autonumber box Content Process participant SR as SpeechRecognition end box HWInference participant SRP as SpeechRecognitionParent end box Main Process participant HWP as HWInferenceParent participant Res as SpeechModelResolver participant MH as nsIMLModelHub (ModelHub) end SR->>SRP: install(["fr"], innerWindowId) Note over SRP: language -> id (dom::LanguagesToSpeechModelId) SRP->>HWP: InstallModel(task, id, innerWindowId,
contentId) Note over SRP,HWP: contentId is supplied by the utility, never sent by content.
A parent-process caller passes 0 for both ids. HWP->>Res: resolve(id) Res-->>HWP: engine/model/revision/filename Note over HWP: window must be owned by contentId
(see Security, below)
progressToken created here, to tell
concurrent installs apart HWP->>Res: authorizeDownload(model, revision, filename,
window, progressToken, callback) Note over Res: already cached, or the user hit Allow
on the model-download doorhanger Res-->>HWP: callback->Resolve(allow) alt allowed HWP->>MH: DownloadModel(engine, task, model, revision, files,
progressToken, progressCallback, completionCallback) MH--)HWP: progress callback(s) MH-->>HWP: success/fail else denied Note over HWP: nothing downloaded end HWP-->>SRP: true/false SRP-->>SR: Promise resolves(installed) ``` ### The testing mock Under `browser.ml.modelHub.testing`, `HWInferenceParent` answers from an in-memory set of "installed" models instead of calling `ModelHub`, so install and availability agree on what has been "downloaded". Resolution and `authorizeDownload()` still run. This is useful e.g. for WPT, for which it is harder to run custom code serving model in CI. This isn't needed for Mochitests, who can pull arbitrarily large model files in there tasks, and run a custom python server to mimick ModelHub repository. This also means end-to-end testing is possible. ## Security The consent decision and the download both live entirely in the trusted parent (main) process, so a compromised content process has no path to install or read an arbitrary model file, nor to trigger a download without the user's consent. - Content-facing protocols (e.g. {searchfox}`PSpeechRecognition `) never mention `model`/`revision`/`filename`. They only carry task-specific, abstract identifiers — for `SpeechRecognition`, BCP-47 language tags. - Turning those into a model id (`dom::LanguagesToSpeechModelId` for speech recognition) reads only a table generated at build time and compiled into the binary; it is not loaded from anything runtime-writable or attacker-writable. That mapping happens wherever the task's actor runs, for `SpeechRecognition`, in the utility process, never in content. Model selection can depend on e.g. checking if hardware acceleration is available, and so is best done in the `HWInference` process. - `HWInferenceParent`, on the main-process side, resolves the id back to the `ModelHub` slug by calling the task's `nsIMLModelResolver`, which reads the very same compiled-in table. So the only attacker-influenced input anywhere on this path is a task-specific abstract identifier, matched against a static compiled-in table, and that id is the only thing that crosses IPC. ### Consent to a model download cannot be faked by content A model download requires the task resolver's authorization, decided **and enforced in the parent (main) process**, never in content or in the utility process: - Content can only *ask*. It sends its request for a model along with the **inner window id** of its requesting document (not a `BrowsingContext` id) to the task's utility-process actor. It never sees, and cannot tweak, a consent answer. - The task's utility-process actor maps the request to a model id and relays `PHWInference::InstallModel` to the main process, attaching the **trusted** `ContentParentId` of the content process that owns the connection — never a value content supplies. That id comes from the manager (`HWInferenceManagerParent::ContentId()`), which got it from `ContentParent` when it brokered the connection. - `HWInferenceParent::RecvInstallModel` (main process) resolves the id via the task's `nsIMLModelResolver`, then resolves the content-supplied inner window id to a `WindowGlobalParent` (`WindowGlobalParent::GetByInnerWindowId`) and denies the request unless that window's `ContentParentId()` matches the trusted `contentId`. A compromised content process cannot name a window it does not own to anchor the prompt on another tab or act for another origin. A parent-process request has no such check to make: the caller is parent-process code, so `contentId`/`innerWindowId` are `0`. - Only then is the download authorized, by the same `nsIMLModelResolver`. Speech recognition's policy is to check whether the model is already cached (`nsIMLModelHub`), allowing with no prompt if so, and otherwise to show a doorhanger; another task can authorize immediately, show a custom prompt, or apply any other policy. Only on a real Allow does `HWInferenceParent` start the download via `nsIMLModelHub::DownloadModel`. ## Logging and tests `MOZ_LOG=HWInference:5` traces the whole facility: connection setup, `RecvInstallModel`/`RecvIsModelInstalled` and the rest of the model path, and actor lifetime, in every process involved. `ModelHub:4` can also be useful. The process and its lifetime rules are covered by gtests in {searchfox}`ipc/glue/test/gtest/TestUtilityProcess.cpp`: Gtest exercise this new process: 'TestUtilityProcess.HWInference*'. The content path, model provisioning and consent are exercised end to end by the speech recognition tests, see [its documentation](/media/SpeechRecognition).