# ltx-2-mlx-swift

Swift/MLX port of Lightricks **LTX-2.3** — a DiT-based foundation model that jointly
generates synchronized video **and** audio. Mirrors the Python oracle
[`dgrauet/ltx-2-mlx`](https://github.com/dgrauet/ltx-2-mlx); integrates into MLXEngine as a
`ModelPackage`.

Unlike the Wan family this is a **standalone substrate** (Gemma-3 text encoder, 128-channel
VAE, joint-AV DiT, BigVGAN + BWE audio) — it does **not** reuse `wan-core`.

## License

- **This port code is licensed [Apache-2.0](LICENSE).** It is our own implementation. Lightricks
  releases their own LTX-2 *inference code* (`ltx-core` / `ltx-pipelines`, per the LTX-Desktop
  `NOTICES.md`) under Apache-2.0 — inference code is not treated as a derivative of the weights.
- **The model weights are NOT included and are NOT Apache-2.0.** LTX-2.3 weights ship under the
  **[LTX-2 Community License](https://huggingface.co/Lightricks/LTX-2.3)** (source-available, with a
  §2 revenue gate and a §A.20 non-compete). Obtain them from Lightricks / `mlx-community` and comply
  with that license — those terms bind any *distribution* of generated output, independent of this code.
- Parity goldens (`parity/goldens/`) are forward-outputs of the weights → weight-derivatives →
  gitignored, never distributed.

## Capabilities

Distilled **text-to-video** (one-stage + two-stage upsampled) and **image-to-video** (first-frame
conditioning via `T2VRequest.initImage`), each across a **bf16 / int8 / int4** quant ladder. Output
is an MP4 with synchronized 48 kHz stereo audio (the joint DiT denoises video + audio together).

Pipeline: `prompt → Gemma-3 (49 hidden states) → connector → joint-AV DiT → distilled Euler
denoise → 128-ch video VAE decode (+ audio VAE → BigVGAN + BWE) → MP4`.

## Parity

Every component is gated numerically against the oracle (per-op cosine ≥ 0.999; most hit ~1.0):
connector, gemma, text-encode, dit-tiny, dit-full, **dit-q8 / dit-q4** (quant), **dit-pertoken**
(i2v per-token timesteps), vae-decode / vae-encode, denoise, e2e, audio-vae-decode, vocoder,
audio-decode, upsampler, upscale-step. (MLX-Swift ↔ MLX-Python is not bit-identical across a
multi-step trajectory — that's expected FP non-determinism; the gate is per-op + perceptual.)

## Memory

Per-stage **load → use → evict** (MLXEngine 1.14 efficiency contract): only the DiT backbone stays
resident through the denoise; the Gemma+Connector text encoder evicts before it, and the VAE-decode
stack / two-stage encoder+upsampler load only around their phases. Because weights are lazy-mmap, the
DiT isn't even materialized during text-encode — encoder and DiT are never co-resident, so the peak is
the DiT denoise **alone**. The manifest declares a **split footprint** (`residentBytes` weights +
`peakActivationBytes` transient); the engine reserves one transient across residents (serialized
inference), so LTX co-resides with other models far better than a single-number floor allows.

Measured in-app (M5 Max / 128 GB, two-stage **704×512×9f**, seed 42; peak scales with resolution×frames):

| Quant | Resident (DiT) | Activation | **Peak** | vs. old all-resident |
|---|---|---|---|---|
| bf16 | 38.9 GB | 13.1 GB | **52.0 GB** | 82.8 GB |
| int8 (q8) | 21.5 GB | 12.1 GB | **33.6 GB** | 62.4 GB |
| int4 (q4) | 12.2 GB | 15.1 GB | **27.3 GB** | 53.3 GB |

Activation is ~dtype-independent (same bf16 compute). int4 is artifact-free but **diverges the distilled
sample** vs bf16/q8 (larger per-step quant error). Re-measure (`RunLTX2 --mem-bench`) before claiming a
larger res/frame envelope fits a tier.

### Memory tiers (`LTX2Configuration.profile`)

Four profiles make LTX-2.3 honestly admissible per tier (`LOW-TIER-PLAN.md`): an **envelope clamp** +
**one/two-stage policy** + **VAE temporal-decode window** + a per-profile activation hint. Low tiers run
**fully sequential** ([Gemma] → [int8 connector] → [DiT denoise] → [chunked VAE decode], each stage
alone; the DiT reloads in ~1.5 s from the page cache, LoRAs re-apply automatically across the reloads).
Measured peaks (low tiers: requests clamped from an oversized 704×512×240; max128: measured at
its full 481f envelope on the heavier i2v path):

| profile | quant | envelope (max) | path | **measured peak** | fits |
|---|---|---|---|---|---|
| `compact24` | int8 streamed · lean decoder | **704×512 × 121f** t2v · 512×288 × 121f conditioned (i2v, IC, a2v) | one-stage | **14.5 GB** t2v at 704×512 (150 s) · **15.9 GB** i2v at 512×288 (76 s) — M5 Pro 24 GB, AB-T-0298 / AB-R-0562 | 24 GB Macs (M5 MBP base) |
| `balanced32` | int4/int8 | 576×320 × 161f | one-stage | **16.1 GB** (115 s) | 32 GB |
| `standard64` | int8 | 704×512 × 161f | two-stage | **37.5 GB** (188 s) | 64 GB |
| `max128` | bf16 | 704×512 × 481f | two-stage | **72.7 GB** (960 s, 481f i2v + adapter LoRA) | 96–128 GB |

16 GB is deliberately unsupported (the int4 DiT alone ≈ a 16 GB governor budget; no smaller LTX-2
checkpoint exists). `nil` profile = unconstrained legacy behavior.

**Contract 1.41.0 (engine 0.55.0, AB-A-0074):** each LTX-2.5 profile also declares its envelope as a
LINE in pixel-frames (`LTX2Configuration.activationScalingHint`, axis `.pixelFrames`, ceiling = the
largest geometry actually measured for that lane) and maps a `T2VRequest` to the pixel-frames of the
geometry it RESOLVES to (`WorkloadDeclaring`, through the same `resolvedGeometry` the run uses; an
a2v request follows its track). The profile still clamps rather than refuses, so the engine's
pre-admission refusal is a backstop — what this buys is `machineFitAdvisory(_:package:workload:)`
answering "will THIS clip fit?" per geometry. The derivation per lane, receipts included, is on the
hint in `LTX2Configuration.swift`; `ActivationScalingTests` holds every line to its measured corners.

## Layout

- `Sources/LTX2` — engine-agnostic functional cores (RoPE, Gemma, Connector, DiT, DenoiseLoop,
  Video/Audio VAE, Vocoder, Upsampler, Positions, LTX2Pipeline).
- `Sources/MLXLTX2` — the MLXEngine `ModelPackage` wrapper (`MLXLTX2Package`, `LTX2Configuration`,
  `FrameCodec`, `ImageInput`).
- `Sources/RunLTX2` — CLI parity-gate driver.
- `parity/dump_*_goldens.py` — oracle golden dumpers (run in the `ltx-2-mlx` uv env).

## Build / gates

Depends on [`ml-explore/mlx-swift-lm`](https://github.com/ml-explore/mlx-swift-lm) as a local
path dep at `../mlx-swift-lm`, checked out at plain upstream `main` — **no fork, no local
patches**. The 49-layer `allHiddenStates` tap lives in this package
(`Sources/LTX2/Gemma3+AllHiddenStates.swift`) and is written against the official
`@_spi(GemmaEncoder)` surface upstream merged in
[#387](https://github.com/ml-explore/mlx-swift-lm/pull/387) (`6608a35`, 2026-07-21).

> ⚠️ **Transitional pin.** No mlx-swift-lm release tag contains #387 yet (latest is 3.31.4,
> which predates the merge), so the dep is still a path dep on a `main` checkout. A main-SHA
> pin is less stable than a tag — don't cut an LTX release off it. When a tag ships, adoption
> is a one-line `Package.swift` edit and nothing else changes.

```bash
# release Xcode 27 (Swift 6.4, macOS 27 SDK) is the active toolchain; no DEVELOPER_DIR override needed (AB-A-0077, 2026-09-17)
xcrun swift run -c release RunLTX2 --dit-full-gate
xcrun swift run -c release RunLTX2 --mem-bench bf16   # split-footprint harness [bf16|int8|int4]
```

> The `--mem-bench` CLI can trip the Metal GPU watchdog on the full two-stage run (beta-OS flakiness);
> the reliable measurement surface is the in-app autorun (`LTX_AUTORUN=1 LTX_QUANT=bf16` in
> LTXVideoTesting), which has the engine's weight prewarm + governor.