# ltx-2-mlx

Pure MLX port of [LTX-2](https://github.com/Lightricks/LTX-2) for Apple Silicon. Three-package monorepo mirroring the reference structure — inference, pipelines, and training — running natively on Metal.

## Features

- **LTX-2.5** — pipeline coverage on 2.5 packs (`--distilled`, `--two-stage`, `--two-stages-hq`, `--one-stage`, `--dfr`, `a2v`, `retake`, `extend`, `ic-lora`, `hdr-ic-lora`), auto-detected from the pack (no new flag), plus **auto-predicted duration** via the DurationHead (omit `-f`, or clamp with `--auto-duration MIN:MAX`). Only `lipdub` waits on a 2.5 Dub-It LoRA; `ic-lora` runs with the IC-LoRAs Lightricks has published for 2.5. See [LTX-2.5 section](#ltx-25).
- **Text-to-Video** — generate video + stereo 48kHz audio from a text prompt
- **Image-to-Video** — animate a reference image; `--image PATH FRAME STRENGTH [auto|replace|guide] [CRF]` is repeatable for multi-anchor I2V, and `FRAME` may be `last` or negative (counted from the end)
- **Audio-to-Video** — generate video conditioned on an audio track (`a2v`, dev + CFG; `a2v --distilled` for the fast distilled path and `a2v --dfr` on the DFR base path (2.5 packs), both experimental)
- **Retake / Extend** — edit existing videos (regenerate segments, add frames); `retake --distilled` / `extend --distilled` run on the distilled model, roughly 10–20× faster (experimental)
- **Interpolation between images** — `generate --two-stage` with each image bound as a guide (`--image PATH FRAME STRENGTH guide`) and `--refine-audio`; this replaces the `keyframe` command, removed following upstream v1.5.0
- **IC-LoRA** — reference video conditioning (depth/pose/edges)
- **HDR IC-LoRA** — single-stage ACEScct SDR-to-HDR (LTX-2.5 packs) producing an HLG BT.2020 10-bit mp4 + EXR frames
- **LipDub** *(experimental)* — lip-dub a reference video by re-syncing visuals to the source audio
- **Two-stage generation** — half-res → neural upscale → refine
- **HQ generation** — res_2s second-order sampler + CFG/STG guidance
- **Prompt Relay** — sequence local prompts over time within one generation (`--segment "text" [LEN]`); a training-free Gaussian penalty gates each prompt's tokens to a slice of the timeline via the video→text and audio→text cross-attentions (`--relay-audio-strength`: default 1.0 on `--distilled` / `--dfr`, 0 on the CFG modes). Works across all generate modes; on CFG modes the masks apply to the conditional pass only.
- **Negative prompt (`--negative-prompt`)** — custom CFG negative on every CFG pipeline (`--one-stage` / `--two-stage` / `--two-stages-hq` / a2v / retake / extend); defaults to the upstream `DEFAULT_NEGATIVE_PROMPT`. Distilled modes have no CFG and reject it, except `generate --distilled` / `--dfr` with `--nag` (experimental Normalized Attention Guidance, applied inside the text cross-attentions).
- **Prompt enhancement** — Gemma 3 12B rewrites short prompts into detailed descriptions
- **Training** — LoRA fine-tuning with flow matching (T2V and V2V strategies)
- **Block streaming (`--low-ram`)** — stream transformer blocks from disk so q8 fits 16 GB Macs and bf16 fits 32 GB Macs (covers every `generate` mode including `--distilled` and `--dfr`, a2v / ic-lora / hdr-ic-lora / lipdub / retake / extend; bind-time LoRA fusion supports custom distilled-lora-strength)
- **Modality tiling (`--tile-frames N --tile-spatial M`)** — split video tokens into spatial+temporal tiles to cap O(N²) attention activations. Combined with `--low-ram`, unblocks long / HD / 4K generations on Mac Studio (64-128 GB) that would otherwise OOM.
- **6 model packs** — bf16 / int8 / int4 for each of LTX-2.3 and LTX-2.5 (fits 16GB–64GB Macs)
- **3 upsamplers** — spatial 2x, spatial 1.5x, temporal 2x

> **Production readiness**: pipelines are classified as Stable / Beta /
> Experimental. See [docs/PIPELINE_MATURITY.md](docs/PIPELINE_MATURITY.md)
> before relying on a pipeline in production code. CLI subcommands flagged
> `[beta]` or `[experimental]` in `--help` may have known quality
> limitations or pre-1.0 LoRA dependencies.

## Requirements

- macOS with Apple Silicon (M1/M2/M3/M4)
- Python 3.11+
- 32GB+ RAM recommended (int8) or 16GB+ with `--low-ram`. 16GB minimum (int4 without streaming)
- ffmpeg (for video encoding)
- HDR (`hdr-ic-lora` only): OpenEXR via the `hdr` extra (`uv sync --extra hdr` in this repo, or `pip install 'ltx-pipelines-mlx[hdr]'`) and an ffmpeg built with `libx265`

## Installation

```bash
git clone https://github.com/dgrauet/ltx-2-mlx.git
cd ltx-2-mlx
uv sync --all-extras
```

## Quick Start

Not sure which pipeline to reach for? See [docs/PIPELINES.md § Which pipeline?](docs/PIPELINES.md#which-pipeline) for a decision tree.

### CLI

```bash
# Text-to-Video — pick a pipeline mode (one of `--two-stage`, `--two-stages-hq`, `--one-stage`, `--distilled`).
# Two-stage is the upstream-recommended production default.
# NOTE: -f/--frames is required on 2.3 packs (the default model below) — it
# only defaults to an auto-predicted duration on LTX-2.5 packs. See "LTX-2.5"
# and "CLI Reference" below.
ltx-2-mlx generate --prompt "A sunset over the ocean" --two-stage -f 97 --frame-rate 24 -o sunset.mp4

# Image-to-Video (any mode supports --image)
ltx-2-mlx generate --prompt "Animate this" --image photo.jpg --two-stage -f 97 --frame-rate 24 -o animated.mp4

# HQ (res_2s sampler, highest quality)
ltx-2-mlx generate --prompt "A scene" --two-stages-hq --stage1-steps 20 -f 97 --frame-rate 24 -o hq.mp4

# Distilled two-stage (fastest, mirrors upstream DistilledPipeline)
ltx-2-mlx generate --prompt "A scene" --distilled -H 720 -W 1280 -f 97 --frame-rate 24 -o distilled.mp4

# One-stage dev + CFG (full target res, mirrors upstream TI2VidOneStagePipeline)
ltx-2-mlx generate --prompt "A scene" --one-stage -f 97 --frame-rate 24 -o one_stage.mp4

# Audio-to-Video
ltx-2-mlx a2v --prompt "Music video" --audio music.wav --frame-rate 24 -o a2v.mp4

# Retake (regenerate frames 1-3 of a video)
ltx-2-mlx retake --prompt "New action" --video source.mp4 --start 1 --end 3 -o retake.mp4

# Extend (add 2 latent frames after)
ltx-2-mlx extend --prompt "Continue the scene" --video source.mp4 --extend-frames 2 -o extended.mp4

# Distilled extend (no CFG, only the new window is denoised; experimental)
ltx-2-mlx extend --distilled --prompt "Continue the scene" --video source.mp4 --extend-frames 4 -o extended_fast.mp4

# Interpolate between two images (guides at the first and last frame)
ltx-2-mlx generate --two-stage --prompt "Smooth transition" --image frame1.png 0 1.0 guide --image frame2.png last 1.0 guide \
    --refine-audio -f 97 --frame-rate 24 -o transition.mp4

# Prompt enhancement
ltx-2-mlx enhance --prompt "a cat" --mode t2v

# Use int4 model (fits 16GB)
ltx-2-mlx generate -p "A cat" --distilled -f 97 --frame-rate 24 -o cat.mp4 --model dgrauet/ltx-2.3-mlx-q4

# Block streaming: bf16 model on 32 GB Mac
ltx-2-mlx generate -p "A cat" --two-stage -f 97 --frame-rate 24 -o cat.mp4 --model dgrauet/ltx-2.3-mlx --low-ram

# Block streaming: q8 model on 16 GB Mac
ltx-2-mlx generate -p "A cat" --distilled -f 97 --frame-rate 24 -o cat.mp4 --model dgrauet/ltx-2.3-mlx-q8 --low-ram

# Block streaming works on every generate mode + a2v / ic-lora / hdr-ic-lora / retake / extend
ltx-2-mlx generate -p "A cat" -f 97 --frame-rate 24 -o cat.mp4 --two-stage --low-ram
ltx-2-mlx generate -p "A cat" -f 97 --frame-rate 24 -o cat.mp4 --two-stages-hq --low-ram
ltx-2-mlx a2v -p "music video" --audio music.wav --frame-rate 24 -o a2v.mp4 --low-ram
ltx-2-mlx ic-lora -p "scene" --lora lora.safetensors 1.0 --video-conditioning depth.mp4 1.0 --frame-rate 24 --low-ram -o out.mp4

# HDR IC-LoRA — SDR mp4 -> HLG master + ACEScg EXR frames (LTX-2.5 pack; gated LoRA repo)
ltx-2-mlx hdr-ic-lora --model dgrauet/ltx-2.5-mlx-q8 --input source_sdr.mp4 \
    --hdr-lora ltx-2.5-22b-ic-lora-sdr-to-hdr-1.0.safetensors \
    --text-embeddings ltx-2.5-22b-ic-lora-sdr-to-hdr-scene-emb.safetensors --low-ram -o out.mp4

# Modality tiling: split video tokens for long/HD scenarios that exceed attention memory.
# Stack with --low-ram for max memory savings on big targets.
ltx-2-mlx generate -p "long scene" --two-stage --low-ram -f 97 --frame-rate 24 \
    --tile-frames 2 --tile-overlap 4 -o long.mp4
ltx-2-mlx generate -p "1080p scene" --two-stages-hq --low-ram -f 97 --frame-rate 24 \
    --tile-spatial 2 --tile-overlap 4 -H 1024 -W 1920 -o hd.mp4

# Model info
ltx-2-mlx info --model dgrauet/ltx-2.3-mlx-q8
```

## LTX-2.5

```bash
ltx-2-mlx generate --distilled --model /path/to/ltx-2.5-mlx-q8 \
    --prompt "a heavy wooden door creaks slowly open" --frame-rate 24 -o out.mp4

# Clamp the auto-predicted duration to 2-4 seconds instead of the default [1, 20]s
ltx-2-mlx generate --distilled --model /path/to/ltx-2.5-mlx-q8 \
    --prompt "a heavy wooden door creaks slowly open" --auto-duration 2:4 --frame-rate 24 -o out.mp4
```

The 2.5 generation is **auto-detected from the model pack** (no new CLI
flag). A HuggingFace repo id (`dgrauet/ltx-2.5-mlx-q8`) or a local pack
directory both work. The pack bundles its own Gemma-4 text encoder
(`text_encoder.safetensors`), so no `mlx-community` Gemma download happens
on this path. `--image` (I2V) works the same as on
2.3.

**`-f/--frames` is optional on 2.5 packs**: omit it and the pack's
`DurationHead` predicts a clip length (seconds) from the encoded prompt
right after text encoding, snapped to the model's frame grid. Pass
`--auto-duration MIN:MAX` (seconds) to override the predictor's clamp
range, or pass `-f` explicitly to bypass prediction entirely (explicit
`-f` always wins). **On 2.3 packs, `-f` stays required** — there is no
`DurationHead` to predict from, and omitting it now raises immediately
(`ValueError: ... Pass num_frames explicitly.`) before any Gemma load.

Sampling: both stages of `--distilled` run euler-ancestral on 2.5 packs
(8 + 3 steps, SDE noise injection; stage-2 noise seeded from `seed + 20000`),
as upstream since v1.4.0. 2.3 packs stay deterministic Euler on both stages.

The dev model + CFG two-stage pipeline also works on 2.5 packs:

```bash
ltx-2-mlx generate --model /path/to/ltx-2.5-mlx-q8 --two-stage \
    --prompt "a heavy wooden door creaks slowly open" --frame-rate 24 -o out.mp4
```

**v1 limits on 2.5 packs**:

| Feature | Status |
|---|---|
| `--two-stage` (dev + CFG) | supported (see above) |
| `--two-stages-hq` (res_2s + CFG) | supported — validated e2e on 2.5 (deterministic, audio at healthy 2.3-level loudness) |
| `DurationHead` / auto-duration (`-f` optional) | supported — `-f` defaults to an auto-predicted duration on `--one-stage`/`--distilled`/`--two-stage`/`--two-stages-hq`/`--dfr`; `--auto-duration MIN:MAX` overrides the clamp. Not available on 2.3 packs (no `DurationHead` weights); `-f` stays required there |
| `a2v` | supported — validated e2e on 2.5 (deterministic, conditioned audio faithfully reconstructed at -36.2 dB). `a2v --distilled` / `a2v --dfr`: Experimental, validated e2e on 2.5 q8 |
| `retake`, `extend` | supported — validated e2e on 2.5 (retake deterministic ×2; extend +N latent frames). `--low-ram` wired (mirrors upstream `offload_mode`): 49-frame retake that OOM'd now peaks at 13.8 GB |
| `ic-lora` | runs with the IC-LoRAs Lightricks has published for 2.5 (`Lightricks/LTX-2.5-22b-IC-LoRA-*`); upstream's plans that keep the IC-LoRA in stage 2 (`stage_2_ic_lora=True`) are not ported |
| `lipdub` | not yet on 2.5 (no 2.5 Dub-It LoRA published) |
| `hdr-ic-lora` | supported, 2.5 only (single-stage ACEScct SDR-to-HDR; Experimental, validated end to end on real weights) |
| `enhance` / `--enhance-prompt` | raises a clear error (Gemma 3-only) |
| `--enable-teacache` | raises a clear error (not calibrated for 2.5) |
| Prompt Relay | supported — validated e2e on 2.5 `--distilled` and the `--dfr` base path; token ranges pinned against the pack's Gemma-4 tokenizer |
| Modality tiling | validated on 2.5 `--distilled` (`--tile-frames 2`, unconditioned and with an end anchor) |
| Diffusion video decoder | opt-in `--video-decoder diffusion` (experimental; auto-tiled above the decode budget, `--diffvae-tile` override); conv stays the default |

`lipdub` lands on 2.5 once Lightricks publishes a 2.5 Dub-It LoRA.

For maximum detail on a 2.5 pack, `generate --dfr` (experimental) runs a base DFR pass with
the official detailing IC-LoRA — see the [`generate --dfr` card](docs/PIPELINES.md#generate---dfr-experimental-ltx-25-packs-only) in the pipelines guide.
That LoRA (`Lightricks/LTX-2.5-22b-IC-LoRA-Pixel-Spatial-Upscaler`) is a **gated** HuggingFace
repo: accept its licence on the model page once, with the account `huggingface-cli login` uses,
before the first `--dfr` run (the run fails before loading any model otherwise).

### Python API

Pick the pipeline class matching your target — every public class
mirrors an upstream Lightricks/LTX-2 pipeline.

Two-stage (recommended for most use cases — dev model + CFG + upscale):

```python
from ltx_pipelines_mlx import TI2VidTwoStagesPipeline

pipe = TI2VidTwoStagesPipeline(model_dir="dgrauet/ltx-2.3-mlx-q8")
pipe.generate_and_save(
    prompt="A sunset over the ocean with waves crashing",
    output_path="sunset.mp4",
    height=480,
    width=704,
    num_frames=97,
    frame_rate=24.0,
    seed=42,
    image="photo.jpg",  # optional I2V
)
```

For other modes:

- `DistilledPipeline` — fastest (distilled half-res + upscale).
- `TI2VidTwoStagesHQPipeline` — highest quality (res_2s + CFG + upscale).
- `TI2VidOneStagePipeline` — full-res CFG, no upscaler dependency.

Audio-to-Video:

```python
from ltx_pipelines_mlx import A2VidPipelineTwoStage

pipe = A2VidPipelineTwoStage(model_dir="dgrauet/ltx-2.3-mlx-q8")
pipe.generate_and_save(
    prompt="A musician performing",
    output_path="a2v.mp4",
    audio_path="music.wav",
    frame_rate=24.0,
)
```

Retake / Extend (single class — extend is folded into `RetakePipeline`):

```python
from ltx_pipelines_mlx import RetakePipeline

pipe = RetakePipeline(model_dir="dgrauet/ltx-2.3-mlx-q8")

# Retake: regenerate latent frames 1-3
video_lat, audio_lat = pipe.retake_from_video(
    prompt="A different scene",
    video_path="source.mp4",
    start_frame=1,
    end_frame=3,
)

# Extend: add 2 latent frames after
video_lat, audio_lat = pipe.extend_from_video(
    prompt="Continue the motion",
    video_path="source.mp4",
    extend_frames=2,
    direction="after",
)
```

## CLI Reference

For the full flag reference, start with [docs/PIPELINES.md § Which pipeline?](docs/PIPELINES.md#which-pipeline) — a decision tree, a card per pipeline, and a flag matrix covering every CLI subcommand, its defaults, and which memory / perf flags apply where.

### Stepwise previews

A generation is opaque until the final VAE decode, which for a long clip can be many
minutes away. These flags decode a short window of latent frames from the in-progress
prediction every N steps and write it as a self-contained animated WebP — a couple of
seconds of real motion at the current denoise quality, so temporal problems show up at
step 4 instead of minute 15. Available on every generating subcommand except `hdr-ic-lora`
(`generate`, `a2v`, `retake`, `extend`, `ic-lora`, `lipdub`).

```
--stepwise-image-output-dir DIR   Write per-step preview clips here
--stepwise-interval N             Preview every N steps (default: 1; last step always previewed)
--stepwise-frames N               Latent frames per preview (default: 8 -> 57 frames, ~2.3s)
--stepwise-frame I                Latent frame the window is centred on (default: middle)
```

```bash
ltx-2-mlx generate -p "a cat walking" -o out.mp4 -f 97 --frame-rate 24 \
  --stepwise-image-output-dir ./previews
```

Produces, for seed 42 on a two-stage run:

```
previews/
  seed_42_s1_step001of030.webp   # 57 frames of motion, this step
  seed_42_s1_step002of030.webp
  ...
  seed_42_s2_step003of003.webp
```

Each file is written **once and never rewritten**, so write cost stays linear in the step
count. Play them back to back for the full progression: the same motion span replayed at
each step, sharpening as the denoise converges.

#### Why a window rather than a single frame

The VAE upsamples time 8x, so `N` latent frames decode to `8N-7` pixel frames. A single
latent frame yields one picture while paying for the whole up-block stack — the worst
point on the cost curve. Measured on a 704x448 clip:

| latent frames | pixel frames | motion @25fps | stage 1 | stage 2 |
|---|---|---|---|---|
| 1 | 1 | — | 0.06s | 0.21s |
| 3 | 17 | 0.68s | 0.49s | 1.90s |
| **8** (default) | **57** | **2.3s** | 1.55s | 6.13s |
| 16 | 121 | 4.8s | 3.23s | 13.0s |

**Cost is independent of clip length** — decoding 8 of 60 latent frames costs the same as
8 of 16 — so previews get proportionally cheaper the longer the clip. On a 15-minute
two-stage run the default adds roughly 7%, and most of that is stage 1, which decodes at
half resolution.

Notes:

- **Memory**: previews keep the VAE decoder resident through denoising, which the
  pipelines otherwise deliberately avoid — the decoder and the transformer are live at
  the same time. On a memory-tight machine that can get the process killed by the OS
  with no error at all, so enabling previews always prints a warning. Combining them
  with `--low-ram` works but is self-defeating, and adds a second line saying so.
- The window is centred on the middle of the clip by default, because frame 0 is the clean
  conditioning image on image-conditioned runs and never changes.
- Multi-stage pipelines tag files `_s1` / `_s2`; the stages run at different resolutions.
- Decoding a window in isolation uses different boundary padding than the full-volume
  decode, so the outermost frames are approximate. Fine for judging motion, not for
  judging final quality.
- A preview failure never aborts a generation: the first one logs a warning and disables
  previews for the rest of the run.

### Environment variables

See [docs/PIPELINES.md § Speed](docs/PIPELINES.md#speed) and [§ Memory](docs/PIPELINES.md#memory) for the watchdog / Gemma-length / low-RAM environment variables and their trade-offs.

`LTX2_COMPUTE_DTYPE=float16` runs the DiT's attention and feed-forward in float16 (opt-in; on an M1 Max with the 2.5 q8 pack a 576×1024×241 `--distilled` I2V render went from 701 s to 563 s).

## Frame Count Reference

The number of frames lives on the `8k + 1` grid (VAE temporal compression 8x); an off-grid request is floored to it with a warning (e.g. 87 → 81). Common values at 24 fps:

| Frames | Duration | Latent frames | Notes |
|--------|----------|---------------|-------|
| 9 | 0.4s | 2 | Minimal, for quick tests |
| 25 | 1.0s | 4 | Short clip |
| 41 | 1.7s | 6 | |
| 49 | 2.0s | 7 | |
| 65 | 2.7s | 9 | |
| 81 | 3.4s | 11 | |
| 97 | 4.0s | 13 | **Default** for `a2v` / `ic-lora` |
| 121 | 5.0s | 16 | |
| 145 | 6.0s | 19 | |
| 161 | 6.7s | 21 | |
| 193 | 8.0s | 25 | Requires 64GB+ RAM |

Higher frame counts require more RAM. With int4 on 32GB, 97 frames at 512x320 is comfortable. Reduce resolution for longer videos.

## Pre-converted Weights

| Variant | HuggingFace | Size | RAM |
|---------|-------------|------|-----|
| bf16 | [dgrauet/ltx-2.3-mlx](https://huggingface.co/dgrauet/ltx-2.3-mlx) | ~42 GB | 64 GB+, or 32 GB with `--low-ram` |
| int8 | [dgrauet/ltx-2.3-mlx-q8](https://huggingface.co/dgrauet/ltx-2.3-mlx-q8) | ~26 GB | 32 GB+, or 16 GB with `--low-ram` |
| int4 | [dgrauet/ltx-2.3-mlx-q4](https://huggingface.co/dgrauet/ltx-2.3-mlx-q4) | ~12 GB | 16 GB+ |
| bf16 (2.5) | [dgrauet/ltx-2.5-mlx](https://huggingface.co/dgrauet/ltx-2.5-mlx) | ~120 GB | `--low-ram` on 32 GB |
| int8 (2.5) | [dgrauet/ltx-2.5-mlx-q8](https://huggingface.co/dgrauet/ltx-2.5-mlx-q8) | ~75 GB | recommended for 2.5 |
| int4 (2.5) | [dgrauet/ltx-2.5-mlx-q4](https://huggingface.co/dgrauet/ltx-2.5-mlx-q4) | ~47 GB | lower quality |

The 2.5 packs carry both the dev and the distilled transformer plus the Gemma-4 text tower, so
their size is the download, not what one pipeline loads.

Weights are pre-converted to MLX format by [mlx-forge](https://github.com/dgrauet/mlx-forge).

## Packages

| Package | Description |
|---------|-------------|
| `ltx-core-mlx` | Model library: DiT, VAE, audio, text encoder, conditioning, guidance |
| `ltx-pipelines-mlx` | Generation pipelines: T2V, I2V, A2V, retake, extend, two-stage, distilled, DFR, IC-LoRA, HDR IC-LoRA, LipDub |
| `ltx-trainer-mlx` | Training: LoRA fine-tuning with flow matching |

## Resources

- [LTX-2](https://github.com/Lightricks/LTX-2) — Lightricks reference (ltx-core + ltx-pipelines + ltx-trainer)
- [mlx-forge](https://github.com/dgrauet/mlx-forge) — weight conversion tool
- Pre-converted weights — HuggingFace collections for [LTX-2.3](https://huggingface.co/collections/dgrauet/ltx-23) and [LTX-2.5](https://huggingface.co/collections/dgrauet/ltx-25-6a90c410ff65a75f8aeae402)
- [MLX](https://github.com/ml-explore/mlx) — Apple Silicon ML framework

## License

MIT