---
layout: '@/layouts/Doc.astro'
title: '🍋 ezpz: distributed PyTorch across any hardware'
date: 2026-01-10
date-created: 2026-01-10
date-modified: today
description: 'A history and overview of `ezpz`, with AMD and Intel PyTorch enablement timelines and why portable distributed training across GPU vendors is finally possible.'
---

For most of PyTorch's first decade, "running PyTorch" effectively meant
"running PyTorch on NVIDIA". Every distributed training script, every
profiler, every example notebook assumed CUDA. If you wanted to run the
same code on AMD or Intel hardware, you were either going to rewrite a
launch script, port a kernel, or maintain a vendor-specific fork — often
all three.

That picture has changed faster than most people realize. In the last
two years, [PyTorch][pytorch] gained native Intel GPU support, AMD
shipped day-zero ROCm builds for every PyTorch release, and Intel's
out-of-tree extension is now finishing its phased shutdown.[^ipex-eol]
You can write one PyTorch script today and run it across NVIDIA, AMD,
and Intel hardware with no code changes — _if_ you handle the
launch / environment / device-init differences.

That last "if" is what [`ezpz`][ezpz] exists to absorb. This post is
mostly about how the vendor landscape got here, and a little about what
that means for the launcher.

[pytorch]: https://pytorch.org
[ezpz]: https://ezpz.cool

## The two timelines

The clearest way to see the shift is side-by-side: AMD's gradual
ROCm-everywhere strategy, and Intel's faster but later push to merge
IPEX into upstream PyTorch.

```mermaid
%%{init: {'themeCSS': '.titleText{color:var(--foreground1)!important;fill:var(--foreground1)!important;font-size:0.95rem!important;font-weight:700;}.taskText{font-weight:600;font-size:0.74rem!important;}.taskText,.taskTextOutsideLeft,.taskTextOutsideRight,.sectionTitle,.tick text{fill:var(--foreground0)!important;}.taskTextOutsideLeft,.taskTextOutsideRight,.sectionTitle{font-size:0.74rem!important;}.tick text{font-size:0.7rem!important;}.taskTextOutsideRight{text-anchor:start;transform:translateX(0.45ch);}.taskTextOutsideLeft{text-anchor:end;transform:translateX(-0.45ch);}.todayMarker{stroke:var(--red)!important;stroke-width:0.12rem;opacity:0.9;}.grid .tick line{stroke:var(--background3)!important;opacity:0.6;}.section0{fill:color-mix(in oklch,var(--background1) 72%,transparent)!important;}.section1{fill:color-mix(in oklch,var(--blue) 38%,transparent)!important;}.active,.done{fill:color-mix(in srgb,var(--blue) 72%,white 28%)!important;}.crit,.milestone{fill:var(--red)!important;stroke:var(--red)!important;}'}}%%
gantt
    title AMD and Intel PyTorch Enablement Timeline
    dateFormat  YYYY
    axisFormat  %Y

    section AMD ROCm and PyTorch
      Torch7 era and early CUDA to HIP ports        :amd1, 2012, 2016
      ROCm 1.0 and HIPIFY tooling                   :amd2, 2016, 2020
      Official PyTorch ROCm Python packages         :amd3, 2021, 2022
      PyTorch Foundation governance participation   :amd4, 2022, 2023
      Triton ecosystem support                      :amd6, 2023, 2024
      MI300x PyTorch guidance                       :amd7, 2024, 2024

    section Intel and PyTorch
      Initial PyTorch contributions                :i2, 2018, 2019
      Intel Extension for PyTorch launch           :i3, 2020, 2024
      VTune ITT API integration in PyTorch         :milestone, i4, 2022, 1d
      PyTorch Foundation Premier membership        :milestone, i5, 2023, 1d
      Prototype native Intel GPU support           :milestone, i6, 2024, 1d
      Solid native Intel GPU support               :milestone, i7, 2025, 1d
      IPEX feature upstreaming completion          :milestone, i8, 2025, 1d
      Intel Extension for PyTorch end of life      :milestone, crit, i9, 2026, 1d
```

Lining the AMD and Intel work up against the actual PyTorch release
cadence is illuminating — most of the integration milestones land on
specific PyTorch versions:

```mermaid
%%{init: {'themeCSS': '.titleText{color:var(--foreground1)!important;fill:var(--foreground1)!important;font-size:0.95rem!important;font-weight:700;}.taskText{font-weight:600;font-size:0.74rem!important;}.taskText,.taskTextOutsideLeft,.taskTextOutsideRight,.sectionTitle,.tick text{fill:var(--foreground0)!important;}.taskTextOutsideLeft,.sectionTitle{font-size:0.74rem!important;}.taskTextOutsideRight{font-size:0.66rem!important;text-anchor:start;transform:translateX(0.2ch);}.tick text{font-size:0.7rem!important;}.taskTextOutsideLeft{text-anchor:end;transform:translateX(-0.45ch);}.todayMarker{stroke:var(--red)!important;stroke-width:0.12rem;opacity:0.9;}.grid .tick line{stroke:var(--background3)!important;opacity:0.6;}.section0{fill:color-mix(in oklch,var(--orange) 30%,transparent)!important;}.section1{fill:color-mix(in oklch,var(--background2) 76%,transparent)!important;}.section2{fill:color-mix(in oklch,var(--blue) 42%,transparent)!important;}.active,.done{fill:color-mix(in srgb,var(--blue) 72%,white 28%)!important;}.crit,.milestone{fill:var(--red)!important;stroke:var(--red)!important;}'}}%%
gantt
    title PyTorch Vendor Integration Timeline AMD vs Intel
    dateFormat  YYYY-MM-DD
    axisFormat  %Y

section AMD
    Installable PyTorch ROCm Python packages         :amd2, 2021-03-04, 1d
    ROCm marked stable                               :amd3, 2022-06-28, 1d

section PyTorch Releases
    1.8                                             :milestone, crit, pt180, 2021-03-04, 1d
    1.12                                            :pt1120, 2022-06-28, 1d
    2.0                                             :milestone, crit, pt200, 2023-03-15, 1d
    2.4                                             :pt24, 2024-07-24, 1d
    2.5                                             :milestone, crit, pt250, 2024-10-17, 1d
    2.6                                             :pt260, 2025-01-29, 1d
    2.7                                             :pt270, 2025-04-23, 1d
    2.8                                             :crit, pt280, 2025-08-06, 1d
    2.9                                             :pt290, 2025-10-15, 1d
    2.10                                            :pt210, 2026-01-15, 1d

section Intel
    Intel GPU improvements begin                        :int2, 2024-07-24, 1d
    Native Intel GPU support in 2.5                    :int3, 2024-10-17, 1d
    Intel GPU eager/compile parity in 2.7              :int4, 2025-04-23, 1d
    Intel XCCL backend in 2.8                           :int5, 2025-04-23, 1d
    IPEX discontinued                                  :int6, 2025-08-06, 2026-03-31
    IPEX end of life                                   :milestone, crit, int7, 2026-03-31, 1d
```

> **Heads up:** Intel's separate IPEX project reaches end-of-life in
> **March 2026** — by then, native PyTorch is the only supported path
> on Intel GPUs.

## AMD: a long, quiet build-up

AMD's path to first-class PyTorch support is a 14-year project that
mostly happened out of view. The pre-history goes back to the Torch7
era — well before PyTorch existed in its current form — and it's not an
accident that ROCm landed on Caffe and Torch7 first. AMD was building
the porting story (HIP, HIPIFY, the C++ dialect, the toolchain) on the
_previous_ generation of frameworks before the new one became
production-default.

That patience paid off in three big jumps:

- **2021 — installable wheels.** Before March 2021, you couldn't just
  `pip install torch` and get an AMD-compatible build. Once the ROCm
  Python packages went official, AMD became a one-line install on
  supported Linux systems — the same UX as CUDA. PyTorch 1.8 was the
  first release with that working out of the box.
- **2022 — governance.** AMD joined the PyTorch Foundation as a
  founding member when the project moved under the Linux Foundation.
  This was the point at which AMD's integration stopped being "a vendor
  patch" and started being a co-owned roadmap.
- **2023 — day-zero.** With PyTorch 2.0, AMD shipped ROCm 6.0 with
  same-day support, including TorchDynamo / TorchInductor on AMD
  hardware. This was the first release where you could pick up a fresh
  PyTorch and have AMD work _immediately_ — no lag, no porting window.

The rest of the timeline is filling in the corners: OpenAI Triton
support arrived in 2023, MI300x guidance in mid-2024, native PyTorch on
Windows for consumer Radeon cards in late 2025. The overall trajectory
is clear: AMD is no longer playing catch-up on the _framework_. The
remaining gaps are about specific kernels, FlashAttention variants,
custom collectives — work that lives in extensions, not in PyTorch
itself.

## Intel: a much faster, much later push

Intel's story is compressed into a much shorter window — basically four
years vs AMD's fourteen — because Intel arrived after the framework had
already standardized. Instead of a slow, parallel ROCm-style stack,
Intel went the out-of-tree extension route first
(IPEX, 2020) and only started the upstream merge in earnest with
PyTorch 2.4 in 2024.

The integration cadence has been remarkably tight:

- **2.4 (Jul 2024)** — first prototype native Intel GPU support
- **2.5 (Oct 2024)** — solid native Intel GPU support landed
- **2.7 (Apr 2025)** — eager + `torch.compile` parity on Intel GPUs
- **2.8 (Aug 2025)** — XCCL collective backend; IPEX active
  development ceases
- **2.10 / Mar 2026** — IPEX project reaches end-of-life

Notable to me: Intel chose to _finish_ upstreaming before retiring the
extension. The IPEX EOL date isn't where the work stops — it's where
the redundancy stops. The features have already moved.

## What this means in practice

If you're writing a new training script today (early 2026), the
boilerplate problem has shifted. You used to spend most of the lifting
on:

1. Picking the right `torch.distributed` backend (`nccl`, `gloo`,
   `xccl`, `rccl`, ...).
2. Knowing which environment variables your launcher expects on this
   particular cluster (`MASTER_ADDR`, `WORLD_SIZE`, `LOCAL_RANK`,
   `PALS_*`, `PMI_*`, `OMPI_*`, `SLURM_*`...).
3. Handling per-vendor device init quirks (`torch.cuda.set_device` vs
   `xpu.set_device` vs `hip.set_device`).
4. Then, finally, the model code.

Steps 1–3 are now _almost_ the same across vendors. The collective
backends mostly map to the right thing automatically. The device
abstraction is unified under `torch.accelerator` (in 2.7+). What's left
is mostly the launch boilerplate — which is what 🍋 [`ezpz`][ezpz]
takes care of:

- `ezpz launch` figures out the launcher (`mpiexec`, `srun`,
  `torchrun`, `deepspeed`) from the environment.
- `ezpz_setup_*` shell helpers normalize the rank/size variables
  across PBS / SLURM / standalone.
- `ezpz yeet` distributes your environment to every node so you don't
  pay the Lustre-import tax — covered in [Running 50k Python
  Processes on Aurora](/posts/2026/05/01).
- The Python entry points stay vendor-agnostic; device init goes
  through one helper that picks `cuda` / `xpu` / `hip` based on what's
  actually available.

The point isn't that `ezpz` is doing anything magical — it's that the
_framework_ finally caught up enough that a small, vendor-agnostic
launcher can exist at all. Five years ago, this post would have been
about writing per-vendor shims. Today it's about deleting them.

## Detailed timelines

For reference, the full chronology:

### AMD

- **Pre-2021 — Torch7 era and CUDA→HIP ports.** Torch7 was released in
  2012 as a precursor to PyTorch (C++ + CUDA). With ROCm 1.0, AMD
  demonstrated CUDA→HIP conversion using HIPIFY, including ports of
  Caffe and Torch7.
- **March 2021** — PyTorch for AMD ROCm becomes officially available
  as a Python package on supported Linux systems.
- **September 2022** — PyTorch joins the Linux Foundation; AMD is a
  founding member of the PyTorch Foundation governing board.
- **April 2023** — AMD ships day-zero support for PyTorch 2.0 within
  the ROCm 6.0 ecosystem, including TorchDynamo/TorchInductor.
- **2023** — OpenAI Triton support extended to AMD GPUs.
- **June 2024** — MI300x PyTorch guidance published, with near
  drop-in compatibility for code written for NVIDIA GPUs.
- **September 2025** — Public preview of PyTorch on Windows for select
  consumer Radeon RX 7000/9000 series GPUs and Ryzen AI APUs (no
  WSL2 needed).
- **October 2024** — How-to guide for Torchtune (PyTorch LLM
  fine-tuning library) on AMD GPUs.
- **November 2025** — AMD Software: PyTorch on Windows Edition 7.1.1
  with ROCm 7.1.1.
- **2026 / post-2026** — MI450X rack-scale solution targeting NVIDIA
  high-end parity in H2 2026; MI500 series in development.

### Intel

- **2018** — Intel begins contributing to upstream PyTorch.
- **2020** — Intel Extension for PyTorch (IPEX) launches as a separate
  package for Intel CPUs and GPUs.
- **October 2022**[^pt113] — PyTorch 1.13 ships with integrated
  Intel VTune ITT API support.
- **August 2023**[^intel-pt-foundation] — Intel joins the PyTorch
  Foundation as a Premier member.
- **July 2024** — PyTorch 2.4 with prototype native Intel GPU support
  (client + data center).
- **April 2025** — PyTorch 2.7 establishes solid Intel GPU support in
  both eager and graph modes (`torch.compile`) on Windows and Linux.
- **August 2025** — IPEX active development ceases following the
  PyTorch 2.8 release; most features are upstreamed.
- **End of March 2026** _(planned)_ — IPEX reaches end-of-life. Use
  native PyTorch directly.

[^ipex-eol]:
    Even now, in 2026, plenty of code is still NVIDIA-centric and is
    rarely designed with multi-platform support in mind — but the
    framework no longer is.

[^pt113]: [PyTorch 1.13 release](https://pytorch.org/blog/pytorch-1-13-release/)

[^intel-pt-foundation]:
    [Intel Joins the PyTorch
    Foundation](https://www.edge-ai-vision.com/2023/08/driving-pytorch-and-ai-everywhere-intel-joins-the-pytorch-foundation/)
