# WinDOM: Self-Family Distillation for Small-Model GUI Grounding

[![arXiv](https://img.shields.io/badge/arXiv-2606.25964-b31b1b.svg)](https://arxiv.org/abs/2606.25964)
[![Website](https://img.shields.io/badge/Website-windom.chlc.top-blue.svg)](https://windom.chlc.top)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)

**Chengheng Li Chen, Zhiqian Zhou, Hao Chen, Nicolas Chauvin** — ICML 2026 submission

**Links:** [Paper (arXiv)](https://arxiv.org/abs/2606.25964) · [Project website](https://windom.chlc.top) · [WinDOM corpus (HF)](https://huggingface.co/datasets/Chengheng/gui-dataset-win11react-v42-500) · [Cleaned split (HF)](https://huggingface.co/datasets/Chengheng/webos-grounding-clean-v2)

We study how far a small (2B) GUI grounding model can be pushed without any
external teacher. We introduce **WinDOM**, a 54,425-record GUI grounding
corpus harvested by driving an open-source Windows 11 web reimplementation
([win11react](https://github.com/blueedgetechno/win11React)) under headless
Playwright and reading ground-truth bounding boxes directly off the DOM. On
top of it we propose **Self-Family Distillation (SFD)** — rejection-sampling
SFT from a frozen same-family 4B teacher prompted with a noisy GT-bbox hint
(SFD-4B), or from an EMA copy of the student itself (self-EMA) — followed by
**GRPO**, where the *saturation depth of the cold-start checkpoint* is an
explicit hyperparameter: under-saturated inits (step 100) beat converged ones
(step 900). SFD-4B + early-init GRPO improves over base Qwen3.5-2B by **+5.4
OOD-mean** accuracy (**+3.5** ScreenSpot-Pro, **+7.0** OSWorld-G, **+5.8**
ScreenSpot-V2); the fully self-referential self-EMA variant nearly matches
the 4B teacher (**65.2 vs 66.3** OOD-mean).

The paper itself is in [paper/](paper/) (`windom_paper.pdf`); the corpus is
documented in [data/README.md](data/README.md).

## Repository layout

```
WinDOM/
├── README.md                  ← this file
├── pyproject.toml / requirements.txt
├── run_from_config.py         ← scraper pipeline entry point (config-driven)
├── paper/                     ← main.tex, main.bib, windom_paper.pdf, styles,
│   └── figures/               ← make_figures.py + shipped figure PDFs + assets/
├── launchers/                 ← 13 run_train_v42_*.sh (training recipes + the
│                                checkpoint-init saturation ablation grid) +
│                                3 template_eval_*.sh (eval wrappers)
├── src/
│   ├── webos_scrapper/        ← WebOSScrapper package: predecessor trajectory
│   │                            pipeline (crawl → annotate → dedup → explore;
│   │                            Strategy pattern in trajectory/runner.py)
│   └── windom/                ← main package: training + evaluation
│       ├── cli.py             ← Facade: `python -m windom <command>` dispatches
│       │                        to the module mains (lazy heavy imports)
│       ├── core/              ← shared lib (prompts / parsing / scoring / data /
│       │                        misc)
│       ├── training/          ← sft.py, sfd_4b.py, sfd_self_ema.py, grpo.py
│       └── evaluation/        ← own_test.py, screenspot.py, osworld_g.py,
│                                test_official_prompt.py (sanity suite)
├── runners/                   ← run_win11_scrapper.py (Playwright CSS-selector
│                                trajectory executor), launch_parallel.py
├── configs/                   ← config_win11react.yml (pipeline config)
├── trajectories/              ← win11_css_500.jsonl (500 CSS-selector trajectories)
├── tests/                     ← test_strategies.py (unit tests, no browser/LLM needed)
├── data/
│   ├── README.md              ← WinDOM corpus documentation (format, pipeline, stats)
│   └── stats.json             ← generation stats
└── scripts/
    ├── download_dataset.py        ← fetches the WinDOM corpus from HuggingFace
    └── download_eval_benchmarks.py← fetches ScreenSpot-V2 / ScreenSpot-Pro
```

## Installation

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install -e .            # both packages (windom + webos_scrapper) + the `windom` CLI
pip install -r requirements.txt

# Scraper only (dataset pipeline): browser backend
playwright install chromium
```

`requirements.txt` covers both the scraper (`playwright`, `openai`, `pyyaml`,
`tqdm`) and the training/eval stack (`torch`, `transformers`, `trl`,
`datasets`, `accelerate`, `huggingface_hub`, `Pillow`; `wandb` is optional —
all launchers pass `--wandb`). Install the torch build matching your CUDA
version. Training and evaluation are single-GPU workloads.

## Reproducing the paper

All launchers resolve the repository root from their own location, export
`PYTHONPATH=$REPO_ROOT/src`, and invoke the unified CLI (`python -m windom
<command>` — same as the `windom` console script from `pip install -e .`), so
they can be run from anywhere. Export `HF_TOKEN` (and `WANDB_API_KEY` for
training), or put them in `~/.env`. Every command also works directly, e.g.
`python -m windom train-sft --help`.

### 1. Download the data

```bash
# WinDOM corpus: raw screenshots + cleaned training split (~54K records)
python3 scripts/download_dataset.py
#   → data/win11react_v42_500/  (incl. gemma4_ready/{train,val,test}.jsonl)

# ScreenSpot-V2 / ScreenSpot-Pro eval benchmarks
python3 scripts/download_eval_benchmarks.py
#   → eval_benchmarks/{screenspot_v2,screenspot_pro}/samples.json
```

OSWorld-G is streamed directly from HuggingFace (`MMInstruction/OSWorld-G`)
by the eval script — nothing to download.

### 2. SFT baseline (cold-start arm + baseline rows)

```bash
bash launchers/run_train_v42_sft.sh
```

Produces `$CKPT_ROOT/v42_sft/checkpoint-{100..1400}` (default
`./checkpoints/`), pushed to `Chengheng/webos-v42-sft`.

### 3. SFD cold-starts

```bash
bash launchers/run_train_v42_4b_hint.sh    # SFD-4B: frozen 4B hint teacher
bash launchers/run_train_v42_self_ema.sh   # self-EMA: teacherless variant
```

Produces `checkpoint-{100..1400}` every 100 steps, pushed to
`Chengheng/webos-v42-4b-hint` / `Chengheng/webos-v42-self-ema`
(`run_train_v42_self_ema_4b.sh` is the 4B student=teacher arm).

### 4. GRPO with saturation-depth choice

Champion recipe (SFD-4B cold-start, under-saturated cp100 init):

```bash
bash launchers/run_train_v42_grpo_4bh.sh
```

The full ablation grid — same command, different `--model`/`--subfolder`
init:

| Init checkpoint | Launcher |
|---|---|
| SFT cp100 / cp900 | `run_train_v42_grpo.sh` / `run_train_v42_grpo_sft_cp900.sh` |
| SFD-4B cp100 / cp900 | `run_train_v42_grpo_4bh.sh` / `run_train_v42_grpo_4bhint_cp900.sh` |
| Self-EMA cp100 / cp900 | `run_train_v42_grpo_ema_cp100.sh` / `run_train_v42_grpo_ema_cp900.sh` |
| Self-EMA-4B cp100 | `run_train_v42_grpo_emacp100_4b.sh` |
| Seed replicates (error bars) | `run_train_v42_grpo_4bh_seed123.sh`, `run_train_v42_grpo_4bh_seed2024.sh` |

Each run saves `checkpoint-{50,100,...}` under `$CKPT_ROOT` and pushes to its
`--hf-repo`. RL hyperparameters (num_generations=8, lr 2e-6, β=0, dapo loss,
temperature 1.3) are in the launchers.

### 5. Evaluate

```bash
# Held-out WinDOM test set
CHECKPOINT=Chengheng/webos-v42-grpo-4bhint-cp900 SUBFOLDER=checkpoint-550 \
    bash launchers/template_eval_own_test.sh

# ScreenSpot-V2 (BENCHMARK=screenspot_pro for ScreenSpot-Pro)
CHECKPOINT=Chengheng/webos-v42-4b-hint SUBFOLDER=checkpoint-200 \
    BENCHMARK=screenspot_v2 bash launchers/template_eval_screenspot.sh

# OSWorld-G
CHECKPOINT=Chengheng/webos-v42-grpo-4bhint-cp900 SUBFOLDER=checkpoint-100 \
    bash launchers/template_eval_osworld_g.sh
```

All evals use greedy decoding, `enable_thinking=False`, and point-in-bbox
click accuracy; each writes `eval_results/<tag>/results.jsonl` + `eval.log`.
Expected metric keys: `click_in_gt` (own test / ScreenSpot) and `hit`
(OSWorld-G). Published checkpoints (`Chengheng/webos-v42-*`) let you skip
steps 2–4 entirely.

### 6. Figures and paper PDF

```bash
cd paper/figures && python3 make_figures.py   # reads eval_results/, writes fig*.pdf
cd ../.. && cd paper && pdflatex main && bibtex main && pdflatex main && pdflatex main
```

### Sanity checks (no GPU needed)

```bash
PYTHONPATH=src python3 -m unittest discover -s tests   # scraper unit tests
PYTHONPATH=src python3 src/windom/evaluation/test_official_prompt.py
                                                       # prompt/parse/score tests
                                                       # (needs training deps installed)
```

## Corpus construction

The WinDOM corpus was built in 6 stages (Playwright scene capture → DOM bbox
scan → pixel filters → VLM labeling → VLM QA → rule-based cleanup), documented
in [data/README.md](data/README.md). **Note:** the final
harvester scripts were never committed to the source repository and are not
included here (they must be recovered from the training cluster); the corpus
itself is fully available on HuggingFace via step 1, and
`src/webos_scrapper/` + `runners/` document the predecessor trajectory
pipeline the harvester grew out of.

## Citation

```bibtex
@article{chen2026windom,
  title   = {WinDOM: Self-Family Distillation for Small-Model GUI Grounding},
  author  = {Chen, Chengheng Li and Zhou, Zhiqian and Chen, Hao and Chauvin, Nicolas},
  journal = {arXiv preprint arXiv:2606.25964},
  year    = {2026}
}
```

## Data & model artifacts

- Corpus: [Chengheng/gui-dataset-win11react-v42-500](https://huggingface.co/datasets/Chengheng/gui-dataset-win11react-v42-500) (raw), [Chengheng/webos-grounding-clean-v2](https://huggingface.co/datasets/Chengheng/webos-grounding-clean-v2) (cleaned)
- Checkpoints: `Chengheng/webos-v42-{sft,4b-hint,self-ema,self-ema-4b}` (cold-starts), `Chengheng/webos-v42-grpo-*` (RL runs)
- Benchmarks: [OS-Copilot/ScreenSpot-v2](https://huggingface.co/datasets/OS-Copilot/ScreenSpot-v2), [likaixin/ScreenSpot-Pro](https://huggingface.co/datasets/likaixin/ScreenSpot-Pro), [MMInstruction/OSWorld-G](https://huggingface.co/datasets/MMInstruction/OSWorld-G)