# Win11React GUI Grounding Dataset (v4.2, 500 scenes)

Generated by the WebOSScrapper pipeline against [win11React](https://github.com/blueedgetechno/win11React).

## Stats

- **54,425 records** (image + bounding box + natural-language instruction)
- **2,013 unique screenshots** across **500 scenes**
- **15 apps**: Explorer, Edge, Settings, Spotify, Calculator, Notepad, Whiteboard, Word, Discord, VS Code, Terminal, Mail, Camera, Microsoft Store, Task Manager
- **6 layouts**: single, corners, mosaic, cascading, split2, many_cascading
- **4 viewports**: 1920×1080, 1920×1200, 2560×1440, 1680×1050
- **~75% accuracy** (audited on 100 random samples; ~25% problem rate, dominated by sidebar mismatches and toolbar-icon offsets)

## Format (ScreenSpot-Pro compatible)

Each line of `screenspot.jsonl` is a JSON object:

```json
{
  "id": "win11react_scene_0042__step03_3apps_msedge_127",
  "img_filename": "outputs/dataset_v42_500/scenes/scene_0042/screenshots/step03_3apps_msedge.png",
  "instruction": "Click the close button of the Edge browser window",
  "bbox": [1872, 0, 1922, 28],
  "img_size": [1920, 1080],
  "application": "win11react",
  "platform": "web",
  "ui_type": "icon",
  "_dom_role": "button",
  "_dom_class": "uicon hvclose",
  "_apps_visible": ["MSEDGE", "WHITEBOARD", "CALCUAPP"]
}
```

### Provenance flags

Records carry tags showing how the label was produced:

- `_relabel_v2: true` — label generated by Gemini 2.5 Flash via `_relabel_dropped_v2.py` (recovered from records the original VLM labeling stage dropped due to API errors)
- `_qa_full_pass: true` — label was reviewed by a second Gemini 2.5 Flash pass; either kept as-is or edited
- `_human_corrected: true` — manually verified by a human reviewer
- `_agent_qa_edited: true` — edited by a Claude vision agent in the multi-agent QA pass

## Pipeline

The dataset was built in 6 stages by `16_collect_dataset.py`
(**not included in this repository** — the final harvester scripts were never
committed to the source repo and must be recovered from the training cluster):

1. **Scene capture (Playwright)** — open N apps in a layout, take a screenshot per intermediate state
2. **DOM bbox scan** (`13_full_dom_scan.py`, not included) — extract every visible interactable element + its bounding box, with per-app occlusion checks
3. **Pixel filters** — drop elements smaller than 14×14, blank crops, oversized container regions
4. **VLM labeling** (Gemini 2.5 Flash) — sees the marked screenshot + DOM hint, produces a 4–14 word ScreenSpot-Pro-style instruction
5. **VLM QA pass** (Gemini 2.5 Flash) — verifies each label by re-examining the marked screenshot; edits or drops mismatches
6. **Rule-based cleanup** — drops records with named-app-on-taskbar (DOM doesn't track slot order reliably), wrong-size window-control bboxes, and quoted-text-vs-DOM mismatches

## Known limitations

- **Taskbar slot misindexing** — the DOM doesn't reliably link taskbar `<div>` slots to their app identity, so labels naming a specific app on the taskbar (e.g., `"Click the Spotify icon on the taskbar"`) were dropped wholesale. Generic taskbar labels (`Start button`, `search icon`, `system tray`) are kept.
- **Window-control swaps** — the maximize/restore icon (two overlapping squares) is occasionally labeled as "minimize". Most of these were caught by the bbox-size filter, but ~5–10% slip through.
- **Sidebar/icon mismatches** — when a sidebar contains many similar icons, the DOM label may not match the visually-rendered one.

## Files

- `screenspot.jsonl` — the dataset (52 MB, 54,425 records)
- `stats.json` — generation stats (scene count, layout/viewport distribution, model used, drop counts)

The screenshots themselves (~2,000 PNGs, ~3.8 GB) are not included in this branch — they are distributed on HuggingFace (`Chengheng/gui-dataset-win11react-v42-500`, fetch via `scripts/download_dataset.py`) and are regenerable from the pipeline. The original regeneration command (script **not included** in this repository, see note above) was:

```bash
python 16_collect_dataset.py \
    --output outputs/dataset_v42_500 \
    --num-scenes 500 \
    --vlm google/gemini-2.5-flash \
    --max-workers 12 \
    --no-llm
```

## License

Same as parent repository.