IGG: A Benchmark for Interactive GUI Grounding
under Visibility Constraints

Kyeong Seon Kim*Jiyeon Son*Tae-Hyun Oh

KAIST *Equal contribution

EMNLP 2026 Main

EMNLP 2026 Main Presentation

Your turn: Just for fun!

Human Leaderboard

GA: tasks solved (%). TT: average time across all four tasks.

Loading records…

RankPlayerGA ↑TT ↓DateYour record

What is IGG?

IGG evaluates GUI grounding when targets must first be revealed through interaction.

Seven visibility constraints before and after interaction: delayed state activation (wait), dynamic target interaction (wait), movable occlusion (drag), scroll-based exploration (scroll), visual disambiguation (move), hidden state activation (click), removable occlusion (click). Center: capabilities needed to reveal the target.
Seven visibility constraints and the interactions that reveal the target.

Task

Agents use screenshots and action history to reveal and click the target in up to eight actions.

Static grounding: target visible, goal localize, action CLICK. IGG: target hidden or inactive, goal reveal then localize, actions SCROLL, MOVE, WAIT, DRAG, CLICK.

Benchmark

601 tasks across 16 web environments modeled on familiar applications.

Task distribution: Viewport 167 scroll-based exploration; Multi-State 89 visual disambiguation, 104 hidden state activation, 53 removable occlusion; Temporal and Manipulation 70 delayed state activation, 58 dynamic target interaction, 60 movable occlusion.
  • Observabilitybring the target into view
  • Disambiguationreveal cues on hover
  • State controlexpand controls or dismiss overlays
  • Temporal controlwait for the right state
  • Google Calendar
  • Google Docs
  • Google Sheets
  • Notion
  • Gmail
  • VS Code Web
  • Figma
  • GitHub
  • Discord
  • Twitter / X
  • Spotify
  • Netflix
  • YouTube
  • Google
  • Walmart
  • Google Maps

Results

Human and model performance

Human participants achieve 94.6% accuracy in our study. On the full benchmark, GPT-5.5 reaches 79.4% and the strongest evaluated open model reaches 31.6%.

Accuracy by constraint subtype. Overall accuracy is 94.6% in the human study, and 79.4% for GPT-5.5 and 30.8% for Qwen3-VL on the full benchmark.
Grounding accuracy (%) by constraint subtype.

Revealing the target

Specifying the reveal action raises Qwen3-VL’s accuracy from 30.8% to 33.9%. Revealing the target for the model raises it to 81.9%.

Qwen3-VL accuracy: unassisted 30.8, reveal action specified 33.9, target revealed 81.9. GPT-5.5 unassisted: 79.4.

Learning to reveal targets

Base models rely mostly on clicks and never use drag.

Action distribution on the validation set: base models mostly click and never drag; trained Qwen3-VL uses drag for 10% of its actions.
Action distribution on the validation set.

SFT followed by GRPO improves accuracy by 17.5–19.2 percentage points across three 8B models. Qwen3-VL’s accuracy on movable occlusions rises from 0% to 62.5%.

Validation accuracy before and after SFT+GRPO: Qwen3-VL 40.0% to 59.2%, InternVL3.5 28.3% to 47.5%, GUI-Owl-1.5 35.0% to 52.5%.
Validation accuracy (%) before and after training.
Qwen3-VL validation accuracy by constraint subtype before and after training. Accuracy on movable occlusions rises from 0% to 62.5%.
Qwen3-VL validation accuracy (%) by constraint subtype.

Why it matters

Hidden controls, overlays, and loading states pose challenges for GUI agents. IGG provides a benchmark to evaluate these challenges and improve how agents handle them.

Where this could help

Let’s grab coffee! ☕

Attending EMNLP 2026 in Budapest?

If you’re working on GUI agents, human-centered AI, accessible interactions, or cultural and individual differences in human-AI interaction, we’d love to connect and exchange ideas!

Always happy to chat over coffee. ☕

ella94.ai@gmail.com

Citation

@inproceedings{kim2026igg,
  title     = {{IGG}: A Benchmark for Interactive {GUI} Grounding under Visibility Constraints},
  author    = {Kim, Kyeong Seon and Son, Jiyeon and Oh, Tae-Hyun},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
  year      = {2026}
}