WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs
Jack Hong
1, [Shilin Yan](https://scholar.google.com/citations?user=2VhjOykAAAAJ&hl=zh-CN&oi=ao)
1†, Jiayin Cai
1, [Xiaolong Jiang](https://scholar.google.com/citations?user=G0Ow8j8AAAAJ&hl=zh-CN&oi=ao)
1, [Yao Hu](https://scholar.google.com/citations?user=LIu7k7wAAAAJ&hl=en)
1, [Weidi Xie](https://scholar.google.com/citations?user=Vtrqj4gAAAAJ&hl=en)
2‡
1Xiaohongshu Inc.
2Shanghai Jiao Tong University
[[🏠 Project Page](https://jaaackhongggg.github.io/WorldSense/)] [[📖 arXiv Paper](https://arxiv.org/pdf/2502.04326)] [[🤗 Dataset](https://huggingface.co/datasets/honglyhly/WorldSense)] [[🏆 Leaderboard](https://jaaackhongggg.github.io/WorldSense/#leaderboard)]
## 📍 Leaderboard
We welcome researchers and developers from the community to submit their models for evaluation and inclusion on the WorldSense [leaderboard](https://jaaackhongggg.github.io/WorldSense/#leaderboard).
To streamline the process, please send an email to **jaaackhong@gmail.com** and CC **tattoo.ysl@gmail.com**.
## 🚀 News
1. [2026-04-28] [Doubao-Seed-2.0-Lite-0428]() is added to [Leardboard](https://jaaackhongggg.github.io/WorldSense/#leaderboard), which achieves a top score of **67.3**!
2. [2026-04-28] [Nemotron3-Nano-Omni](https://arxiv.org/abs/2604.24954) is added to [Leardboard](https://jaaackhongggg.github.io/WorldSense/#leaderboard).
3. [2026-04-01] [Gemini-3.1-Pro](https://deepmind.google/models/gemini/pro/), [Qwen-3.5-Omni-Plus](https://qwen.ai/blog?id=qwen3.5-omni) and [Qwen-3.5-Omni-Flash](https://qwen.ai/blog?id=qwen3.5-omni) are added to [Leardboard](https://jaaackhongggg.github.io/WorldSense/#leaderboard). Gemini-3.1-Pro achieves a top score of **65.5**!
4. [2026-01-26] 🎉🎉🎉 WorldSense is accepted by ICLR 2026.
5. [2025-09-09] [Gemini-2.5-Pro](https://deepmind.google/models/gemini/pro/) and [Gemini-2.5-Flash](https://deepmind.google/models/gemini/flash/) are added to [Leardboard](https://jaaackhongggg.github.io/WorldSense/#leaderboard). Gemini-2.5-Pro achieves a top score of **65.1**!
6. [2025-04-18] [VITA-1.5](https://arxiv.org/pdf/2501.01957) and [Qwen2.5 Omni](https://qwenlm.github.io/blog/qwen2.5-omni/) are added to [Leardboard](https://jaaackhongggg.github.io/WorldSense/#leaderboard).
7. [2025-02-07] We release [WorldSense](https://huggingface.co/datasets/honglyhly/WorldSense), the first benchmark for real-world omnimodal understanding of MLLMs.
## 👀 WorldSense Overview
we introduce **WorldSense**, the **first** benchmark to assess the multi-modal video understanding, that simultaneously encompasses _visual, audio, and text_ inputs. In contrast to existing benchmarks, our **WorldSense** has several features:
* **Collaboration of omni-modality**. We design the evaluation tasks to feature a strong coupling of audio and video, requiring models to effectively utilize the **synergistic perception of omni-modality**;
* **Diversity of videos and tasks**. WorldSense encompasses a diverse collection of **1,662** audio-visual synchronised videos, systematically categorized into **8** primary domains and **67** fine-grained subcategories to cover the broad scenarios, and **3,172** multi-choice QA pairs across **26** distinct tasks to enable the comprehensive evaluation;
* **High-quality annotations**. All the QA pairs are manually labeled by 80 expert annotators with multiple rounds of correction to ensure quality.
Based on our **WorldSense**, we extensively evaluate various state-of-the-art models. The experimental results indicate that existing models face significant challenges in understanding real-world scenarios (48% best accuracy). We hope our **WorldSense** can provide a platform for evaluating the ability in constructing and understanding coherent contexts from omni-modality.
