## 🧠 Abstract
Large multimodal models (LMMs) have demonstrated strong OCR recognition capabilities, yet remain vulnerable to adversarial visual text that is readable to humans but challenging for models to localize and recognize. Existing OCR benchmarks mainly focus on natural or document-style text, while adversarial OCR evaluations remain limited in scale, task coverage, or region-aware evaluation.
In this paper, we formulate adversarial OCR as a **grounded OCR perception** task and introduce **AdvSpot**, the first benchmark for grounded adversarial OCR evaluation. AdvSpot comprises 390 images with region-level annotations, spanning 5 primary categories and 13 fine-grained adversarial OCR types. To address this challenge, we propose **ArmorOCR**, a two-stage training framework for robust adversarial OCR perception. ArmorOCR first acquires missing adversarial OCR perception from privileged transformed observations through On-Policy Self-Distillation (OPSD), and then refines grounded OCR perception through Group Relative Policy Optimization (GRPO) with task-conditioned rewards for localization, recognition, full spotting, and visual question answering (VQA).
Extensive experiments on AdvSpot, other adversarial OCR benchmarks, and general OCR benchmarks demonstrate that ArmorOCR consistently improves adversarial OCR perception while preserving competitive general OCR capability.
## 🚀 Release
- [2026/09/23] 🔥 Released the **AdvSpot benchmark data** ([inclusionAI/advspot-public](https://huggingface.co/datasets/inclusionAI/advspot-public)) and updated `advspot_infer.py` (faster prefetched data loading).
- [2026/09/01] 🔥 Released the **GGUF quantized weights** ([inclusionAI/ArmorOCR-GGUF](https://huggingface.co/inclusionAI/ArmorOCR-GGUF)).
- [2026/08/24] 🔥 Released the AdvSpot evaluation/inference script (`advspot_infer.py`).
- [2026/08/10] 🔥 Released the inference code and examples.
## 📦 AdvSpot Benchmark
AdvSpot is the first grounded adversarial OCR perception benchmark, providing:
- **383 images** with region-level annotations (bounding boxes, transcriptions, perception-type labels, region-grounded VQA pairs) → **389 grounded VQA pairs**.
> 📊 **Public release vs. paper.** The paper reports 390 images / 397 VQA pairs. This public release is a **desensitized** version (sensitive samples removed, with replacements synthesized to keep every subtype at 30–50 samples) → 383 images / 389 pairs. See the dataset card for details.
- **5 primary categories** and **13 fine-grained adversarial OCR types** organized by underlying perception failure mechanisms:
- Spatial Manipulation (Rotated / Mirrored / Tiny Text)
- Glyph Variation (Stylized / Handwritten Text)
- Visual Encoding (Symbol / Dot / Line Encoding)
- Contextual Blending (AIGC Fusion / Low Contrast / Pattern Overlay)
- Imaging Degradation (Capture / Post-processing Artifacts)
- **Region-grounded evaluation** with VQA accuracy and IoU metrics.
**Taxonomy of adversarial OCR perception in AdvSpot.**
**Representative adversarial OCR examples across the 13 fine-grained types.**
The AdvSpot benchmark data is now available at 👉 [inclusionAI/advspot-public](https://huggingface.co/datasets/inclusionAI/advspot-public) (`data_public.jsonl` + `images/`; 383 images / 389 grounded VQA pairs; a desensitized public release). See [Evaluate on the AdvSpot benchmark](#3-evaluate-on-the-advspot-benchmark) for the evaluation/inference script (`advspot_infer.py`).
## ⚙️ Installation
```bash
conda create -n armorocr python=3.11
conda activate armorocr
pip install pillow==12.0.0
pip install torch==2.8.0 torchvision==0.23.0
pip install transformers==4.57.1 accelerate==1.12.0
```
> ⚠️ **Note:** ArmorOCR is built on the [Qwen3-VL](https://github.com/qwenlm/qwen3-vl) series. Please make sure your `transformers` version satisfies the minimum requirement of Qwen3-VL (see [Qwen3-VL README](https://github.com/qwenlm/qwen3-vl#requirements) for details).
## 🔍 Inference
### 1. Download model weights
Download the pre-trained weights from 👉 [inclusionAI/ArmorOCR](https://huggingface.co/inclusionAI/ArmorOCR) on Hugging Face.
> We also provide quantized GGUF checkpoints at 👉 [inclusionAI/ArmorOCR-GGUF](https://huggingface.co/inclusionAI/ArmorOCR-GGUF), along with a minimal `serve_gguf.sh` + inference example in that repo.
> The GGUF repo also includes an evaluation comparison table on AdvSpot (base vs. Q8_0 vs. Q4_K_M).
### 2. Run inference on the provided examples
```bash
python infer.py
```
Before running, please modify `YOUR_MODEL_PATH` in `infer.py` to point to your local checkpoint directory.
### 3. Evaluate on the AdvSpot benchmark
We provide `advspot_infer.py` for running ArmorOCR (and other Qwen3-VL-series models) on the AdvSpot benchmark and computing region-grounded evaluation metrics (VQA accuracy + IoU).
```bash
# 1) Download the benchmark data from https://huggingface.co/datasets/inclusionAI/advspot-public
# (data_public.jsonl + images/) into a local directory, e.g. ./advspot-public
python advspot_infer.py \
--model YOUR_MODEL_PATH \
--data-file ./advspot-public/data_public.jsonl \
--output-dir ./outputs \
--num-gpus 1
```
Outputs include predictions (`
.json` / `.csv`), region-grounded metrics (`#metrics.json`), and badcases (`#badcases.csv`). Use `--num-gpus > 1` for data-parallel inference, `--model-arch Qwen3-VL-MoE` for MoE checkpoints, and `--resume` to continue from a previous run. Run `python advspot_infer.py --help` for the full list of options.
## 🖼️ Examples
We provide four adversarial OCR examples in the `examples/` folder. Each example corresponds to a different adversarial pattern, with the model's perception analysis and final answer visualized in the corresponding case-study figure.
| Example | Input | Model Output (case study) |
|:---:|:---:|:---:|
| 1 | `examples/example_1.png` |
|
| 2 | `examples/example_2.png` |
|
| 3 | `examples/example_3.png` |
|
| 4 | `examples/example_4.jpeg` |
|
