# ArmorOCR: Grounded Adversarial Visual Perception via Observation-Transferred Self-Distillation

## 🧠 Abstract Large multimodal models (LMMs) have demonstrated strong OCR recognition capabilities, yet remain vulnerable to adversarial visual text that is readable to humans but challenging for models to localize and recognize. Existing OCR benchmarks mainly focus on natural or document-style text, while adversarial OCR evaluations remain limited in scale, task coverage, or region-aware evaluation. In this paper, we formulate adversarial OCR as a **grounded OCR perception** task and introduce **AdvSpot**, the first benchmark for grounded adversarial OCR evaluation. AdvSpot comprises 390 images with region-level annotations, spanning 5 primary categories and 13 fine-grained adversarial OCR types. To address this challenge, we propose **ArmorOCR**, a two-stage training framework for robust adversarial OCR perception. ArmorOCR first acquires missing adversarial OCR perception from privileged transformed observations through On-Policy Self-Distillation (OPSD), and then refines grounded OCR perception through Group Relative Policy Optimization (GRPO) with task-conditioned rewards for localization, recognition, full spotting, and visual question answering (VQA). Extensive experiments on AdvSpot, other adversarial OCR benchmarks, and general OCR benchmarks demonstrate that ArmorOCR consistently improves adversarial OCR perception while preserving competitive general OCR capability. ## 🚀 Release - [2026/09/01] 🔥 Released the **GGUF quantized weights** ([inclusionAI/ArmorOCR-GGUF](https://huggingface.co/inclusionAI/ArmorOCR-GGUF)). - [2026/08/24] 🔥 Released the AdvSpot evaluation/inference script (`advspot_infer.py`). - [2026/08/10] 🔥 Released the inference code and examples. - [Coming Soon] 🔥 AdvSpot benchmark data. ## 📦 AdvSpot Benchmark (Coming Soon) AdvSpot is the first grounded adversarial OCR perception benchmark, providing: - **390 images** with region-level annotations (bounding boxes, transcriptions, perception-type labels, region-grounded VQA pairs). - **5 primary categories** and **13 fine-grained adversarial OCR types** organized by underlying perception failure mechanisms: - Spatial Manipulation (Rotated / Mirrored / Tiny Text) - Glyph Variation (Stylized / Handwritten Text) - Visual Encoding (Symbol / Dot / Line Encoding) - Contextual Blending (AIGC Fusion / Low Contrast / Pattern Overlay) - Imaging Degradation (Capture / Post-processing Artifacts) - **Region-grounded evaluation** with VQA accuracy and IoU metrics. The benchmark data will be released in the near future. The evaluation/inference script (`advspot_infer.py`) is already available — see [Evaluate on the AdvSpot benchmark](#3-evaluate-on-the-advspot-benchmark). ## ⚙️ Installation ```bash conda create -n armorocr python=3.11 conda activate armorocr pip install pillow==12.0.0 pip install torch==2.8.0 torchvision==0.23.0 pip install transformers==4.57.1 accelerate==1.12.0 ``` > ⚠️ **Note:** ArmorOCR is built on the [Qwen3-VL](https://github.com/qwenlm/qwen3-vl) series. Please make sure your `transformers` version satisfies the minimum requirement of Qwen3-VL (see [Qwen3-VL README](https://github.com/qwenlm/qwen3-vl#requirements) for details). ## 🔍 Inference ### 1. Download model weights Download the pre-trained weights from 👉 [inclusionAI/ArmorOCR](https://huggingface.co/inclusionAI/ArmorOCR) on Hugging Face. > We also provide quantized GGUF checkpoints at 👉 [inclusionAI/ArmorOCR-GGUF](https://huggingface.co/inclusionAI/ArmorOCR-GGUF), along with a minimal `serve_gguf.sh` + inference example in that repo. > The GGUF repo also includes an evaluation comparison table on AdvSpot (base vs. Q8_0 vs. Q4_K_M). ### 2. Run inference on the provided examples ```bash python infer.py ``` Before running, please modify `YOUR_MODEL_PATH` in `infer.py` to point to your local checkpoint directory. ### 3. Evaluate on the AdvSpot benchmark We provide `advspot_infer.py` for running ArmorOCR (and other Qwen3-VL-series models) on the AdvSpot benchmark and computing region-grounded evaluation metrics (VQA accuracy + IoU). ```bash python advspot_infer.py \ --model YOUR_MODEL_PATH \ --data-file PATH_TO_ADVSPOT.json \ --output-dir ./outputs \ --num-gpus 1 ``` Outputs include predictions (`.json` / `.csv`), region-grounded metrics (`#metrics.json`), and badcases (`#badcases.csv`). Use `--num-gpus > 1` for data-parallel inference, `--model-arch Qwen3-VL-MoE` for MoE checkpoints, and `--resume` to continue from a previous run. Run `python advspot_infer.py --help` for the full list of options. ## 🖼️ Examples We provide four adversarial OCR examples in the `examples/` folder. Each example corresponds to a different adversarial pattern, with the model's perception analysis and final answer visualized in the corresponding case-study figure.

| Example | Input | Model Output (case study) | |:---:|:---:|:---:| | 1 | `examples/example_1.png` | | | 2 | `examples/example_2.png` | | | 3 | `examples/example_3.png` | | | 4 | `examples/example_4.jpeg` | |

## 🤝 Acknowledgement This work would not have been possible without the following excellent projects: 1. [Qwen3-VL](https://github.com/qwenlm/qwen3-vl) — the base vision-language model family ArmorOCR is built upon. 2. [ms-swift](https://github.com/modelscope/ms-swift) — the training framework used for ArmorOCR's two-stage post-training. 3. [llama.cpp](https://github.com/ggml-org/llama.cpp) — used for the GGUF quantized release and `llama-server` inference. ## 🔗 Other Work Welcome to follow our other work: [clh124/Awesome-Hard-OCR-LMM](https://github.com/clh124/Awesome-Hard-OCR-LMM).