```
### End-to-End Evaluation
End-to-end evaluation assesses the model's accuracy in parsing PDF page content. The evaluation uses the model's Markdown output of the entire PDF page parsing results as the prediction. The Overall metric is calculated as:
$$\text{Overall} = \frac{(1-\textit{Text Edit Distance}) \times 100 + \textit{Table TEDS} +\textit{Formula CDM}}{3}$$
Comprehensive evaluation of document parsing on OmniDocBench (v1.5)
| Model Type |
Methods |
Size |
Overall↑ |
TextEdit↓ |
FormulaCDM↑ |
TableTEDS↑ |
TableTEDS-S↑ |
Read OrderEdit↓ |
Specialized
VLMs |
| PaddleOCR-VL-1.5 |
0.9B |
94.50 |
0.035 |
94.21 |
92.76 |
95.79 |
0.042 |
| GLM-OCR |
0.9B |
94.35 |
0.045 |
93.65 |
93.89 |
96.50 |
0.047 |
| Youtu-Parsing |
2.5B |
93.37 |
0.042 |
91.22 |
93.10 |
96.47 |
0.026 |
| PaddleOCR-VL |
0.9B |
92.86 |
0.035 |
91.22 |
90.89 |
94.76 |
0.043 |
| Logics-Parsing-v2 |
4B |
92.56 |
0.043 |
91.41 |
90.54 |
93.85 |
0.044 |
| FireRed-OCR |
2B |
92.07 |
0.035 |
90.98 |
88.72 |
92.38 |
0.041 |
| MinerU2.5 |
1.2B |
90.93 |
0.045 |
88.86 |
88.44 |
92.42 |
0.044 |
| HunyuanOCR |
1B |
90.57 |
0.085 |
86.01 |
94.19 |
95.96 |
0.082 |
| OpenDoc |
0.1B |
90.57 |
0.043 |
87.70 |
88.30 |
92.24 |
0.050 |
| FD-RL |
4B |
90.20 |
0.053 |
88.52 |
87.43 |
92.19 |
0.063 |
| DeepSeek-OCR-2 |
3B |
89.17 |
0.049 |
86.85 |
85.60 |
90.06 |
0.060 |
| MonkeyOCR-pro-3B |
3B |
88.85 |
0.075 |
87.25 |
86.78 |
90.63 |
0.128 |
| Dolphin-v2 |
3B |
88.71 |
0.073 |
87.26 |
86.20 |
89.77 |
0.064 |
| OCRVerse |
4B |
88.55 |
0.058 |
86.91 |
84.55 |
88.45 |
0.071 |
| dots.ocr |
3B |
88.41 |
0.048 |
83.22 |
86.78 |
90.62 |
0.053 |
| MonkeyOCR-3B |
3B |
87.13 |
0.075 |
87.45 |
81.39 |
85.92 |
0.129 |
| Deepseek-OCR |
3B |
87.01 |
0.073 |
83.37 |
84.97 |
88.80 |
0.086 |
| MonkeyOCR-pro-1.2B |
1.2B |
86.96 |
0.084 |
85.02 |
84.24 |
89.02 |
0.130 |
| Nanonets-OCR-s |
3B |
85.59 |
0.093 |
85.90 |
80.14 |
85.57 |
0.108 |
| MinerU2-VLM |
0.9B |
85.56 |
0.078 |
80.95 |
83.54 |
87.66 |
0.086 |
| Dolphin-1.5 |
0.3B |
85.07 |
0.095 |
81.10 |
83.63 |
87.27 |
0.082 |
| olmOCR |
7B |
81.79 |
0.096 |
86.04 |
68.92 |
74.77 |
0.121 |
| POINTS-Reader |
3B |
80.98 |
0.134 |
79.20 |
77.13 |
81.66 |
0.145 |
| Mistral OCR |
- |
78.83 |
0.164 |
82.84 |
70.03 |
78.04 |
0.144 |
| OCRFlux |
3B |
74.82 |
0.193 |
68.03 |
75.75 |
80.23 |
0.202 |
| Dolphin |
0.3B |
74.67 |
0.125 |
67.85 |
68.70 |
77.77 |
0.124 |
General
VLMs |
| Ovis2.6-30B-A3B |
30B |
92.36 |
0.037 |
90.32 |
90.46 |
94.00 |
0.046 |
| Gemini-3 Flash |
- |
90.37 |
0.065 |
89.56 |
88.01 |
93.79 |
0.071 |
| Gemini-3 Pro |
- |
90.17 |
0.062 |
88.79 |
87.83 |
93.32 |
0.074 |
| Kimi K2.5 |
1T |
89.33 |
0.065 |
86.92 |
87.57 |
91.82 |
0.084 |
| Qwen3-VL-235B |
235B |
89.15 |
0.069 |
88.14 |
86.21 |
90.55 |
0.068 |
| Gemini-2.5 Pro |
- |
88.03 |
0.075 |
85.82 |
85.71 |
90.29 |
0.097 |
| Qwen2.5-VL |
72B |
87.02 |
0.094 |
88.27 |
82.15 |
86.22 |
0.102 |
| GPT5.2 |
- |
85.75 |
0.124 |
86.93 |
82.76 |
88.25 |
0.106 |
| InternVL3.5 |
241B |
82.67 |
0.142 |
87.23 |
75.00 |
81.28 |
0.125 |
| InternVL3 |
78B |
80.33 |
0.131 |
83.42 |
70.64 |
77.74 |
0.113 |
| GPT-4o |
- |
75.02 |
0.217 |
79.70 |
67.07 |
76.09 |
0.148 |
Pipeline
Tools |
| PP-StructureV3 |
- |
86.73 |
0.073 |
85.79 |
81.68 |
89.48 |
0.073 |
| Mathpix |
- |
80.11 |
0.168 |
84.75 |
72.43 |
79.25 |
0.165 |
| Mineru2-pipeline |
- |
75.51 |
0.209 |
76.55 |
70.90 |
79.11 |
0.225 |
| Marker-1.8.2 |
- |
71.30 |
0.206 |
76.66 |
57.88 |
71.17 |
0.250 |
More detailed attribute-level evaluation results are shown in the paper. Or you can use the [tools/generate_result_tables.ipynb](./tools/generate_result_tables.ipynb) to generate the result leaderboard.
#### End-to-End Evaluation Method - end2end
End-to-end evaluation consists of two approaches:
- `end2end`: This method uses OmniDocBench's JSON files as Ground Truth. For config file reference, see: [end2end](./configs/end2end.yaml)
- `md2md`: This method uses OmniDocBench's markdown format as Ground Truth. Details will be discussed in the next section *markdown-to-markdown evaluation*.
We recommend using the `end2end` evaluation approach since it preserves the category and attribute information of samples, enabling special category ignore operations and attribute-level result output.
The `end2end` evaluation can assess four dimensions. We provide an example of end2end evaluation results in [result](./result), including:
- Text paragraphs
- Display formulas
- Tables
- Reading order
【Field explanations for end2end.yaml】
The configuration of `end2end.yaml` is as follows:
```YAML
end2end_eval: # Specify task name, common for end-to-end evaluation
metrics: # Configure metrics to use
text_block: # Configuration for text paragraphs
metric:
- Edit_dist # Normalized Edit Distance
- BLEU
- METEOR
display_formula: # Configuration for display formulas
metric:
- Edit_dist
- CDM # Only supports exporting format required for CDM evaluation, stored in results
table: # Configuration for tables
metric:
- TEDS
- Edit_dist
reading_order: # Configuration for reading order
metric:
- Edit_dist
dataset: # Dataset configuration
dataset_name: end2end_dataset # Dataset name, no need to modify
ground_truth:
data_path: ./demo_data/omnidocbench_demo/OmniDocBench_demo.json # Path to OmniDocBench
prediction:
data_path: ./demo_data/end2end # Folder path for model's PDF page parsing markdown results
match_method: quick_match # Matching method, options: no_split/no_split/quick_match
filter: # Page-level filtering
language: english # Page attributes and corresponding tags to evaluate
```
The `data_path` under `prediction` is the folder path containing the model's PDF page parsing results. The folder contains markdown files for each page, with filenames matching the image names but replacing the `.jpg` extension with `.md`.
[CDM](https://github.com/opendatalab/UniMERNet/tree/main/cdm) now supports direct evaluation, which requires you to set up the CDM environment according to the [README](./metrics/cdm/README.md) and then call `CDM` directly in the config file. In addition, we still support exporting the JSON format required for CDM evaluation as before: simply add the `CDM_plain` field in the metric configuration, and the output will be organized into the CDM input format and stored in the [result](./result) directory.
For end-to-end evaluation, the config allows selecting different matching methods. There are three matching approaches:
- `no_split`: Does not split or match text blocks, but rather combines them into a single markdown for calculation. This method will not output attribute-level results or reading order results.
- `simple_match`: Performs only paragraph segmentation using double line breaks, then directly matches one-to-one with GT without any truncation or merging.
- `quick_match`: Builds on paragraph segmentation by adding truncation and merging operations to reduce the impact of paragraph segmentation differences on final results, using *Adjacency Search Match* for truncation and merging. In version 1.5, the evaluation method has been fully upgraded to a *Hybrid Matching* approach, allowing formulas and text to be matched with each other, which reduces the score impact caused by models outputting formulas in unicode format.
We recommend using `quick_match` for better matching results. However, if the model's paragraph segmentation is accurate, `simple_match` can be used for faster evaluation. The matching method is configured through the `match_method` field under `dataset` in the config.
The `filter` field allows filtering the dataset. For example, setting `filter` to `language: english` under `dataset` will evaluate only pages in English. See the *Dataset Introduction* section for more page attributes. Comment out the `filter` fields to evaluate the full dataset.
#### End-to-end Evaluation Method - md2md
The markdown-to-markdown evaluation uses the model's markdown output of the entire PDF page parsing as the Prediction, and OmniDocBench's markdown format as the Ground Truth. Please refer to the config file: [md2md](./configs/md2md.yaml). We recommend using the `end2end` approach from the previous section to evaluate with OmniDocBench, as it preserves rich attribute annotations and ignore logic. However, we still provide the `md2md` evaluation method to align with existing evaluation approaches.
The `md2md` evaluation can assess four dimensions:
- Text paragraphs
- Display formulas
- Tables
- Reading order
【Field explanations for md2md.yaml】
The configuration of `md2md.yaml` is as follows:
```YAML
end2end_eval: # Specify task name, common for end-to-end evaluation
metrics: # Configure metrics to use
text_block: # Configuration for text paragraphs
metric:
- Edit_dist # Normalized Edit Distance
- BLEU
- METEOR
display_formula: # Configuration for display formulas
metric:
- Edit_dist
- CDM # Only supports exporting format required for CDM evaluation, stored in results
table: # Configuration for tables
metric:
- TEDS
- Edit_dist
reading_order: # Configuration for reading order
metric:
- Edit_dist
dataset: # Dataset configuration
dataset_name: md2md_dataset # Dataset name, no need to modify
ground_truth: # Configuration for ground truth dataset
data_path: ./demo_data/omnidocbench_demo/mds # Path to OmniDocBench markdown folder
page_info: ./demo_data/omnidocbench_demo/OmniDocBench_demo.json # Path to OmniDocBench JSON file, mainly used to get page-level attributes
prediction: # Configuration for model predictions
data_path: ./demo_data/end2end # Folder path for model's PDF page parsing markdown results
match_method: quick_match # Matching method, options: no_split/no_split/quick_match
filter: # Page-level filtering
language: english # Page attributes and corresponding tags to evaluate
```
The `data_path` under `prediction` is the folder path for the model's PDF page parsing results, which contains markdown files corresponding to each page. The filenames match the image names, with only the `.jpg` extension replaced with `.md`.
The `data_path` under `ground_truth` is the path to OmniDocBench's markdown folder, with filenames corresponding one-to-one with the model's PDF page parsing markdown results. The `page_info` path under `ground_truth` is the path to OmniDocBench's JSON file, mainly used to obtain page-level attributes. If page-level attribute evaluation results are not needed, this field can be commented out. However, without configuring the `page_info` field under `ground_truth`, the `filter` related functionality cannot be used.
For explanations of other fields in the config, please refer to the *End-to-end Evaluation - end2end* section.
### Formula Recognition Evaluation
OmniDocBench contains bounding box information for formulas on each PDF page along with corresponding formula recognition annotations, making it suitable as a benchmark for formula recognition evaluation. Formulas include display formulas (`equation_isolated`) and inline formulas (`equation_inline`). Currently, this repo provides examples for evaluating display formulas.
| Models |
CDM |
ExpRate@CDM |
BLEU |
Norm Edit |
| GOT-OCR |
74.1 |
28.0 |
55.07 |
0.290 |
| Mathpix |
86.6 |
2.8 |
66.56 |
0.322 |
| Pix2Tex |
73.9 |
39.5 |
46.00 |
0.337 |
| UniMERNet-B |
85.0 |
60.2 |
60.84 |
0.238 |
| GPT4o |
86.8 |
65.5 |
45.17 |
0.282 |
| InternVL2-Llama3-76B |
67.4 |
54.5 |
47.63 |
0.308 |
| Qwen2-VL-72B |
83.8 |
55.4 |
53.71 |
0.285 |
Component-level formula recognition evaluation on OmniDocBench (v1.0) formula subset.
Formula recognition evaluation can be configured according to [formula_recognition](./configs/formula_recognition.yaml).
【Field explanations for formula_recognition.yaml】
The configuration of `formula_recognition.yaml` is as follows:
```YAML
recogition_eval: # Specify task name, common for all recognition-related tasks
metrics: # Configure metrics to use
- Edit_dist # Normalized Edit Distance
- CDM # Only supports exporting formats required for CDM evaluation, stored in results
dataset: # Dataset configuration
dataset_name: omnidocbench_single_module_dataset # Dataset name, no need to modify if following specified input format
ground_truth: # Ground truth dataset configuration
data_path: ./demo_data/recognition/OmniDocBench_demo_formula.json # JSON file containing both ground truth and model prediction results
data_key: latex # Field name storing Ground Truth, for OmniDocBench, formula recognition results are stored in latex field
category_filter: ['equation_isolated'] # Categories used for evaluation, in formula recognition, the category_name is equation_isolated
prediction: # Model prediction configuration
data_key: pred # Field name storing model prediction results, this is user-defined
category_type: formula # category_type is mainly used for selecting data preprocessing strategy, options: formula/text
```
For the `metrics` section, in addition to the supported metrics, it also supports exporting formats required for [CDM](https://github.com/opendatalab/UniMERNet/tree/main/cdm) evaluation. Simply configure the CDM field in metrics to organize the output into CDM input format, which will be stored in [result](./result).
For the `dataset` section, the data format in the `ground_truth` `data_path` remains consistent with OmniDocBench, with just a custom field added under the corresponding formula sample to store the model's prediction results. The field storing prediction information is specified through the `data_key` under the `prediction` field in `dataset`, such as `pred`. For more details about OmniDocBench's file structure, please refer to the "Dataset Introduction" section. The input format for model results can be found in [OmniDocBench_demo_formula](./demo_data/recognition/OmniDocBench_demo_formula.json), which follows this format:
```JSON
[{
"layout_dets": [ // List of page elements
{
"category_type": "equation_isolated", // OmniDocBench category name
"poly": [ // OmniDocBench position info, coordinates for top-left, top-right, bottom-right, bottom-left corners (x,y)
136.0,
781.0,
340.0,
781.0,
340.0,
806.0,
136.0,
806.0
],
... // Other OmniDocBench fields
"latex": "$xxx$", // LaTeX formula will be written here
"pred": "$xxx$", // !! Model prediction result stored here, user-defined new field at same level as ground truth
...
],
"page_info": {...}, // OmniDocBench page information
"extra": {...} // OmniDocBench annotation relationship information
},
...
]
```
Here is a model inference script for reference:
```PYTHON
import os
import json
from PIL import Image
def poly2bbox(poly):
L = poly[0]
U = poly[1]
R = poly[2]
D = poly[5]
L, R = min(L, R), max(L, R)
U, D = min(U, D), max(U, D)
bbox = [L, U, R, D]
return bbox
question = "\nPlease convert this cropped image directly into latex."
with open('./demo_data/omnidocbench_demo/OmniDocBench_demo.json', 'r') as f:
samples = json.load(f)
for sample in samples:
img_name = os.path.basename(sample['page_info']['image_path'])
img_path = os.path.join('./Docparse/images', img_name)
img = Image.open(img_path)
if not os.path.exists(img_path):
print('No exist: ', img_name)
continue
for i, anno in enumerate(sample['layout_dets']):
if anno['category_type'] != 'equation_isolated': # Filter out equation_isolated category for evaluation
continue
bbox = poly2bbox(anno['poly'])
im = img.crop(bbox).convert('RGB')
response = model.chat(im, question) # Modify the way the image is passed in according to the model
anno['pred'] = response # Directly add a new field to store the model's inference results under the corresponding annotation
with open('./demo_data/recognition/OmniDocBench_demo_formula.json', 'w', encoding='utf-8') as f:
json.dump(samples, f, ensure_ascii=False)
```
### Text OCR Evaluation
OmniDocBench contains bounding box information and corresponding text recognition annotations for all text in each PDF page, making it suitable as a benchmark for OCR evaluation. The text annotations include both block-level and span-level annotations, both of which can be used for evaluation. This repo currently provides an example of block-level evaluation, which evaluates OCR at the text paragraph level.
| Model Type |
Model |
Language |
Text background |
Text Rotate |
| EN |
ZH |
Mixed |
White |
Single |
Multi |
Normal |
Rotate90 |
Rotate270 |
Horizontal |
Pipeline Tools
&
Expert Vision
Models |
PaddleOCR |
0.071 |
0.055 |
0.118 |
0.060 |
0.038 |
0.085 |
0.060 |
0.015 |
0.285 |
0.021 |
| OpenOCR |
0.07 |
0.068 |
0.106 |
0.069 |
0.058 |
0.081 |
0.069 |
0.038 |
0.891 |
0.025 |
| Tesseract-OCR |
0.096 |
0.551 |
0.250 |
0.439 |
0.328 |
0.331 |
0.426 |
0.117 |
0.969 |
0.984 |
| EasyOCR |
0.26 |
0.398 |
0.445 |
0.366 |
0.287 |
0.388 |
0.36 |
0.97 |
0.997 |
0.926 |
| Surya |
0.057 |
0.123 |
0.164 |
0.093 |
0.186 |
0.235 |
0.104 |
0.634 |
0.767 |
0.255 |
| Mathpix |
0.033 |
0.240 |
0.261 |
0.185 |
0.121 |
0.166 |
0.180 |
0.038 |
0.185 |
0.638 |
| GOT-OCR |
0.041 |
0.112 |
0.135 |
0.092 |
0.052 |
0.155 |
0.091 |
0.562 |
0.966 |
0.097 |
Vision Language
Models |
Qwen2-VL-72B |
0.072 |
0.274 |
0.286 |
0.234 |
0.155 |
0.148 |
0.223 |
0.273 |
0.721 |
0.067 |
| InternVL2-76B |
0.074 |
0.155 |
0.242 |
0.113 |
0.352 |
0.269 |
0.132 |
0.610 |
0.907 |
0.595 |
| GPT4o |
0.020 |
0.224 |
0.125 |
0.167 |
0.140 |
0.220 |
0.168 |
0.115 |
0.718 |
0.132 |
Component-level OCR text recognition evaluation on OmniDocBench (v1.0) text subset.
OCR text recognition evaluation can be configured according to [ocr](./configs/ocr.yaml).
【The field explanation of ocr.yaml】
The configuration file for `ocr.yaml` is as follows:
```YAML
recogition_eval: # Specify task name, common for all recognition-related tasks
metrics: # Configure metrics to use
- Edit_dist # Normalized Edit Distance
- BLEU
- METEOR
dataset: # Dataset configuration
dataset_name: omnidocbench_single_module_dataset # Dataset name, no need to modify if following the specified input format
ground_truth: # Ground truth dataset configuration
data_path: ./demo_data/recognition/OmniDocBench_demo_text_ocr.json # JSON file containing both ground truth and model prediction results
data_key: text # Field name storing Ground Truth, for OmniDocBench, text recognition results are stored in the text field, all block level annotations containing text field will participate in evaluation
prediction: # Model prediction configuration
data_key: pred # Field name storing model prediction results, this is user-defined
category_type: text # category_type is mainly used for selecting data preprocessing strategy, options: formula/text
```
For the `dataset` section, the input `ground_truth` `data_path` follows the same data format as OmniDocBench, with just a new custom field added under samples containing the text field to store the model's prediction results. The field storing prediction information is specified through the `data_key` under the `prediction` field in `dataset`, for example `pred`. The input format of the dataset can be referenced in [OmniDocBench_demo_text_ocr](./demo_data/recognition/OmniDocBench_demo_text_ocr.json), and the meanings of various fields can be found in the examples provided in the *Formula Recognition Evaluation* section.
Here is a reference model inference script for your consideration:
```PYTHON
import os
import json
from PIL import Image
def poly2bbox(poly):
L = poly[0]
U = poly[1]
R = poly[2]
D = poly[5]
L, R = min(L, R), max(L, R)
U, D = min(U, D), max(U, D)
bbox = [L, U, R, D]
return bbox
question = "\nPlease convert this cropped image directly into latex."
with open('./demo_data/omnidocbench_demo/OmniDocBench_demo.json', 'r') as f:
samples = json.load(f)
for sample in samples:
img_name = os.path.basename(sample['page_info']['image_path'])
img_path = os.path.join('./Docparse/images', img_name)
img = Image.open(img_path)
if not os.path.exists(img_path):
print('No exist: ', img_name)
continue
for i, anno in enumerate(sample['layout_dets']):
if not anno.get('text'): # Filter out annotations containing the text field from OmniDocBench for model inference
continue
bbox = poly2bbox(anno['poly'])
im = img.crop(bbox).convert('RGB')
response = model.chat(im, question) # Modify the way the image is passed in according to the model
anno['pred'] = response # Directly add a new field to store the model's inference results under the corresponding annotation
with open('./demo_data/recognition/OmniDocBench_demo_text_ocr.json', 'w', encoding='utf-8') as f:
json.dump(samples, f, ensure_ascii=False)
```
### Table Recognition Evaluation
OmniDocBench contains bounding box information for tables on each PDF page along with corresponding table recognition annotations, making it suitable as a benchmark for table recognition evaluation. The table annotations are available in both HTML and LaTeX formats, with this repository currently providing examples for HTML format evaluation.
| Model Type |
Model |
Language |
Table Frame Type |
Special Situation |
Overall |
| EN |
ZH |
Mixed |
Full |
Omission |
Three |
Zero |
Merge Cell(+/-) |
Formula(+/-) |
Colorful(+/-) |
Rotate(+/-) |
| OCR-based Models |
PaddleOCR |
76.8 |
71.8 |
80.1 |
67.9 |
74.3 |
81.1 |
74.5 |
70.6/75.2 |
71.3/74.1 |
72.7/74.0 |
23.3/74.6 |
73.6 |
| RapidTable |
80.0 |
83.2 |
91.2 |
83.0 |
79.7 |
83.4 |
78.4 |
77.1/85.4 |
76.7/83.9 |
77.6/84.9 |
25.2/83.7 |
82.5 |
| Expert VLMs |
StructEqTable |
72.8 |
75.9 |
83.4 |
72.9 |
76.2 |
76.9 |
88 |
64.5/81 |
69.2/76.6 |
72.8/76.4 |
30.5/76.2 |
75.8 |
| GOT-OCR |
72.2 |
75.5 |
85.4 |
73.1 |
72.7 |
78.2 |
75.7 |
65.0/80.2 |
64.3/77.3 |
70.8/76.9 |
8.5/76.3 |
74.9 |
| General VLMs |
Qwen2-VL-7B |
70.2 |
70.7 |
82.4 |
70.2 |
62.8 |
74.5 |
80.3 |
60.8/76.5 |
63.8/72.6 |
71.4/70.8 |
20.0/72.1 |
71.0 |
| InternVL2-8B |
70.9 |
71.5 |
77.4 |
69.5 |
69.2 |
74.8 |
75.8 |
58.7/78.4 |
62.4/73.6 |
68.2/73.1 |
20.4/72.6 |
71.5 |
Component-level Table Recognition evaluation on OmniDocBench(v1.0) table subset. (+/-) means with/without special situation.
Table recognition evaluation can be configured according to [table_recognition](./configs/table_recognition.yaml).
**For tables predicted to be in LaTeX format, the [latexml](https://math.nist.gov/~BMiller/LaTeXML/) tool will be used to convert LaTeX to HTML before evaluation. The evaluation code will automatically perform format conversion, and users need to preinstall [latexml](https://math.nist.gov/~BMiller/LaTeXML/)**
【The field explanation of table_recognition.yaml】
The configuration file for `table_recognition.yaml` is as follows:
```YAML
recogition_eval: # Specify task name, common for all recognition-related tasks
metrics: # Configure metrics to use
- TEDS # Tree Edit Distance based Similarity
- Edit_dist # Normalized Edit Distance
dataset: # Dataset configuration
dataset_name: omnidocbench_single_module_dataset # Dataset name, no need to modify if following specified input format
ground_truth: # Configuration for ground truth dataset
data_path: ./demo_data/recognition/OmniDocBench_demo_table.json # JSON file containing both ground truth and model prediction results
data_key: html # Field name storing Ground Truth, for OmniDocBench, table recognition results are stored in html and latex fields, change to latex when evaluating latex format tables
category_filter: table # Category for evaluation, in table recognition, the category_name is table
prediction: # Configuration for model prediction results
data_key: pred # Field name storing model prediction results, this is user-defined
category_type: table # category_type is mainly used for data preprocessing strategy selection
```
For the `dataset` section, the data format in the `ground_truth`'s `data_path` remains consistent with OmniDocBench, with only a custom field added under the corresponding table sample to store the model's prediction result. The field storing prediction information is specified through `data_key` under the `prediction` field in `dataset`, such as `pred`. For more details about OmniDocBench's file structure, please refer to the "Dataset Introduction" section. The input format for model results can be found in [OmniDocBench_demo_table](./demo_data/recognition/OmniDocBench_demo_table.json), which follows this format:
```JSON
[{
"layout_dets": [ // List of page elements
{
"category_type": "table", // OmniDocBench category name
"poly": [ // OmniDocBench position info: x,y coordinates for top-left, top-right, bottom-right, bottom-left corners
136.0,
781.0,
340.0,
781.0,
340.0,
806.0,
136.0,
806.0
],
... // Other OmniDocBench fields
"latex": "$xxx$", // Table LaTeX annotation goes here
"html": "$xxx$", // Table HTML annotation goes here
"pred": "$xxx$", // !! Model prediction result stored here, user-defined new field at same level as ground truth
...
],
"page_info": {...}, // OmniDocBench page information
"extra": {...} // OmniDocBench annotation relationship information
},
...
]
```
Here is a model inference script for reference:
```PYTHON
import os
import json
from PIL import Image
def poly2bbox(poly):
L = poly[0]
U = poly[1]
R = poly[2]
D = poly[5]
L, R = min(L, R), max(L, R)
U, D = min(U, D), max(U, D)
bbox = [L, U, R, D]
return bbox
question = "\nPlease convert this cropped image directly into html format of table."
with open('./demo_data/omnidocbench_demo/OmniDocBench_demo.json', 'r') as f:
samples = json.load(f)
for sample in samples:
img_name = os.path.basename(sample['page_info']['image_path'])
img_path = os.path.join('./demo_data/omnidocbench_demo/images', img_name)
img = Image.open(img_path)
if not os.path.exists(img_path):
print('No exist: ', img_name)
continue
for i, anno in enumerate(sample['layout_dets']):
if anno['category_type'] != 'table': # Filter out the table category for evaluation
continue
bbox = poly2bbox(anno['poly'])
im = img.crop(bbox).convert('RGB')
response = model.chat(im, question) # Need to modify the way the image is passed in depending on the model
anno['pred'] = response # Directly add a new field to store the model's inference result at the same level as the ground truth
with open('./demo_data/recognition/OmniDocBench_demo_table.json', 'w', encoding='utf-8') as f:
json.dump(samples, f, ensure_ascii=False)
```
### Layout Detection
OmniDocBench contains bounding box information for all document components on each PDF page, making it suitable as a benchmark for layout detection task evaluation.
| Model |
Backbone |
Params |
Book |
Slides |
Research
Report |
Textbook |
Exam
Paper |
Magazine |
Academic
Literature |
Notes |
Newspaper |
Average |
| DiT-L |
ViT-L |
361.6M |
43.44 |
13.72 |
45.85 |
15.45 |
3.40 |
29.23 |
66.13 |
0.21 |
23.65 |
26.90 |
| LayoutLMv3 |
RoBERTa-B |
138.4M |
42.12 |
13.63 |
43.22 |
21.00 |
5.48 |
31.81 |
64.66 |
0.80 |
30.84 |
28.84 |
| DocLayout-YOLO |
v10m |
19.6M |
43.71 |
48.71 |
72.83 |
42.67 |
35.40 |
51.44 |
64.64 |
9.54 |
57.54 |
47.38 |
| SwinDocSegmenter |
Swin-L |
223M |
42.91 |
28.20 |
47.29 |
32.44 |
20.81 |
52.35 |
48.54 |
12.38 |
38.06 |
35.89 |
| GraphKD |
R101 |
44.5M |
39.03 |
16.18 |
39.92 |
22.82 |
14.31 |
37.61 |
44.43 |
5.71 |
23.86 |
27.10 |
| DOCX-Chain |
- |
- |
30.86 |
11.71 |
39.62 |
19.23 |
10.67 |
23.00 |
41.60 |
1.80 |
16.96 |
21.27 |
Component-level layout detection evaluation on OmniDocBench (v1.0) layout subset: mAP results by PDF page type.
Layout detection config file reference [layout_detection](./configs/layout_detection.yaml), data format reference [detection_prediction](./demo_data/detection/detection_prediction.json).
【The field explanation of layout_detection.yaml】
Here is the configuration file for `layout_detection.yaml`:
```YAML
detection_eval: # Specify task name, common for all detection-related tasks
metrics:
- COCODet # Detection task related metrics, mainly mAP, mAR etc.
dataset:
dataset_name: detection_dataset_simple_format # Dataset name, no need to modify if following specified input format
ground_truth:
data_path: ./demo_data/omnidocbench_demo/OmniDocBench_demo.json # Path to OmniDocBench JSON file
prediction:
data_path: ./demo_data/detection/detection_prediction.json # Path to model prediction result JSON file
filter: # Page level filtering
data_source: exam_paper # Page attributes and corresponding tags to be evaluated
categories:
eval_cat: # Categories participating in final evaluation
block_level: # Block level categories, see OmniDocBench evaluation set introduction for details
- title # Title
- text # Text
- abandon # Includes headers, footers, page numbers, and page annotations
- figure # Image
- figure_caption # Image caption
- table # Table
- table_caption # Table caption
- table_footnote # Table footnote
- isolate_formula # Display formula (this is a layout display formula, lower priority than 14)
- formula_caption # Display formula label
gt_cat_mapping: # Mapping table from ground truth to final evaluation categories, key is ground truth category, value is final evaluation category name
figure_footnote: figure_footnote
figure_caption: figure_caption
page_number: abandon
header: abandon
page_footnote: abandon
table_footnote: table_footnote
code_txt: figure
equation_caption: formula_caption
equation_isolated: isolate_formula
table: table
refernece: text
table_caption: table_caption
figure: figure
title: title
text_block: text
footer: abandon
pred_cat_mapping: # Mapping table from prediction to final evaluation categories, key is prediction category, value is final evaluation category name
title : title
plain text: text
abandon: abandon
figure: figure
figure_caption: figure_caption
table: table
table_caption: table_caption
table_footnote: table_footnote
isolate_formula: isolate_formula
formula_caption: formula_caption
```
The `filter` field can be used to filter the dataset. For example, setting the `filter` field under `dataset` to `data_source: exam_paper` will filter for pages with data type exam_paper. For more page attributes, please refer to the "Evaluation Set Introduction" section. If you want to evaluate the full dataset, comment out the `filter` related fields.
The `data_path` under the `prediction` section in `dataset` takes the model's prediction as input, with the following data format:
```JSON
{
"results": [
{
"image_name": "docstructbench_llm-raw-scihub-o.O-adsc.201190003.pdf_6", // image name
"bbox": [53.892921447753906, 909.8675537109375, 808.5555419921875, 1006.2714233398438], // bounding box coordinates, representing x,y coordinates of top-left and bottom-right corners
"category_id": 1, // category ID number
"score": 0.9446213841438293 // confidence score
},
... // all bounding boxes are flattened in a single list
],
"categories": {"0": "title", "1": "plain text", "2": "abandon", ...} // mapping between category IDs and category names
```
### Formula Detection
OmniDocBench contains bounding box information for each formula on each PDF page, making it suitable as a benchmark for formula detection task evaluation.
The format for formula detection is essentially the same as layout detection. Formulas include both inline and display formulas. In this section, we provide a config example that can evaluate detection results for both display formulas and inline formulas simultaneously. Formula detection can be configured according to [formula_detection](./configs/formula_detection.yaml).
【The field explanation of formula_detection.yaml】
Here is the configuration file for `formula_detection.yaml`:
```YAML
detection_eval: # Specify task name, common for all detection-related tasks
metrics:
- COCODet # Detection task related metrics, mainly mAP, mAR etc.
dataset:
dataset_name: detection_dataset_simple_format # Dataset name, no need to modify if following specified input format
ground_truth:
data_path: ./demo_data/omnidocbench_demo/OmniDocBench_demo.json # Path to OmniDocBench JSON file
prediction:
data_path: ./demo_data/detection/detection_prediction.json # Path to model prediction JSON file
filter: # Page-level filtering
data_source: exam_paper # Page attributes and corresponding tags to evaluate
categories:
eval_cat: # Categories participating in final evaluation
block_level: # Block level categories, see OmniDocBench dataset intro for details
- isolate_formula # Display formula
span_level: # Span level categories, see OmniDocBench dataset intro for details
- inline_formula # Inline formula
gt_cat_mapping: # Mapping table from ground truth to final evaluation categories, key is ground truth category, value is final evaluation category name
equation_isolated: isolate_formula
equation_inline: inline_formula
pred_cat_mapping: # Mapping table from prediction to final evaluation categories, key is prediction category, value is final evaluation category name
interline_formula: isolate_formula
inline_formula: inline_formula
```
Please refer to the `Layout Detection` section for parameter explanations and dataset format. The main difference between formula detection and layout detection is that under the `eval_cat` category that participates in the final evaluation, a `span_level` category `inline_formula` has been added. Both span_level and block_level categories will participate together in the evaluation.
## Tools
We provide several tools in the `tools` directory:
- [json2md](./tools/json2md.py) for converting OmniDocBench from JSON format to Markdown format;
- [visualization](./tools/visualization.py) for visualizing OmniDocBench JSON files;
- [generate_result_tables](./tools/generate_result_tables.py) for generating the result leaderboard of the evaluation;
- The [model_infer](./tools/model_infer) folder provides some model inference scripts for reference. Please use after configuring the model environment. Including:
- `_img2md.py` for calling the models to convert images to Markdown format;
- `_ocr.py` is to invoke the models for text recognition of block-level document text paragraphs;
- `_formula.py` is used to call the models for formula recognition of display formulas (`equation_isolated`);
## The evaluation model information
### End2End
### Text Recognition
### Layout
### Formula
### Table
## TODO
- [ ] Integration of `match_full` algorithm
- [ ] Optimization of matching post-processing for model-specific output formats
- [ ] Addition of Unicode mapping table for special characters
## Known Issues
- Some models occasionally produce non-standard output formats (e.g., recognizing multi-column text as tables, or formulas as Unicode text), leading to matching failures. This can be optimized through post-processing of model output formats
- Due to varying symbol recognition capabilities across different models, some symbols are recognized inconsistently (e.g., list identifiers). Currently, only Chinese and English text are included in text evaluation. A Unicode mapping table will be added later for optimization
We welcome everyone to use the OmniDocBench dataset and provide valuable feedback and suggestions to help us continuously improve the dataset quality and evaluation tools. For any comments or suggestions, please feel free to open an issue and we will respond promptly. If you have evaluation scheme optimizations, you can submit a PR and we will review and update in a timely manner.
## Acknowledgement
- Thank [2077AI](https://2077ai.com) for supporting the dataset annotation.
- [PubTabNet](https://github.com/ibm-aur-nlp/PubTabNet) for TEDS metric calculation
- [latexml](https://github.com/brucemiller/LaTeXML) LaTeX to HTML conversion tool
- [Tester](https://github.com/intsig-textin/markdown_tester) Markdown table to HTML conversion tool
## Copyright Statement
The PDFs are collected from public online channels and community user contributions. Content that is not allowed for distribution has been removed. The dataset is for research purposes only and not for commercial use. If there are any copyright concerns, please contact OpenDataLab@pjlab.org.cn.
## Citation
```bibtex
@misc{ouyang2024omnidocbenchbenchmarkingdiversepdf,
title={OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations},
author={Linke Ouyang and Yuan Qu and Hongbin Zhou and Jiawei Zhu and Rui Zhang and Qunshu Lin and Bin Wang and Zhiyuan Zhao and Man Jiang and Xiaomeng Zhao and Jin Shi and Fan Wu and Pei Chu and Minghao Liu and Zhenxiang Li and Chao Xu and Bo Zhang and Botian Shi and Zhongying Tu and Conghui He},
year={2024},
eprint={2412.07626},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2412.07626},
}
```