# Benchmark results
## Introductoin
- Benchmark results using the weights from [Detectron model zoo]() are basically the same. Hence, I won't list them here.
- All the testing results below are on COCO2017 val-set, using the models trained from scratch on COCO2017 train-set.
- Evaluation scores are rounded to 3 decimal places and presented in the format of percentage.
- Multiple evaluation on using pytorch may be listed. Each represents different training trial with same settings.
### COCO evaluation metrics
Average Precision |
AP50:95 | AP at IoU=0.50:0.05:0.95 |
AP50 | AP at IoU=0.50 |
AP75 | AP at IoU=0.75 |
APs | AP for small objects: area < 322 |
APm | AP for medium objects: 322 < area < 962 |
APl | AP for large objects: area > 962 |
Average Recall |
AR1 | AR given 1 detection per image |
AR10 | AR given 10 detection per image |
AR100 | AR given 100 detection per image |
ARs | AR for small objects: area < 322 |
ARm | AR for medium objects: 322 < area < 962 |
ARl | AR for large objects: area > 962 |
## Faster-RCNN
### e2e_faster_rcnn-R-50-FPN_1x
- Training command:
```
python tools/train_net_step.py \
--dataset coco2017 --cfg configs/e2e_faster_rcnn_R-50-FPN_1x.yaml \
--bs 8 --iter_size 2 --use_tfboard
```
on two 1080ti GPUs.
Box |
source |
AP50:95 |
AP50 |
AP75 |
APs |
APm |
APl |
AR1 |
AR10 |
AR100 |
ARs |
ARm |
ARl |
PyTorch |
37.1 |
59.1 |
40.0 |
21.5 |
39.8 |
48.3 |
30.8 |
48.0 |
50.3 |
31.4 |
53.9 |
63.6 |
Detectron |
36.7 |
58.4 |
39.6 |
21.1 |
39.8 |
48.1 |
30.6 |
48.0 |
50.4 |
31.8 |
54.1 |
63.4 |
- Total loss comparison:
![img](demo/loss_cmp_of_e2e_faster_rcnn_R-50-FPN_1x.jpg)
## Mask-RCNN
- Training command:
```
python train_net_step.py \
--dataset coco2017 --cfg configs/e2e_mask_rcnn_R-50-FPN_1x.yml --use_tfboard
```
on four M40 GPUs.
Box |
source |
AP50:95 |
AP50 |
AP75 |
APs |
APm |
APl |
AR1 |
AR10 |
AR100 |
ARs |
ARm |
ARl |
PyTorch |
37.7 |
59.1 |
41.0 |
21.5 |
40.7 |
49.6 |
31.3 |
48.7 |
51.1 |
32.7 |
54.4 |
64.9 |
PyTorch |
37.6 |
59.1 |
40.9 |
21.6 |
40.7 |
49.0 |
31.2 |
49.0 |
51.4 |
32.3 |
55.0 |
64.7 |
Detectron |
37.7 |
59.2 |
40.9 |
21.4 |
40.8 |
49.7 |
31.3 |
48.9 |
51.2 |
32.3 |
54.8 |
64.8 |
Mask |
source |
AP50:95 |
AP50 |
AP75 |
APs |
APm |
APl |
AR1 |
AR10 |
AR100 |
ARs |
ARm |
ARl |
PyTorch |
33.7 |
55.5 |
35.8 |
14.9 |
36.3 |
50.4 |
29.1 |
44.2 |
46.1 |
26.7 |
49.7 |
62.2 |
PyTorch |
33.8 |
55.7 |
35.5 |
15.3 |
36.3 |
50.5 |
29.2 |
44.5 |
46.4 |
26.2 |
50.0 |
62.5 |
Detectron |
33.9 |
55.8 |
35.8 |
14.9 |
36.3 |
50.9 |
29.2 |
44.4 |
46.2 |
26.2 |
50.1 |
62.0 |
- Total loss comparison:
![img](demo/loss_cmp_of_e2e_mask_rcnn_R-50-FPN_1x.jpg)
## Keypoint-RCNN
- Training command:
```
python tools/train_net_step.py \
--dataset keypoints_coco2017 --cfg configs/e2e_keypoint_rcnn_R-50-FPN_1x.yaml \
--bs 8 --iter_size 2 --use_tfboard
```
on four 1080 GPUs
Box |
source |
AP50:95 |
AP50 |
AP75 |
APs |
APm |
APl |
AR1 |
AR10 |
AR100 |
ARs |
ARm |
ARl |
PyTorch |
52.2 |
81.9 |
56.4 |
35.3 |
59.6 |
68.4 |
18.3 |
53.1 |
61.2 |
47.0 |
66.9 |
75.9 |
PyTorch |
53.5 |
82.8 |
58.4 |
36.7 |
61.2 |
69.5 |
18.6 |
54.2 |
62.2 |
47.8 |
68.3 |
76.7 |
Detectron |
53.6 |
82.8 |
58.3 |
36.5 |
61.2 |
69.7 |
18.7 |
54.3 |
62.2 |
47.6 |
68.3 |
76.8 |
Keypoint |
source |
AP50:95 |
AP50 |
AP75 |
APm |
APl |
AR1 |
AR10 |
AR100 |
ARm |
ARl |
PyTorch |
62.8 |
85.5 |
68.3 |
57.0 |
71.9 |
69.7 |
90.1 |
74.9 |
63.7 |
78.1 |
PyTorch |
63.9 |
86.0 |
69.2 |
58.5 |
72.7 |
70.6 |
90.7 |
75.8 |
65.0 |
78.6 |
Detectron |
64.2 |
86.4 |
69.9 |
58.5 |
73.4 |
70.7 |
90.9 |
75.9 |
64.9 |
79.0 |
- Total loss comparison:
![img](demo/loss_cmp_of_e2e_keypoint_rcnn_R-50-FPN_1x.jpg)