Update eval numbers to post-crop Hidden Trespass at validation-selected operating point
Browse files
README.md
CHANGED
|
@@ -53,13 +53,14 @@ operating point).
|
|
| 53 |
from paddlex import create_model
|
| 54 |
|
| 55 |
model = create_model(model_name="PP-DocLayout-L", model_dir="inference")
|
| 56 |
-
for res in model.predict("page.jpg", threshold=0.
|
| 57 |
res.print() # boxes with label in {header, text-area, footnote, footer}
|
| 58 |
res.save_to_img("out/")
|
| 59 |
```
|
| 60 |
|
| 61 |
A ready-made `infer.py` (batch, YOLO-format output) is included in this repo.
|
| 62 |
-
Recommended global operating confidence: **0.
|
|
|
|
| 63 |
|
| 64 |
## Evaluation (TiBLAD v4, 833-page test)
|
| 65 |
|
|
@@ -67,36 +68,40 @@ Recommended global operating confidence: **0.68** (the best-mean-F1 point).
|
|
| 67 |
|---|---|---|---|
|
| 68 |
| license | AGPL-3.0 | **Apache-2.0** | Apache-2.0 |
|
| 69 |
| base model | RT-DETR-l (Ultralytics) | **PP-DocLayout-L (PaddleOCR, RT-DETR-L)** | RF-DETR-L (Roboflow) |
|
| 70 |
-
| mean F1 (canonical 3-class) | 0.
|
| 71 |
-
| header-footer F1 | 0.
|
| 72 |
-
| text-area F1 | 0.999 | **0.
|
| 73 |
-
| footnote F1 | 0.
|
| 74 |
| mean AP@0.50 | 0.974 | **0.959** | 0.925 |
|
| 75 |
| mean AP@[0.50:0.95] | 0.786 | **0.781** | 0.667 |
|
| 76 |
| shared-class mAP@[.50:.95] (DocLayNet-aligned) | 0.650 | **0.641** | 0.604 |
|
| 77 |
-
| Hidden Trespass — header/footer | 0.
|
| 78 |
-
| Hidden Trespass — footnote | 0.
|
| 79 |
| COTe (Trespass) | 0.975 (0.001) | **0.978 (0.000)** | 0.974 (0.002) |
|
| 80 |
-
| operating confidence | 0.
|
| 81 |
|
| 82 |
-
*"operating confidence" is the single global best-mean-F1 confidence
|
| 83 |
-
|
|
|
|
| 84 |
|
| 85 |
-
**Hidden Trespass** =
|
| 86 |
-
**
|
| 87 |
-
|
| 88 |
-
|
|
|
|
|
|
|
| 89 |
|
| 90 |
## Which checkpoint to pick
|
| 91 |
|
| 92 |
| checkpoint | license | mean F1 | shared mAP | footnote HT |
|
| 93 |
|---|---|---|---|---|
|
| 94 |
-
| TiBLA-RTDETR (primary) | AGPL-3.0 | 0.
|
| 95 |
-
| **TiBLA-PP-DocLayout-L** | Apache-2.0 | 0.
|
| 96 |
-
| TiBLA-RFDETR | Apache-2.0 | 0.
|
| 97 |
|
| 98 |
-
RT-DETR-l
|
| 99 |
-
|
|
|
|
| 100 |
lighter PyTorch-native Apache-2.0 option.
|
| 101 |
|
| 102 |
## Citation
|
|
|
|
| 53 |
from paddlex import create_model
|
| 54 |
|
| 55 |
model = create_model(model_name="PP-DocLayout-L", model_dir="inference")
|
| 56 |
+
for res in model.predict("page.jpg", threshold=0.61): # recommended global conf
|
| 57 |
res.print() # boxes with label in {header, text-area, footnote, footer}
|
| 58 |
res.save_to_img("out/")
|
| 59 |
```
|
| 60 |
|
| 61 |
A ready-made `infer.py` (batch, YOLO-format output) is included in this repo.
|
| 62 |
+
Recommended global operating confidence: **0.61** (the validation-selected
|
| 63 |
+
best-mean-F1 point).
|
| 64 |
|
| 65 |
## Evaluation (TiBLAD v4, 833-page test)
|
| 66 |
|
|
|
|
| 68 |
|---|---|---|---|
|
| 69 |
| license | AGPL-3.0 | **Apache-2.0** | Apache-2.0 |
|
| 70 |
| base model | RT-DETR-l (Ultralytics) | **PP-DocLayout-L (PaddleOCR, RT-DETR-L)** | RF-DETR-L (Roboflow) |
|
| 71 |
+
| mean F1 (canonical 3-class) | 0.952 | **0.955** | 0.921 |
|
| 72 |
+
| header-footer F1 | 0.954 | **0.953** | 0.947 |
|
| 73 |
+
| text-area F1 | 0.999 | **0.998** | 0.996 |
|
| 74 |
+
| footnote F1 | 0.902 | **0.914** | 0.821 |
|
| 75 |
| mean AP@0.50 | 0.974 | **0.959** | 0.925 |
|
| 76 |
| mean AP@[0.50:0.95] | 0.786 | **0.781** | 0.667 |
|
| 77 |
| shared-class mAP@[.50:.95] (DocLayNet-aligned) | 0.650 | **0.641** | 0.604 |
|
| 78 |
+
| Hidden Trespass — header/footer | 0.009 | **0.004** | 0.021 |
|
| 79 |
+
| Hidden Trespass — footnote | 0.043 | **0.037** | 0.178 |
|
| 80 |
| COTe (Trespass) | 0.975 (0.001) | **0.978 (0.000)** | 0.974 (0.002) |
|
| 81 |
+
| operating confidence | 0.64 | **0.61** | 0.47 |
|
| 82 |
|
| 83 |
+
*"operating confidence" is the single global best-mean-F1 confidence, selected on
|
| 84 |
+
the leak-free validation split and frozen for test (no test-set tuning). COCO AP
|
| 85 |
+
rows are threshold-free (all detections above the fixed 0.05 floor).*
|
| 86 |
|
| 87 |
+
**Hidden Trespass** = peripheral (header/footer/footnote) ground-truth **area**
|
| 88 |
+
that survives in the *actual OCR body crop* `C = E \ P`, where `E` is the predicted
|
| 89 |
+
`text-area` envelope and `P` is the union of the predicted peripheral boxes the
|
| 90 |
+
pipeline subtracts; area-based, micro-averaged over the test set. Lower is better
|
| 91 |
+
(less peripheral text bled into the OCR region). Formal definition in the
|
| 92 |
+
[paper](https://github.com/buda-base/papers).
|
| 93 |
|
| 94 |
## Which checkpoint to pick
|
| 95 |
|
| 96 |
| checkpoint | license | mean F1 | shared mAP | footnote HT |
|
| 97 |
|---|---|---|---|---|
|
| 98 |
+
| TiBLA-RTDETR (primary) | AGPL-3.0 | 0.952 | 0.650 | 0.043 |
|
| 99 |
+
| **TiBLA-PP-DocLayout-L** | Apache-2.0 | 0.955 | 0.641 | 0.037 |
|
| 100 |
+
| TiBLA-RFDETR | Apache-2.0 | 0.921 | 0.604 | 0.178 |
|
| 101 |
|
| 102 |
+
RT-DETR-l leads on mAP, shared-class mAP and the 5-seed mean F1 (0.961 ± 0.009),
|
| 103 |
+
but its weights are AGPL-3.0 (Ultralytics). If you need a permissive license,
|
| 104 |
+
PP-DocLayout-L is an Apache-2.0 match (statistically on par on F1); RF-DETR is a
|
| 105 |
lighter PyTorch-native Apache-2.0 option.
|
| 106 |
|
| 107 |
## Citation
|