Eroux commited on
Commit
ce0e254
·
verified ·
1 Parent(s): da894bb

Update eval numbers to post-crop Hidden Trespass at validation-selected operating point

Browse files
Files changed (1) hide show
  1. README.md +25 -20
README.md CHANGED
@@ -53,13 +53,14 @@ operating point).
53
  from paddlex import create_model
54
 
55
  model = create_model(model_name="PP-DocLayout-L", model_dir="inference")
56
- for res in model.predict("page.jpg", threshold=0.68): # recommended global conf
57
  res.print() # boxes with label in {header, text-area, footnote, footer}
58
  res.save_to_img("out/")
59
  ```
60
 
61
  A ready-made `infer.py` (batch, YOLO-format output) is included in this repo.
62
- Recommended global operating confidence: **0.68** (the best-mean-F1 point).
 
63
 
64
  ## Evaluation (TiBLAD v4, 833-page test)
65
 
@@ -67,36 +68,40 @@ Recommended global operating confidence: **0.68** (the best-mean-F1 point).
67
  |---|---|---|---|
68
  | license | AGPL-3.0 | **Apache-2.0** | Apache-2.0 |
69
  | base model | RT-DETR-l (Ultralytics) | **PP-DocLayout-L (PaddleOCR, RT-DETR-L)** | RF-DETR-L (Roboflow) |
70
- | mean F1 (canonical 3-class) | 0.959 | **0.958** | 0.927 |
71
- |   header-footer F1 | 0.952 | **0.951** | 0.949 |
72
- |   text-area F1 | 0.999 | **0.997** | 0.996 |
73
- |   footnote F1 | 0.925 | **0.925** | 0.835 |
74
  | mean AP@0.50 | 0.974 | **0.959** | 0.925 |
75
  | mean AP@[0.50:0.95] | 0.786 | **0.781** | 0.667 |
76
  | shared-class mAP@[.50:.95] (DocLayNet-aligned) | 0.650 | **0.641** | 0.604 |
77
- | Hidden Trespass — header/footer | 0.008 | **0.003** | 0.020 |
78
- | Hidden Trespass — footnote | 0.037 | **0.037** | 0.216 |
79
  | COTe (Trespass) | 0.975 (0.001) | **0.978 (0.000)** | 0.974 (0.002) |
80
- | operating confidence | 0.74 | **0.68** | 0.26 |
81
 
82
- *"operating confidence" is the single global best-mean-F1 confidence used for the
83
- reported F1.*
 
84
 
85
- **Hidden Trespass** = the missed peripheral (header/footer/footnote) ground-truth
86
- **area** that falls inside the predicted `text-area` crop; area-based,
87
- micro-averaged over the test set. Lower is better (less clutter bled into the OCR
88
- region). Formal definition in the [paper](https://github.com/buda-base/papers).
 
 
89
 
90
  ## Which checkpoint to pick
91
 
92
  | checkpoint | license | mean F1 | shared mAP | footnote HT |
93
  |---|---|---|---|---|
94
- | TiBLA-RTDETR (primary) | AGPL-3.0 | 0.959 | 0.650 | 0.037 |
95
- | **TiBLA-PP-DocLayout-L** | Apache-2.0 | 0.958 | 0.641 | 0.037 |
96
- | TiBLA-RFDETR | Apache-2.0 | 0.927 | 0.604 | 0.216 |
97
 
98
- RT-DETR-l has the top scores but its weights are AGPL-3.0 (Ultralytics). If you
99
- need a permissive license, PP-DocLayout-L matches it at Apache-2.0; RF-DETR is a
 
100
  lighter PyTorch-native Apache-2.0 option.
101
 
102
  ## Citation
 
53
  from paddlex import create_model
54
 
55
  model = create_model(model_name="PP-DocLayout-L", model_dir="inference")
56
+ for res in model.predict("page.jpg", threshold=0.61): # recommended global conf
57
  res.print() # boxes with label in {header, text-area, footnote, footer}
58
  res.save_to_img("out/")
59
  ```
60
 
61
  A ready-made `infer.py` (batch, YOLO-format output) is included in this repo.
62
+ Recommended global operating confidence: **0.61** (the validation-selected
63
+ best-mean-F1 point).
64
 
65
  ## Evaluation (TiBLAD v4, 833-page test)
66
 
 
68
  |---|---|---|---|
69
  | license | AGPL-3.0 | **Apache-2.0** | Apache-2.0 |
70
  | base model | RT-DETR-l (Ultralytics) | **PP-DocLayout-L (PaddleOCR, RT-DETR-L)** | RF-DETR-L (Roboflow) |
71
+ | mean F1 (canonical 3-class) | 0.952 | **0.955** | 0.921 |
72
+ |   header-footer F1 | 0.954 | **0.953** | 0.947 |
73
+ |   text-area F1 | 0.999 | **0.998** | 0.996 |
74
+ |   footnote F1 | 0.902 | **0.914** | 0.821 |
75
  | mean AP@0.50 | 0.974 | **0.959** | 0.925 |
76
  | mean AP@[0.50:0.95] | 0.786 | **0.781** | 0.667 |
77
  | shared-class mAP@[.50:.95] (DocLayNet-aligned) | 0.650 | **0.641** | 0.604 |
78
+ | Hidden Trespass — header/footer | 0.009 | **0.004** | 0.021 |
79
+ | Hidden Trespass — footnote | 0.043 | **0.037** | 0.178 |
80
  | COTe (Trespass) | 0.975 (0.001) | **0.978 (0.000)** | 0.974 (0.002) |
81
+ | operating confidence | 0.64 | **0.61** | 0.47 |
82
 
83
+ *"operating confidence" is the single global best-mean-F1 confidence, selected on
84
+ the leak-free validation split and frozen for test (no test-set tuning). COCO AP
85
+ rows are threshold-free (all detections above the fixed 0.05 floor).*
86
 
87
+ **Hidden Trespass** = peripheral (header/footer/footnote) ground-truth **area**
88
+ that survives in the *actual OCR body crop* `C = E \ P`, where `E` is the predicted
89
+ `text-area` envelope and `P` is the union of the predicted peripheral boxes the
90
+ pipeline subtracts; area-based, micro-averaged over the test set. Lower is better
91
+ (less peripheral text bled into the OCR region). Formal definition in the
92
+ [paper](https://github.com/buda-base/papers).
93
 
94
  ## Which checkpoint to pick
95
 
96
  | checkpoint | license | mean F1 | shared mAP | footnote HT |
97
  |---|---|---|---|---|
98
+ | TiBLA-RTDETR (primary) | AGPL-3.0 | 0.952 | 0.650 | 0.043 |
99
+ | **TiBLA-PP-DocLayout-L** | Apache-2.0 | 0.955 | 0.641 | 0.037 |
100
+ | TiBLA-RFDETR | Apache-2.0 | 0.921 | 0.604 | 0.178 |
101
 
102
+ RT-DETR-l leads on mAP, shared-class mAP and the 5-seed mean F1 (0.961 ± 0.009),
103
+ but its weights are AGPL-3.0 (Ultralytics). If you need a permissive license,
104
+ PP-DocLayout-L is an Apache-2.0 match (statistically on par on F1); RF-DETR is a
105
  lighter PyTorch-native Apache-2.0 option.
106
 
107
  ## Citation