chest2vec
/

chest2vec_labeler

@@ -132,47 +132,33 @@ Per-label threshold tuning lifts macro-F1 by ~4–6 points over the fixed-0.5 th
 ## Evaluation
-**How these numbers were produced:** `run_para_v2_eval.sh` runs the model in **direct-paragraph**
-mode (full report, max_len 512) and writes `eval_ctrate_test_direct.json` (public) and
-`eval_sample1000_private.json` (private). Metric = **macro-F1** of the **positive class** (softmax
-probability of class 2) at **threshold 0.33**. Because the all-labels macro is dragged down by
-sparse-tail labels, the **headline restricts to leaf labels with ≥30 positive examples** in that
-eval set — **53 of the evaluated leaves on the public set, 29 on the private set**. Upper/anatomy
-rows are the hierarchy roll-up.
-**CT-RATE revised test (public, 1,464 reports)** — from `chest2vec/chest2vec_labels` test split:
-| Level | # labels | macro-F1 @0.33 | macro-AUC |
-|---|--:|--:|--:|
-| leaf (≥30 positives) | 53 | **0.875** | 0.989 |
-| leaf (all evaluated) | 131 | 0.749 | — |
-| upper (≥30 positives) | 27 | 0.938 | 0.994 |
-| anatomy | 10 | 0.956 | 0.993 |
-**Private evaluation set (1,000 reports)** — a held-out internal set, not released:
-| Level | # labels | macro-F1 @0.33 | macro-AUC |
 |---|--:|--:|--:|
-| leaf (≥30 positives) | 29 | **0.766** | 0.972 |
-| leaf (all evaluated) | 60 | 0.731 | — |
-| upper (≥30 positives) | 19 | 0.837 | — |
-| anatomy | 10 | 0.869 | — |
-**Per-label best F1** (threshold swept per label to maximize F1; macro over leaf labels with ≥30
-positives, via `model.per_label_best_f1`):
-| Eval set | macro best-F1 (≥30) | macro-F1 @0.5 (≥30) | macro best-F1 (all evaluated) |
-|---|--:|--:|--:|
-| CT-RATE public | **0.907** | 0.866 | 0.844 |
-| Private | **0.820** | 0.761 | 0.795 |
-F1-optimal thresholds vary widely by label (~0.04–0.75), so per-label tuning recovers ~4–6 macro-F1
-points over a single global threshold.
-Leaf macro-AUC barely moves public→private (**0.989 → 0.972**), i.e. label ranking transfers to
-the unseen set; the F1 gap is mostly threshold / labeling-convention, not a domain failure.
-Separately, a radiologist spot-checked **966** reports of the public test labels (857 fully
-accepted / 60 imperfect-but-acceptable / 49 failed; see the [dataset card](https://huggingface.co/datasets/chest2vec/chest2vec_labels)).
 ## Caveats

 ## Evaluation
+Reported as macro-averages over per-label leaf metrics, with 95% bootstrap percentile confidence
+intervals. Compare only within a row.
+- **AUROC** — macro one-versus-rest AUROC on the positive-class probability. **Threshold-free.**
+- **Three-class F1** — macro F1 from the ternary head (`negative` / `uncertain` / `positive`).
+  Uses `argmax`; no threshold involved.
+- **Binary F1** — macro F1 with `uncertain` mapped to `negative`. Per-label decision thresholds
+  are selected on the CT-RATE tuning partition (not on the reported eval set).
+Two label subsets are reported per eval set:
+- **≥30 positives** — leaves with at least 30 positive test instances (headline).
+- **≥1 positive** — all evaluated leaves (of the 137, 129 had ≥1 positive in CT-RATE and 120 in
+  SNUH).
+| Label set | AUROC | Three-class F1 | Binary F1 |
 |---|--:|--:|--:|
+| **CT-RATE (in distribution)** | | | |
+| 53 (≥30 pos) | **0.989** (0.988, 0.991) | 0.686 (0.667, 0.692) | **0.888** (0.879, 0.896) |
+| 129 (≥1 pos) | 0.988 (0.985, 0.991) | 0.610 (0.592, 0.612) | 0.769 (0.731, 0.777) |
+| **SNUH (external)** | | | |
+| 29 (≥30 pos) | **0.971** (0.967, 0.975) | 0.590 (0.579, 0.598) | **0.766** (0.749, 0.780) |
+| 120 (≥1 pos) | 0.972 (0.968, 0.975) | 0.541 (0.523, 0.543) | 0.652 (0.601, 0.660) |
+Leaf macro-AUROC barely moves in distribution → external (**0.989 → 0.971**): label ranking
+transfers to unseen data; the residual F1 gap reflects labeling-convention drift, not a domain
+failure. A radiologist spot-checked 966 reports of the public test labels (857 fully accepted /
+60 imperfect-but-acceptable / 49 failed; see the [dataset card](https://huggingface.co/datasets/chest2vec/chest2vec_labels)).
 ## Caveats