README.md

---
license: cc-by-nc-sa-4.0
language:
- en
base_model:
- Qwen/Qwen3-Embedding-0.6B
datasets:
- chest2vec/chest2vec_labels
pipeline_tag: text-classification
library_name: transformers
tags:
- radiology
- chest-ct
- report-labeling
- multi-label
- ct-rate
- chexbert-style-f1
---

# chest2vec CT Report Labeler (0.6B)

A weakly-supervised **multi-label classifier** that reads a free-text **chest-CT report** and
predicts a **137-leaf chest-imaging taxonomy**, with a **ternary** status per label
(*negative / uncertain / positive*).

It also provides a **CheXbert / SRR-BERT-style report-comparison F1**: label a list of
ground-truth reports and a list of generated/predicted reports, then score them against each
other (micro / macro / weighted F1) — useful for evaluating radiology report generation.

- **Base architecture:** [`Qwen/Qwen3-Embedding-0.6B`](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) (Apache-2.0)
- **Adaptation:** LoRA (r=16, α=32) **merged into the weights** + last-token (EOS) pooling + L2-norm + a linear ternary head (`1024 → 137 × 3`)
- **Self-contained:** the full model (encoder + head) ships in `model.safetensors`. Loading does **not** download Qwen3-Embedding weights — the architecture is rebuilt from the bundled config and our weights are loaded in. Tokenizer is bundled too.
- **Params:** ~596M · weights in float32
- **Training labels:** [`chest2vec/chest2vec_labels`](https://huggingface.co/datasets/chest2vec/chest2vec_labels) (revised CT-RATE, 137-leaf taxonomy)

## Label space

The model head predicts **137 leaf labels**. They roll up through the chest-imaging hierarchy
into **38 upper/container groups** and **10 anatomy sections** (the `label_hierarchy` in
`config.json`), so predictions and report-comparison F1 can be reported at leaf, upper, or
anatomy granularity.

- The model outputs all **137** leaves. In the training data, **136** of them have at least one
  positive example; the single exception is **`IVC filter`** (kept for taxonomy completeness,
  but it had no positives, so the model effectively never predicts it).
- The exact label list is in `config.json` (`labels`). Full definitions and per-split counts are
  in the **[chest2vec/chest2vec_labels](https://huggingface.co/datasets/chest2vec/chest2vec_labels)**
  dataset's [`LABEL_HIERARCHY.md`](https://huggingface.co/datasets/chest2vec/chest2vec_labels/blob/main/LABEL_HIERARCHY.md).

This model was **trained and evaluated on the
[chest2vec/chest2vec_labels](https://huggingface.co/datasets/chest2vec/chest2vec_labels)
dataset** (revised CT-RATE, 137-leaf taxonomy).

**Ternary head** — `softmax(logits, dim=-1)` over class indices `[0, 1, 2]`:

| class index | meaning | value |
|---:|---|---:|
| 0 | negative | 0 |
| 1 | uncertain | -1 |
| 2 | positive | 1 |

A label is reported **positive** when `P(class=2) ≥ threshold` (default **0.5**).

## Usage

```python
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("chest2vec/chest2vec_labeler", trust_remote_code=True).eval()
tok   = AutoTokenizer.from_pretrained("chest2vec/chest2vec_labeler", trust_remote_code=True)

reports = ["Bibasilar atelectasis with small bilateral pleural effusions. Cardiomegaly. Coronary artery calcification."]

# 1) human-readable positive labels per report
print(model.label_reports(reports, tokenizer=tok))
# [{'Subsegmental / linear atelectasis': 'positive', 'Pleural effusion': 'positive',
#   'Cardiomegaly': 'positive', 'Coronary artery calcification': 'positive'}]

# 2) full prediction matrices
out = model.predict(reports, tokenizer=tok, threshold=0.5, return_ternary=True)
out["labels"]      # list of 137 label names
out["proba"]       # [N, 137] P(positive)
out["positive"]    # [N, 137] in {0,1}
out["ternary"]     # [N, 137] in {-1,0,1}
```

### CheXbert / SRR-BERT-style report comparison

Label both ground-truth and predicted reports, then compute label-level F1 (GT-labels treated
as truth):

```python
res = model.score_reports(gt_reports, pred_reports, tokenizer=tok)   # equal-length lists
# scores are reported at three hierarchy levels:
for level in ("leaf", "upper", "anatomy"):
    b = res[level]
    print(level, b["n_labels"], b["micro"]["f1"], b["macro"]["f1"], b["weighted"]["f1"])
print(res["leaf"]["per_label"]["Pleural effusion"])   # {'precision':..,'recall':..,'f1':..,'support_gt':..}

# or one-liner that loads the model for you:
from modeling_chest2vec_labeler import report_f1
report_f1(gt_reports, pred_reports, tokenizer=tok)
```

Each level (`leaf` = 137 labels, `upper` = 38 container groups, `anatomy` = 10 sections) returns
`micro` / `macro` / `weighted` precision/recall/F1 plus `per_label`. Upper/anatomy scores are the
max-over-children roll-up of the leaf predictions (`model.aggregate_hierarchy(...)`). Coarser
levels are easier to match, so upper/anatomy F1 are typically higher than leaf.

### Per-label best F1 (threshold tuning)

The default decision threshold is a single global value, but the F1-optimal threshold differs per
label. To get the **best achievable F1 per label** (and the threshold that achieves it) against a
ground-truth label set:

```python
# gt: a DataFrame with the 137 label columns (ternary; positive == 1), or a binary array
res = model.per_label_best_f1(reports, gt, tokenizer=tok, level="leaf", min_pos=30)
res["macro_best_f1_min_pos"]                 # macro best-F1 over labels with >= min_pos positives
res["per_label"]["Pleural effusion"]         # {'best_f1':.., 'best_threshold':.., 'n_pos':..}
```

Per-label threshold tuning lifts macro-F1 by ~4–6 points over the fixed-0.5 threshold (see below).

## Inputs & conventions

- Input is the **findings** text (the model was trained on CT-RATE findings + their refined
  section-structured form). Reports are formatted internally as
  `Instruct: Given the following chest CT report, extract the presence/absence of entities\nQuery: <report>`,
  truncated to **512** tokens, with an EOS token appended and left-padding.
- For best fidelity, run in float32 (default). bf16 is fine for throughput with negligible drift.

## Evaluation

Reported as macro-averages over per-label leaf metrics, with 95% bootstrap percentile confidence
intervals. Compare only within a row.

- **AUROC** — macro one-versus-rest AUROC on the positive-class probability. **Threshold-free.**
- **Three-class F1** — macro F1 from the ternary head (`negative` / `uncertain` / `positive`).
  Uses `argmax`; no threshold involved.
- **Binary F1** — macro F1 with `uncertain` mapped to `negative`. Per-label decision thresholds
  are selected on the CT-RATE tuning partition (not on the reported eval set).

Two label subsets are reported per eval set:
- **≥30 positives** — leaves with at least 30 positive test instances (headline).
- **≥1 positive** — all evaluated leaves (of the 137, 129 had ≥1 positive in CT-RATE and 120 in
  SNUH).

| Label set | AUROC | Three-class F1 | Binary F1 |
|---|--:|--:|--:|
| **CT-RATE (in distribution)** | | | |
| 53 (≥30 pos) | **0.989** (0.988, 0.991) | 0.686 (0.667, 0.692) | **0.888** (0.879, 0.896) |
| 129 (≥1 pos) | 0.988 (0.985, 0.991) | 0.610 (0.592, 0.612) | 0.769 (0.731, 0.777) |
| **SNUH (external)** | | | |
| 29 (≥30 pos) | **0.971** (0.967, 0.975) | 0.590 (0.579, 0.598) | **0.766** (0.749, 0.780) |
| 120 (≥1 pos) | 0.972 (0.968, 0.975) | 0.541 (0.523, 0.543) | 0.652 (0.601, 0.660) |

Leaf macro-AUROC barely moves in distribution → external (**0.989 → 0.971**): label ranking
transfers to unseen data; the residual F1 gap reflects labeling-convention drift, not a domain
failure. A radiologist spot-checked 966 reports of the public test labels (857 fully accepted /
60 imperfect-but-acceptable / 49 failed; see the [dataset card](https://huggingface.co/datasets/chest2vec/chest2vec_labels)).

## Caveats

- **Weakly supervised** — trained on LLM-generated labels (not radiologist ground truth) derived
  from report **text**, not images. Not a medical device; not for clinical use.
- `IVC filter` is in the taxonomy for completeness but had no training positives.
- `score_reports` measures **label agreement** between two reports as judged by this labeler;
  like CheXbert-F1 it inherits the labeler's own error modes.

## License & attribution

Released under **CC-BY-NC-SA-4.0**. Built on **`Qwen/Qwen3-Embedding-0.6B`** (Apache-2.0) and
trained using labels derived from **[CT-RATE](https://huggingface.co/datasets/ibrahimhamamci/CT-RATE)**
(CC-BY-NC-SA-4.0). **If you use this model, cite the CT-RATE paper** (arXiv:2403.17834) and
acknowledge Qwen3-Embedding. See the [dataset card](https://huggingface.co/datasets/chest2vec/chest2vec_labels)
for the full citation.