README.md

---

license: apache-2.0
library_name: onnx
pipeline_tag: image-classification
tags:
  - medical-imaging
  - modality-classification
  - dicom
  - onnx
  - int8
---


# ModalityScan

Single-label imaging-modality classification for medical images. One shared ConvNet backbone, two
softmax heads (`family`, `modality`), shipped as a static-INT8 ONNX graph.

Predicts **what kind of image this is**, not what is in it. Anatomy is a different model.

## What it outputs

```

input   pixel_values     float32  [batch, 3, 224, 224]   NCHW, ImageNet-normalized, letterboxed

outputs family_logits    float32  [batch, 6]

        modality_logits  float32  [batch, 20]

```

Softmax, temperature and thresholds live **outside** the graph so calibration and policy stay
configuration. `modality-taxonomy.json` travels with the weights - a 20-element
logits vector is 20 meaningless numbers without it.

## Classes

| class | family | tier | train frames | sources | P | R | F1 | support |
|---|---|---|---|---|---|---|---|---|
| `CT` | cross_sectional | model_trained | 5,996 | 3 | 0.8469 | 0.8059 | 0.8259 | 2,684 |
| `MR` | cross_sectional | experimental | 404 | 1 | 0.2476 | 1.0 | 0.3969 | 52 |

| `US` | ultrasound | model_trained | 3,779 | 2 | 0.861 | 0.9558 | 0.9059 | 3,123 |
| `CR` | projection_radiography | experimental | 0 | 0 | 0.0 | 0.0 | 0.0 | 0 |

| `DX` | projection_radiography | model_trained | 6,000 | 2 | 0.9956 | 0.9814 | 0.9884 | 3,442 |

| `MG` | projection_radiography | experimental | 0 | 0 | 0.0 | 0.0 | 0.0 | 0 |
| `RF` | projection_radiography | not_trained | 0 | 0 | n/a | n/a | n/a | 0 |
| `XA` | projection_radiography | not_trained | 0 | 0 | n/a | n/a | n/a | 0 |
| `PT` | cross_sectional | experimental | 0 | 0 | 0.0 | 0.0 | 0.0 | 0 |

| `NM` | cross_sectional | experimental | 0 | 0 | 0.0 | 0.0 | 0.0 | 0 |
| `OCT` | optical | experimental | 6,000 | 1 | 0.9977 | 1.0 | 0.9989 | 881 |
| `FUNDUS` | optical | experimental | 1,600 | 1 | 1.0 | 1.0 | 1.0 | 246 |
| `DERMOSCOPY` | optical | experimental | 6,000 | 1 | 0.5773 | 0.9943 | 0.7305 | 879 |
| `ENDOSCOPY` | optical | experimental | 0 | 0 | 0.0 | 0.0 | 0.0 | 0 |
| `HISTOPATHOLOGY` | microscopy | experimental | 6,000 | 1 | 0.2853 | 1.0 | 0.4439 | 926 |
| `MICROSCOPY` | microscopy | not_trained | 6,000 | 2 | n/a | n/a | n/a | 3,479 |

| `PHOTO` | non_diagnostic | experimental | 0 | 0 | 0.0 | 0.0 | 0.0 | 0 |
| `RENDER_3D` | non_diagnostic | experimental | 0 | 0 | 0.0 | 0.0 | 0.0 | 0 |

| `DOCUMENT` | non_diagnostic | not_trained | 0 | 0 | n/a | n/a | n/a | 0 |

| `WAVEFORM` | non_diagnostic | not_trained | 0 | 0 | n/a | n/a | n/a | 0 |



**Tier is a statement about evidence, not difficulty.** `model_trained` means the corpus had at
least two independent sources and enough patients to make the number falsifiable. `experimental`
means it did not. `not_trained` classes are never emitted by the decoder, so no metrics are
reported for them.

Some `not_trained` classes *do* have training frames. They were withdrawn after
measurement rather than never attempted, and the decoder never emits them:

* **`MICROSCOPY` is not claimed.** Withdrawn from the claim after measurement. Held-out-source recall was 97/3,479 = 2.8%, with 60% of held-out frames predicted HISTOPATHOLOGY. Its two sources are not interchangeable samples of one modality: TissueMNIST (trained on) is grayscale kidney cortex, BloodMNIST (held out) is colour-stained blood smears, so training never saw colour microscopy. The slot stays in the 20-wide output because that is a published contract; the decoder never emits it.

## Results

Headline: **macro-F1 0.9067** over the `model_trained` classes
(`CT`, `DX`, `US`), balanced policy, on the held-out test split.

Accuracy on frames whose true class is one the model claims: **0.9218**.
Accuracy over *every* labelled frame including classes the model does not claim:
**0.7446** - the difference is almost entirely withdrawn-class frames,
which are counted as errors here by design.

| benchmark tier | rows | accuracy | macro-F1 |
|---|---|---|---|
| held_out_source | 11,000 | 0.6618 | 0.9021 |
| in_distribution | 4,712 | 0.9378 | 0.9109 |



In-distribution macro-F1 is 0.9109 and held-out-source macro-F1 is 0.9021. The gap is small, which is the evidence that the model reads the image rather than the dataset it came from.



Coarse and fine accuracy are identical here: no collapse group has two populated classes in this corpus, so the group step of the cascade never fires. The cascade is still live - it is simply untested by this evaluation.



Open-set rejection: **not measured**. The corpus contains no `open_set` rows, so the energy cut was fitted for false-rejection rate only and its true detection rate is unknown. `out_of_distribution: true` is provisional - do not rely on it in production until it has been measured against real non-diagnostic material.

Expected calibration error on the test split: **0.0991**
(calibrated). Temperature was fitted on validation; the test
split is harder because it carries every held-out source, so this is the pessimistic figure.

## The abstention cascade

When no single class clears its threshold the decoder falls back, and always reports how far it
fell:

| `granularity` | meaning |
|---|---|
| `modality` | a specific modality, e.g. `CT` |
| `group` | a confusable group, e.g. `XR_PLAIN` when CR and DX are not separable |
| `family` | an acquisition family, e.g. `projection_radiography` |
| `none` | abstained, or rejected as out-of-distribution |

A caller routing to a CT pipeline should require `granularity == "modality"`. A caller deciding
whether an image is worth processing at all is satisfied by `family`.

## Limitations

* **MR sequence, contrast phase and anatomy are out of scope.** This model says `MR`; it does not
  say `T2 FLAIR`, and it does not say `brain`.
* **Secondary Capture is not a modality.** An SC object is a screenshot of something else. The
  model sees pixels and will answer for what was on screen.
* **Compound figures have no single answer.** A literature figure with CT and MR panels side by
  side violates the single-label assumption this model is built on.
* **Digitised screen-film mammography is over-represented** in the `MG` training data relative to
  modern full-field digital mammography.
* Scores are calibrated. Read the measured precision and
  recall above, not the score, when choosing a policy.

## Quick start (Python + onnxruntime)

No clone, no framework - the artifact and its config files are all you need.

```bash

pip install onnxruntime pillow numpy huggingface_hub

```

```python

import json, numpy as np, onnxruntime as ort

from PIL import Image

from huggingface_hub import hf_hub_download



REPO = "vectorsense/modalityscan"

sess = ort.InferenceSession(hf_hub_download(REPO, "model.int8.onnx"),

                            providers=["CPUExecutionProvider"])

tax  = json.load(open(hf_hub_download(REPO, "modality-taxonomy.json"), encoding="utf-8"))

book = json.load(open(hf_hub_download(REPO, "modality-thresholds.json"), encoding="utf-8"))

pre  = json.load(open(hf_hub_download(REPO, "preprocessor.json"), encoding="utf-8"))



MODALITY = tax["levels"]["modality"]["index"]

FAMILY   = tax["levels"]["family"]["index"]

TIER     = {label: tier

            for tier in ("model_trained", "experimental", "not_trained")

            for label in tax["tiers"][tier]["types"]}



mean = np.array(pre["image_mean"], np.float32)[:, None, None]

std  = np.array(pre["image_std"],  np.float32)[:, None, None]



def pixel_values(path, size=224):

    # Letterbox, never centre-crop. The field-of-view outline - ultrasound sector, circular

    # fundus aperture, endoscope vignette - is a top-tier modality cue; cropping discards it.

    image = Image.open(path).convert("RGB")

    scale = size / max(image.size)

    small = image.resize((max(1, round(image.width * scale)),

                          max(1, round(image.height * scale))), Image.BILINEAR)

    canvas = Image.new("RGB", (size, size), (0, 0, 0))

    canvas.paste(small, ((size - small.width) // 2, (size - small.height) // 2))

    array = np.asarray(canvas, np.float32) / 255.0

    return ((array.transpose(2, 0, 1) - mean) / std)[None]



def softmax(x):

    e = np.exp(x - x.max())

    return e / e.sum()



family_logits, modality_logits = sess.run(

    ["family_logits", "modality_logits"], {"pixel_values": pixel_values("frame.png")}

)



# Temperature lives outside the graph so calibration stays configuration, not a re-export.

T = book["calibration"]["temperature"]

p_mod = softmax(modality_logits[0] / T["modality"])

p_fam = softmax(family_logits[0] / T["family"])



policy = book["default_policy"]

floor  = book["policies"][policy]["modality"]

over   = book["per_class_overrides"].get(policy, {})



# Energy over logits, not softmax: softmax is confident on garbage by construction.

energy = -float(np.log(np.exp(modality_logits[0] / T["modality"]).sum()))

if energy > book["ood"]["energy_threshold"][policy]:

    print("out of distribution ->", book["ood"]["fallback_label"])

else:

    # `not_trained` classes keep their slot because the output width is a published contract,

    # but they are not claims. A raw argmax will eventually return one of them.

    best = next(i for i in np.argsort(-p_mod) if TIER.get(MODALITY[i]) != "not_trained")

    label, score = MODALITY[best], float(p_mod[best])

    if score >= over.get(label, floor):

        print(f"modality: {label} {score:.3f}  tier={TIER[label]}")

    else:

        print(f"family:   {FAMILY[int(p_fam.argmax())]} {p_fam.max():.3f}  (cascade fell back)")

```

Two lines carry most of the correctness. **Letterbox rather than centre-crop** - the field-of-view
outline is signal, not padding. **Skip `not_trained` classes** - a raw `argmax` will eventually

hand you a label this model does not stand behind.



## Run it as a REST API



The companion repo ships a FastAPI service with the cascade, the calibrated thresholds, the

energy rejection and series aggregation already wired up.



```bash

pip install fastapi "uvicorn[standard]" onnxruntime pillow numpy

uvicorn service.app:app --port 5003      # auto-loads model.int8.onnx from ./models

```



```bash

curl -s http://127.0.0.1:5003/classify_modality \

  -H "Content-Type: application/json" \

  -d '{"frames":[{"id":"series-1/f000","image_base64":"iVBORw0KGgo..."}],

       "options":{"policy":"balanced"}}'

```



The same call from Python - send a PNG or JPEG, get parsed JSON back:



```python

import base64, requests



URL = "http://127.0.0.1:5003/classify_modality"



def classify(path, policy="balanced"):

    payload = {

        "frames": [{"id": "series-1/f000",

                    "image_base64": base64.b64encode(open(path, "rb").read()).decode()}],

        "options": {"policy": policy, "top_k": 3},

    }

    response = requests.post(URL, json=payload, timeout=30)

    response.raise_for_status()     # unknown request fields return 422, never a silent default

    return response.json()



frame = classify("frame.png")["results"][0]

print(frame["modality"]["label"], frame["modality"]["granularity"], frame["modality"]["score"])

print("out of distribution:", frame["out_of_distribution"])

```



A whole DICOM series in one request, which is the cheapest accuracy in the serving path:



```python

payload = {

    "frames": [{"id": f"series-1/f{i:03d}",

                "image_base64": base64.b64encode(open(p, "rb").read()).decode()}

               for i, p in enumerate(frame_paths)],

    "options": {"policy": "balanced", "aggregate_by": "series"},

}

print(requests.post(URL, json=payload, timeout=60).json()["aggregate"])

```



Endpoints: `GET /`, `POST /classify_modality`, `GET /health/ready`, `GET /metadata`.

`GET /metadata` reports the tier of every class at runtime, so a caller can tell a claimed class

from an experimental one without reading this card.



Prefer the service over the raw-ONNX snippet unless you have a reason not to: it already applies

the calibrated temperature, the per-class thresholds, the abstention cascade and the rejection

path. The snippet reimplements those in miniature and will drift from the config.



## Usage via the library



```python

from modalityscan import ModalityClassifier



classifier = ModalityClassifier(model_dir="models")

results, series = classifier.classify(

    [{"id": "series-1/f000", "image_path": "frame.png"}],

    policy="balanced",

    aggregate_by="series",

)

print(results[0].modality.label, results[0].modality.granularity)

```



Modality is constant across a DICOM series, so passing several frames with a shared `series_id`

and `aggregate_by="series"` is the cheapest accuracy available in the serving path.



## Attribution



Training data is used under the licences recorded in `ATTRIBUTION.md`. Repository: `vectorsense/modalityscan`.