README.md · vectorsense/organscan at main

Publish OrganScan

c0abae0 verified about 1 month ago

18.2 kB

---
license: apache-2.0
tags:
- image-classification
- multi-label-image-classification
- medical-imaging
- medical
- anatomy
- ultrasound
- computed-tomography
- radiology
- dicom
- onnx
- onnxruntime
- quantized
- int8
- edge
pipeline_tag: image-classification
library_name: onnx
base_model: timm/mobilenetv4_conv_small.e2400_r224_in1k
metrics:
- accuracy
- f1
---

# OrganScan — anatomy classification from DICOM frames

Given a frame rendered from DICOM pixel data — ultrasound cine, CT, MR or X-ray — OrganScan returns
the **modality**, the **body region**, and **every organ visible in the frame**. It ships as a
CPU-only **INT8 ONNX** artifact of 2.9 MB that runs fully offline, on a phone-class ARM core.

It is a **hierarchical, multi-label** classifier, not a flat organ classifier. One shared backbone,
three sigmoid heads.

| | |
|---|---|
| **Backbone** | `mobilenetv4_conv_small.e2400_r224_in1k` (ImageNet-init, fine-tuned) |
| **Heads** | modality (5) · region (8) · organ (30), independent sigmoids |
| **Input** | `pixel_values` float32 `[batch, 3, 224, 224]`, NCHW, ImageNet-normalized |
| **Outputs** | `modality_logits`, `region_logits`, `organ_logits` — **logits, not probabilities** |
| **Artifact** | INT8 ONNX, 2.9 MB, CPU-only, no network at inference |
| **Latency** | ~26.878 ms / frame on CPU via `onnxruntime` |
| **Batch** | dynamic; H/W fixed at 224 so ORT/NNAPI kernels stay optimised |
| **Modalities** | US · CT · MR · XR |
| **Taxonomy** | `1.0.0` — a head width change is a version bump, never a silent edit |

**What it is for:** routing and triage of imaging frames — *which study is this, and what is in
view* — so a downstream pipeline can pick the right model, the right window, or the right reader.

**What it is not:** a diagnostic device. It reports anatomy, never pathology, and holds no
regulatory clearance of any kind.

## Why multi-label and hierarchical

An RUQ ultrasound view shows liver *and* right kidney *and* diaphragm. A softmax is normalised, so
raising `liver` mathematically forces `kidney` down — it would train the model to suppress something
visibly present. Independent sigmoids let `liver=0.91, kidney_right=0.86` coexist.

The three levels exist because labels arrive at different granularities. A DICOM series with only
`Modality` supervises one level; a TotalSegmentator slice supervises all three. A per-class masked
loss lets every frame contribute to whichever levels — and whichever *classes* — it actually knows
about. A flat organ classifier would have to discard most of the corpus.

## Results — test split, policy `balanced`

| Level | Classes scored | macro-AP |
|---|---:|---:|
| modality | 4 | **1.000** |
| region | 8 | **0.980** |
| organ | 30 | **0.917** |

### Per organ

| Organ | AP | Precision | Recall | F1 | Support |
|---|---:|---:|---:|---:|---:|
| `BREAST` | 1.000 | 1.000 | 1.000 | **1.000** | 155 |
| `UTERUS` | 1.000 | 0.978 | 1.000 | **0.989** | 132 |
| `LUNG_RIGHT` | 0.992 | 0.959 | 0.986 | **0.972** | 2,148 |
| `LUNG_LEFT` | 0.992 | 0.953 | 0.987 | **0.969** | 2,154 |
| `AORTA_ABDOMINAL` | 0.989 | 0.894 | 0.989 | **0.939** | 1,784 |
| `BRAIN` | 0.986 | 0.831 | 0.997 | **0.906** | 773 |
| `AORTA_THORACIC` | 0.984 | 0.852 | 0.997 | **0.919** | 766 |
| `LIVER` | 0.983 | 0.921 | 0.956 | **0.938** | 2,806 |
| `COLON` | 0.983 | 0.901 | 0.961 | **0.930** | 2,092 |
| `FEMUR_LEFT` | 0.978 | 0.915 | 0.949 | **0.932** | 926 |
| `FEMUR_RIGHT` | 0.976 | 0.924 | 0.953 | **0.938** | 894 |
| `HEART` | 0.976 | 0.925 | 0.927 | **0.926** | 1,178 |
| `KIDNEY_RIGHT` | 0.970 | 0.804 | 0.964 | **0.877** | 1,369 |
| `KIDNEY_LEFT` | 0.969 | 0.823 | 0.966 | **0.889** | 1,404 |
| `OVARY` | 0.967 | 0.675 | 0.986 | **0.801** | 141 |
| `SMALL_BOWEL` | 0.962 | 0.816 | 0.971 | **0.887** | 1,515 |
| `TRACHEA` | 0.962 | 0.792 | 0.957 | **0.867** | 740 |
| `INFERIOR_VENA_CAVA` | 0.959 | 0.857 | 0.966 | **0.908** | 2,249 |
| `SPLEEN` | 0.956 | 0.864 | 0.909 | **0.886** | 1,631 |
| `PANCREAS` | 0.950 | 0.776 | 0.966 | **0.861** | 1,349 |
| `STOMACH` | 0.932 | 0.807 | 0.910 | **0.855** | 1,173 |
| `BLADDER` | 0.931 | 0.830 | 0.882 | **0.856** | 748 |
| `DUODENUM` | 0.929 | 0.625 | 0.979 | **0.763** | 875 |
| `PORTAL_VEIN` | 0.821 | 0.522 | 0.961 | **0.676** | 309 |
| `THYROID` | 0.811 | 0.552 | 0.896 | **0.683** | 212 |
| `ESOPHAGUS` | 0.801 | 0.555 | 0.963 | **0.704** | 1,103 |
| `PROSTATE` | 0.790 | 0.520 | 0.878 | **0.653** | 74 |
| `GALLBLADDER` | 0.689 | 0.431 | 0.940 | **0.591** | 446 |
| `ADRENAL_LEFT` | 0.643 | 0.320 | 0.919 | **0.475** | 174 |
| `ADRENAL_RIGHT` | 0.633 | 0.358 | 0.905 | **0.513** | 147 |

### The numbers that matter more than the headline

| Slice | Rows | Organ macro-AP |
|---|---:|---:|
| in_distribution | 15,395 | 0.917 |

**Loop level** (1,413 loops): organ macro-AP **0.945**. This is the number a user experiences for ultrasound.

**Laterality:** the model picks the wrong side on 0.7% of frames where it identifies
the organ correctly. That error class is reported separately on purpose — `kidney_left ->
kidney_right` is a minor error, `thyroid -> bladder` is not, and averaging them hides the expensive
one.

## Class tiers — what is actually supported

Not every class is equal, and this card says so rather than advertising one number.

| Tier | Count | Classes |
|---|---:|---|
| Tier | Count | Classes |
|---|---:|---|
| **model_trained** | 0 | — |
| **weak_label_only** | 0 | — |
| **experimental** | 0 | — |
| **blocked** | 0 | — |

`GET /metadata` on the service reports the tier for every class, so a caller can tell a validated
class from an experimental one at runtime.

## Quick start (Python + onnxruntime)

```bash
pip install onnxruntime pillow numpy huggingface_hub
```

```python
import json, numpy as np, onnxruntime as ort
from PIL import Image
from huggingface_hub import hf_hub_download

REPO = "vectorsense/organscan"
sess = ort.InferenceSession(hf_hub_download(REPO, "model.int8.onnx"),
                            providers=["CPUExecutionProvider"])
tax  = json.load(open(hf_hub_download(REPO, "anatomy-tiers.json"), encoding="utf-8"))
pre  = json.load(open(hf_hub_download(REPO, "preprocessor.json"), encoding="utf-8"))

ORGANS  = tax["levels"]["organ"]["index"]
REGIONS = tax["levels"]["region"]["index"]
MODS    = tax["levels"]["modality"]["index"]
PARENT  = [REGIONS.index(tax["parents"]["organ_to_region"][o]) for o in ORGANS]
mean = np.array(pre["image_mean"], np.float32)[:, None, None]
std  = np.array(pre["image_std"],  np.float32)[:, None, None]

def pixel_values(path, size=224):
    image = Image.open(path).convert("RGB")
    scale = size / min(image.size)
    image = image.resize((round(image.width * scale), round(image.height * scale)), Image.BILINEAR)
    left, top = (image.width - size) // 2, (image.height - size) // 2
    crop = np.asarray(image.crop((left, top, left + size, top + size)), np.float32) / 255.0
    return ((crop.transpose(2, 0, 1) - mean) / std)[None]

def sigmoid(x): return 1.0 / (1.0 + np.exp(-x))

m, r, o = sess.run(["modality_logits", "region_logits", "organ_logits"],
                   {"pixel_values": pixel_values("frame.png")})
p_mod, p_reg, p_org = sigmoid(m[0]), sigmoid(r[0]), sigmoid(o[0])

# Chain down the hierarchy so a child can never outrank its parent.
p_org = p_org * p_reg[PARENT]

# Per-class and per-modality thresholds, not a flat cut. Ultrasound labels one organ per frame
# and masks the rest, so the organ head carries a CT co-occurrence prior into US frames; the
# per-modality gate is what holds it back. A flat 0.5 over-reports badly on ultrasound.
book = json.load(open(hf_hub_download(REPO, "anatomy-thresholds.json"), encoding="utf-8"))
policy = "balanced"
modality = MODS[p_mod.argmax()]
default = book["policies"][policy]["organ"]
per_class = book.get("per_class_overrides", {}).get(policy, {})
per_modality = book.get("per_modality_overrides", {}).get(policy, {}).get(modality, {})

def threshold(label):
    return per_modality.get(label, per_class.get(label, default))

print("modality:", modality, round(float(p_mod.max()), 3))
print("region:  ", REGIONS[p_reg.argmax()], round(float(p_reg.max()), 3))
for i in np.argsort(-p_org)[:5]:
    if p_org[i] >= threshold(ORGANS[i]):
        print(f"  {ORGANS[i]:16s} {p_org[i]:.3f}")
```

**Apply the chaining.** Without it the organ head can report `bladder` on a chest X-ray. It costs one
multiply.

## Run it as a REST API

The companion repo ships a FastAPI service with the policy presets, hierarchy chaining and
loop-level aggregation already wired up.

```bash
pip install fastapi "uvicorn[standard]" onnxruntime pillow numpy
uvicorn service.app:app --port 5002      # auto-loads model.int8.onnx from ./models
```

```bash
curl -s http://127.0.0.1:5002/classify_anatomy \
  -H "Content-Type: application/json" \
  -d '{"frames":[{"id":"loop-1/f000","image_base64":"iVBORw0KGgo..."}],
       "options":{"policy":"balanced","aggregate_by":"loop"}}'
```

The same call from Python, which is what most callers actually want — send a PNG or JPEG, get the
parsed JSON back:

```python
import base64, json, requests

def classify(path, url="http://127.0.0.1:5002/classify_anatomy", policy="balanced"):
    payload = {
        "frames": [{"id": "loop-1/f000",
                    "image_base64": base64.b64encode(open(path, "rb").read()).decode()}],
        "options": {"policy": policy, "top_k": 5},
    }
    response = requests.post(url, json=payload, timeout=30)
    response.raise_for_status()          # unknown request fields return 422, not a silent default
    return response.json()

result = classify("frame.png")
frame = result["results"][0]
print(frame["modality"]["label"], frame["region"]["label"])
for organ in frame["organs"]:
    print(f"  {organ['label']:16s} {organ['score']:.3f}")
print("model", result["model_version"], "taxonomy", result["taxonomy_version"])
```

A whole cine loop in one request, which is how ultrasound should be served:

```python
payload = {
    "frames": [{"id": f"loop-1/f{i:03d}",
                "image_base64": base64.b64encode(open(p, "rb").read()).decode()}
               for i, p in enumerate(frame_paths)],       # 8-16 frames at temporal stride
    "options": {"policy": "balanced", "aggregate_by": "loop"},
}
print(requests.post(url, json=payload, timeout=60).json()["aggregate"])
```

Endpoints: `GET /`, `POST /classify_anatomy`, `GET /health/ready`, `GET /metadata`.

Prefer this over the raw-ONNX snippet above unless you have a reason not to: the service already
applies the calibrated per-class and per-modality thresholds, the hierarchy chaining and the
abstention rules. The raw snippet uses a flat 0.5 cut, which on ultrasound over-reports badly.

## Loop-level aggregation

For ultrasound this is the single most valuable thing in the serving path. Sample 8–16 frames at
temporal stride across the cine, send them as one request with `aggregate_by: "loop"`, and the
service returns a per-loop verdict using the **mean of the top-half frame scores**. Probe-in-air and
transition frames drag a plain mean down; the top-half mean is worth several points for one batched
forward pass.

## Policy presets and abstention

`strict` / `balanced` / `precision` set per-level thresholds; override per request with
`options.score_threshold`. Published results use `balanced`.

`indeterminate` is a first-class outcome. `region=abdomen, organs=[]` — "abdominal ultrasound, organ
indeterminate" — is a legitimate and actionable answer. A model that confidently says "kidney" on a
probe-in-air frame is worse than useless.

Scores are **not calibrated probabilities**. 0.90 does not mean 90% correct. See
`anatomy-thresholds.json`.

## Training record

Full provenance lives in this repo under `training/` — best checkpoint, configs, corpus manifest,
source licences, corpus validation report. Trained on 82,354 frames from 7,483
patients.

## Limitations

- **Not a diagnostic device.** It identifies what is in the picture. It says nothing about whether
  what is in the picture is normal.
- **Ultrasound organ coverage is thin.** Only one licence-clear public abdominal ultrasound organ
  dataset exists (563 patients, one site, one vendor family). Public data can carry the modality and
  region heads; a production ultrasound organ head needs internal DICOM.
- **Cardiac echo is effectively absent.** EchoNet forbids derivative works and CAMUS publishes no
  licence, so the `cardiac` region is trained from CT only.
- **Vendor and site shift is the dominant failure mode**, more than anatomy. Validate on your own
  scanners before relying on this.
- **Burned-in annotation.** If you feed uncropped ultrasound frames whose vendor UI prints the organ
  name, you are measuring OCR, not anatomy. Crop to `SequenceOfUltrasoundRegions` (0018,6011).
- **No fetal organ labels.** `obstetric` is a region with no organ children by design.

## Licence

**Model weights and code: `apache-2.0`.** Every contributing source is CC BY 4.0, which requires attribution only, so the weights carry no copyleft or field-of-use restriction.

The weights are a derivative of their training data, so the training-data licences reach through to
this artifact. The table below lists every source that contributed at least one frame, and
`ATTRIBUTION.md` in this repo carries the full notice.

| Source | Creators | Licence | Frames used |
|---|---|---|---:|
| [TotalSegmentator CT dataset v3.0.0](https://zenodo.org/records/22688904) | Jakob Wasserthal et al. | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 61,902 |
| [AMOS22: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://zenodo.org/records/7262581) | Yuanfeng Ji et al. | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 11,480 |
| [FETAL_PLANES_DB: Common maternal-fetal ultrasound images](https://zenodo.org/records/3904280) | Xavier P. Burgos-Artizzu et al. | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 10,748 |
| [MedMNIST v2 (ChestMNIST, 224px)](https://zenodo.org/records/10519652) | Jiancheng Yang et al. | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 7,924 |
| [MedMNIST v2 (OrganSMNIST, 224px)](https://zenodo.org/records/10519652) | Jiancheng Yang et al. | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 4,000 |
| [MedMNIST v2 (OrganAMNIST, 224px)](https://zenodo.org/records/10519652) | Jiancheng Yang et al. | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 3,914 |
| [MedMNIST v2 (OrganCMNIST, 224px)](https://zenodo.org/records/10519652) | Jiancheng Yang et al. | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 3,902 |
| [Abdominal Ultrasound Image Dataset for Organ Classification and Disease Detection](https://scholarsjunction.msstate.edu/research-data/5/) | Sifat Zina Karim | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 3,380 |
| [MedMNIST v2 (BreastMNIST, 224px)](https://zenodo.org/records/10519652) | Jiancheng Yang et al. | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 779 |

**All source material was modified.** Frames were sliced from volumes or cine loops, windowed,
resized to 224×224, capped per patient, and relabelled into the OrganScan taxonomy. Per-source detail
is in `ATTRIBUTION.md` and `training/sources.json`.

### If you redistribute this model

You must pass on the attribution. Copy `ATTRIBUTION.md` alongside the weights, keep this licence
section intact, and state that you modified the model if you did. The simplest compliant thing is to
link back to this repository.



### Attribution

```
OrganScan is trained on the following datasets, all of which were modified
(sliced to 2D, windowed, resized to 224px, and relabelled):

- TotalSegmentator CT dataset v3.0.0
  Jakob Wasserthal, Hanns-Christian Breit, Manfred T. Meyer, Maurice Pradella, Daniel Hinck, Alexander W. Sauter, Tobias Heye, Daniel T. Boll, Joshy Cyriac, Shan Yang, Michael Bach, Martin Segeroth - CC-BY-4.0
  https://zenodo.org/records/22688904
- AMOS22: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
  Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhang, Wanling Ma, Xiang Wan, Ping Luo - CC-BY-4.0
  https://zenodo.org/records/7262581
- FETAL_PLANES_DB: Common maternal-fetal ultrasound images
  Xavier P. Burgos-Artizzu, David Coronado-Gutierrez, Brenda Valenzuela-Alcaraz, Elisenda Bonet-Carne, Elisenda Eixarch, Fatima Crispi, Eduard Gratacós - CC-BY-4.0
  https://zenodo.org/records/3904280
- MedMNIST v2 (ChestMNIST, 224px)
  Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0
  https://zenodo.org/records/10519652
- MedMNIST v2 (OrganSMNIST, 224px)
  Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0
  https://zenodo.org/records/10519652
- MedMNIST v2 (OrganAMNIST, 224px)
  Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0
  https://zenodo.org/records/10519652
- MedMNIST v2 (OrganCMNIST, 224px)
  Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0
  https://zenodo.org/records/10519652
- Abdominal Ultrasound Image Dataset for Organ Classification and Disease Detection
  Sifat Zina Karim - CC-BY-4.0
  https://scholarsjunction.msstate.edu/research-data/5/
- MedMNIST v2 (BreastMNIST, 224px)
  Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0
  https://zenodo.org/records/10519652
```

## Author

Arnab Pal — [LinkedIn](https://www.linkedin.com/in/pal-arnab) &middot; [GitHub](https://github.com/palarnab)

	---
	license: apache-2.0
	tags:
	- image-classification
	- multi-label-image-classification
	- medical-imaging
	- medical
	- anatomy
	- ultrasound
	- computed-tomography
	- radiology
	- dicom
	- onnx
	- onnxruntime
	- quantized
	- int8
	- edge
	pipeline_tag: image-classification
	library_name: onnx
	base_model: timm/mobilenetv4_conv_small.e2400_r224_in1k
	metrics:
	- accuracy
	- f1
	---

	# OrganScan — anatomy classification from DICOM frames

	Given a frame rendered from DICOM pixel data — ultrasound cine, CT, MR or X-ray — OrganScan returns
	the modality, the body region, and every organ visible in the frame. It ships as a
	CPU-only INT8 ONNX artifact of 2.9 MB that runs fully offline, on a phone-class ARM core.

	It is a hierarchical, multi-label classifier, not a flat organ classifier. One shared backbone,
	three sigmoid heads.

	\| \| \|
	\|---\|---\|
	\| Backbone \| `mobilenetv4_conv_small.e2400_r224_in1k` (ImageNet-init, fine-tuned) \|
	\| Heads \| modality (5) · region (8) · organ (30), independent sigmoids \|
	\| Input \| `pixel_values` float32 `[batch, 3, 224, 224]`, NCHW, ImageNet-normalized \|
	\| Outputs \| `modality_logits`, `region_logits`, `organ_logits` — logits, not probabilities \|
	\| Artifact \| INT8 ONNX, 2.9 MB, CPU-only, no network at inference \|
	\| Latency \| ~26.878 ms / frame on CPU via `onnxruntime` \|
	\| Batch \| dynamic; H/W fixed at 224 so ORT/NNAPI kernels stay optimised \|
	\| Modalities \| US · CT · MR · XR \|
	\| Taxonomy \| `1.0.0` — a head width change is a version bump, never a silent edit \|

	What it is for: routing and triage of imaging frames — *which study is this, and what is in
	view* — so a downstream pipeline can pick the right model, the right window, or the right reader.

	What it is not: a diagnostic device. It reports anatomy, never pathology, and holds no
	regulatory clearance of any kind.

	## Why multi-label and hierarchical

	An RUQ ultrasound view shows liver and right kidney and diaphragm. A softmax is normalised, so
	raising `liver` mathematically forces `kidney` down — it would train the model to suppress something
	visibly present. Independent sigmoids let `liver=0.91, kidney_right=0.86` coexist.

	The three levels exist because labels arrive at different granularities. A DICOM series with only
	`Modality` supervises one level; a TotalSegmentator slice supervises all three. A per-class masked
	loss lets every frame contribute to whichever levels — and whichever classes — it actually knows
	about. A flat organ classifier would have to discard most of the corpus.

	## Results — test split, policy `balanced`

	\| Level \| Classes scored \| macro-AP \|
	\|---\|---:\|---:\|
	\| modality \| 4 \| 1.000 \|
	\| region \| 8 \| 0.980 \|
	\| organ \| 30 \| 0.917 \|

	### Per organ

	\| Organ \| AP \| Precision \| Recall \| F1 \| Support \|
	\|---\|---:\|---:\|---:\|---:\|---:\|
	\| `BREAST` \| 1.000 \| 1.000 \| 1.000 \| 1.000 \| 155 \|
	\| `UTERUS` \| 1.000 \| 0.978 \| 1.000 \| 0.989 \| 132 \|
	\| `LUNG_RIGHT` \| 0.992 \| 0.959 \| 0.986 \| 0.972 \| 2,148 \|
	\| `LUNG_LEFT` \| 0.992 \| 0.953 \| 0.987 \| 0.969 \| 2,154 \|
	\| `AORTA_ABDOMINAL` \| 0.989 \| 0.894 \| 0.989 \| 0.939 \| 1,784 \|
	\| `BRAIN` \| 0.986 \| 0.831 \| 0.997 \| 0.906 \| 773 \|
	\| `AORTA_THORACIC` \| 0.984 \| 0.852 \| 0.997 \| 0.919 \| 766 \|
	\| `LIVER` \| 0.983 \| 0.921 \| 0.956 \| 0.938 \| 2,806 \|
	\| `COLON` \| 0.983 \| 0.901 \| 0.961 \| 0.930 \| 2,092 \|
	\| `FEMUR_LEFT` \| 0.978 \| 0.915 \| 0.949 \| 0.932 \| 926 \|
	\| `FEMUR_RIGHT` \| 0.976 \| 0.924 \| 0.953 \| 0.938 \| 894 \|
	\| `HEART` \| 0.976 \| 0.925 \| 0.927 \| 0.926 \| 1,178 \|
	\| `KIDNEY_RIGHT` \| 0.970 \| 0.804 \| 0.964 \| 0.877 \| 1,369 \|
	\| `KIDNEY_LEFT` \| 0.969 \| 0.823 \| 0.966 \| 0.889 \| 1,404 \|
	\| `OVARY` \| 0.967 \| 0.675 \| 0.986 \| 0.801 \| 141 \|
	\| `SMALL_BOWEL` \| 0.962 \| 0.816 \| 0.971 \| 0.887 \| 1,515 \|
	\| `TRACHEA` \| 0.962 \| 0.792 \| 0.957 \| 0.867 \| 740 \|
	\| `INFERIOR_VENA_CAVA` \| 0.959 \| 0.857 \| 0.966 \| 0.908 \| 2,249 \|
	\| `SPLEEN` \| 0.956 \| 0.864 \| 0.909 \| 0.886 \| 1,631 \|
	\| `PANCREAS` \| 0.950 \| 0.776 \| 0.966 \| 0.861 \| 1,349 \|
	\| `STOMACH` \| 0.932 \| 0.807 \| 0.910 \| 0.855 \| 1,173 \|
	\| `BLADDER` \| 0.931 \| 0.830 \| 0.882 \| 0.856 \| 748 \|
	\| `DUODENUM` \| 0.929 \| 0.625 \| 0.979 \| 0.763 \| 875 \|
	\| `PORTAL_VEIN` \| 0.821 \| 0.522 \| 0.961 \| 0.676 \| 309 \|
	\| `THYROID` \| 0.811 \| 0.552 \| 0.896 \| 0.683 \| 212 \|
	\| `ESOPHAGUS` \| 0.801 \| 0.555 \| 0.963 \| 0.704 \| 1,103 \|
	\| `PROSTATE` \| 0.790 \| 0.520 \| 0.878 \| 0.653 \| 74 \|
	\| `GALLBLADDER` \| 0.689 \| 0.431 \| 0.940 \| 0.591 \| 446 \|
	\| `ADRENAL_LEFT` \| 0.643 \| 0.320 \| 0.919 \| 0.475 \| 174 \|
	\| `ADRENAL_RIGHT` \| 0.633 \| 0.358 \| 0.905 \| 0.513 \| 147 \|

	### The numbers that matter more than the headline

	\| Slice \| Rows \| Organ macro-AP \|
	\|---\|---:\|---:\|
	\| in_distribution \| 15,395 \| 0.917 \|

	Loop level (1,413 loops): organ macro-AP 0.945. This is the number a user experiences for ultrasound.

	Laterality: the model picks the wrong side on 0.7% of frames where it identifies
	the organ correctly. That error class is reported separately on purpose — `kidney_left ->
	kidney_right` is a minor error, `thyroid -> bladder` is not, and averaging them hides the expensive
	one.

	## Class tiers — what is actually supported

	Not every class is equal, and this card says so rather than advertising one number.

	\| Tier \| Count \| Classes \|
	\|---\|---:\|---\|
	\| Tier \| Count \| Classes \|
	\|---\|---:\|---\|
	\| model_trained \| 0 \| — \|
	\| weak_label_only \| 0 \| — \|
	\| experimental \| 0 \| — \|
	\| blocked \| 0 \| — \|

	`GET /metadata` on the service reports the tier for every class, so a caller can tell a validated
	class from an experimental one at runtime.

	## Quick start (Python + onnxruntime)

	```bash
	pip install onnxruntime pillow numpy huggingface_hub
	```

	```python
	import json, numpy as np, onnxruntime as ort
	from PIL import Image
	from huggingface_hub import hf_hub_download

	REPO = "vectorsense/organscan"
	sess = ort.InferenceSession(hf_hub_download(REPO, "model.int8.onnx"),
	providers=["CPUExecutionProvider"])
	tax = json.load(open(hf_hub_download(REPO, "anatomy-tiers.json"), encoding="utf-8"))
	pre = json.load(open(hf_hub_download(REPO, "preprocessor.json"), encoding="utf-8"))

	ORGANS = tax["levels"]["organ"]["index"]
	REGIONS = tax["levels"]["region"]["index"]
	MODS = tax["levels"]["modality"]["index"]
	PARENT = [REGIONS.index(tax["parents"]["organ_to_region"][o]) for o in ORGANS]
	mean = np.array(pre["image_mean"], np.float32)[:, None, None]
	std = np.array(pre["image_std"], np.float32)[:, None, None]

	def pixel_values(path, size=224):
	image = Image.open(path).convert("RGB")
	scale = size / min(image.size)
	image = image.resize((round(image.width * scale), round(image.height * scale)), Image.BILINEAR)
	left, top = (image.width - size) // 2, (image.height - size) // 2
	crop = np.asarray(image.crop((left, top, left + size, top + size)), np.float32) / 255.0
	return ((crop.transpose(2, 0, 1) - mean) / std)[None]

	def sigmoid(x): return 1.0 / (1.0 + np.exp(-x))

	m, r, o = sess.run(["modality_logits", "region_logits", "organ_logits"],
	{"pixel_values": pixel_values("frame.png")})
	p_mod, p_reg, p_org = sigmoid(m[0]), sigmoid(r[0]), sigmoid(o[0])

	# Chain down the hierarchy so a child can never outrank its parent.
	p_org = p_org * p_reg[PARENT]

	# Per-class and per-modality thresholds, not a flat cut. Ultrasound labels one organ per frame
	# and masks the rest, so the organ head carries a CT co-occurrence prior into US frames; the
	# per-modality gate is what holds it back. A flat 0.5 over-reports badly on ultrasound.
	book = json.load(open(hf_hub_download(REPO, "anatomy-thresholds.json"), encoding="utf-8"))
	policy = "balanced"
	modality = MODS[p_mod.argmax()]
	default = book["policies"][policy]["organ"]
	per_class = book.get("per_class_overrides", {}).get(policy, {})
	per_modality = book.get("per_modality_overrides", {}).get(policy, {}).get(modality, {})

	def threshold(label):
	return per_modality.get(label, per_class.get(label, default))

	print("modality:", modality, round(float(p_mod.max()), 3))
	print("region: ", REGIONS[p_reg.argmax()], round(float(p_reg.max()), 3))
	for i in np.argsort(-p_org)[:5]:
	if p_org[i] >= threshold(ORGANS[i]):
	print(f" {ORGANS[i]:16s} {p_org[i]:.3f}")
	```

	Apply the chaining. Without it the organ head can report `bladder` on a chest X-ray. It costs one
	multiply.

	## Run it as a REST API

	The companion repo ships a FastAPI service with the policy presets, hierarchy chaining and
	loop-level aggregation already wired up.

	```bash
	pip install fastapi "uvicorn[standard]" onnxruntime pillow numpy
	uvicorn service.app:app --port 5002 # auto-loads model.int8.onnx from ./models
	```

	```bash
	curl -s http://127.0.0.1:5002/classify_anatomy \
	-H "Content-Type: application/json" \
	-d '{"frames":[{"id":"loop-1/f000","image_base64":"iVBORw0KGgo..."}],
	"options":{"policy":"balanced","aggregate_by":"loop"}}'
	```

	The same call from Python, which is what most callers actually want — send a PNG or JPEG, get the
	parsed JSON back:

	```python
	import base64, json, requests

	def classify(path, url="http://127.0.0.1:5002/classify_anatomy", policy="balanced"):
	payload = {
	"frames": [{"id": "loop-1/f000",
	"image_base64": base64.b64encode(open(path, "rb").read()).decode()}],
	"options": {"policy": policy, "top_k": 5},
	}
	response = requests.post(url, json=payload, timeout=30)
	response.raise_for_status() # unknown request fields return 422, not a silent default
	return response.json()

	result = classify("frame.png")
	frame = result["results"][0]
	print(frame["modality"]["label"], frame["region"]["label"])
	for organ in frame["organs"]:
	print(f" {organ['label']:16s} {organ['score']:.3f}")
	print("model", result["model_version"], "taxonomy", result["taxonomy_version"])
	```

	A whole cine loop in one request, which is how ultrasound should be served:

	```python
	payload = {
	"frames": [{"id": f"loop-1/f{i:03d}",
	"image_base64": base64.b64encode(open(p, "rb").read()).decode()}
	for i, p in enumerate(frame_paths)], # 8-16 frames at temporal stride
	"options": {"policy": "balanced", "aggregate_by": "loop"},
	}
	print(requests.post(url, json=payload, timeout=60).json()["aggregate"])
	```

	Endpoints: `GET /`, `POST /classify_anatomy`, `GET /health/ready`, `GET /metadata`.

	Prefer this over the raw-ONNX snippet above unless you have a reason not to: the service already
	applies the calibrated per-class and per-modality thresholds, the hierarchy chaining and the
	abstention rules. The raw snippet uses a flat 0.5 cut, which on ultrasound over-reports badly.

	## Loop-level aggregation

	For ultrasound this is the single most valuable thing in the serving path. Sample 8–16 frames at
	temporal stride across the cine, send them as one request with `aggregate_by: "loop"`, and the
	service returns a per-loop verdict using the mean of the top-half frame scores. Probe-in-air and
	transition frames drag a plain mean down; the top-half mean is worth several points for one batched
	forward pass.

	## Policy presets and abstention

	`strict` / `balanced` / `precision` set per-level thresholds; override per request with
	`options.score_threshold`. Published results use `balanced`.

	`indeterminate` is a first-class outcome. `region=abdomen, organs=[]` — "abdominal ultrasound, organ
	indeterminate" — is a legitimate and actionable answer. A model that confidently says "kidney" on a
	probe-in-air frame is worse than useless.

	Scores are not calibrated probabilities. 0.90 does not mean 90% correct. See
	`anatomy-thresholds.json`.

	## Training record

	Full provenance lives in this repo under `training/` — best checkpoint, configs, corpus manifest,
	source licences, corpus validation report. Trained on 82,354 frames from 7,483
	patients.

	## Limitations

	- Not a diagnostic device. It identifies what is in the picture. It says nothing about whether
	what is in the picture is normal.
	- Ultrasound organ coverage is thin. Only one licence-clear public abdominal ultrasound organ
	dataset exists (563 patients, one site, one vendor family). Public data can carry the modality and
	region heads; a production ultrasound organ head needs internal DICOM.
	- Cardiac echo is effectively absent. EchoNet forbids derivative works and CAMUS publishes no
	licence, so the `cardiac` region is trained from CT only.
	- Vendor and site shift is the dominant failure mode, more than anatomy. Validate on your own
	scanners before relying on this.
	- Burned-in annotation. If you feed uncropped ultrasound frames whose vendor UI prints the organ
	name, you are measuring OCR, not anatomy. Crop to `SequenceOfUltrasoundRegions` (0018,6011).
	- No fetal organ labels. `obstetric` is a region with no organ children by design.

	## Licence

	Model weights and code: `apache-2.0`. Every contributing source is CC BY 4.0, which requires attribution only, so the weights carry no copyleft or field-of-use restriction.

	The weights are a derivative of their training data, so the training-data licences reach through to
	this artifact. The table below lists every source that contributed at least one frame, and
	`ATTRIBUTION.md` in this repo carries the full notice.

	\| Source \| Creators \| Licence \| Frames used \|
	\|---\|---\|---\|---:\|
	\| [TotalSegmentator CT dataset v3.0.0](https://zenodo.org/records/22688904) \| Jakob Wasserthal et al. \| [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) \| 61,902 \|
	\| [AMOS22: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://zenodo.org/records/7262581) \| Yuanfeng Ji et al. \| [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) \| 11,480 \|
	\| [FETAL_PLANES_DB: Common maternal-fetal ultrasound images](https://zenodo.org/records/3904280) \| Xavier P. Burgos-Artizzu et al. \| [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) \| 10,748 \|
	\| [MedMNIST v2 (ChestMNIST, 224px)](https://zenodo.org/records/10519652) \| Jiancheng Yang et al. \| [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) \| 7,924 \|
	\| [MedMNIST v2 (OrganSMNIST, 224px)](https://zenodo.org/records/10519652) \| Jiancheng Yang et al. \| [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) \| 4,000 \|
	\| [MedMNIST v2 (OrganAMNIST, 224px)](https://zenodo.org/records/10519652) \| Jiancheng Yang et al. \| [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) \| 3,914 \|
	\| [MedMNIST v2 (OrganCMNIST, 224px)](https://zenodo.org/records/10519652) \| Jiancheng Yang et al. \| [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) \| 3,902 \|
	\| [Abdominal Ultrasound Image Dataset for Organ Classification and Disease Detection](https://scholarsjunction.msstate.edu/research-data/5/) \| Sifat Zina Karim \| [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) \| 3,380 \|
	\| [MedMNIST v2 (BreastMNIST, 224px)](https://zenodo.org/records/10519652) \| Jiancheng Yang et al. \| [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) \| 779 \|

	All source material was modified. Frames were sliced from volumes or cine loops, windowed,
	resized to 224×224, capped per patient, and relabelled into the OrganScan taxonomy. Per-source detail
	is in `ATTRIBUTION.md` and `training/sources.json`.

	### If you redistribute this model

	You must pass on the attribution. Copy `ATTRIBUTION.md` alongside the weights, keep this licence
	section intact, and state that you modified the model if you did. The simplest compliant thing is to
	link back to this repository.



	### Attribution

	```
	OrganScan is trained on the following datasets, all of which were modified
	(sliced to 2D, windowed, resized to 224px, and relabelled):

	- TotalSegmentator CT dataset v3.0.0
	Jakob Wasserthal, Hanns-Christian Breit, Manfred T. Meyer, Maurice Pradella, Daniel Hinck, Alexander W. Sauter, Tobias Heye, Daniel T. Boll, Joshy Cyriac, Shan Yang, Michael Bach, Martin Segeroth - CC-BY-4.0
	https://zenodo.org/records/22688904
	- AMOS22: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
	Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhang, Wanling Ma, Xiang Wan, Ping Luo - CC-BY-4.0
	https://zenodo.org/records/7262581
	- FETAL_PLANES_DB: Common maternal-fetal ultrasound images
	Xavier P. Burgos-Artizzu, David Coronado-Gutierrez, Brenda Valenzuela-Alcaraz, Elisenda Bonet-Carne, Elisenda Eixarch, Fatima Crispi, Eduard Gratacós - CC-BY-4.0
	https://zenodo.org/records/3904280
	- MedMNIST v2 (ChestMNIST, 224px)
	Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0
	https://zenodo.org/records/10519652
	- MedMNIST v2 (OrganSMNIST, 224px)
	Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0
	https://zenodo.org/records/10519652
	- MedMNIST v2 (OrganAMNIST, 224px)
	Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0
	https://zenodo.org/records/10519652
	- MedMNIST v2 (OrganCMNIST, 224px)
	Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0
	https://zenodo.org/records/10519652
	- Abdominal Ultrasound Image Dataset for Organ Classification and Disease Detection
	Sifat Zina Karim - CC-BY-4.0
	https://scholarsjunction.msstate.edu/research-data/5/
	- MedMNIST v2 (BreastMNIST, 224px)
	Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0
	https://zenodo.org/records/10519652
	```

	## Author

	Arnab Pal — [LinkedIn](https://www.linkedin.com/in/pal-arnab) · [GitHub](https://github.com/palarnab)