OrganScan — anatomy classification from DICOM frames

Given a frame rendered from DICOM pixel data — ultrasound cine, CT, MR or X-ray — OrganScan returns the modality, the body region, and every organ visible in the frame. It ships as a CPU-only INT8 ONNX artifact of 2.9 MB that runs fully offline, on a phone-class ARM core.

It is a hierarchical, multi-label classifier, not a flat organ classifier. One shared backbone, three sigmoid heads.

Backbone	mobilenetv4_conv_small.e2400_r224_in1k (ImageNet-init, fine-tuned)
Heads	modality (5) · region (8) · organ (30), independent sigmoids
Input	pixel_values float32 [batch, 3, 224, 224], NCHW, ImageNet-normalized
Outputs	modality_logits, region_logits, organ_logits — logits, not probabilities
Artifact	INT8 ONNX, 2.9 MB, CPU-only, no network at inference
Latency	~26.878 ms / frame on CPU via onnxruntime
Batch	dynamic; H/W fixed at 224 so ORT/NNAPI kernels stay optimised
Modalities	US · CT · MR · XR
Taxonomy	1.0.0 — a head width change is a version bump, never a silent edit

What it is for: routing and triage of imaging frames — which study is this, and what is in view — so a downstream pipeline can pick the right model, the right window, or the right reader.

What it is not: a diagnostic device. It reports anatomy, never pathology, and holds no regulatory clearance of any kind.

Why multi-label and hierarchical

An RUQ ultrasound view shows liver and right kidney and diaphragm. A softmax is normalised, so raising liver mathematically forces kidney down — it would train the model to suppress something visibly present. Independent sigmoids let liver=0.91, kidney_right=0.86 coexist.

The three levels exist because labels arrive at different granularities. A DICOM series with only Modality supervises one level; a TotalSegmentator slice supervises all three. A per-class masked loss lets every frame contribute to whichever levels — and whichever classes — it actually knows about. A flat organ classifier would have to discard most of the corpus.

Results — test split, policy balanced

Level	Classes scored	macro-AP
modality	4	1.000
region	8	0.980
organ	30	0.917

Per organ

Organ	AP	Precision	Recall	F1	Support
BREAST	1.000	1.000	1.000	1.000	155
UTERUS	1.000	0.978	1.000	0.989	132
LUNG_RIGHT	0.992	0.959	0.986	0.972	2,148
LUNG_LEFT	0.992	0.953	0.987	0.969	2,154
AORTA_ABDOMINAL	0.989	0.894	0.989	0.939	1,784
BRAIN	0.986	0.831	0.997	0.906	773
AORTA_THORACIC	0.984	0.852	0.997	0.919	766
LIVER	0.983	0.921	0.956	0.938	2,806
COLON	0.983	0.901	0.961	0.930	2,092
FEMUR_LEFT	0.978	0.915	0.949	0.932	926
FEMUR_RIGHT	0.976	0.924	0.953	0.938	894
HEART	0.976	0.925	0.927	0.926	1,178
KIDNEY_RIGHT	0.970	0.804	0.964	0.877	1,369
KIDNEY_LEFT	0.969	0.823	0.966	0.889	1,404
OVARY	0.967	0.675	0.986	0.801	141
SMALL_BOWEL	0.962	0.816	0.971	0.887	1,515
TRACHEA	0.962	0.792	0.957	0.867	740
INFERIOR_VENA_CAVA	0.959	0.857	0.966	0.908	2,249
SPLEEN	0.956	0.864	0.909	0.886	1,631
PANCREAS	0.950	0.776	0.966	0.861	1,349
STOMACH	0.932	0.807	0.910	0.855	1,173
BLADDER	0.931	0.830	0.882	0.856	748
DUODENUM	0.929	0.625	0.979	0.763	875
PORTAL_VEIN	0.821	0.522	0.961	0.676	309
THYROID	0.811	0.552	0.896	0.683	212
ESOPHAGUS	0.801	0.555	0.963	0.704	1,103
PROSTATE	0.790	0.520	0.878	0.653	74
GALLBLADDER	0.689	0.431	0.940	0.591	446
ADRENAL_LEFT	0.643	0.320	0.919	0.475	174
ADRENAL_RIGHT	0.633	0.358	0.905	0.513	147

The numbers that matter more than the headline

Slice	Rows	Organ macro-AP
in_distribution	15,395	0.917

Loop level (1,413 loops): organ macro-AP 0.945. This is the number a user experiences for ultrasound.

Laterality: the model picks the wrong side on 0.7% of frames where it identifies the organ correctly. That error class is reported separately on purpose — kidney_left -> kidney_right is a minor error, thyroid -> bladder is not, and averaging them hides the expensive one.

Class tiers — what is actually supported

Not every class is equal, and this card says so rather than advertising one number.

Tier	Count	Classes
Tier	Count	Classes
---	---:	---
model_trained	0	—
weak_label_only	0	—
experimental	0	—
blocked	0	—

GET /metadata on the service reports the tier for every class, so a caller can tell a validated class from an experimental one at runtime.

Quick start (Python + onnxruntime)

pip install onnxruntime pillow numpy huggingface_hub

import json, numpy as np, onnxruntime as ort
from PIL import Image
from huggingface_hub import hf_hub_download

REPO = "vectorsense/organscan"
sess = ort.InferenceSession(hf_hub_download(REPO, "model.int8.onnx"),
                            providers=["CPUExecutionProvider"])
tax  = json.load(open(hf_hub_download(REPO, "anatomy-tiers.json"), encoding="utf-8"))
pre  = json.load(open(hf_hub_download(REPO, "preprocessor.json"), encoding="utf-8"))

ORGANS  = tax["levels"]["organ"]["index"]
REGIONS = tax["levels"]["region"]["index"]
MODS    = tax["levels"]["modality"]["index"]
PARENT  = [REGIONS.index(tax["parents"]["organ_to_region"][o]) for o in ORGANS]
mean = np.array(pre["image_mean"], np.float32)[:, None, None]
std  = np.array(pre["image_std"],  np.float32)[:, None, None]

def pixel_values(path, size=224):
    image = Image.open(path).convert("RGB")
    scale = size / min(image.size)
    image = image.resize((round(image.width * scale), round(image.height * scale)), Image.BILINEAR)
    left, top = (image.width - size) // 2, (image.height - size) // 2
    crop = np.asarray(image.crop((left, top, left + size, top + size)), np.float32) / 255.0
    return ((crop.transpose(2, 0, 1) - mean) / std)[None]

def sigmoid(x): return 1.0 / (1.0 + np.exp(-x))

m, r, o = sess.run(["modality_logits", "region_logits", "organ_logits"],
                   {"pixel_values": pixel_values("frame.png")})
p_mod, p_reg, p_org = sigmoid(m[0]), sigmoid(r[0]), sigmoid(o[0])

# Chain down the hierarchy so a child can never outrank its parent.
p_org = p_org * p_reg[PARENT]

# Per-class and per-modality thresholds, not a flat cut. Ultrasound labels one organ per frame
# and masks the rest, so the organ head carries a CT co-occurrence prior into US frames; the
# per-modality gate is what holds it back. A flat 0.5 over-reports badly on ultrasound.
book = json.load(open(hf_hub_download(REPO, "anatomy-thresholds.json"), encoding="utf-8"))
policy = "balanced"
modality = MODS[p_mod.argmax()]
default = book["policies"][policy]["organ"]
per_class = book.get("per_class_overrides", {}).get(policy, {})
per_modality = book.get("per_modality_overrides", {}).get(policy, {}).get(modality, {})

def threshold(label):
    return per_modality.get(label, per_class.get(label, default))

print("modality:", modality, round(float(p_mod.max()), 3))
print("region:  ", REGIONS[p_reg.argmax()], round(float(p_reg.max()), 3))
for i in np.argsort(-p_org)[:5]:
    if p_org[i] >= threshold(ORGANS[i]):
        print(f"  {ORGANS[i]:16s} {p_org[i]:.3f}")

Apply the chaining. Without it the organ head can report bladder on a chest X-ray. It costs one multiply.

Run it as a REST API

The companion repo ships a FastAPI service with the policy presets, hierarchy chaining and loop-level aggregation already wired up.

pip install fastapi "uvicorn[standard]" onnxruntime pillow numpy
uvicorn service.app:app --port 5002      # auto-loads model.int8.onnx from ./models

curl -s http://127.0.0.1:5002/classify_anatomy \
  -H "Content-Type: application/json" \
  -d '{"frames":[{"id":"loop-1/f000","image_base64":"iVBORw0KGgo..."}],
       "options":{"policy":"balanced","aggregate_by":"loop"}}'

The same call from Python, which is what most callers actually want — send a PNG or JPEG, get the parsed JSON back:

import base64, json, requests

def classify(path, url="http://127.0.0.1:5002/classify_anatomy", policy="balanced"):
    payload = {
        "frames": [{"id": "loop-1/f000",
                    "image_base64": base64.b64encode(open(path, "rb").read()).decode()}],
        "options": {"policy": policy, "top_k": 5},
    }
    response = requests.post(url, json=payload, timeout=30)
    response.raise_for_status()          # unknown request fields return 422, not a silent default
    return response.json()

result = classify("frame.png")
frame = result["results"][0]
print(frame["modality"]["label"], frame["region"]["label"])
for organ in frame["organs"]:
    print(f"  {organ['label']:16s} {organ['score']:.3f}")
print("model", result["model_version"], "taxonomy", result["taxonomy_version"])

A whole cine loop in one request, which is how ultrasound should be served:

payload = {
    "frames": [{"id": f"loop-1/f{i:03d}",
                "image_base64": base64.b64encode(open(p, "rb").read()).decode()}
               for i, p in enumerate(frame_paths)],       # 8-16 frames at temporal stride
    "options": {"policy": "balanced", "aggregate_by": "loop"},
}
print(requests.post(url, json=payload, timeout=60).json()["aggregate"])

Endpoints: GET /, POST /classify_anatomy, GET /health/ready, GET /metadata.

Prefer this over the raw-ONNX snippet above unless you have a reason not to: the service already applies the calibrated per-class and per-modality thresholds, the hierarchy chaining and the abstention rules. The raw snippet uses a flat 0.5 cut, which on ultrasound over-reports badly.

Loop-level aggregation

For ultrasound this is the single most valuable thing in the serving path. Sample 8–16 frames at temporal stride across the cine, send them as one request with aggregate_by: "loop", and the service returns a per-loop verdict using the mean of the top-half frame scores. Probe-in-air and transition frames drag a plain mean down; the top-half mean is worth several points for one batched forward pass.

Policy presets and abstention

strict / balanced / precision set per-level thresholds; override per request with options.score_threshold. Published results use balanced.

indeterminate is a first-class outcome. region=abdomen, organs=[] — "abdominal ultrasound, organ indeterminate" — is a legitimate and actionable answer. A model that confidently says "kidney" on a probe-in-air frame is worse than useless.

Scores are not calibrated probabilities. 0.90 does not mean 90% correct. See anatomy-thresholds.json.

Training record

Full provenance lives in this repo under training/ — best checkpoint, configs, corpus manifest, source licences, corpus validation report. Trained on 82,354 frames from 7,483 patients.

Limitations

• Not a diagnostic device. It identifies what is in the picture. It says nothing about whether what is in the picture is normal.
• Ultrasound organ coverage is thin. Only one licence-clear public abdominal ultrasound organ dataset exists (563 patients, one site, one vendor family). Public data can carry the modality and region heads; a production ultrasound organ head needs internal DICOM.
• Cardiac echo is effectively absent. EchoNet forbids derivative works and CAMUS publishes no licence, so the cardiac region is trained from CT only.
• Vendor and site shift is the dominant failure mode, more than anatomy. Validate on your own scanners before relying on this.
• Burned-in annotation. If you feed uncropped ultrasound frames whose vendor UI prints the organ name, you are measuring OCR, not anatomy. Crop to SequenceOfUltrasoundRegions (0018,6011).
• No fetal organ labels. obstetric is a region with no organ children by design.

Licence

Model weights and code: apache-2.0. Every contributing source is CC BY 4.0, which requires attribution only, so the weights carry no copyleft or field-of-use restriction.

The weights are a derivative of their training data, so the training-data licences reach through to this artifact. The table below lists every source that contributed at least one frame, and ATTRIBUTION.md in this repo carries the full notice.

Source	Creators	Licence	Frames used
TotalSegmentator CT dataset v3.0.0	Jakob Wasserthal et al.	CC-BY-4.0	61,902
AMOS22: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation	Yuanfeng Ji et al.	CC-BY-4.0	11,480
FETAL_PLANES_DB: Common maternal-fetal ultrasound images	Xavier P. Burgos-Artizzu et al.	CC-BY-4.0	10,748
MedMNIST v2 (ChestMNIST, 224px)	Jiancheng Yang et al.	CC-BY-4.0	7,924
MedMNIST v2 (OrganSMNIST, 224px)	Jiancheng Yang et al.	CC-BY-4.0	4,000
MedMNIST v2 (OrganAMNIST, 224px)	Jiancheng Yang et al.	CC-BY-4.0	3,914
MedMNIST v2 (OrganCMNIST, 224px)	Jiancheng Yang et al.	CC-BY-4.0	3,902
Abdominal Ultrasound Image Dataset for Organ Classification and Disease Detection	Sifat Zina Karim	CC-BY-4.0	3,380
MedMNIST v2 (BreastMNIST, 224px)	Jiancheng Yang et al.	CC-BY-4.0	779

All source material was modified. Frames were sliced from volumes or cine loops, windowed, resized to 224×224, capped per patient, and relabelled into the OrganScan taxonomy. Per-source detail is in ATTRIBUTION.md and training/sources.json.

If you redistribute this model

You must pass on the attribution. Copy ATTRIBUTION.md alongside the weights, keep this licence section intact, and state that you modified the model if you did. The simplest compliant thing is to link back to this repository.

Attribution

OrganScan is trained on the following datasets, all of which were modified
(sliced to 2D, windowed, resized to 224px, and relabelled):

- TotalSegmentator CT dataset v3.0.0
  Jakob Wasserthal, Hanns-Christian Breit, Manfred T. Meyer, Maurice Pradella, Daniel Hinck, Alexander W. Sauter, Tobias Heye, Daniel T. Boll, Joshy Cyriac, Shan Yang, Michael Bach, Martin Segeroth - CC-BY-4.0
  https://zenodo.org/records/22688904
- AMOS22: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
  Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhang, Wanling Ma, Xiang Wan, Ping Luo - CC-BY-4.0
  https://zenodo.org/records/7262581
- FETAL_PLANES_DB: Common maternal-fetal ultrasound images
  Xavier P. Burgos-Artizzu, David Coronado-Gutierrez, Brenda Valenzuela-Alcaraz, Elisenda Bonet-Carne, Elisenda Eixarch, Fatima Crispi, Eduard Gratacós - CC-BY-4.0
  https://zenodo.org/records/3904280
- MedMNIST v2 (ChestMNIST, 224px)
  Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0
  https://zenodo.org/records/10519652
- MedMNIST v2 (OrganSMNIST, 224px)
  Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0
  https://zenodo.org/records/10519652
- MedMNIST v2 (OrganAMNIST, 224px)
  Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0
  https://zenodo.org/records/10519652
- MedMNIST v2 (OrganCMNIST, 224px)
  Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0
  https://zenodo.org/records/10519652
- Abdominal Ultrasound Image Dataset for Organ Classification and Disease Detection
  Sifat Zina Karim - CC-BY-4.0
  https://scholarsjunction.msstate.edu/research-data/5/
- MedMNIST v2 (BreastMNIST, 224px)
  Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0
  https://zenodo.org/records/10519652

Author

Arnab Pal — LinkedIn · GitHub