--- license: apache-2.0 tags: - image-classification - multi-label-image-classification - medical-imaging - medical - anatomy - ultrasound - computed-tomography - radiology - dicom - onnx - onnxruntime - quantized - int8 - edge pipeline_tag: image-classification library_name: onnx base_model: timm/mobilenetv4_conv_small.e2400_r224_in1k metrics: - accuracy - f1 --- # OrganScan — anatomy classification from DICOM frames Given a frame rendered from DICOM pixel data — ultrasound cine, CT, MR or X-ray — OrganScan returns the **modality**, the **body region**, and **every organ visible in the frame**. It ships as a CPU-only **INT8 ONNX** artifact of 2.9 MB that runs fully offline, on a phone-class ARM core. It is a **hierarchical, multi-label** classifier, not a flat organ classifier. One shared backbone, three sigmoid heads. | | | |---|---| | **Backbone** | `mobilenetv4_conv_small.e2400_r224_in1k` (ImageNet-init, fine-tuned) | | **Heads** | modality (5) · region (8) · organ (30), independent sigmoids | | **Input** | `pixel_values` float32 `[batch, 3, 224, 224]`, NCHW, ImageNet-normalized | | **Outputs** | `modality_logits`, `region_logits`, `organ_logits` — **logits, not probabilities** | | **Artifact** | INT8 ONNX, 2.9 MB, CPU-only, no network at inference | | **Latency** | ~26.878 ms / frame on CPU via `onnxruntime` | | **Batch** | dynamic; H/W fixed at 224 so ORT/NNAPI kernels stay optimised | | **Modalities** | US · CT · MR · XR | | **Taxonomy** | `1.0.0` — a head width change is a version bump, never a silent edit | **What it is for:** routing and triage of imaging frames — *which study is this, and what is in view* — so a downstream pipeline can pick the right model, the right window, or the right reader. **What it is not:** a diagnostic device. It reports anatomy, never pathology, and holds no regulatory clearance of any kind. ## Why multi-label and hierarchical An RUQ ultrasound view shows liver *and* right kidney *and* diaphragm. A softmax is normalised, so raising `liver` mathematically forces `kidney` down — it would train the model to suppress something visibly present. Independent sigmoids let `liver=0.91, kidney_right=0.86` coexist. The three levels exist because labels arrive at different granularities. A DICOM series with only `Modality` supervises one level; a TotalSegmentator slice supervises all three. A per-class masked loss lets every frame contribute to whichever levels — and whichever *classes* — it actually knows about. A flat organ classifier would have to discard most of the corpus. ## Results — test split, policy `balanced` | Level | Classes scored | macro-AP | |---|---:|---:| | modality | 4 | **1.000** | | region | 8 | **0.980** | | organ | 30 | **0.917** | ### Per organ | Organ | AP | Precision | Recall | F1 | Support | |---|---:|---:|---:|---:|---:| | `BREAST` | 1.000 | 1.000 | 1.000 | **1.000** | 155 | | `UTERUS` | 1.000 | 0.978 | 1.000 | **0.989** | 132 | | `LUNG_RIGHT` | 0.992 | 0.959 | 0.986 | **0.972** | 2,148 | | `LUNG_LEFT` | 0.992 | 0.953 | 0.987 | **0.969** | 2,154 | | `AORTA_ABDOMINAL` | 0.989 | 0.894 | 0.989 | **0.939** | 1,784 | | `BRAIN` | 0.986 | 0.831 | 0.997 | **0.906** | 773 | | `AORTA_THORACIC` | 0.984 | 0.852 | 0.997 | **0.919** | 766 | | `LIVER` | 0.983 | 0.921 | 0.956 | **0.938** | 2,806 | | `COLON` | 0.983 | 0.901 | 0.961 | **0.930** | 2,092 | | `FEMUR_LEFT` | 0.978 | 0.915 | 0.949 | **0.932** | 926 | | `FEMUR_RIGHT` | 0.976 | 0.924 | 0.953 | **0.938** | 894 | | `HEART` | 0.976 | 0.925 | 0.927 | **0.926** | 1,178 | | `KIDNEY_RIGHT` | 0.970 | 0.804 | 0.964 | **0.877** | 1,369 | | `KIDNEY_LEFT` | 0.969 | 0.823 | 0.966 | **0.889** | 1,404 | | `OVARY` | 0.967 | 0.675 | 0.986 | **0.801** | 141 | | `SMALL_BOWEL` | 0.962 | 0.816 | 0.971 | **0.887** | 1,515 | | `TRACHEA` | 0.962 | 0.792 | 0.957 | **0.867** | 740 | | `INFERIOR_VENA_CAVA` | 0.959 | 0.857 | 0.966 | **0.908** | 2,249 | | `SPLEEN` | 0.956 | 0.864 | 0.909 | **0.886** | 1,631 | | `PANCREAS` | 0.950 | 0.776 | 0.966 | **0.861** | 1,349 | | `STOMACH` | 0.932 | 0.807 | 0.910 | **0.855** | 1,173 | | `BLADDER` | 0.931 | 0.830 | 0.882 | **0.856** | 748 | | `DUODENUM` | 0.929 | 0.625 | 0.979 | **0.763** | 875 | | `PORTAL_VEIN` | 0.821 | 0.522 | 0.961 | **0.676** | 309 | | `THYROID` | 0.811 | 0.552 | 0.896 | **0.683** | 212 | | `ESOPHAGUS` | 0.801 | 0.555 | 0.963 | **0.704** | 1,103 | | `PROSTATE` | 0.790 | 0.520 | 0.878 | **0.653** | 74 | | `GALLBLADDER` | 0.689 | 0.431 | 0.940 | **0.591** | 446 | | `ADRENAL_LEFT` | 0.643 | 0.320 | 0.919 | **0.475** | 174 | | `ADRENAL_RIGHT` | 0.633 | 0.358 | 0.905 | **0.513** | 147 | ### The numbers that matter more than the headline | Slice | Rows | Organ macro-AP | |---|---:|---:| | in_distribution | 15,395 | 0.917 | **Loop level** (1,413 loops): organ macro-AP **0.945**. This is the number a user experiences for ultrasound. **Laterality:** the model picks the wrong side on 0.7% of frames where it identifies the organ correctly. That error class is reported separately on purpose — `kidney_left -> kidney_right` is a minor error, `thyroid -> bladder` is not, and averaging them hides the expensive one. ## Class tiers — what is actually supported Not every class is equal, and this card says so rather than advertising one number. | Tier | Count | Classes | |---|---:|---| | Tier | Count | Classes | |---|---:|---| | **model_trained** | 0 | — | | **weak_label_only** | 0 | — | | **experimental** | 0 | — | | **blocked** | 0 | — | `GET /metadata` on the service reports the tier for every class, so a caller can tell a validated class from an experimental one at runtime. ## Quick start (Python + onnxruntime) ```bash pip install onnxruntime pillow numpy huggingface_hub ``` ```python import json, numpy as np, onnxruntime as ort from PIL import Image from huggingface_hub import hf_hub_download REPO = "vectorsense/organscan" sess = ort.InferenceSession(hf_hub_download(REPO, "model.int8.onnx"), providers=["CPUExecutionProvider"]) tax = json.load(open(hf_hub_download(REPO, "anatomy-tiers.json"), encoding="utf-8")) pre = json.load(open(hf_hub_download(REPO, "preprocessor.json"), encoding="utf-8")) ORGANS = tax["levels"]["organ"]["index"] REGIONS = tax["levels"]["region"]["index"] MODS = tax["levels"]["modality"]["index"] PARENT = [REGIONS.index(tax["parents"]["organ_to_region"][o]) for o in ORGANS] mean = np.array(pre["image_mean"], np.float32)[:, None, None] std = np.array(pre["image_std"], np.float32)[:, None, None] def pixel_values(path, size=224): image = Image.open(path).convert("RGB") scale = size / min(image.size) image = image.resize((round(image.width * scale), round(image.height * scale)), Image.BILINEAR) left, top = (image.width - size) // 2, (image.height - size) // 2 crop = np.asarray(image.crop((left, top, left + size, top + size)), np.float32) / 255.0 return ((crop.transpose(2, 0, 1) - mean) / std)[None] def sigmoid(x): return 1.0 / (1.0 + np.exp(-x)) m, r, o = sess.run(["modality_logits", "region_logits", "organ_logits"], {"pixel_values": pixel_values("frame.png")}) p_mod, p_reg, p_org = sigmoid(m[0]), sigmoid(r[0]), sigmoid(o[0]) # Chain down the hierarchy so a child can never outrank its parent. p_org = p_org * p_reg[PARENT] # Per-class and per-modality thresholds, not a flat cut. Ultrasound labels one organ per frame # and masks the rest, so the organ head carries a CT co-occurrence prior into US frames; the # per-modality gate is what holds it back. A flat 0.5 over-reports badly on ultrasound. book = json.load(open(hf_hub_download(REPO, "anatomy-thresholds.json"), encoding="utf-8")) policy = "balanced" modality = MODS[p_mod.argmax()] default = book["policies"][policy]["organ"] per_class = book.get("per_class_overrides", {}).get(policy, {}) per_modality = book.get("per_modality_overrides", {}).get(policy, {}).get(modality, {}) def threshold(label): return per_modality.get(label, per_class.get(label, default)) print("modality:", modality, round(float(p_mod.max()), 3)) print("region: ", REGIONS[p_reg.argmax()], round(float(p_reg.max()), 3)) for i in np.argsort(-p_org)[:5]: if p_org[i] >= threshold(ORGANS[i]): print(f" {ORGANS[i]:16s} {p_org[i]:.3f}") ``` **Apply the chaining.** Without it the organ head can report `bladder` on a chest X-ray. It costs one multiply. ## Run it as a REST API The companion repo ships a FastAPI service with the policy presets, hierarchy chaining and loop-level aggregation already wired up. ```bash pip install fastapi "uvicorn[standard]" onnxruntime pillow numpy uvicorn service.app:app --port 5002 # auto-loads model.int8.onnx from ./models ``` ```bash curl -s http://127.0.0.1:5002/classify_anatomy \ -H "Content-Type: application/json" \ -d '{"frames":[{"id":"loop-1/f000","image_base64":"iVBORw0KGgo..."}], "options":{"policy":"balanced","aggregate_by":"loop"}}' ``` The same call from Python, which is what most callers actually want — send a PNG or JPEG, get the parsed JSON back: ```python import base64, json, requests def classify(path, url="http://127.0.0.1:5002/classify_anatomy", policy="balanced"): payload = { "frames": [{"id": "loop-1/f000", "image_base64": base64.b64encode(open(path, "rb").read()).decode()}], "options": {"policy": policy, "top_k": 5}, } response = requests.post(url, json=payload, timeout=30) response.raise_for_status() # unknown request fields return 422, not a silent default return response.json() result = classify("frame.png") frame = result["results"][0] print(frame["modality"]["label"], frame["region"]["label"]) for organ in frame["organs"]: print(f" {organ['label']:16s} {organ['score']:.3f}") print("model", result["model_version"], "taxonomy", result["taxonomy_version"]) ``` A whole cine loop in one request, which is how ultrasound should be served: ```python payload = { "frames": [{"id": f"loop-1/f{i:03d}", "image_base64": base64.b64encode(open(p, "rb").read()).decode()} for i, p in enumerate(frame_paths)], # 8-16 frames at temporal stride "options": {"policy": "balanced", "aggregate_by": "loop"}, } print(requests.post(url, json=payload, timeout=60).json()["aggregate"]) ``` Endpoints: `GET /`, `POST /classify_anatomy`, `GET /health/ready`, `GET /metadata`. Prefer this over the raw-ONNX snippet above unless you have a reason not to: the service already applies the calibrated per-class and per-modality thresholds, the hierarchy chaining and the abstention rules. The raw snippet uses a flat 0.5 cut, which on ultrasound over-reports badly. ## Loop-level aggregation For ultrasound this is the single most valuable thing in the serving path. Sample 8–16 frames at temporal stride across the cine, send them as one request with `aggregate_by: "loop"`, and the service returns a per-loop verdict using the **mean of the top-half frame scores**. Probe-in-air and transition frames drag a plain mean down; the top-half mean is worth several points for one batched forward pass. ## Policy presets and abstention `strict` / `balanced` / `precision` set per-level thresholds; override per request with `options.score_threshold`. Published results use `balanced`. `indeterminate` is a first-class outcome. `region=abdomen, organs=[]` — "abdominal ultrasound, organ indeterminate" — is a legitimate and actionable answer. A model that confidently says "kidney" on a probe-in-air frame is worse than useless. Scores are **not calibrated probabilities**. 0.90 does not mean 90% correct. See `anatomy-thresholds.json`. ## Training record Full provenance lives in this repo under `training/` — best checkpoint, configs, corpus manifest, source licences, corpus validation report. Trained on 82,354 frames from 7,483 patients. ## Limitations - **Not a diagnostic device.** It identifies what is in the picture. It says nothing about whether what is in the picture is normal. - **Ultrasound organ coverage is thin.** Only one licence-clear public abdominal ultrasound organ dataset exists (563 patients, one site, one vendor family). Public data can carry the modality and region heads; a production ultrasound organ head needs internal DICOM. - **Cardiac echo is effectively absent.** EchoNet forbids derivative works and CAMUS publishes no licence, so the `cardiac` region is trained from CT only. - **Vendor and site shift is the dominant failure mode**, more than anatomy. Validate on your own scanners before relying on this. - **Burned-in annotation.** If you feed uncropped ultrasound frames whose vendor UI prints the organ name, you are measuring OCR, not anatomy. Crop to `SequenceOfUltrasoundRegions` (0018,6011). - **No fetal organ labels.** `obstetric` is a region with no organ children by design. ## Licence **Model weights and code: `apache-2.0`.** Every contributing source is CC BY 4.0, which requires attribution only, so the weights carry no copyleft or field-of-use restriction. The weights are a derivative of their training data, so the training-data licences reach through to this artifact. The table below lists every source that contributed at least one frame, and `ATTRIBUTION.md` in this repo carries the full notice. | Source | Creators | Licence | Frames used | |---|---|---|---:| | [TotalSegmentator CT dataset v3.0.0](https://zenodo.org/records/22688904) | Jakob Wasserthal et al. | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 61,902 | | [AMOS22: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://zenodo.org/records/7262581) | Yuanfeng Ji et al. | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 11,480 | | [FETAL_PLANES_DB: Common maternal-fetal ultrasound images](https://zenodo.org/records/3904280) | Xavier P. Burgos-Artizzu et al. | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 10,748 | | [MedMNIST v2 (ChestMNIST, 224px)](https://zenodo.org/records/10519652) | Jiancheng Yang et al. | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 7,924 | | [MedMNIST v2 (OrganSMNIST, 224px)](https://zenodo.org/records/10519652) | Jiancheng Yang et al. | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 4,000 | | [MedMNIST v2 (OrganAMNIST, 224px)](https://zenodo.org/records/10519652) | Jiancheng Yang et al. | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 3,914 | | [MedMNIST v2 (OrganCMNIST, 224px)](https://zenodo.org/records/10519652) | Jiancheng Yang et al. | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 3,902 | | [Abdominal Ultrasound Image Dataset for Organ Classification and Disease Detection](https://scholarsjunction.msstate.edu/research-data/5/) | Sifat Zina Karim | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 3,380 | | [MedMNIST v2 (BreastMNIST, 224px)](https://zenodo.org/records/10519652) | Jiancheng Yang et al. | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | 779 | **All source material was modified.** Frames were sliced from volumes or cine loops, windowed, resized to 224×224, capped per patient, and relabelled into the OrganScan taxonomy. Per-source detail is in `ATTRIBUTION.md` and `training/sources.json`. ### If you redistribute this model You must pass on the attribution. Copy `ATTRIBUTION.md` alongside the weights, keep this licence section intact, and state that you modified the model if you did. The simplest compliant thing is to link back to this repository. ### Attribution ``` OrganScan is trained on the following datasets, all of which were modified (sliced to 2D, windowed, resized to 224px, and relabelled): - TotalSegmentator CT dataset v3.0.0 Jakob Wasserthal, Hanns-Christian Breit, Manfred T. Meyer, Maurice Pradella, Daniel Hinck, Alexander W. Sauter, Tobias Heye, Daniel T. Boll, Joshy Cyriac, Shan Yang, Michael Bach, Martin Segeroth - CC-BY-4.0 https://zenodo.org/records/22688904 - AMOS22: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhang, Wanling Ma, Xiang Wan, Ping Luo - CC-BY-4.0 https://zenodo.org/records/7262581 - FETAL_PLANES_DB: Common maternal-fetal ultrasound images Xavier P. Burgos-Artizzu, David Coronado-Gutierrez, Brenda Valenzuela-Alcaraz, Elisenda Bonet-Carne, Elisenda Eixarch, Fatima Crispi, Eduard Gratacós - CC-BY-4.0 https://zenodo.org/records/3904280 - MedMNIST v2 (ChestMNIST, 224px) Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0 https://zenodo.org/records/10519652 - MedMNIST v2 (OrganSMNIST, 224px) Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0 https://zenodo.org/records/10519652 - MedMNIST v2 (OrganAMNIST, 224px) Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0 https://zenodo.org/records/10519652 - MedMNIST v2 (OrganCMNIST, 224px) Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0 https://zenodo.org/records/10519652 - Abdominal Ultrasound Image Dataset for Organ Classification and Disease Detection Sifat Zina Karim - CC-BY-4.0 https://scholarsjunction.msstate.edu/research-data/5/ - MedMNIST v2 (BreastMNIST, 224px) Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni - CC-BY-4.0 https://zenodo.org/records/10519652 ``` ## Author Arnab Pal — [LinkedIn](https://www.linkedin.com/in/pal-arnab) · [GitHub](https://github.com/palarnab)