File size: 7,498 Bytes
a41010e dba8722 a41010e dba8722 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 | ---
license: other
license_name: mixed-upstream-licenses
license_link: https://huggingface.co/QingboKang/SonoIntent-Models/blob/main/NOTICE.md
pipeline_tag: image-segmentation
tags:
- ultrasound
- medical-imaging
- reasoning-segmentation
- custom_code
---
# SonoIntent-Models
Fine-tuned checkpoints and inference code for two reasoning segmentation models
on **SonoIntent**, a clinical objective-driven ultrasound reasoning
segmentation benchmark.
## Benchmark
SonoIntent pairs each ultrasound image with a clinical question, a reference
reasoning that resolves the question to a target structure, and a binary target
mask. The full benchmark contains 130,928 image-question samples from 83,324
images across 30 public ultrasound sources. The paper's frozen test split has
22,870 tasks.
The public pixel release,
[`QingboKang/SonoIntent-Annotations`](https://huggingface.co/datasets/QingboKang/SonoIntent-Annotations),
contains 16,869 tasks from seven CC BY sources (HC18, PSFHS, BrEaST, CCAUI,
CUBS, FASS, Segthy). The six examples in `examples/cases/` come from its
HC18/PSFHS test rows.
## Models and paper results
Both checkpoints are the exact weights used for the paper's full 22,870-task
test run. No retraining, weight modification or result recomputation was done
for this release.
| Model | Checkpoint size | Overall Dice | TargetAcc | TA-Dice |
|---|---:|---:|---:|---:|
| VRS-HQ | 14.58 GB (bf16) | 50.82% | 40.18% | 33.50% |
| Sa2VA-14B | 30.74 GB (fp16) | 48.40% | 98.59% | 47.51% |
Overall Dice is computed over the full test split. TargetAcc and TA-Dice
follow the benchmark's target-accuracy and target-aware Dice definitions.
The six released examples are illustrative and are not a benchmark subset;
their Dice values are not comparable to the table above.
## Repository layout
```text
SonoIntent-Models/
βββ README.md
βββ NOTICE.md
βββ LICENSES/ # Apache-2.0 and Llama 2 Community License texts
βββ VRS-HQ/
β βββ checkpoint/ # fine-tuned HF checkpoint (config, tokenizer, 3 safetensors shards)
β βββ code/ # vendored model code needed to load the checkpoint
β βββ test.py
β βββ requirements.txt
β βββ README.md
βββ Sa2VA-14B/
β βββ checkpoint/ # fine-tuned HF checkpoint with remote code (7 safetensors shards)
β βββ test.py
β βββ requirements.txt
β βββ README.md
βββ examples/
βββ cases/example_001..006/{image.png,gt_mask.png,sample.json}
βββ outputs/{VRS-HQ,Sa2VA-14B}/example_00N/{predicted_mask.png,comparison.png,result.json,summary.json}
```
Each model directory is self-contained and can be loaded independently.
## Hardware and dependencies
Python 3.10+, PyTorch 2.1+ and the versions in each `requirements.txt`.
The release was validated with CPU inference (2x Intel Xeon 6530, 64 cores /
128 threads, 503 GB RAM; bf16 for VRS-HQ, fp16 for Sa2VA-14B). Both scripts accept CUDA
devices; the GPU paths are implemented but were not part of this validation.
Approximate CPU memory: VRS-HQ 15 GB, Sa2VA-14B 31 GB. A GPU with 24 GB
(VRS-HQ) or 40 GB (Sa2VA-14B, or two 24 GB GPUs via `--vision-device`/
`--llm-device`) is sufficient.
## Download
```bash
huggingface-cli download QingboKang/SonoIntent-Models --repo-type model --local-dir SonoIntent-Models
cd SonoIntent-Models
```
`examples/` is part of the repository, so no separate dataset download is
needed for the released cases. VRS-HQ also downloads
`openai/clip-vit-large-patch14` from the Hub at first run.
## Single example
```bash
# VRS-HQ, one case
python VRS-HQ/test.py --limit 1
# Sa2VA-14B, one case
python Sa2VA-14B/test.py --limit 1
```
Both commands default to the checkpoint in their own directory, the six cases
in `examples/cases`, and outputs in `examples/outputs/<model>`, independent of
the working directory.
Force CPU or choose devices explicitly:
```bash
python VRS-HQ/test.py --device cpu --dtype bfloat16 --threads 64 --limit 1
python Sa2VA-14B/test.py --vision-device cpu --llm-device cpu --dtype float16 --threads 64 --limit 1
```
## Batch over all six cases
```bash
python VRS-HQ/test.py \
--cases examples/cases \
--output examples/outputs/VRS-HQ
python Sa2VA-14B/test.py \
--cases examples/cases \
--output examples/outputs/Sa2VA-14B
```
`--limit N` restricts the run to the first N cases. `--max-new-tokens`
defaults to 384, matching the paper.
## Outputs
For every case the scripts write to `examples/outputs/<model>/<case>/`:
- `predicted_mask.png` β binary prediction at the original image resolution.
- `comparison.png` β four panels: ultrasound image, ground truth, predicted
mask, prediction overlay; the header repeats the clinical question and
target, the footer gives model, Dice and sample ID.
- `result.json` β sample_id, public_id, dataset, source, split, question,
target, generated text and reasoning, mask validity, Dice/IoU/precision/
recall, pixel counts and the input SHA-256 values.
A `summary.json` in the model output directory lists all cases and the mean
Dice over the released examples.
## Relation to the dataset
The six cases are real test rows of the frozen SonoIntent benchmark, taken
from the public HC18 and PSFHS test splits. `image.png` and `gt_mask.png` are
byte-identical to the released `SonoIntent-Annotations` assets; their SHA-256
values are in each `sample.json` and `image_index.csv`. The examples were
chosen before inference to cover fetal head and pubic symphysis, including one
PSFHS image with two different clinical questions and targets, and both
typical and difficult cases.
## Licenses and restrictions
See [NOTICE.md](NOTICE.md) for the per-component license table and
`LICENSES/` for the Apache-2.0 and Llama 2 Community License texts. In short:
- The fine-tuned weights are distributed under the licenses of their upstream
releases: VRS-HQ (Apache-2.0, with Chat-UniVi/Llama-2 components) and
Sa2VA-14B (Apache-2.0).
- The test scripts are original code. Vendored model code retains its upstream
license; the VRS-HQ GitHub repository does not declare a license, so that
component is included only to reproduce the published inference and remains
the property of its authors.
- The example images and masks are CC BY 4.0 (HC18, PSFHS) and retain their
source citations.
- Not a medical device; research use only.
## Citation
The SonoIntent paper citation will be added when it is available. Until then,
cite the dataset repository and the source datasets used by the examples:
```bibtex
@misc{sonointent2026,
title = {SonoIntent: Clinical Objective-Driven Ultrasound Reasoning Segmentation},
author = {Kang, Qingbo and others},
year = {2026},
note = {Dataset: https://huggingface.co/datasets/QingboKang/SonoIntent}
}
@article{vandenheuvel2018hc18,
title = {Automated measurement of fetal head circumference using 2D ultrasound images},
author = {van den Heuvel, Thomas L. A. and others},
journal = {PLoS ONE},
volume = {13},
number = {8},
pages = {e0200412},
year = {2018},
doi = {10.1371/journal.pone.0200412}
}
@article{lu2024psfhs,
title = {PSFHS: Intrapartum ultrasound image dataset for AI-based segmentation of pubic symphysis and fetal head},
author = {Lu, Yang and others},
journal = {Scientific Data},
volume = {11},
pages = {436},
year = {2024},
doi = {10.1038/s41597-024-03266-4}
}
```
|