SonoIntent-Models / README.md
QingboKang's picture
Add model cards, inference code and six example cases
dba8722 verified
|
Raw History Blame Contribute Delete
7.5 kB
---
license: other
license_name: mixed-upstream-licenses
license_link: https://huggingface.co/QingboKang/SonoIntent-Models/blob/main/NOTICE.md
pipeline_tag: image-segmentation
tags:
- ultrasound
- medical-imaging
- reasoning-segmentation
- custom_code
---
# SonoIntent-Models
Fine-tuned checkpoints and inference code for two reasoning segmentation models
on **SonoIntent**, a clinical objective-driven ultrasound reasoning
segmentation benchmark.
## Benchmark
SonoIntent pairs each ultrasound image with a clinical question, a reference
reasoning that resolves the question to a target structure, and a binary target
mask. The full benchmark contains 130,928 image-question samples from 83,324
images across 30 public ultrasound sources. The paper's frozen test split has
22,870 tasks.
The public pixel release,
[`QingboKang/SonoIntent-Annotations`](https://huggingface.co/datasets/QingboKang/SonoIntent-Annotations),
contains 16,869 tasks from seven CC BY sources (HC18, PSFHS, BrEaST, CCAUI,
CUBS, FASS, Segthy). The six examples in `examples/cases/` come from its
HC18/PSFHS test rows.
## Models and paper results
Both checkpoints are the exact weights used for the paper's full 22,870-task
test run. No retraining, weight modification or result recomputation was done
for this release.
| Model | Checkpoint size | Overall Dice | TargetAcc | TA-Dice |
|---|---:|---:|---:|---:|
| VRS-HQ | 14.58 GB (bf16) | 50.82% | 40.18% | 33.50% |
| Sa2VA-14B | 30.74 GB (fp16) | 48.40% | 98.59% | 47.51% |
Overall Dice is computed over the full test split. TargetAcc and TA-Dice
follow the benchmark's target-accuracy and target-aware Dice definitions.
The six released examples are illustrative and are not a benchmark subset;
their Dice values are not comparable to the table above.
## Repository layout
```text
SonoIntent-Models/
β”œβ”€β”€ README.md
β”œβ”€β”€ NOTICE.md
β”œβ”€β”€ LICENSES/ # Apache-2.0 and Llama 2 Community License texts
β”œβ”€β”€ VRS-HQ/
β”‚ β”œβ”€β”€ checkpoint/ # fine-tuned HF checkpoint (config, tokenizer, 3 safetensors shards)
β”‚ β”œβ”€β”€ code/ # vendored model code needed to load the checkpoint
β”‚ β”œβ”€β”€ test.py
β”‚ β”œβ”€β”€ requirements.txt
β”‚ └── README.md
β”œβ”€β”€ Sa2VA-14B/
β”‚ β”œβ”€β”€ checkpoint/ # fine-tuned HF checkpoint with remote code (7 safetensors shards)
β”‚ β”œβ”€β”€ test.py
β”‚ β”œβ”€β”€ requirements.txt
β”‚ └── README.md
└── examples/
β”œβ”€β”€ cases/example_001..006/{image.png,gt_mask.png,sample.json}
└── outputs/{VRS-HQ,Sa2VA-14B}/example_00N/{predicted_mask.png,comparison.png,result.json,summary.json}
```
Each model directory is self-contained and can be loaded independently.
## Hardware and dependencies
Python 3.10+, PyTorch 2.1+ and the versions in each `requirements.txt`.
The release was validated with CPU inference (2x Intel Xeon 6530, 64 cores /
128 threads, 503 GB RAM; bf16 for VRS-HQ, fp16 for Sa2VA-14B). Both scripts accept CUDA
devices; the GPU paths are implemented but were not part of this validation.
Approximate CPU memory: VRS-HQ 15 GB, Sa2VA-14B 31 GB. A GPU with 24 GB
(VRS-HQ) or 40 GB (Sa2VA-14B, or two 24 GB GPUs via `--vision-device`/
`--llm-device`) is sufficient.
## Download
```bash
huggingface-cli download QingboKang/SonoIntent-Models --repo-type model --local-dir SonoIntent-Models
cd SonoIntent-Models
```
`examples/` is part of the repository, so no separate dataset download is
needed for the released cases. VRS-HQ also downloads
`openai/clip-vit-large-patch14` from the Hub at first run.
## Single example
```bash
# VRS-HQ, one case
python VRS-HQ/test.py --limit 1
# Sa2VA-14B, one case
python Sa2VA-14B/test.py --limit 1
```
Both commands default to the checkpoint in their own directory, the six cases
in `examples/cases`, and outputs in `examples/outputs/<model>`, independent of
the working directory.
Force CPU or choose devices explicitly:
```bash
python VRS-HQ/test.py --device cpu --dtype bfloat16 --threads 64 --limit 1
python Sa2VA-14B/test.py --vision-device cpu --llm-device cpu --dtype float16 --threads 64 --limit 1
```
## Batch over all six cases
```bash
python VRS-HQ/test.py \
--cases examples/cases \
--output examples/outputs/VRS-HQ
python Sa2VA-14B/test.py \
--cases examples/cases \
--output examples/outputs/Sa2VA-14B
```
`--limit N` restricts the run to the first N cases. `--max-new-tokens`
defaults to 384, matching the paper.
## Outputs
For every case the scripts write to `examples/outputs/<model>/<case>/`:
- `predicted_mask.png` β€” binary prediction at the original image resolution.
- `comparison.png` β€” four panels: ultrasound image, ground truth, predicted
mask, prediction overlay; the header repeats the clinical question and
target, the footer gives model, Dice and sample ID.
- `result.json` β€” sample_id, public_id, dataset, source, split, question,
target, generated text and reasoning, mask validity, Dice/IoU/precision/
recall, pixel counts and the input SHA-256 values.
A `summary.json` in the model output directory lists all cases and the mean
Dice over the released examples.
## Relation to the dataset
The six cases are real test rows of the frozen SonoIntent benchmark, taken
from the public HC18 and PSFHS test splits. `image.png` and `gt_mask.png` are
byte-identical to the released `SonoIntent-Annotations` assets; their SHA-256
values are in each `sample.json` and `image_index.csv`. The examples were
chosen before inference to cover fetal head and pubic symphysis, including one
PSFHS image with two different clinical questions and targets, and both
typical and difficult cases.
## Licenses and restrictions
See [NOTICE.md](NOTICE.md) for the per-component license table and
`LICENSES/` for the Apache-2.0 and Llama 2 Community License texts. In short:
- The fine-tuned weights are distributed under the licenses of their upstream
releases: VRS-HQ (Apache-2.0, with Chat-UniVi/Llama-2 components) and
Sa2VA-14B (Apache-2.0).
- The test scripts are original code. Vendored model code retains its upstream
license; the VRS-HQ GitHub repository does not declare a license, so that
component is included only to reproduce the published inference and remains
the property of its authors.
- The example images and masks are CC BY 4.0 (HC18, PSFHS) and retain their
source citations.
- Not a medical device; research use only.
## Citation
The SonoIntent paper citation will be added when it is available. Until then,
cite the dataset repository and the source datasets used by the examples:
```bibtex
@misc{sonointent2026,
title = {SonoIntent: Clinical Objective-Driven Ultrasound Reasoning Segmentation},
author = {Kang, Qingbo and others},
year = {2026},
note = {Dataset: https://huggingface.co/datasets/QingboKang/SonoIntent}
}
@article{vandenheuvel2018hc18,
title = {Automated measurement of fetal head circumference using 2D ultrasound images},
author = {van den Heuvel, Thomas L. A. and others},
journal = {PLoS ONE},
volume = {13},
number = {8},
pages = {e0200412},
year = {2018},
doi = {10.1371/journal.pone.0200412}
}
@article{lu2024psfhs,
title = {PSFHS: Intrapartum ultrasound image dataset for AI-based segmentation of pubic symphysis and fetal head},
author = {Lu, Yang and others},
journal = {Scientific Data},
volume = {11},
pages = {436},
year = {2024},
doi = {10.1038/s41597-024-03266-4}
}
```