--- license: other license_name: mixed-upstream-licenses license_link: https://huggingface.co/QingboKang/SonoIntent-Models/blob/main/NOTICE.md pipeline_tag: image-segmentation tags: - ultrasound - medical-imaging - reasoning-segmentation - custom_code --- # SonoIntent-Models Fine-tuned checkpoints and inference code for two reasoning segmentation models on **SonoIntent**, a clinical objective-driven ultrasound reasoning segmentation benchmark. ## Benchmark SonoIntent pairs each ultrasound image with a clinical question, a reference reasoning that resolves the question to a target structure, and a binary target mask. The full benchmark contains 130,928 image-question samples from 83,324 images across 30 public ultrasound sources. The paper's frozen test split has 22,870 tasks. The public pixel release, [`QingboKang/SonoIntent-Annotations`](https://huggingface.co/datasets/QingboKang/SonoIntent-Annotations), contains 16,869 tasks from seven CC BY sources (HC18, PSFHS, BrEaST, CCAUI, CUBS, FASS, Segthy). The six examples in `examples/cases/` come from its HC18/PSFHS test rows. ## Models and paper results Both checkpoints are the exact weights used for the paper's full 22,870-task test run. No retraining, weight modification or result recomputation was done for this release. | Model | Checkpoint size | Overall Dice | TargetAcc | TA-Dice | |---|---:|---:|---:|---:| | VRS-HQ | 14.58 GB (bf16) | 50.82% | 40.18% | 33.50% | | Sa2VA-14B | 30.74 GB (fp16) | 48.40% | 98.59% | 47.51% | Overall Dice is computed over the full test split. TargetAcc and TA-Dice follow the benchmark's target-accuracy and target-aware Dice definitions. The six released examples are illustrative and are not a benchmark subset; their Dice values are not comparable to the table above. ## Repository layout ```text SonoIntent-Models/ ├── README.md ├── NOTICE.md ├── LICENSES/ # Apache-2.0 and Llama 2 Community License texts ├── VRS-HQ/ │ ├── checkpoint/ # fine-tuned HF checkpoint (config, tokenizer, 3 safetensors shards) │ ├── code/ # vendored model code needed to load the checkpoint │ ├── test.py │ ├── requirements.txt │ └── README.md ├── Sa2VA-14B/ │ ├── checkpoint/ # fine-tuned HF checkpoint with remote code (7 safetensors shards) │ ├── test.py │ ├── requirements.txt │ └── README.md └── examples/ ├── cases/example_001..006/{image.png,gt_mask.png,sample.json} └── outputs/{VRS-HQ,Sa2VA-14B}/example_00N/{predicted_mask.png,comparison.png,result.json,summary.json} ``` Each model directory is self-contained and can be loaded independently. ## Hardware and dependencies Python 3.10+, PyTorch 2.1+ and the versions in each `requirements.txt`. The release was validated with CPU inference (2x Intel Xeon 6530, 64 cores / 128 threads, 503 GB RAM; bf16 for VRS-HQ, fp16 for Sa2VA-14B). Both scripts accept CUDA devices; the GPU paths are implemented but were not part of this validation. Approximate CPU memory: VRS-HQ 15 GB, Sa2VA-14B 31 GB. A GPU with 24 GB (VRS-HQ) or 40 GB (Sa2VA-14B, or two 24 GB GPUs via `--vision-device`/ `--llm-device`) is sufficient. ## Download ```bash huggingface-cli download QingboKang/SonoIntent-Models --repo-type model --local-dir SonoIntent-Models cd SonoIntent-Models ``` `examples/` is part of the repository, so no separate dataset download is needed for the released cases. VRS-HQ also downloads `openai/clip-vit-large-patch14` from the Hub at first run. ## Single example ```bash # VRS-HQ, one case python VRS-HQ/test.py --limit 1 # Sa2VA-14B, one case python Sa2VA-14B/test.py --limit 1 ``` Both commands default to the checkpoint in their own directory, the six cases in `examples/cases`, and outputs in `examples/outputs/`, independent of the working directory. Force CPU or choose devices explicitly: ```bash python VRS-HQ/test.py --device cpu --dtype bfloat16 --threads 64 --limit 1 python Sa2VA-14B/test.py --vision-device cpu --llm-device cpu --dtype float16 --threads 64 --limit 1 ``` ## Batch over all six cases ```bash python VRS-HQ/test.py \ --cases examples/cases \ --output examples/outputs/VRS-HQ python Sa2VA-14B/test.py \ --cases examples/cases \ --output examples/outputs/Sa2VA-14B ``` `--limit N` restricts the run to the first N cases. `--max-new-tokens` defaults to 384, matching the paper. ## Outputs For every case the scripts write to `examples/outputs///`: - `predicted_mask.png` — binary prediction at the original image resolution. - `comparison.png` — four panels: ultrasound image, ground truth, predicted mask, prediction overlay; the header repeats the clinical question and target, the footer gives model, Dice and sample ID. - `result.json` — sample_id, public_id, dataset, source, split, question, target, generated text and reasoning, mask validity, Dice/IoU/precision/ recall, pixel counts and the input SHA-256 values. A `summary.json` in the model output directory lists all cases and the mean Dice over the released examples. ## Relation to the dataset The six cases are real test rows of the frozen SonoIntent benchmark, taken from the public HC18 and PSFHS test splits. `image.png` and `gt_mask.png` are byte-identical to the released `SonoIntent-Annotations` assets; their SHA-256 values are in each `sample.json` and `image_index.csv`. The examples were chosen before inference to cover fetal head and pubic symphysis, including one PSFHS image with two different clinical questions and targets, and both typical and difficult cases. ## Licenses and restrictions See [NOTICE.md](NOTICE.md) for the per-component license table and `LICENSES/` for the Apache-2.0 and Llama 2 Community License texts. In short: - The fine-tuned weights are distributed under the licenses of their upstream releases: VRS-HQ (Apache-2.0, with Chat-UniVi/Llama-2 components) and Sa2VA-14B (Apache-2.0). - The test scripts are original code. Vendored model code retains its upstream license; the VRS-HQ GitHub repository does not declare a license, so that component is included only to reproduce the published inference and remains the property of its authors. - The example images and masks are CC BY 4.0 (HC18, PSFHS) and retain their source citations. - Not a medical device; research use only. ## Citation The SonoIntent paper citation will be added when it is available. Until then, cite the dataset repository and the source datasets used by the examples: ```bibtex @misc{sonointent2026, title = {SonoIntent: Clinical Objective-Driven Ultrasound Reasoning Segmentation}, author = {Kang, Qingbo and others}, year = {2026}, note = {Dataset: https://huggingface.co/datasets/QingboKang/SonoIntent} } @article{vandenheuvel2018hc18, title = {Automated measurement of fetal head circumference using 2D ultrasound images}, author = {van den Heuvel, Thomas L. A. and others}, journal = {PLoS ONE}, volume = {13}, number = {8}, pages = {e0200412}, year = {2018}, doi = {10.1371/journal.pone.0200412} } @article{lu2024psfhs, title = {PSFHS: Intrapartum ultrasound image dataset for AI-based segmentation of pubic symphysis and fetal head}, author = {Lu, Yang and others}, journal = {Scientific Data}, volume = {11}, pages = {436}, year = {2024}, doi = {10.1038/s41597-024-03266-4} } ```