| --- |
| library_name: transformers |
| pipeline_tag: image-feature-extraction |
| base_model: facebook/dinov3-vitb16-pretrain-lvd1689m |
| license: other |
| license_name: dinov3-license |
| license_link: https://github.com/facebookresearch/dinov3/blob/main/LICENSE.md |
| tags: |
| - medical-imaging |
| - hand-xray |
| - foundation-model |
| - neurips-2026 |
| --- |
| |
| # HandXFM |
|
|
| **HandXFM: Semantic-Structural Distillation for Hand Radiograph Foundation Models** |
| **Accepted at NeurIPS 2026** · [Paper](https://neurips.cc/virtual/2026/loc/sydney/poster/150797) |
|
|
| HandXFM is a foundation model for hand X-ray analysis. Built on DINOv3 ViT-B/16, it learns transferable visual representations through semantic-structural distillation from BiomedCLIP and a chest X-ray ViT. The model can serve as a backbone for bone age estimation, rheumatoid arthritis scoring, abnormality classification, joint detection, bone segmentation, and visual question answering. |
|
|
| This repository provides the **pretrained HandXFM vision encoder** (without the pretraining teacher models or downstream task-specific heads). |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from PIL import Image |
| from transformers import AutoImageProcessor, AutoModel |
| |
| model_id = "Yuesai/HandXFM" |
| processor = AutoImageProcessor.from_pretrained(model_id) |
| model = AutoModel.from_pretrained(model_id).eval() |
| |
| image = Image.open("hand_xray.png").convert("RGB") |
| inputs = processor(images=image, return_tensors="pt") |
| |
| with torch.no_grad(): |
| tokens = model(**inputs).last_hidden_state |
| |
| # DINOv3 token order: [CLS, register tokens, patch tokens] |
| num_register_tokens = getattr(model.config, "num_register_tokens", 0) |
| cls_embedding = tokens[:, 0, :] # [B, 768] |
| patch_embeddings = tokens[:, 1 + num_register_tokens:, :] # [B, 196, 768] for 224x224 |
| |
| # Optional: mean-pool CLS + patch tokens (excluding register tokens) |
| features = torch.cat([cls_embedding.unsqueeze(1), patch_embeddings], dim=1) |
| pooled_embedding = features.mean(dim=1) # [B, 768] |
| |
| print("CLS:", cls_embedding.shape) |
| print("Patches:", patch_embeddings.shape) |
| print("Pooled:", pooled_embedding.shape) |
| ``` |
|
|
| Use `cls_embedding` for global image features, `patch_embeddings` for spatial features, or `pooled_embedding` for a single pooled representation. Register tokens are excluded from patch features. |
|
|
| **Intended use:** Research on hand radiographs. Not validated for direct clinical diagnosis or treatment decisions. The model weights are distributed subject to the [DINOv3 License](https://github.com/facebookresearch/dinov3/blob/main/LICENSE.md); see the accompanying license file. |
|
|
| ## Citation |
|
|
| If you use HandXFM in your research, please cite: |
|
|
| ```bibtex |
| @inproceedings{long2026handxfm, |
| title = {{HandXFM}: Semantic-Structural Distillation for Hand Radiograph Foundation Models}, |
| author = {Long, Yuxi and Feng, Ganlin and Chen, Lianghong and O'Neil, Liam J. and Hitchon, Carol A. and Hu, Pingzhao}, |
| booktitle = {Advances in Neural Information Processing Systems}, |
| year = {2026}, |
| url = {https://neurips.cc/virtual/2026/loc/sydney/poster/150797} |
| } |
| ``` |
|
|
