---
library_name: transformers
pipeline_tag: image-feature-extraction
base_model: facebook/dinov3-vitb16-pretrain-lvd1689m
license: other
license_name: dinov3-license
license_link: https://github.com/facebookresearch/dinov3/blob/main/LICENSE.md
tags:
  - medical-imaging
  - hand-xray
  - foundation-model
  - neurips-2026
---

# HandXFM

**HandXFM: Semantic-Structural Distillation for Hand Radiograph Foundation Models**  
**Accepted at NeurIPS 2026** · [Paper](https://neurips.cc/virtual/2026/loc/sydney/poster/150797)

HandXFM is a foundation model for hand X-ray analysis. Built on DINOv3 ViT-B/16, it learns transferable visual representations through semantic-structural distillation from BiomedCLIP and a chest X-ray ViT. The model can serve as a backbone for bone age estimation, rheumatoid arthritis scoring, abnormality classification, joint detection, bone segmentation, and visual question answering.

This repository provides the **pretrained HandXFM vision encoder** (without the pretraining teacher models or downstream task-specific heads).

## Usage

```python
import torch
from PIL import Image
from transformers import AutoImageProcessor, AutoModel

model_id = "Yuesai/HandXFM"
processor = AutoImageProcessor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id).eval()

image = Image.open("hand_xray.png").convert("RGB")
inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    tokens = model(**inputs).last_hidden_state

# DINOv3 token order: [CLS, register tokens, patch tokens]
num_register_tokens = getattr(model.config, "num_register_tokens", 0)
cls_embedding = tokens[:, 0, :]                         # [B, 768]
patch_embeddings = tokens[:, 1 + num_register_tokens:, :]  # [B, 196, 768] for 224x224

# Optional: mean-pool CLS + patch tokens (excluding register tokens)
features = torch.cat([cls_embedding.unsqueeze(1), patch_embeddings], dim=1)
pooled_embedding = features.mean(dim=1)                  # [B, 768]

print("CLS:", cls_embedding.shape)
print("Patches:", patch_embeddings.shape)
print("Pooled:", pooled_embedding.shape)
```

Use `cls_embedding` for global image features, `patch_embeddings` for spatial features, or `pooled_embedding` for a single pooled representation. Register tokens are excluded from patch features.

**Intended use:** Research on hand radiographs. Not validated for direct clinical diagnosis or treatment decisions. The model weights are distributed subject to the [DINOv3 License](https://github.com/facebookresearch/dinov3/blob/main/LICENSE.md); see the accompanying license file.

## Citation

If you use HandXFM in your research, please cite:

```bibtex
@inproceedings{long2026handxfm,
  title     = {{HandXFM}: Semantic-Structural Distillation for Hand Radiograph Foundation Models},
  author    = {Long, Yuxi and Feng, Ganlin and Chen, Lianghong and O'Neil, Liam J. and Hitchon, Carol A. and Hu, Pingzhao},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  url       = {https://neurips.cc/virtual/2026/loc/sydney/poster/150797}
}
```