| --- |
| library_name: transformers |
| license: other |
| license_name: dinov3-license |
| license_link: LICENSE.md |
| tags: |
| - medical |
| - radiology |
| - vision-language |
| - chest-x-ray |
| - glint |
| - custom_code |
| - arxiv:2606.03180 |
| base_model: facebook/dinov3-vitl16-pretrain-lvd1689m |
| --- |
| |
| # GLINT-CXR |
|
|
| <p> |
| 📝 <a href="https://arxiv.org/abs/2606.03180">Paper</a> • |
| 💻 <a href="https://github.com/deepnoid-ai/GLINT">Code</a> • |
| 🌐 <a href="https://deepnoid-ai.github.io/GLINT/">Project page</a> • |
| 🤗 <a href="https://huggingface.co/Deepnoid/GLINT-CT">GLINT-CT</a> |
| </p> |
|
|
| GLINT-CXR is the chest X-ray model of **GLINT** (*GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained |
| Radiology Representations*, NeurIPS 2026). |
|
|
| **Built with DINOv3.** |
|
|
| - **Vision encoder:** DINOv3 ViT-L/16, input 1024×1024 (4096 patches) |
| - **Text encoder:** `sentence-transformers/all-mpnet-base-v2` (fine-tuned) |
| - **Alignment:** linear projection (1024→768) + LayerNorm, mean-pooled text + LayerNorm, SGA similarity head |
| - **Training:** MIMIC-CXR, with Sparsely Gated Alignment (SGA) and Dense Feature Regularization (DFR) from a frozen |
| DINOv3 ViT-7B teacher |
|
|
| > ⚠️ GLINT is released for research purposes only and is not intended for clinical use, including the diagnosis or treatment of any medical condition. |
|
|
| ## Usage |
|
|
| Both examples use the bundled `inference.py` / `utils.py`; run them from the downloaded repo. |
|
|
| ### Full pipeline |
|
|
| ```bash |
| hf download Deepnoid/GLINT-CXR --local-dir glint-cxr && cd glint-cxr |
| python inference.py --image cxr.png --text "There is a right pleural effusion." |
| # prints the sentence probability and saves the heat-map overlay to similarity_map.png |
| ``` |
|
|
| ### AutoModel |
|
|
| ```python |
| import torch |
| from transformers import AutoModel, AutoTokenizer, AutoImageProcessor |
| from utils import model_inference, save_similarity_map # bundled helpers |
| |
| repo = "Deepnoid/GLINT-CXR" |
| device = "cuda" if torch.cuda.is_available() else "cpu" |
| |
| model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).to(device).eval() |
| tokenizer = AutoTokenizer.from_pretrained(repo) |
| image_processor = AutoImageProcessor.from_pretrained(repo) |
| |
| prob, similarity_map = model_inference( |
| "cxr.png", "There is a right pleural effusion.", |
| model=model, tokenizer=tokenizer, image_processor=image_processor, |
| ) |
| save_similarity_map("cxr.png", similarity_map, "similarity_map.png") |
| ``` |
|
|
| - **Outputs:** `prob` is the sentence probability (sigmoid); `similarity_map` is the per-pixel map in [0, 1]. |
| - **Inputs:** PNG / JPG (8- or 16-bit) or DICOM, min-max stretched to 8 bit by `utils.load_image` as in evaluation. |
| - **Underlying call:** `model.compute_logits(pixel_values, [encoded_text])` returns `logits` and `similarity_scores` |
| (64×64 patches). |
|
|
| Vision encoder only (`vision_only=True`): |
|
|
| ```python |
| import torch |
| from transformers import AutoModel, AutoImageProcessor |
| from utils import load_image # bundled helper |
| |
| repo = "Deepnoid/GLINT-CXR" |
| vision = AutoModel.from_pretrained(repo, trust_remote_code=True, vision_only=True, dtype=torch.float32).eval() |
| image_processor = AutoImageProcessor.from_pretrained(repo) |
| |
| pixel_values = image_processor(load_image("cxr.png"), return_tensors="pt")["pixel_values"] |
| with torch.no_grad(): |
| out = vision(pixel_values) |
| print(out.last_hidden_state.shape) # (1, 4101, 1024): CLS + 4 registers + 4096 patches |
| ``` |
|
|
| No gated DINOv3 download is needed; the model is built from this repo alone. |
|
|
| ## Requirements |
|
|
| `transformers==4.56.1`, `torch`, `torchvision`, `entmax`; `utils.py` also uses `opencv-python` (overlay) and `pydicom` |
| (DICOM). |
|
|
| > ⚠️ Keep `transformers==4.56.1`: 5.x silently skips the DINOv3 vision weights, leaving a random encoder. |
|
|
| ## Notes |
|
|
| - Training and fine-tuning code: [GitHub](https://github.com/deepnoid-ai/GLINT); split ID lists: [`data/`](./tree/main/data). |
| - `dfr.py` and its projector weights are used only in training; the DINOv3 ViT-7B teacher is not included. |
| - `vision_only=True` still downloads the full checkpoint; `save_pretrained` on the loaded model keeps a vision-only copy. |
|
|
| ## Acknowledgments |
|
|
| This work was supported by the Technology Innovation Program (RS-2025-02221011, |
| Development of Medical-Specialized Multimodal Hyperscale Generative AI Technology |
| for Global Integration) funded by the Ministry of Trade, Industry & Energy (MOTIE, |
| South Korea), and by the Advanced GPU Utilization Support Program funded by the |
| Korean government (Ministry of Science and ICT) (Project No. 02-26-91-0330). |
|
|
| ## License |
|
|
| - **Model weights:** derivative of Meta's [DINOv3](https://github.com/facebookresearch/dinov3) ViT-L/16 (LVD-1689M) |
| weights, © Meta Platforms, Inc., under the [DINOv3 License](LICENSE.md). |
| - Using or distributing them means accepting its terms; redistributions must include the license. |
| - **Code (`*.py`):** [CC BY-NC 4.0](LICENSES/CC-BY-NC-4.0.txt), as in the [GitHub repository](https://github.com/deepnoid-ai/GLINT). |
| - **Text encoder:** fine-tuned from |
| [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) |
| (Apache-2.0, modified; `LICENSES/Apache-2.0.txt`). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{park2026glint, |
| title = {{GLINT}: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations}, |
| author = {Park, Jonggwon and Lee, Seongeun and Park, Junhyun and |
| Yun, Hannah and Kim, Hyunwoong and Jeong, Sohyun and |
| Kang, Hyewon and Yoon, Byungmu and Choi, Kyoyun}, |
| booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems}, |
| year = {2026}, |
| url = {https://arxiv.org/abs/2606.03180} |
| } |
| ``` |
|
|
