Blame for README.md

---
library_name: transformers
license: other
license_name: dinov3-license
license_link: LICENSE.md
tags:
- medical
- radiology
- vision-language
- chest-x-ray
- glint
- custom_code
- arxiv:2606.03180
base_model: facebook/dinov3-vitl16-pretrain-lvd1689m
---

# GLINT-CXR

<p>
📝 <a href="https://arxiv.org/abs/2606.03180">Paper</a> •
💻 <a href="https://github.com/deepnoid-ai/GLINT">Code</a> •
🌐 <a href="https://deepnoid-ai.github.io/GLINT/">Project page</a> •
🤗 <a href="https://huggingface.co/Deepnoid/GLINT-CT">GLINT-CT</a>
</p>

GLINT-CXR is the chest X-ray model of **GLINT** (*GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained
Radiology Representations*, NeurIPS 2026).

**Built with DINOv3.**

- **Vision encoder:** DINOv3 ViT-L/16, input 1024×1024 (4096 patches)
- **Text encoder:** `sentence-transformers/all-mpnet-base-v2` (fine-tuned)
- **Alignment:** linear projection (1024→768) + LayerNorm, mean-pooled text + LayerNorm, SGA similarity head
- **Training:** MIMIC-CXR, with Sparsely Gated Alignment (SGA) and Dense Feature Regularization (DFR) from a frozen
  DINOv3 ViT-7B teacher

> ⚠️ GLINT is released for research purposes only and is not intended for clinical use, including the diagnosis or treatment of any medical condition.

## Usage

Both examples use the bundled `inference.py` / `utils.py`; run them from the downloaded repo.

### Full pipeline

```bash
hf download Deepnoid/GLINT-CXR --local-dir glint-cxr && cd glint-cxr
python inference.py --image cxr.png --text "There is a right pleural effusion."
# prints the sentence probability and saves the heat-map overlay to similarity_map.png
```

### AutoModel

```python
import torch
from transformers import AutoModel, AutoTokenizer, AutoImageProcessor
from utils import model_inference, save_similarity_map   # bundled helpers

repo = "Deepnoid/GLINT-CXR"
device = "cuda" if torch.cuda.is_available() else "cpu"

model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).to(device).eval()
tokenizer = AutoTokenizer.from_pretrained(repo)
image_processor = AutoImageProcessor.from_pretrained(repo)

prob, similarity_map = model_inference(
    "cxr.png", "There is a right pleural effusion.",
    model=model, tokenizer=tokenizer, image_processor=image_processor,
)
save_similarity_map("cxr.png", similarity_map, "similarity_map.png")
```

- **Outputs:** `prob` is the sentence probability (sigmoid); `similarity_map` is the per-pixel map in [0, 1].
- **Inputs:** PNG / JPG (8- or 16-bit) or DICOM, min-max stretched to 8 bit by `utils.load_image` as in evaluation.
- **Underlying call:** `model.compute_logits(pixel_values, [encoded_text])` returns `logits` and `similarity_scores`
  (64×64 patches).

Vision encoder only (`vision_only=True`):

```python
import torch
from transformers import AutoModel, AutoImageProcessor
from utils import load_image   # bundled helper

repo = "Deepnoid/GLINT-CXR"
vision = AutoModel.from_pretrained(repo, trust_remote_code=True, vision_only=True, dtype=torch.float32).eval()
image_processor = AutoImageProcessor.from_pretrained(repo)

pixel_values = image_processor(load_image("cxr.png"), return_tensors="pt")["pixel_values"]
with torch.no_grad():
    out = vision(pixel_values)
print(out.last_hidden_state.shape)    # (1, 4101, 1024): CLS + 4 registers + 4096 patches
```

No gated DINOv3 download is needed; the model is built from this repo alone.

## Requirements

`transformers==4.56.1`, `torch`, `torchvision`, `entmax`; `utils.py` also uses `opencv-python` (overlay) and `pydicom`
(DICOM).

> ⚠️ Keep `transformers==4.56.1`: 5.x silently skips the DINOv3 vision weights, leaving a random encoder.

## Notes

- Training and fine-tuning code: [GitHub](https://github.com/deepnoid-ai/GLINT); split ID lists: [`data/`](./tree/main/data).
- `dfr.py` and its projector weights are used only in training; the DINOv3 ViT-7B teacher is not included.
- `vision_only=True` still downloads the full checkpoint; `save_pretrained` on the loaded model keeps a vision-only copy.

## Acknowledgments

This work was supported by the Technology Innovation Program (RS-2025-02221011,
Development of Medical-Specialized Multimodal Hyperscale Generative AI Technology
for Global Integration) funded by the Ministry of Trade, Industry & Energy (MOTIE,
South Korea), and by the Advanced GPU Utilization Support Program funded by the
Korean government (Ministry of Science and ICT) (Project No. 02-26-91-0330).

## License

- **Model weights:** derivative of Meta's [DINOv3](https://github.com/facebookresearch/dinov3) ViT-L/16 (LVD-1689M)
  weights, © Meta Platforms, Inc., under the [DINOv3 License](LICENSE.md).
  - Using or distributing them means accepting its terms; redistributions must include the license.
- **Code (`*.py`):** [CC BY-NC 4.0](LICENSES/CC-BY-NC-4.0.txt), as in the [GitHub repository](https://github.com/deepnoid-ai/GLINT).
- **Text encoder:** fine-tuned from
  [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)
  (Apache-2.0, modified; `LICENSES/Apache-2.0.txt`).

## Citation

```bibtex
@inproceedings{park2026glint,
  title     = {{GLINT}: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations},
  author    = {Park, Jonggwon and Lee, Seongeun and Park, Junhyun and
               Yun, Hannah and Kim, Hyunwoong and Jeong, Sohyun and
               Kang, Hyewon and Yoon, Byungmu and Choi, Kyoyun},
  booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
  year      = {2026},
  url       = {https://arxiv.org/abs/2606.03180}
}
```