File size: 2,710 Bytes
d3b69a3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | ---
license: mit
library_name: open_clip
pipeline_tag: zero-shot-image-classification
tags:
- clip
- biomedical
- vision-language
- scientific-figures
- panel2patch
---
# Panel2Patch: From Panel to Pixel (ViT-L/14)
The released vision-language model of **"From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature"**
(CVPR 2026, [arXiv:2512.02566](https://arxiv.org/abs/2512.02566)). Code, data index and evaluation:
[github.com/Flaick/Panel2Patch](https://github.com/Flaick/Panel2Patch).
Panel2Patch turns biomedical figures from the literature into three levels of image-text supervision without human annotation
(whole figure, sub-panel, marked region) and pretrains a CLIP model on all three. This checkpoint is the model reported as
"Ours" in Table 3 of the paper: OpenCLIP `ViT-L-14`, initialised from BMC-CLIP, text tower frozen, the last 5 image blocks trainable,
epoch 3 of the 20-epoch cosine schedule (run `unlock5_withrawdata`, every epoch of which is in
[KKYYKK/biomedica_fg_models](https://huggingface.co/KKYYKK/biomedica_fg_models)).
## Files
| file | content |
|---|---|
| `panel2patch.pt` | `{"state_dict", "epoch": 3, "name"}` plus the training optimizer states; identical to `biomedica_fg_models/unlock5_withrawdata/epoch_3.pt` |
## Loading
```python
import open_clip, torch
model, _, preprocess = open_clip.create_model_and_transforms("ViT-L-14", pretrained=None)
sd = torch.load("panel2patch.pt", map_location="cpu")["state_dict"]
sd = {k[7:] if k.startswith("module.") else k: v for k, v in sd.items()}
missing, unexpected = model.load_state_dict(sd, strict=False) # training-only modules are ignored
assert not missing
tokenizer = open_clip.get_tokenizer("ViT-L-14")
```
## Zero-shot results (Table 3 of the paper, top-1 accuracy, %)
| model | Biology | Dermatology | Microscopy | Ophthalmology | Pathology | Radiology | Avg |
|---|---:|---:|---:|---:|---:|---:|---:|
| BMC-CLIP | 34.08 | 65.81 | 50.09 | 36.74 | 41.21 | 59.15 | 47.85 |
| Panel2Patch (this checkpoint, paper) | 38.53 | 65.19 | 54.00 | 35.78 | 42.91 | 65.07 | 50.25 |
| Panel2Patch (this checkpoint, re-evaluated with the released code) | 38.58 | 65.16 | 54.01 | 35.77 | 43.68 | 65.01 | 50.37 |
The evaluation protocol (EVVLM closed VQA, two caption styles pooled) and the per-benchmark numbers are in the GitHub repository
(`eval/zeroshot_eval.py`, `eval/table3.py`).
## Citation
```bibtex
@inproceedings{yuan2026panel2pixel,
title = {From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature},
author = {Yuan, Kun and others},
booktitle = {CVPR},
year = {2026},
note = {arXiv:2512.02566}
}
```
|