---
license: mit
library_name: open_clip
pipeline_tag: zero-shot-image-classification
tags:
  - clip
  - biomedical
  - vision-language
  - scientific-figures
  - panel2patch
---

# Panel2Patch: From Panel to Pixel (ViT-L/14)

The released vision-language model of **"From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature"**
(CVPR 2026, [arXiv:2512.02566](https://arxiv.org/abs/2512.02566)). Code, data index and evaluation:
[github.com/Flaick/Panel2Patch](https://github.com/Flaick/Panel2Patch).

Panel2Patch turns biomedical figures from the literature into three levels of image-text supervision without human annotation
(whole figure, sub-panel, marked region) and pretrains a CLIP model on all three. This checkpoint is the model reported as
"Ours" in Table 3 of the paper: OpenCLIP `ViT-L-14`, initialised from BMC-CLIP, text tower frozen, the last 5 image blocks trainable,
epoch 3 of the 20-epoch cosine schedule (run `unlock5_withrawdata`, every epoch of which is in
[KKYYKK/biomedica_fg_models](https://huggingface.co/KKYYKK/biomedica_fg_models)).

## Files

| file | content |
|---|---|
| `panel2patch.pt` | `{"state_dict", "epoch": 3, "name"}` plus the training optimizer states; identical to `biomedica_fg_models/unlock5_withrawdata/epoch_3.pt` |

## Loading

```python
import open_clip, torch
model, _, preprocess = open_clip.create_model_and_transforms("ViT-L-14", pretrained=None)
sd = torch.load("panel2patch.pt", map_location="cpu")["state_dict"]
sd = {k[7:] if k.startswith("module.") else k: v for k, v in sd.items()}
missing, unexpected = model.load_state_dict(sd, strict=False)   # training-only modules are ignored
assert not missing
tokenizer = open_clip.get_tokenizer("ViT-L-14")
```

## Zero-shot results (Table 3 of the paper, top-1 accuracy, %)

| model | Biology | Dermatology | Microscopy | Ophthalmology | Pathology | Radiology | Avg |
|---|---:|---:|---:|---:|---:|---:|---:|
| BMC-CLIP | 34.08 | 65.81 | 50.09 | 36.74 | 41.21 | 59.15 | 47.85 |
| Panel2Patch (this checkpoint, paper) | 38.53 | 65.19 | 54.00 | 35.78 | 42.91 | 65.07 | 50.25 |
| Panel2Patch (this checkpoint, re-evaluated with the released code) | 38.58 | 65.16 | 54.01 | 35.77 | 43.68 | 65.01 | 50.37 |

The evaluation protocol (EVVLM closed VQA, two caption styles pooled) and the per-benchmark numbers are in the GitHub repository
(`eval/zeroshot_eval.py`, `eval/table3.py`).

## Citation

```bibtex
@inproceedings{yuan2026panel2pixel,
  title     = {From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature},
  author    = {Yuan, Kun and others},
  booktitle = {CVPR},
  year      = {2026},
  note      = {arXiv:2512.02566}
}
```