Panel2Patch: From Panel to Pixel (ViT-L/14)

The released vision-language model of "From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature" (CVPR 2026, arXiv:2512.02566). Code, data index and evaluation: github.com/Flaick/Panel2Patch.

Panel2Patch turns biomedical figures from the literature into three levels of image-text supervision without human annotation (whole figure, sub-panel, marked region) and pretrains a CLIP model on all three. This checkpoint is the model reported as "Ours" in Table 3 of the paper: OpenCLIP ViT-L-14, initialised from BMC-CLIP, text tower frozen, the last 5 image blocks trainable, epoch 3 of the 20-epoch cosine schedule (run unlock5_withrawdata, every epoch of which is in KKYYKK/biomedica_fg_models).

Files

file	content
panel2patch.pt	{"state_dict", "epoch": 3, "name"} plus the training optimizer states; identical to biomedica_fg_models/unlock5_withrawdata/epoch_3.pt

Loading

import open_clip, torch
model, _, preprocess = open_clip.create_model_and_transforms("ViT-L-14", pretrained=None)
sd = torch.load("panel2patch.pt", map_location="cpu")["state_dict"]
sd = {k[7:] if k.startswith("module.") else k: v for k, v in sd.items()}
missing, unexpected = model.load_state_dict(sd, strict=False)   # training-only modules are ignored
assert not missing
tokenizer = open_clip.get_tokenizer("ViT-L-14")

Zero-shot results (Table 3 of the paper, top-1 accuracy, %)

model	Biology	Dermatology	Microscopy	Ophthalmology	Pathology	Radiology	Avg
BMC-CLIP	34.08	65.81	50.09	36.74	41.21	59.15	47.85
Panel2Patch (this checkpoint, paper)	38.53	65.19	54.00	35.78	42.91	65.07	50.25
Panel2Patch (this checkpoint, re-evaluated with the released code)	38.58	65.16	54.01	35.77	43.68	65.01	50.37

The evaluation protocol (EVVLM closed VQA, two caption styles pooled) and the per-benchmark numbers are in the GitHub repository (eval/zeroshot_eval.py, eval/table3.py).

Citation

@inproceedings{yuan2026panel2pixel,
  title     = {From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature},
  author    = {Yuan, Kun and others},
  booktitle = {CVPR},
  year      = {2026},
  note      = {arXiv:2512.02566}
}