| --- |
| license: mit |
| library_name: open_clip |
| pipeline_tag: zero-shot-image-classification |
| tags: |
| - clip |
| - biomedical |
| - vision-language |
| - scientific-figures |
| - panel2patch |
| --- |
| |
| # Panel2Patch: From Panel to Pixel (ViT-L/14) |
|
|
| The released vision-language model of **"From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature"** |
| (CVPR 2026, [arXiv:2512.02566](https://arxiv.org/abs/2512.02566)). Code, data index and evaluation: |
| [github.com/Flaick/Panel2Patch](https://github.com/Flaick/Panel2Patch). |
|
|
| Panel2Patch turns biomedical figures from the literature into three levels of image-text supervision without human annotation |
| (whole figure, sub-panel, marked region) and pretrains a CLIP model on all three. This checkpoint is the model reported as |
| "Ours" in Table 3 of the paper: OpenCLIP `ViT-L-14`, initialised from BMC-CLIP, text tower frozen, the last 5 image blocks trainable, |
| epoch 3 of the 20-epoch cosine schedule (run `unlock5_withrawdata`, every epoch of which is in |
| [KKYYKK/biomedica_fg_models](https://huggingface.co/KKYYKK/biomedica_fg_models)). |
|
|
| ## Files |
|
|
| | file | content | |
| |---|---| |
| | `panel2patch.pt` | `{"state_dict", "epoch": 3, "name"}` plus the training optimizer states; identical to `biomedica_fg_models/unlock5_withrawdata/epoch_3.pt` | |
|
|
| ## Loading |
|
|
| ```python |
| import open_clip, torch |
| model, _, preprocess = open_clip.create_model_and_transforms("ViT-L-14", pretrained=None) |
| sd = torch.load("panel2patch.pt", map_location="cpu")["state_dict"] |
| sd = {k[7:] if k.startswith("module.") else k: v for k, v in sd.items()} |
| missing, unexpected = model.load_state_dict(sd, strict=False) # training-only modules are ignored |
| assert not missing |
| tokenizer = open_clip.get_tokenizer("ViT-L-14") |
| ``` |
|
|
| ## Zero-shot results (Table 3 of the paper, top-1 accuracy, %) |
|
|
| | model | Biology | Dermatology | Microscopy | Ophthalmology | Pathology | Radiology | Avg | |
| |---|---:|---:|---:|---:|---:|---:|---:| |
| | BMC-CLIP | 34.08 | 65.81 | 50.09 | 36.74 | 41.21 | 59.15 | 47.85 | |
| | Panel2Patch (this checkpoint, paper) | 38.53 | 65.19 | 54.00 | 35.78 | 42.91 | 65.07 | 50.25 | |
| | Panel2Patch (this checkpoint, re-evaluated with the released code) | 38.58 | 65.16 | 54.01 | 35.77 | 43.68 | 65.01 | 50.37 | |
|
|
| The evaluation protocol (EVVLM closed VQA, two caption styles pooled) and the per-benchmark numbers are in the GitHub repository |
| (`eval/zeroshot_eval.py`, `eval/table3.py`). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{yuan2026panel2pixel, |
| title = {From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature}, |
| author = {Yuan, Kun and others}, |
| booktitle = {CVPR}, |
| year = {2026}, |
| note = {arXiv:2512.02566} |
| } |
| ``` |
|
|
