Panel2Patch release: ViT-L/14 checkpoint and model card

README.md

@@ -0,0 +1,64 @@
+---
+license: mit
+library_name: open_clip
+pipeline_tag: zero-shot-image-classification
+tags:
+  - clip
+  - biomedical
+  - vision-language
+  - scientific-figures
+  - panel2patch
+---
+
+# Panel2Patch: From Panel to Pixel (ViT-L/14)
+
+The released vision-language model of **"From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature"**
+(CVPR 2026, [arXiv:2512.02566](https://arxiv.org/abs/2512.02566)). Code, data index and evaluation:
+[github.com/Flaick/Panel2Patch](https://github.com/Flaick/Panel2Patch).
+
+Panel2Patch turns biomedical figures from the literature into three levels of image-text supervision without human annotation
+(whole figure, sub-panel, marked region) and pretrains a CLIP model on all three. This checkpoint is the model reported as
+"Ours" in Table 3 of the paper: OpenCLIP `ViT-L-14`, initialised from BMC-CLIP, text tower frozen, the last 5 image blocks trainable,
+epoch 3 of the 20-epoch cosine schedule (run `unlock5_withrawdata`, every epoch of which is in
+[KKYYKK/biomedica_fg_models](https://huggingface.co/KKYYKK/biomedica_fg_models)).
+
+## Files
+
+| file | content |
+|---|---|
+| `panel2patch.pt` | `{"state_dict", "epoch": 3, "name"}` plus the training optimizer states; identical to `biomedica_fg_models/unlock5_withrawdata/epoch_3.pt` |
+
+## Loading
+
+```python
+import open_clip, torch
+model, _, preprocess = open_clip.create_model_and_transforms("ViT-L-14", pretrained=None)
+sd = torch.load("panel2patch.pt", map_location="cpu")["state_dict"]
+sd = {k[7:] if k.startswith("module.") else k: v for k, v in sd.items()}
+missing, unexpected = model.load_state_dict(sd, strict=False)   # training-only modules are ignored
+assert not missing
+tokenizer = open_clip.get_tokenizer("ViT-L-14")
+```
+
+## Zero-shot results (Table 3 of the paper, top-1 accuracy, %)
+
+| model | Biology | Dermatology | Microscopy | Ophthalmology | Pathology | Radiology | Avg |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| BMC-CLIP | 34.08 | 65.81 | 50.09 | 36.74 | 41.21 | 59.15 | 47.85 |
+| Panel2Patch (this checkpoint, paper) | 38.53 | 65.19 | 54.00 | 35.78 | 42.91 | 65.07 | 50.25 |
+| Panel2Patch (this checkpoint, re-evaluated with the released code) | 38.58 | 65.16 | 54.01 | 35.77 | 43.68 | 65.01 | 50.37 |
+
+The evaluation protocol (EVVLM closed VQA, two caption styles pooled) and the per-benchmark numbers are in the GitHub repository
+(`eval/zeroshot_eval.py`, `eval/table3.py`).
+
+## Citation
+
+```bibtex
+@inproceedings{yuan2026panel2pixel,
+  title     = {From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature},
+  author    = {Yuan, Kun and others},
+  booktitle = {CVPR},
+  year      = {2026},
+  note      = {arXiv:2512.02566}
+}
+```

panel2patch.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d3f310838284d36a88fd04872670849ad41da80dd7c09ced26631c66c6772d1c
+size 3222346361