File size: 7,252 Bytes
6e93787 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 | ---
library_name: pytorch
pipeline_tag: visual-question-answering
license: llama3.2 # TODO: confirm (Llama-3.2 derivative; see Licensing)
language:
- en
base_model:
- meta-llama/Llama-3.2-1B
datasets:
- devpatel1012/polished-vlm-dataset
tags:
- medical
- biomedical
- surgical
- visual-question-answering
- vqa
- multimodal
- lora
- cvt
- clip-vit
---
# MedOptm — medical VLM checkpoints
Trained weights for **MedOptm**, a compact medical **Vision-Language Model**:
a frozen vision encoder feeds visual tokens through a **TokenPacker** projector
into **Llama-3.2-1B**, adapted with LoRA. You give it a medical/surgical image
and a question; it answers in natural language.
- 💻 **Code (required to load these weights):** <https://github.com/Devpatel1012/MedOptm-VLM>
- 🗂️ **Training dataset:** <https://huggingface.co/datasets/devpatel1012/polished-vlm-dataset>
> ⚠️ **Research use only — not for clinical use.** Answers are derived from
> biomedical figure captions, not written or reviewed by clinicians, and can be
> wrong. Do not use this to make or support medical decisions.
---
## Two models in this repo
Both are 1B-class and trained on the **same** curated Surgical-VQA data — they
differ **only in the vision encoder**, which is the whole point of the comparison.
| Model | Vision encoder | How the vision side is trained | Checkpoint prefix |
|---|---|---|---|
| **MedOptm-CvT** *(flagship)* | CvT `microsoft/cvt-w24-384-22k` | last encoder stage is **fully fine-tuned** | `phase1_alignment_*`, `phase2_vqa_lora_*` |
| **MedOptm-ViT** *(efficient)* | CLIP-ViT-L/14-336 | encoder adapted with **LoRA** (base frozen) | `clipvit_phase1_alignment_*`, `clipvit_phase2_vqa_lora_*` |
Each model was trained in **two phases**:
- **Phase 1 — `*_alignment_*`**: aligns the projector on image–caption pairs.
- **Phase 2 — `*_vqa_lora_*`**: instruction-tunes on VQA (adds LoRA to the LLM).
**Which file to use:** the **Phase 2 `*_vqa_lora_best.pt`** of whichever model you
want — it is the final, self-contained model (it already includes the aligned
projector from Phase 1). `*_periodic.pt` files are just resume snapshots; the
Phase 1 files are the intermediate alignment stage and are not needed for inference.
> These checkpoints store **only the trained parts** — the projector, the LoRA
> adapters (and, for the CvT model, the fine-tuned top encoder stage). The frozen
> base weights (`meta-llama/Llama-3.2-1B` and the vision encoder) are **not**
> included: they're downloaded separately when you build the model with the repo
> code.
---
## Results
On the curated Surgical-VQA held-out test split. Higher is better; MedOptm models
are **1B-class**.
| Model | Params | ROUGE-L | BLEU | METEOR | BERTScore-F1 |
|---|---|---|---|---|---|
| **MedOptm-CvT (Curated)** 🥇 | 1.58 B | **0.289** | **0.114** | **0.324** | **0.877** |
| MedOptm-ViT (Curated) 🥈 | 1.58 B | 0.282 | 0.112 | 0.298 | 0.875 |
| LLaVA-Med Mistral | 7.57 B | 0.219 | 0.025 | 0.267 | 0.869 |
| SmolVLM Instruct | 2.25 B | 0.147 | 0.015 | 0.155 | 0.864 |
| LLaVA-1.5 (general) | 7.06 B | 0.139 | 0.009 | 0.167 | 0.854 |
**Headline:** the 1.58 B MedOptm-CvT model **beats LLaVA-Med 7B** (0.289 vs 0.219
ROUGE-L, non-overlapping 95% CIs) at roughly **1/5 the size**.
### Efficiency
| Model | Peak VRAM | GFLOPs / sample | Tokens/s | ROUGE-L / GFLOP |
|---|---|---|---|---|
| MedOptm-CvT (Curated) | 10.28 GB | 1977 | 57.9 | 0.000146 |
| **MedOptm-ViT (Curated)** | **6.54 GB** | **619** | **68.7** | **0.000455** |
**Trade-off:** MedOptm-ViT gives up only ~2.6% ROUGE-L for **~1/3 the FLOPs**,
the lowest VRAM, and the highest throughput — i.e. it is ~3× more compute-efficient
per unit of quality. Pick **CvT for best quality**, **ViT for best efficiency**.
<sub>Energy/FLOP figures are estimated (constant-GPU-draw × wall-clock), not power-meter measured.</sub>
---
## Why the CvT checkpoint is much larger than the ViT one
The CvT model **fully fine-tunes** its top vision-encoder stage (hundreds of
millions of full-precision weights), whereas the ViT model adapts its encoder
with tiny **LoRA** layers. On top of that, these `.pt` files also store the
**AdamW optimizer state** (two fp32 momentum buffers per trained parameter),
which is the bulk of the size for the CvT model. That's why, e.g.,
`phase1_alignment_best.pt` (CvT) is ~2 GB while `clipvit_phase1_alignment_best.pt`
(ViT) is ~177 MB. The optimizer state is only needed to *resume* training — it can
be stripped for inference to shrink the files substantially.
---
## How to use
These are not standalone HF models; load them with the project code.
```bash
git clone https://github.com/Devpatel1012/MedOptm-VLM.git
cd MedOptm-VLM && pip install -e .
# download this checkpoints repo (e.g. with huggingface_hub) alongside it
```
```python
import torch
from medoptm.models.vit_vlm import ViTCustomVLM # or: from medoptm.models.cvt_vlm import CustomVLM
model = ViTCustomVLM(
hf_token="<your_hf_token>",
vision_model_name="openai/clip-vit-large-patch14-336",
llm_model_name="meta-llama/Llama-3.2-1B",
image_size=336, scale_factor=2,
vision_lora_r=16, vision_lora_alpha=32, vision_lora_dropout=0.05,
)
model.setup_llm_lora(lora_r=16, lora_alpha=32, lora_dropout=0.05)
ckpt = torch.load("clipvit_phase2_vqa_lora_best.pt", map_location="cpu", weights_only=True)
model.load_trainable_parameters(ckpt["model_state"])
model.eval()
answer = model.generate(images=[img], text_prompts=["Where were the pedicle screws placed?"])
```
For the CvT model, use `CustomVLM` with `vision_model_name="microsoft/cvt-w24-384-22k"`,
`scale_factor=1`, and `lora_r=64, lora_alpha=256, lora_dropout=0.10` (the
`cvt_curated` preset). **Match each checkpoint to the right class and config** —
the exact per-model settings are in `medoptm/config.py`.
---
## Training data
Trained on [`devpatel1012/polished-vlm-dataset`](https://huggingface.co/datasets/devpatel1012/polished-vlm-dataset)
— image–question–answer triples derived from **BIOMEDICA** figure captions
(BIOMEDICA is built from PubMed Central Open Access). See that dataset's card for
provenance and licensing.
## Limitations
- **Not clinically validated.** Research artifact only.
- Results are measured on this project's curated Surgical-VQA test split;
numbers may not transfer to other data.
- Absolute metric values are modest (medical VQA is hard, references are short);
the comparison is meaningful **relative** to the baselines on the same split.
- Inherits any biases/gaps of the underlying BIOMEDICA data (published figures
skew toward illustrative cases).
## Licensing
These weights are a **derivative of `meta-llama/Llama-3.2-1B`**, so the **Llama 3.2
Community License** applies and the model must carry a **"Built with Llama"**
notice. The vision encoders (CvT / CLIP) and the training data carry their own
terms. Confirm and set the `license` field above accordingly before relying on it
commercially.
## Citation
```bibtex
@misc{medoptm2026,
title = {MedOptm: a compact medical vision-language model},
author = {Dev Patel},
year = {2026},
howpublished = {\url{https://github.com/Devpatel1012/MedOptm-VLM}}
}
```
|