File size: 7,252 Bytes
6e93787
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
---
library_name: pytorch
pipeline_tag: visual-question-answering
license: llama3.2          # TODO: confirm (Llama-3.2 derivative; see Licensing)
language:
  - en
base_model:
  - meta-llama/Llama-3.2-1B
datasets:
  - devpatel1012/polished-vlm-dataset
tags:
  - medical
  - biomedical
  - surgical
  - visual-question-answering
  - vqa
  - multimodal
  - lora
  - cvt
  - clip-vit
---

# MedOptm — medical VLM checkpoints

Trained weights for **MedOptm**, a compact medical **Vision-Language Model**:
a frozen vision encoder feeds visual tokens through a **TokenPacker** projector
into **Llama-3.2-1B**, adapted with LoRA. You give it a medical/surgical image
and a question; it answers in natural language.

- 💻 **Code (required to load these weights):** <https://github.com/Devpatel1012/MedOptm-VLM>
- 🗂️ **Training dataset:** <https://huggingface.co/datasets/devpatel1012/polished-vlm-dataset>

> ⚠️ **Research use only — not for clinical use.** Answers are derived from
> biomedical figure captions, not written or reviewed by clinicians, and can be
> wrong. Do not use this to make or support medical decisions.

---

## Two models in this repo

Both are 1B-class and trained on the **same** curated Surgical-VQA data — they
differ **only in the vision encoder**, which is the whole point of the comparison.

| Model | Vision encoder | How the vision side is trained | Checkpoint prefix |
|---|---|---|---|
| **MedOptm-CvT** *(flagship)* | CvT `microsoft/cvt-w24-384-22k` | last encoder stage is **fully fine-tuned** | `phase1_alignment_*`, `phase2_vqa_lora_*` |
| **MedOptm-ViT** *(efficient)* | CLIP-ViT-L/14-336 | encoder adapted with **LoRA** (base frozen) | `clipvit_phase1_alignment_*`, `clipvit_phase2_vqa_lora_*` |

Each model was trained in **two phases**:

- **Phase 1 — `*_alignment_*`**: aligns the projector on image–caption pairs.
- **Phase 2 — `*_vqa_lora_*`**: instruction-tunes on VQA (adds LoRA to the LLM).

**Which file to use:** the **Phase 2 `*_vqa_lora_best.pt`** of whichever model you
want — it is the final, self-contained model (it already includes the aligned
projector from Phase 1). `*_periodic.pt` files are just resume snapshots; the
Phase 1 files are the intermediate alignment stage and are not needed for inference.

> These checkpoints store **only the trained parts** — the projector, the LoRA
> adapters (and, for the CvT model, the fine-tuned top encoder stage). The frozen
> base weights (`meta-llama/Llama-3.2-1B` and the vision encoder) are **not**
> included: they're downloaded separately when you build the model with the repo
> code.

---

## Results

On the curated Surgical-VQA held-out test split. Higher is better; MedOptm models
are **1B-class**.

| Model | Params | ROUGE-L | BLEU | METEOR | BERTScore-F1 |
|---|---|---|---|---|---|
| **MedOptm-CvT (Curated)** 🥇 | 1.58 B | **0.289** | **0.114** | **0.324** | **0.877** |
| MedOptm-ViT (Curated) 🥈 | 1.58 B | 0.282 | 0.112 | 0.298 | 0.875 |
| LLaVA-Med Mistral | 7.57 B | 0.219 | 0.025 | 0.267 | 0.869 |
| SmolVLM Instruct | 2.25 B | 0.147 | 0.015 | 0.155 | 0.864 |
| LLaVA-1.5 (general) | 7.06 B | 0.139 | 0.009 | 0.167 | 0.854 |

**Headline:** the 1.58 B MedOptm-CvT model **beats LLaVA-Med 7B** (0.289 vs 0.219
ROUGE-L, non-overlapping 95% CIs) at roughly **1/5 the size**.

### Efficiency

| Model | Peak VRAM | GFLOPs / sample | Tokens/s | ROUGE-L / GFLOP |
|---|---|---|---|---|
| MedOptm-CvT (Curated) | 10.28 GB | 1977 | 57.9 | 0.000146 |
| **MedOptm-ViT (Curated)** | **6.54 GB** | **619** | **68.7** | **0.000455** |

**Trade-off:** MedOptm-ViT gives up only ~2.6% ROUGE-L for **~1/3 the FLOPs**,
the lowest VRAM, and the highest throughput — i.e. it is ~3× more compute-efficient
per unit of quality. Pick **CvT for best quality**, **ViT for best efficiency**.

<sub>Energy/FLOP figures are estimated (constant-GPU-draw × wall-clock), not power-meter measured.</sub>

---

## Why the CvT checkpoint is much larger than the ViT one

The CvT model **fully fine-tunes** its top vision-encoder stage (hundreds of
millions of full-precision weights), whereas the ViT model adapts its encoder
with tiny **LoRA** layers. On top of that, these `.pt` files also store the
**AdamW optimizer state** (two fp32 momentum buffers per trained parameter),
which is the bulk of the size for the CvT model. That's why, e.g.,
`phase1_alignment_best.pt` (CvT) is ~2 GB while `clipvit_phase1_alignment_best.pt`
(ViT) is ~177 MB. The optimizer state is only needed to *resume* training — it can
be stripped for inference to shrink the files substantially.

---

## How to use

These are not standalone HF models; load them with the project code.

```bash
git clone https://github.com/Devpatel1012/MedOptm-VLM.git
cd MedOptm-VLM && pip install -e .
# download this checkpoints repo (e.g. with huggingface_hub) alongside it
```

```python
import torch
from medoptm.models.vit_vlm import ViTCustomVLM   # or: from medoptm.models.cvt_vlm import CustomVLM

model = ViTCustomVLM(
    hf_token="<your_hf_token>",
    vision_model_name="openai/clip-vit-large-patch14-336",
    llm_model_name="meta-llama/Llama-3.2-1B",
    image_size=336, scale_factor=2,
    vision_lora_r=16, vision_lora_alpha=32, vision_lora_dropout=0.05,
)
model.setup_llm_lora(lora_r=16, lora_alpha=32, lora_dropout=0.05)

ckpt = torch.load("clipvit_phase2_vqa_lora_best.pt", map_location="cpu", weights_only=True)
model.load_trainable_parameters(ckpt["model_state"])
model.eval()

answer = model.generate(images=[img], text_prompts=["Where were the pedicle screws placed?"])
```

For the CvT model, use `CustomVLM` with `vision_model_name="microsoft/cvt-w24-384-22k"`,
`scale_factor=1`, and `lora_r=64, lora_alpha=256, lora_dropout=0.10` (the
`cvt_curated` preset). **Match each checkpoint to the right class and config** —
the exact per-model settings are in `medoptm/config.py`.

---

## Training data

Trained on [`devpatel1012/polished-vlm-dataset`](https://huggingface.co/datasets/devpatel1012/polished-vlm-dataset)
— image–question–answer triples derived from **BIOMEDICA** figure captions
(BIOMEDICA is built from PubMed Central Open Access). See that dataset's card for
provenance and licensing.

## Limitations

- **Not clinically validated.** Research artifact only.
- Results are measured on this project's curated Surgical-VQA test split;
  numbers may not transfer to other data.
- Absolute metric values are modest (medical VQA is hard, references are short);
  the comparison is meaningful **relative** to the baselines on the same split.
- Inherits any biases/gaps of the underlying BIOMEDICA data (published figures
  skew toward illustrative cases).

## Licensing

These weights are a **derivative of `meta-llama/Llama-3.2-1B`**, so the **Llama 3.2
Community License** applies and the model must carry a **"Built with Llama"**
notice. The vision encoders (CvT / CLIP) and the training data carry their own
terms. Confirm and set the `license` field above accordingly before relying on it
commercially.

## Citation

```bibtex
@misc{medoptm2026,
  title  = {MedOptm: a compact medical vision-language model},
  author = {Dev Patel},
  year   = {2026},
  howpublished = {\url{https://github.com/Devpatel1012/MedOptm-VLM}}
}
```