MedOptm-Checkpoints / README.md
devpatel1012's picture
Upload README.md with huggingface_hub
6e93787 verified
|
Raw History Blame Contribute Delete
7.25 kB
---
library_name: pytorch
pipeline_tag: visual-question-answering
license: llama3.2 # TODO: confirm (Llama-3.2 derivative; see Licensing)
language:
- en
base_model:
- meta-llama/Llama-3.2-1B
datasets:
- devpatel1012/polished-vlm-dataset
tags:
- medical
- biomedical
- surgical
- visual-question-answering
- vqa
- multimodal
- lora
- cvt
- clip-vit
---
# MedOptm — medical VLM checkpoints
Trained weights for **MedOptm**, a compact medical **Vision-Language Model**:
a frozen vision encoder feeds visual tokens through a **TokenPacker** projector
into **Llama-3.2-1B**, adapted with LoRA. You give it a medical/surgical image
and a question; it answers in natural language.
- 💻 **Code (required to load these weights):** <https://github.com/Devpatel1012/MedOptm-VLM>
- 🗂️ **Training dataset:** <https://huggingface.co/datasets/devpatel1012/polished-vlm-dataset>
> ⚠️ **Research use only — not for clinical use.** Answers are derived from
> biomedical figure captions, not written or reviewed by clinicians, and can be
> wrong. Do not use this to make or support medical decisions.
---
## Two models in this repo
Both are 1B-class and trained on the **same** curated Surgical-VQA data — they
differ **only in the vision encoder**, which is the whole point of the comparison.
| Model | Vision encoder | How the vision side is trained | Checkpoint prefix |
|---|---|---|---|
| **MedOptm-CvT** *(flagship)* | CvT `microsoft/cvt-w24-384-22k` | last encoder stage is **fully fine-tuned** | `phase1_alignment_*`, `phase2_vqa_lora_*` |
| **MedOptm-ViT** *(efficient)* | CLIP-ViT-L/14-336 | encoder adapted with **LoRA** (base frozen) | `clipvit_phase1_alignment_*`, `clipvit_phase2_vqa_lora_*` |
Each model was trained in **two phases**:
- **Phase 1 — `*_alignment_*`**: aligns the projector on image–caption pairs.
- **Phase 2 — `*_vqa_lora_*`**: instruction-tunes on VQA (adds LoRA to the LLM).
**Which file to use:** the **Phase 2 `*_vqa_lora_best.pt`** of whichever model you
want — it is the final, self-contained model (it already includes the aligned
projector from Phase 1). `*_periodic.pt` files are just resume snapshots; the
Phase 1 files are the intermediate alignment stage and are not needed for inference.
> These checkpoints store **only the trained parts** — the projector, the LoRA
> adapters (and, for the CvT model, the fine-tuned top encoder stage). The frozen
> base weights (`meta-llama/Llama-3.2-1B` and the vision encoder) are **not**
> included: they're downloaded separately when you build the model with the repo
> code.
---
## Results
On the curated Surgical-VQA held-out test split. Higher is better; MedOptm models
are **1B-class**.
| Model | Params | ROUGE-L | BLEU | METEOR | BERTScore-F1 |
|---|---|---|---|---|---|
| **MedOptm-CvT (Curated)** 🥇 | 1.58 B | **0.289** | **0.114** | **0.324** | **0.877** |
| MedOptm-ViT (Curated) 🥈 | 1.58 B | 0.282 | 0.112 | 0.298 | 0.875 |
| LLaVA-Med Mistral | 7.57 B | 0.219 | 0.025 | 0.267 | 0.869 |
| SmolVLM Instruct | 2.25 B | 0.147 | 0.015 | 0.155 | 0.864 |
| LLaVA-1.5 (general) | 7.06 B | 0.139 | 0.009 | 0.167 | 0.854 |
**Headline:** the 1.58 B MedOptm-CvT model **beats LLaVA-Med 7B** (0.289 vs 0.219
ROUGE-L, non-overlapping 95% CIs) at roughly **1/5 the size**.
### Efficiency
| Model | Peak VRAM | GFLOPs / sample | Tokens/s | ROUGE-L / GFLOP |
|---|---|---|---|---|
| MedOptm-CvT (Curated) | 10.28 GB | 1977 | 57.9 | 0.000146 |
| **MedOptm-ViT (Curated)** | **6.54 GB** | **619** | **68.7** | **0.000455** |
**Trade-off:** MedOptm-ViT gives up only ~2.6% ROUGE-L for **~1/3 the FLOPs**,
the lowest VRAM, and the highest throughput — i.e. it is ~3× more compute-efficient
per unit of quality. Pick **CvT for best quality**, **ViT for best efficiency**.
<sub>Energy/FLOP figures are estimated (constant-GPU-draw × wall-clock), not power-meter measured.</sub>
---
## Why the CvT checkpoint is much larger than the ViT one
The CvT model **fully fine-tunes** its top vision-encoder stage (hundreds of
millions of full-precision weights), whereas the ViT model adapts its encoder
with tiny **LoRA** layers. On top of that, these `.pt` files also store the
**AdamW optimizer state** (two fp32 momentum buffers per trained parameter),
which is the bulk of the size for the CvT model. That's why, e.g.,
`phase1_alignment_best.pt` (CvT) is ~2 GB while `clipvit_phase1_alignment_best.pt`
(ViT) is ~177 MB. The optimizer state is only needed to *resume* training — it can
be stripped for inference to shrink the files substantially.
---
## How to use
These are not standalone HF models; load them with the project code.
```bash
git clone https://github.com/Devpatel1012/MedOptm-VLM.git
cd MedOptm-VLM && pip install -e .
# download this checkpoints repo (e.g. with huggingface_hub) alongside it
```
```python
import torch
from medoptm.models.vit_vlm import ViTCustomVLM # or: from medoptm.models.cvt_vlm import CustomVLM
model = ViTCustomVLM(
hf_token="<your_hf_token>",
vision_model_name="openai/clip-vit-large-patch14-336",
llm_model_name="meta-llama/Llama-3.2-1B",
image_size=336, scale_factor=2,
vision_lora_r=16, vision_lora_alpha=32, vision_lora_dropout=0.05,
)
model.setup_llm_lora(lora_r=16, lora_alpha=32, lora_dropout=0.05)
ckpt = torch.load("clipvit_phase2_vqa_lora_best.pt", map_location="cpu", weights_only=True)
model.load_trainable_parameters(ckpt["model_state"])
model.eval()
answer = model.generate(images=[img], text_prompts=["Where were the pedicle screws placed?"])
```
For the CvT model, use `CustomVLM` with `vision_model_name="microsoft/cvt-w24-384-22k"`,
`scale_factor=1`, and `lora_r=64, lora_alpha=256, lora_dropout=0.10` (the
`cvt_curated` preset). **Match each checkpoint to the right class and config** —
the exact per-model settings are in `medoptm/config.py`.
---
## Training data
Trained on [`devpatel1012/polished-vlm-dataset`](https://huggingface.co/datasets/devpatel1012/polished-vlm-dataset)
— image–question–answer triples derived from **BIOMEDICA** figure captions
(BIOMEDICA is built from PubMed Central Open Access). See that dataset's card for
provenance and licensing.
## Limitations
- **Not clinically validated.** Research artifact only.
- Results are measured on this project's curated Surgical-VQA test split;
numbers may not transfer to other data.
- Absolute metric values are modest (medical VQA is hard, references are short);
the comparison is meaningful **relative** to the baselines on the same split.
- Inherits any biases/gaps of the underlying BIOMEDICA data (published figures
skew toward illustrative cases).
## Licensing
These weights are a **derivative of `meta-llama/Llama-3.2-1B`**, so the **Llama 3.2
Community License** applies and the model must carry a **"Built with Llama"**
notice. The vision encoders (CvT / CLIP) and the training data carry their own
terms. Confirm and set the `license` field above accordingly before relying on it
commercially.
## Citation
```bibtex
@misc{medoptm2026,
title = {MedOptm: a compact medical vision-language model},
author = {Dev Patel},
year = {2026},
howpublished = {\url{https://github.com/Devpatel1012/MedOptm-VLM}}
}
```