devpatel1012 commited on
Commit
6e93787
·
verified ·
1 Parent(s): 4eaa0e6

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +182 -0
README.md ADDED
@@ -0,0 +1,182 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: pytorch
3
+ pipeline_tag: visual-question-answering
4
+ license: llama3.2 # TODO: confirm (Llama-3.2 derivative; see Licensing)
5
+ language:
6
+ - en
7
+ base_model:
8
+ - meta-llama/Llama-3.2-1B
9
+ datasets:
10
+ - devpatel1012/polished-vlm-dataset
11
+ tags:
12
+ - medical
13
+ - biomedical
14
+ - surgical
15
+ - visual-question-answering
16
+ - vqa
17
+ - multimodal
18
+ - lora
19
+ - cvt
20
+ - clip-vit
21
+ ---
22
+
23
+ # MedOptm — medical VLM checkpoints
24
+
25
+ Trained weights for **MedOptm**, a compact medical **Vision-Language Model**:
26
+ a frozen vision encoder feeds visual tokens through a **TokenPacker** projector
27
+ into **Llama-3.2-1B**, adapted with LoRA. You give it a medical/surgical image
28
+ and a question; it answers in natural language.
29
+
30
+ - 💻 **Code (required to load these weights):** <https://github.com/Devpatel1012/MedOptm-VLM>
31
+ - 🗂️ **Training dataset:** <https://huggingface.co/datasets/devpatel1012/polished-vlm-dataset>
32
+
33
+ > ⚠️ **Research use only — not for clinical use.** Answers are derived from
34
+ > biomedical figure captions, not written or reviewed by clinicians, and can be
35
+ > wrong. Do not use this to make or support medical decisions.
36
+
37
+ ---
38
+
39
+ ## Two models in this repo
40
+
41
+ Both are 1B-class and trained on the **same** curated Surgical-VQA data — they
42
+ differ **only in the vision encoder**, which is the whole point of the comparison.
43
+
44
+ | Model | Vision encoder | How the vision side is trained | Checkpoint prefix |
45
+ |---|---|---|---|
46
+ | **MedOptm-CvT** *(flagship)* | CvT `microsoft/cvt-w24-384-22k` | last encoder stage is **fully fine-tuned** | `phase1_alignment_*`, `phase2_vqa_lora_*` |
47
+ | **MedOptm-ViT** *(efficient)* | CLIP-ViT-L/14-336 | encoder adapted with **LoRA** (base frozen) | `clipvit_phase1_alignment_*`, `clipvit_phase2_vqa_lora_*` |
48
+
49
+ Each model was trained in **two phases**:
50
+
51
+ - **Phase 1 — `*_alignment_*`**: aligns the projector on image–caption pairs.
52
+ - **Phase 2 — `*_vqa_lora_*`**: instruction-tunes on VQA (adds LoRA to the LLM).
53
+
54
+ **Which file to use:** the **Phase 2 `*_vqa_lora_best.pt`** of whichever model you
55
+ want — it is the final, self-contained model (it already includes the aligned
56
+ projector from Phase 1). `*_periodic.pt` files are just resume snapshots; the
57
+ Phase 1 files are the intermediate alignment stage and are not needed for inference.
58
+
59
+ > These checkpoints store **only the trained parts** — the projector, the LoRA
60
+ > adapters (and, for the CvT model, the fine-tuned top encoder stage). The frozen
61
+ > base weights (`meta-llama/Llama-3.2-1B` and the vision encoder) are **not**
62
+ > included: they're downloaded separately when you build the model with the repo
63
+ > code.
64
+
65
+ ---
66
+
67
+ ## Results
68
+
69
+ On the curated Surgical-VQA held-out test split. Higher is better; MedOptm models
70
+ are **1B-class**.
71
+
72
+ | Model | Params | ROUGE-L | BLEU | METEOR | BERTScore-F1 |
73
+ |---|---|---|---|---|---|
74
+ | **MedOptm-CvT (Curated)** 🥇 | 1.58 B | **0.289** | **0.114** | **0.324** | **0.877** |
75
+ | MedOptm-ViT (Curated) 🥈 | 1.58 B | 0.282 | 0.112 | 0.298 | 0.875 |
76
+ | LLaVA-Med Mistral | 7.57 B | 0.219 | 0.025 | 0.267 | 0.869 |
77
+ | SmolVLM Instruct | 2.25 B | 0.147 | 0.015 | 0.155 | 0.864 |
78
+ | LLaVA-1.5 (general) | 7.06 B | 0.139 | 0.009 | 0.167 | 0.854 |
79
+
80
+ **Headline:** the 1.58 B MedOptm-CvT model **beats LLaVA-Med 7B** (0.289 vs 0.219
81
+ ROUGE-L, non-overlapping 95% CIs) at roughly **1/5 the size**.
82
+
83
+ ### Efficiency
84
+
85
+ | Model | Peak VRAM | GFLOPs / sample | Tokens/s | ROUGE-L / GFLOP |
86
+ |---|---|---|---|---|
87
+ | MedOptm-CvT (Curated) | 10.28 GB | 1977 | 57.9 | 0.000146 |
88
+ | **MedOptm-ViT (Curated)** | **6.54 GB** | **619** | **68.7** | **0.000455** |
89
+
90
+ **Trade-off:** MedOptm-ViT gives up only ~2.6% ROUGE-L for **~1/3 the FLOPs**,
91
+ the lowest VRAM, and the highest throughput — i.e. it is ~3× more compute-efficient
92
+ per unit of quality. Pick **CvT for best quality**, **ViT for best efficiency**.
93
+
94
+ <sub>Energy/FLOP figures are estimated (constant-GPU-draw × wall-clock), not power-meter measured.</sub>
95
+
96
+ ---
97
+
98
+ ## Why the CvT checkpoint is much larger than the ViT one
99
+
100
+ The CvT model **fully fine-tunes** its top vision-encoder stage (hundreds of
101
+ millions of full-precision weights), whereas the ViT model adapts its encoder
102
+ with tiny **LoRA** layers. On top of that, these `.pt` files also store the
103
+ **AdamW optimizer state** (two fp32 momentum buffers per trained parameter),
104
+ which is the bulk of the size for the CvT model. That's why, e.g.,
105
+ `phase1_alignment_best.pt` (CvT) is ~2 GB while `clipvit_phase1_alignment_best.pt`
106
+ (ViT) is ~177 MB. The optimizer state is only needed to *resume* training — it can
107
+ be stripped for inference to shrink the files substantially.
108
+
109
+ ---
110
+
111
+ ## How to use
112
+
113
+ These are not standalone HF models; load them with the project code.
114
+
115
+ ```bash
116
+ git clone https://github.com/Devpatel1012/MedOptm-VLM.git
117
+ cd MedOptm-VLM && pip install -e .
118
+ # download this checkpoints repo (e.g. with huggingface_hub) alongside it
119
+ ```
120
+
121
+ ```python
122
+ import torch
123
+ from medoptm.models.vit_vlm import ViTCustomVLM # or: from medoptm.models.cvt_vlm import CustomVLM
124
+
125
+ model = ViTCustomVLM(
126
+ hf_token="<your_hf_token>",
127
+ vision_model_name="openai/clip-vit-large-patch14-336",
128
+ llm_model_name="meta-llama/Llama-3.2-1B",
129
+ image_size=336, scale_factor=2,
130
+ vision_lora_r=16, vision_lora_alpha=32, vision_lora_dropout=0.05,
131
+ )
132
+ model.setup_llm_lora(lora_r=16, lora_alpha=32, lora_dropout=0.05)
133
+
134
+ ckpt = torch.load("clipvit_phase2_vqa_lora_best.pt", map_location="cpu", weights_only=True)
135
+ model.load_trainable_parameters(ckpt["model_state"])
136
+ model.eval()
137
+
138
+ answer = model.generate(images=[img], text_prompts=["Where were the pedicle screws placed?"])
139
+ ```
140
+
141
+ For the CvT model, use `CustomVLM` with `vision_model_name="microsoft/cvt-w24-384-22k"`,
142
+ `scale_factor=1`, and `lora_r=64, lora_alpha=256, lora_dropout=0.10` (the
143
+ `cvt_curated` preset). **Match each checkpoint to the right class and config** —
144
+ the exact per-model settings are in `medoptm/config.py`.
145
+
146
+ ---
147
+
148
+ ## Training data
149
+
150
+ Trained on [`devpatel1012/polished-vlm-dataset`](https://huggingface.co/datasets/devpatel1012/polished-vlm-dataset)
151
+ — image–question–answer triples derived from **BIOMEDICA** figure captions
152
+ (BIOMEDICA is built from PubMed Central Open Access). See that dataset's card for
153
+ provenance and licensing.
154
+
155
+ ## Limitations
156
+
157
+ - **Not clinically validated.** Research artifact only.
158
+ - Results are measured on this project's curated Surgical-VQA test split;
159
+ numbers may not transfer to other data.
160
+ - Absolute metric values are modest (medical VQA is hard, references are short);
161
+ the comparison is meaningful **relative** to the baselines on the same split.
162
+ - Inherits any biases/gaps of the underlying BIOMEDICA data (published figures
163
+ skew toward illustrative cases).
164
+
165
+ ## Licensing
166
+
167
+ These weights are a **derivative of `meta-llama/Llama-3.2-1B`**, so the **Llama 3.2
168
+ Community License** applies and the model must carry a **"Built with Llama"**
169
+ notice. The vision encoders (CvT / CLIP) and the training data carry their own
170
+ terms. Confirm and set the `license` field above accordingly before relying on it
171
+ commercially.
172
+
173
+ ## Citation
174
+
175
+ ```bibtex
176
+ @misc{medoptm2026,
177
+ title = {MedOptm: a compact medical vision-language model},
178
+ author = {Dev Patel},
179
+ year = {2026},
180
+ howpublished = {\url{https://github.com/Devpatel1012/MedOptm-VLM}}
181
+ }
182
+ ```