|
Download README.md from devpatel1012/MedOptm-Checkpoints: direct link, hf CLI and curl.
- Browser
- Download file 7.25 kB
-
https://huggingface.co/devpatel1012/MedOptm-Checkpoints/resolve/main/README.md
- Command line
-
hf download hf://devpatel1012/MedOptm-Checkpoints/README.md
-
curl -L -o README.md https://huggingface.co/devpatel1012/MedOptm-Checkpoints/resolve/main/README.md
7.25 kB
| library_name: pytorch | |
| pipeline_tag: visual-question-answering | |
| license: llama3.2 # TODO: confirm (Llama-3.2 derivative; see Licensing) | |
| language: | |
| - en | |
| base_model: | |
| - meta-llama/Llama-3.2-1B | |
| datasets: | |
| - devpatel1012/polished-vlm-dataset | |
| tags: | |
| - medical | |
| - biomedical | |
| - surgical | |
| - visual-question-answering | |
| - vqa | |
| - multimodal | |
| - lora | |
| - cvt | |
| - clip-vit | |
| # MedOptm — medical VLM checkpoints | |
| Trained weights for **MedOptm**, a compact medical **Vision-Language Model**: | |
| a frozen vision encoder feeds visual tokens through a **TokenPacker** projector | |
| into **Llama-3.2-1B**, adapted with LoRA. You give it a medical/surgical image | |
| and a question; it answers in natural language. | |
| - 💻 **Code (required to load these weights):** <https://github.com/Devpatel1012/MedOptm-VLM> | |
| - 🗂️ **Training dataset:** <https://huggingface.co/datasets/devpatel1012/polished-vlm-dataset> | |
| > ⚠️ **Research use only — not for clinical use.** Answers are derived from | |
| > biomedical figure captions, not written or reviewed by clinicians, and can be | |
| > wrong. Do not use this to make or support medical decisions. | |
| --- | |
| ## Two models in this repo | |
| Both are 1B-class and trained on the **same** curated Surgical-VQA data — they | |
| differ **only in the vision encoder**, which is the whole point of the comparison. | |
| | Model | Vision encoder | How the vision side is trained | Checkpoint prefix | | |
| |---|---|---|---| | |
| | **MedOptm-CvT** *(flagship)* | CvT `microsoft/cvt-w24-384-22k` | last encoder stage is **fully fine-tuned** | `phase1_alignment_*`, `phase2_vqa_lora_*` | | |
| | **MedOptm-ViT** *(efficient)* | CLIP-ViT-L/14-336 | encoder adapted with **LoRA** (base frozen) | `clipvit_phase1_alignment_*`, `clipvit_phase2_vqa_lora_*` | | |
| Each model was trained in **two phases**: | |
| - **Phase 1 — `*_alignment_*`**: aligns the projector on image–caption pairs. | |
| - **Phase 2 — `*_vqa_lora_*`**: instruction-tunes on VQA (adds LoRA to the LLM). | |
| **Which file to use:** the **Phase 2 `*_vqa_lora_best.pt`** of whichever model you | |
| want — it is the final, self-contained model (it already includes the aligned | |
| projector from Phase 1). `*_periodic.pt` files are just resume snapshots; the | |
| Phase 1 files are the intermediate alignment stage and are not needed for inference. | |
| > These checkpoints store **only the trained parts** — the projector, the LoRA | |
| > adapters (and, for the CvT model, the fine-tuned top encoder stage). The frozen | |
| > base weights (`meta-llama/Llama-3.2-1B` and the vision encoder) are **not** | |
| > included: they're downloaded separately when you build the model with the repo | |
| > code. | |
| --- | |
| ## Results | |
| On the curated Surgical-VQA held-out test split. Higher is better; MedOptm models | |
| are **1B-class**. | |
| | Model | Params | ROUGE-L | BLEU | METEOR | BERTScore-F1 | | |
| |---|---|---|---|---|---| | |
| | **MedOptm-CvT (Curated)** 🥇 | 1.58 B | **0.289** | **0.114** | **0.324** | **0.877** | | |
| | MedOptm-ViT (Curated) 🥈 | 1.58 B | 0.282 | 0.112 | 0.298 | 0.875 | | |
| | LLaVA-Med Mistral | 7.57 B | 0.219 | 0.025 | 0.267 | 0.869 | | |
| | SmolVLM Instruct | 2.25 B | 0.147 | 0.015 | 0.155 | 0.864 | | |
| | LLaVA-1.5 (general) | 7.06 B | 0.139 | 0.009 | 0.167 | 0.854 | | |
| **Headline:** the 1.58 B MedOptm-CvT model **beats LLaVA-Med 7B** (0.289 vs 0.219 | |
| ROUGE-L, non-overlapping 95% CIs) at roughly **1/5 the size**. | |
| ### Efficiency | |
| | Model | Peak VRAM | GFLOPs / sample | Tokens/s | ROUGE-L / GFLOP | | |
| |---|---|---|---|---| | |
| | MedOptm-CvT (Curated) | 10.28 GB | 1977 | 57.9 | 0.000146 | | |
| | **MedOptm-ViT (Curated)** | **6.54 GB** | **619** | **68.7** | **0.000455** | | |
| **Trade-off:** MedOptm-ViT gives up only ~2.6% ROUGE-L for **~1/3 the FLOPs**, | |
| the lowest VRAM, and the highest throughput — i.e. it is ~3× more compute-efficient | |
| per unit of quality. Pick **CvT for best quality**, **ViT for best efficiency**. | |
| <sub>Energy/FLOP figures are estimated (constant-GPU-draw × wall-clock), not power-meter measured.</sub> | |
| --- | |
| ## Why the CvT checkpoint is much larger than the ViT one | |
| The CvT model **fully fine-tunes** its top vision-encoder stage (hundreds of | |
| millions of full-precision weights), whereas the ViT model adapts its encoder | |
| with tiny **LoRA** layers. On top of that, these `.pt` files also store the | |
| **AdamW optimizer state** (two fp32 momentum buffers per trained parameter), | |
| which is the bulk of the size for the CvT model. That's why, e.g., | |
| `phase1_alignment_best.pt` (CvT) is ~2 GB while `clipvit_phase1_alignment_best.pt` | |
| (ViT) is ~177 MB. The optimizer state is only needed to *resume* training — it can | |
| be stripped for inference to shrink the files substantially. | |
| --- | |
| ## How to use | |
| These are not standalone HF models; load them with the project code. | |
| ```bash | |
| git clone https://github.com/Devpatel1012/MedOptm-VLM.git | |
| cd MedOptm-VLM && pip install -e . | |
| # download this checkpoints repo (e.g. with huggingface_hub) alongside it | |
| ``` | |
| ```python | |
| import torch | |
| from medoptm.models.vit_vlm import ViTCustomVLM # or: from medoptm.models.cvt_vlm import CustomVLM | |
| model = ViTCustomVLM( | |
| hf_token="<your_hf_token>", | |
| vision_model_name="openai/clip-vit-large-patch14-336", | |
| llm_model_name="meta-llama/Llama-3.2-1B", | |
| image_size=336, scale_factor=2, | |
| vision_lora_r=16, vision_lora_alpha=32, vision_lora_dropout=0.05, | |
| ) | |
| model.setup_llm_lora(lora_r=16, lora_alpha=32, lora_dropout=0.05) | |
| ckpt = torch.load("clipvit_phase2_vqa_lora_best.pt", map_location="cpu", weights_only=True) | |
| model.load_trainable_parameters(ckpt["model_state"]) | |
| model.eval() | |
| answer = model.generate(images=[img], text_prompts=["Where were the pedicle screws placed?"]) | |
| ``` | |
| For the CvT model, use `CustomVLM` with `vision_model_name="microsoft/cvt-w24-384-22k"`, | |
| `scale_factor=1`, and `lora_r=64, lora_alpha=256, lora_dropout=0.10` (the | |
| `cvt_curated` preset). **Match each checkpoint to the right class and config** — | |
| the exact per-model settings are in `medoptm/config.py`. | |
| --- | |
| ## Training data | |
| Trained on [`devpatel1012/polished-vlm-dataset`](https://huggingface.co/datasets/devpatel1012/polished-vlm-dataset) | |
| — image–question–answer triples derived from **BIOMEDICA** figure captions | |
| (BIOMEDICA is built from PubMed Central Open Access). See that dataset's card for | |
| provenance and licensing. | |
| ## Limitations | |
| - **Not clinically validated.** Research artifact only. | |
| - Results are measured on this project's curated Surgical-VQA test split; | |
| numbers may not transfer to other data. | |
| - Absolute metric values are modest (medical VQA is hard, references are short); | |
| the comparison is meaningful **relative** to the baselines on the same split. | |
| - Inherits any biases/gaps of the underlying BIOMEDICA data (published figures | |
| skew toward illustrative cases). | |
| ## Licensing | |
| These weights are a **derivative of `meta-llama/Llama-3.2-1B`**, so the **Llama 3.2 | |
| Community License** applies and the model must carry a **"Built with Llama"** | |
| notice. The vision encoders (CvT / CLIP) and the training data carry their own | |
| terms. Confirm and set the `license` field above accordingly before relying on it | |
| commercially. | |
| ## Citation | |
| ```bibtex | |
| @misc{medoptm2026, | |
| title = {MedOptm: a compact medical vision-language model}, | |
| author = {Dev Patel}, | |
| year = {2026}, | |
| howpublished = {\url{https://github.com/Devpatel1012/MedOptm-VLM}} | |
| } | |
| ``` | |