Endo_edge Qwen3-VL-8B LoRA + trained vision encoder, step 1250

LoRA r=32 adapters plus the separately-exported trained vision tower (PEFT does not include it). Lowest-validation surviving checkpoint; step 750 had lower val loss but was rotated away.

endoedge_qwen3vl8b_lora_vis_r32/step-1250/README.md

@@ -0,0 +1,93 @@
+# Endo_edge — Qwen3-VL-8B-Instruct, LoRA + trained vision encoder (step 1250)
+
+Instruction fine-tune of `Qwen/Qwen3-VL-8B-Instruct` that reads a GI endoscopy frame
+and emits a unified JSON report. Trained on GastroVision + HyperKvasir + LIMUC
+(29,938 images; 20,836 train / 4,530 val).
+
+## ⚠️ Read this before loading
+
+**The adapter alone is not the model.** The vision encoder was trained too, and PEFT
+does not export it — `adapter_model.safetensors` contains 504 LoRA tensors and
+**zero** vision tensors. The 351 trained vision tensors (576.4 M params) are in
+`vision_tower.safetensors` and must be loaded separately, or you silently get the
+stock Qwen vision encoder and lose most of what was trained.
+
+Verified against the base checkpoint: every probed vision tensor differs
+(e.g. `blocks.26.mlp.linear_fc2.weight`, max |Δ| 0.040).
+
+## Files
+
+| file | contents |
+|---|---|
+| `adapter/` | LoRA r=32 α=64 on q/k/v/o/gate/up/down — 504 tensors, 87.3 M params |
+| `vision_tower.safetensors` | trained vision encoder, bf16 — 351 tensors, 576.4 M params |
+| `training_meta.json` | hyperparameters and the full train/val loss history |
+| `checkpoint_meta.json` | step / epoch stamp from the trainer |
+
+## Loading
+
+```python
+import torch
+from transformers import AutoProcessor, AutoModelForImageTextToText
+from safetensors.torch import load_file
+from peft import PeftModel
+
+BASE = "Qwen/Qwen3-VL-8B-Instruct"
+HERE = "."            # this folder
+
+model = AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
+
+# 1. trained vision encoder  (keys are already HF-native: model.visual.*)
+missing, unexpected = model.load_state_dict(load_file(f"{HERE}/vision_tower.safetensors"), strict=False)
+assert not unexpected, unexpected          # 351 keys must all land
+
+# 2. LoRA adapters on the language tower
+model = PeftModel.from_pretrained(model, f"{HERE}/adapter")
+model.eval()
+
+processor = AutoProcessor.from_pretrained(BASE, min_pixels=128*128, max_pixels=768*768)
+```
+
+`max_pixels = 768*768` caps the image at ≤576 visual tokens and matches training.
+
+## Output format
+
+The model emits **only the keys the image establishes** — absent keys are omitted,
+never returned as `null`. Targets in training carried 2, 4, 5 or 6 of these nine keys:
+
+`abnormality_present`, `abnormality_type`, `grade`, `instrument_present`,
+`landmark`, `landmark_certain`, `prep_quality`, `procedure`, `tract`
+
+```json
+{"instrument_present": true, "landmark_certain": false}
+```
+
+`instrument_present` and `landmark_certain` are always present.
+`landmark_certain: false` means the anatomy was **not annotated**, not that anatomy is absent.
+
+## Training
+
+bf16, effective batch 44 (micro 22 × accum 2), 3 epochs, 1587 steps, `adamw_8bit`.
+Two parameter groups: adapters at lr 1e-4, the full-rank vision tower at 2e-5.
+Batches are balanced round-robin across the three corpora, and 75 % of each batch
+uses a prompt that names the specific keys that sample supports.
+
+| step | train | val |
+|---|---|---|
+| 250 | 0.0143 | 0.0101 |
+| 500 | 0.0180 | 0.0104 |
+| **750** | 0.0129 | **0.0072** |
+| 1000 | 0.0076 | 0.0086 |
+| **1250** | 0.0053 | **0.0092** ← this checkpoint |
+| 1500 | 0.0059 | 0.0097 |
+
+Validation bottoms out at step 750 and rises after, while train loss keeps falling —
+the run overfits in the second half. **The step-750 checkpoint no longer exists**
+(rotated away by `keep_last_checkpoints=3`); 1250 is the lowest-validation checkpoint
+that survived.
+
+## Intended use
+
+Research artifact. Not a medical device, not validated for clinical use, and not
+evaluated on any external cohort — the three training corpora are the only data it
+has seen, and the source annotations are single-label per image.

endoedge_qwen3vl8b_lora_vis_r32/step-1250/adapter/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-VL-8B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "down_proj",
+    "gate_proj",
+    "k_proj",
+    "q_proj",
+    "v_proj",
+    "o_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

endoedge_qwen3vl8b_lora_vis_r32/step-1250/adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:426758c27b645487074ed199afe56d4f779ee1c912efaaba37b89026528e71df
+size 349251312

endoedge_qwen3vl8b_lora_vis_r32/step-1250/checkpoint_meta.json

@@ -0,0 +1,6 @@
+{
+  "step": 1250,
+  "epoch": 0,
+  "run": "lora_vis_r32",
+  "time": 1788112823.6126652
+}

endoedge_qwen3vl8b_lora_vis_r32/step-1250/training_meta.json

@@ -0,0 +1,28 @@
+{
+  "run": "lora_vis_r32",
+  "base_model": "Qwen/Qwen3-VL-8B-Instruct",
+  "step": 1250,
+  "total_steps": 1587,
+  "epochs": 3,
+  "effective_batch": 44,
+  "micro_batch": 22,
+  "grad_accum": 2,
+  "precision": "bf16",
+  "optimizer": "adamw_8bit",
+  "lr_adapters": 1e-4,
+  "lr_vision": 2e-5,
+  "lora": {"r": 32, "alpha": 64, "dropout": 0.05,
+           "target_modules": ["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"]},
+  "trainable_params": {"adapters": 87293952, "vision_tower": 576400000, "total": 663693952},
+  "train_images": 20836,
+  "val_images": 4530,
+  "validation_loss_by_step": {
+    "250": 0.0101, "500": 0.0104, "750": 0.0072,
+    "1000": 0.0086, "1250": 0.0092, "1500": 0.0097
+  },
+  "train_loss_by_step": {
+    "250": 0.0143, "500": 0.0180, "750": 0.0129,
+    "1000": 0.0076, "1250": 0.0053, "1500": 0.0059
+  },
+  "note": "step 750 had the lowest validation loss but its checkpoint was rotated away by keep_last_checkpoints=3; 1250 is the lowest-validation checkpoint that still exists"
+}

endoedge_qwen3vl8b_lora_vis_r32/step-1250/vision_tower.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a665bd3c70a58373f95f97823023fb4d0b00621a957eaa0b0ef44f172472ce5d
+size 1152816104