File size: 4,666 Bytes
5a3a92b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 | ---
license: apache-2.0
base_model:
- Qwen/Qwen2.5-VL-3B-Instruct
- Qwen/Qwen2.5-VL-7B-Instruct
library_name: peft
pipeline_tag: image-text-to-text
tags:
- lora
- peft
- vlm
- mapdr
- traffic-sign
- sign-to-lane
- autonomous-driving
---
# SignVLM — LoRA adapters
LoRA adapters for **SignVLM**: vision-faithful sign-to-lane rule binding on
MapDR, built on `Qwen2.5-VL-{3B,7B}-Instruct`.
**Code & full reproduction recipe**: <https://github.com/ray90100/SignVLM-public>
Each subfolder is one adapter. SFT subfolders hold a single `final/`-style
adapter; DPO and GRPO subfolders each hold a 4-checkpoint trajectory
(`step_60/`, `step_100/`, `step_140/`, `step_200/`) so you can reproduce
the full training-curve plot from the paper.
## Quick start
Install the repo per its README §2 (Python 3.10 + the pinned
`requirements.txt`), download the matching base model, then:
```python
from peft import PeftModel
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
base = Qwen2_5_VLForConditionalGeneration.from_pretrained(
"Qwen/Qwen2.5-VL-7B-Instruct",
torch_dtype="bfloat16",
device_map="cuda",
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")
# Paper main adapter (Qwen2.5-VL-7B + LoRA SFT + CAVP, seed 42).
model = PeftModel.from_pretrained(
base,
"ray90100/SignVLM-public",
subfolder="sft-7B-CAVP-p0.3-s42",
)
model.eval()
```
For a DPO / GRPO trajectory checkpoint, point `subfolder` at the specific
step, e.g. `subfolder="dpo-7B-tokenmask-beta5.0/step_140"`.
### Download just one adapter to disk
```bash
huggingface-cli download ray90100/SignVLM-public \
--include "sft-7B-CAVP-p0.3-s42/*" \
--local-dir ./ckpts/
```
Then pass the local path to the eval / training scripts in the GitHub
repo, e.g.
```bash
python scripts/eval_sft.py --adapter ckpts/sft-7B-CAVP-p0.3-s42
python scripts/eval_canonical.py runs/sft/.../eval_<...> --rule-version v3
```
## Adapter index
All adapters use LoRA rank 64, `max_image_pixels = 802816`, bf16, AdamW,
trained on 6× RTX 4090 (24 GB).
### Stage 1 — SFT
| Subfolder | Base | Training | Paper reference |
|---|---|---|---|
| `sft-7B-CAVP-p0.3-s42` | Qwen2.5-VL-7B | `PERTURB_MODE=conflict PERTURB_PROB=0.3`, seed 42 | **Main result.** Tab 1 / Tab 3 / Tab 4 SignVLM-CAVP row; the headline Overall F1 ≈ 0.80 figure cited in the GitHub README §4.3 |
| `sft-7B-CAVP-p0.3-s43` | Qwen2.5-VL-7B | same, seed 43 | Tab 1 dual-seed |
| `sft-7B-no_CAVP-s42` | Qwen2.5-VL-7B | `PERTURB_MODE=none`, seed 42 | Tab 1 "SFT no-CAVP" row (illustrates the −61 pt collapse under panel-conflict input) |
| `sft-7B-noise0.3-s42` | Qwen2.5-VL-7B | `PERTURB_MODE=noise PERTURB_PROB=0.3`, seed 42 | Legacy noise ablation (input-fidelity training, not vision-prior robustness) |
| `sft-3B-CAVP-p0.3-s42` | Qwen2.5-VL-3B | `PERTURB_MODE=conflict PERTURB_PROB=0.3`, seed 42 | Tab 1 / Tab 3 capacity-conditional finding (most stable 3B run) |
| `sft-3B-CAVP-p0.5-s42` | Qwen2.5-VL-3B | `PERTURB_MODE=conflict PERTURB_PROB=0.5`, seed 42 | Tab 3 — at high conflict intensity matches 7B-p0.3 within noise floor |
### Stage 2 — DPO (experimental, results withheld)
Token-mask DPO (`--token-mask-dpo`) with KL anchor + adapter swap
(`β=5.0`, `lr=5e-6`, 2 epochs), initialised from the matching SFT-CAVP-p0.3
adapter. Each subfolder contains `step_60/`, `step_100/`, `step_140/`,
`step_200/`.
| Subfolder | Init adapter |
|---|---|
| `dpo-7B-tokenmask-beta5.0` | `sft-7B-CAVP-p0.3-s42` |
| `dpo-3B-tokenmask-beta5.0` | `sft-3B-CAVP-p0.3-s42` |
### Stage 2 — GRPO (experimental, results withheld)
GRPO with per-field-graded reward, initialised from
`sft-3B-CAVP-p0.3-s42`. Same 4-step trajectory layout.
| Subfolder | Reward weighting |
|---|---|
| `grpo-3B-uniform` | uniform across conflict types |
| `grpo-3B-B-variant` | direction-weighted `(0.15, 0.55, 0.15, 0.15)` |
## Limitations
- All adapters are trained on **MapDR** (urban / expressway dash-cam,
roadside pillar signs in mainland China). Behaviour on other regions or
on overhead gantry signs is not characterised.
- Stage 2 (DPO / GRPO) checkpoints are released for transparency; the
corresponding paper numbers are withheld pending review.
- The base Qwen2.5-VL checkpoints are **not** included here — download
them from their official Hugging Face repos.
## License
Adapters are released under Apache-2.0. The underlying MapDR dataset is
CC BY-NC-SA 4.0 (non-commercial). Cite the MapDR paper if you publish
results using these adapters.
## Citation
Paper under review. Placeholder in the GitHub repo's README §10 will be
updated once accepted.
|