File size: 2,702 Bytes
83407d4 8f8ad2c 83407d4 8f8ad2c 83407d4 8f8ad2c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | ---
license: apache-2.0
base_model: Qwen/Qwen3-VL-2B-Thinking
datasets:
- SeonghoonYu/Masking-KD-Rollouts
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- knowledge-distillation
- multimodal-reasoning
- qwen3_vl
---
# Masking-KD (Qwen3-VL-2B-Thinking)
Checkpoint for **Hide to See: Reasoning-prefix Masking for Visual-anchored Thinking in VLM Distillation** (NeurIPS 2026).
[Qwen3-VL-2B-Thinking](https://huggingface.co/Qwen/Qwen3-VL-2B-Thinking) distilled from [Qwen3-VL-8B-Thinking](https://huggingface.co/Qwen/Qwen3-VL-8B-Thinking) with Masking-KD.
Paper: https://arxiv.org/abs/2605.11651 | Code: https://github.com/Seonghoon-Yu/Masking-KD
## Training
| | |
|---|---|
| Student / Teacher | Qwen3-VL-2B-Thinking / Qwen3-VL-8B-Thinking |
| Data | [SeonghoonYu/Masking-KD-Rollouts](https://huggingface.co/datasets/SeonghoonYu/Masking-KD-Rollouts): 19,387 correct greedy teacher rollouts on ViRL39K |
| Schedule | 2 epochs (76 steps), global batch size 512, learning rate 1e-6 |
## Usage
The model was trained with the reasoning instruction below appended to each question.
```python
from transformers import AutoModelForImageTextToText, AutoProcessor
model = AutoModelForImageTextToText.from_pretrained("SeonghoonYu/Masking-KD", dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained("SeonghoonYu/Masking-KD")
question = "Find x."
instruction = (
"You first think through the reasoning process as an internal monologue, enclosed within <think> </think> tags. "
"Then, provide your final answer enclosed within \\boxed{}."
)
messages = [{
"role": "user",
"content": [
{"type": "image", "image": "path/to/image.png"},
{"type": "text", "text": f"{question}\n\n{instruction}"},
],
}]
inputs = processor.apply_chat_template(
messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt"
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=4096, do_sample=False)
print(processor.decode(outputs[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```
To reproduce the benchmark evaluation (Geo3K, MathVista, We-Math, MMK12, MathVerse, LogicVista, MMMU-Pro):
```bash
git clone https://github.com/Seonghoon-Yu/Masking-KD && cd Masking-KD
python evaluation/prepare_data.py
bash scripts/eval.sh SeonghoonYu/Masking-KD
```
## Citation
```bibtex
@inproceedings{yu2026hide,
title = {Hide to See: Reasoning-prefix Masking for Visual-anchored Thinking in VLM Distillation},
author = {Yu, Seonghoon and Nam, Dongjun and Lee, Byung-Kwan and Son, Jeany},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
}
```
|