Add model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model:
|
| 4 |
+
- Qwen/Qwen3.6-35B-A3B
|
| 5 |
+
- Qwen/Qwen3.8-27B
|
| 6 |
+
- lmms-lab/LLaVA-Video-7B-Qwen2
|
| 7 |
+
- lmms-lab/llava-onevision-qwen2-7b-ov
|
| 8 |
+
base_model_relation: adapter
|
| 9 |
+
datasets:
|
| 10 |
+
- lmms-lab/LLaVA-Video-178K
|
| 11 |
+
tags:
|
| 12 |
+
- vcompactor
|
| 13 |
+
- vision-token-compression
|
| 14 |
+
- video-understanding
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
# VCompactor
|
| 18 |
+
|
| 19 |
+
[VCompactor](https://github.com/DingYikang/longvlm/tree/release) learns compact vision tokens for frozen VLMs: they replace the original vision tokens in the VLM's
|
| 20 |
+
input, and the VLM itself is not modified. Each subfolder holds the compactor (`config.json` + `model.safetensors`,
|
| 21 |
+
fp32) for one frozen VLM.
|
| 22 |
+
|
| 23 |
+
| Subfolder | Frozen VLM | Frames | Compact tokens | Accuracy | Uncompressed |
|
| 24 |
+
|---|---|:-:|:-:|:-:|:-:|
|
| 25 |
+
| `llava-onevision-7b-64tok` | [LLaVA-OneVision-7B](https://huggingface.co/lmms-lab/llava-onevision-qwen2-7b-ov) | 32 | 64 | 56.3 | 58.4 |
|
| 26 |
+
| `llava-onevision-7b-matryoshka` | [LLaVA-OneVision-7B](https://huggingface.co/lmms-lab/llava-onevision-qwen2-7b-ov) | 32 | 32 / 64 / 128 / 320 / 640 | 54.7 / 55.8 / 57.0 / 57.4 / 57.5 | 58.4 |
|
| 27 |
+
| `llava-video-7b-128tok` | [LLaVA-Video-7B](https://huggingface.co/lmms-lab/LLaVA-Video-7B-Qwen2) | 64 | 128 | 58.1 | 60.7 |
|
| 28 |
+
| `qwen3.6-35b-a3b-64tok` | [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) | 64 | 64 | 64.8 | 70.6 |
|
| 29 |
+
| `qwen3.8-27b-64tok` | [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) | 64 | 64 | 58.7 | 64.1 |
|
| 30 |
+
|
| 31 |
+
Accuracy is the average over MVBench, EgoSchema, LongVideoBench and Video-MME (uncompressed: 6,272 vision tokens,
|
| 32 |
+
10,816 for LLaVA-Video). The Matryoshka compactor serves all five budgets with one encoder (`TOKENS_PER_STEP` = 1 / 2 /
|
| 33 |
+
4 / 10 / 20 tokens per frame) and transfers to images: POPE accuracy 81.11 / 83.89 / 85.63 / 86.12 and MME perception
|
| 34 |
+
1081.4 / 1140.0 / 1346.9 / 1375.3 at 2 / 4 / 10 / 20 tokens per image crop. Qwen3.8 results restrict the first
|
| 35 |
+
generated token to A-E (`ANSWER_LETTERS=ABCDE`).
|
| 36 |
+
|
| 37 |
+
## Usage
|
| 38 |
+
|
| 39 |
+
Install the [VCompactor code](https://github.com/DingYikang/longvlm/tree/release) (`requirements.txt` for LLaVA, `requirements-qwen.txt` for Qwen) and pass
|
| 40 |
+
`dingyikang/VCompactor/<subfolder>`; only that subfolder is downloaded.
|
| 41 |
+
|
| 42 |
+
```python
|
| 43 |
+
from vcompactor import VCompactor
|
| 44 |
+
|
| 45 |
+
compactor = VCompactor.from_pretrained("dingyikang/VCompactor/llava-onevision-7b-64tok", device="cuda")
|
| 46 |
+
```
|
| 47 |
+
|
| 48 |
+
```bash
|
| 49 |
+
MODEL=<LLaVA-OneVision-7B directory> COMPACTOR=dingyikang/VCompactor/llava-onevision-7b-64tok bash scripts/eval.sh
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
## License
|
| 53 |
+
|
| 54 |
+
Apache-2.0. The compactors were trained with the frozen VLMs above (Apache-2.0) on videos and questions from
|
| 55 |
+
LLaVA-Video-178K; follow the terms of that dataset.
|