|
Download README.md from dingyikang/VCompactor: direct link, hf CLI and curl.
- Browser
- Download file 2.61 kB
-
https://huggingface.co/dingyikang/VCompactor/resolve/main/README.md
- Command line
-
hf download hf://dingyikang/VCompactor/README.md
-
curl -L -o README.md https://huggingface.co/dingyikang/VCompactor/resolve/main/README.md
2.61 kB
| license: apache-2.0 | |
| base_model: | |
| - Qwen/Qwen3.6-35B-A3B | |
| - Qwen/Qwen3.8-27B | |
| - lmms-lab/LLaVA-Video-7B-Qwen2 | |
| - lmms-lab/llava-onevision-qwen2-7b-ov | |
| base_model_relation: adapter | |
| datasets: | |
| - lmms-lab/LLaVA-Video-178K | |
| tags: | |
| - vcompactor | |
| - vision-token-compression | |
| - video-understanding | |
| # VCompactor | |
| [VCompactor](https://github.com/DingYikang/longvlm/tree/release) learns compact vision tokens for frozen VLMs: they replace the original vision tokens in the VLM's | |
| input, and the VLM itself is not modified. Each subfolder holds the compactor (`config.json` + `model.safetensors`, | |
| fp32) for one frozen VLM. | |
| | Subfolder | Frozen VLM | Frames | Compact tokens | Accuracy | Uncompressed | | |
| |---|---|:-:|:-:|:-:|:-:| | |
| | `llava-onevision-7b-64tok` | [LLaVA-OneVision-7B](https://huggingface.co/lmms-lab/llava-onevision-qwen2-7b-ov) | 32 | 64 | 56.3 | 58.4 | | |
| | `llava-onevision-7b-matryoshka` | [LLaVA-OneVision-7B](https://huggingface.co/lmms-lab/llava-onevision-qwen2-7b-ov) | 32 | 32 / 64 / 128 / 320 / 640 | 54.7 / 55.8 / 57.0 / 57.4 / 57.5 | 58.4 | | |
| | `llava-video-7b-128tok` | [LLaVA-Video-7B](https://huggingface.co/lmms-lab/LLaVA-Video-7B-Qwen2) | 64 | 128 | 58.1 | 60.7 | | |
| | `qwen3.6-35b-a3b-64tok` | [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) | 64 | 64 | 64.8 | 70.6 | | |
| | `qwen3.8-27b-64tok` | [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) | 64 | 64 | 58.7 | 64.1 | | |
| Accuracy is the average over MVBench, EgoSchema, LongVideoBench and Video-MME (uncompressed: 6,272 vision tokens, | |
| 10,816 for LLaVA-Video). The Matryoshka compactor serves all five budgets with one encoder (`TOKENS_PER_STEP` = 1 / 2 / | |
| 4 / 10 / 20 tokens per frame) and transfers to images: POPE accuracy 81.11 / 83.89 / 85.63 / 86.12 and MME perception | |
| 1081.4 / 1140.0 / 1346.9 / 1375.3 at 2 / 4 / 10 / 20 tokens per image crop. Qwen3.8 results restrict the first | |
| generated token to A-E (`ANSWER_LETTERS=ABCDE`). | |
| ## Usage | |
| Install the [VCompactor code](https://github.com/DingYikang/longvlm/tree/release) (`requirements.txt` for LLaVA, `requirements-qwen.txt` for Qwen) and pass | |
| `dingyikang/VCompactor/<subfolder>`; only that subfolder is downloaded. | |
| ```python | |
| from vcompactor import VCompactor | |
| compactor = VCompactor.from_pretrained("dingyikang/VCompactor/llava-onevision-7b-64tok", device="cuda") | |
| ``` | |
| ```bash | |
| MODEL=<LLaVA-OneVision-7B directory> COMPACTOR=dingyikang/VCompactor/llava-onevision-7b-64tok bash scripts/eval.sh | |
| ``` | |
| ## License | |
| Apache-2.0. The compactors were trained with the frozen VLMs above (Apache-2.0) on videos and questions from | |
| LLaVA-Video-178K; follow the terms of that dataset. | |