dingyikang commited on
Commit
0a19d9e
·
verified ·
1 Parent(s): 7889e22

Add model card

Browse files
Files changed (1) hide show
  1. README.md +55 -0
README.md ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model:
4
+ - Qwen/Qwen3.6-35B-A3B
5
+ - Qwen/Qwen3.8-27B
6
+ - lmms-lab/LLaVA-Video-7B-Qwen2
7
+ - lmms-lab/llava-onevision-qwen2-7b-ov
8
+ base_model_relation: adapter
9
+ datasets:
10
+ - lmms-lab/LLaVA-Video-178K
11
+ tags:
12
+ - vcompactor
13
+ - vision-token-compression
14
+ - video-understanding
15
+ ---
16
+
17
+ # VCompactor
18
+
19
+ [VCompactor](https://github.com/DingYikang/longvlm/tree/release) learns compact vision tokens for frozen VLMs: they replace the original vision tokens in the VLM's
20
+ input, and the VLM itself is not modified. Each subfolder holds the compactor (`config.json` + `model.safetensors`,
21
+ fp32) for one frozen VLM.
22
+
23
+ | Subfolder | Frozen VLM | Frames | Compact tokens | Accuracy | Uncompressed |
24
+ |---|---|:-:|:-:|:-:|:-:|
25
+ | `llava-onevision-7b-64tok` | [LLaVA-OneVision-7B](https://huggingface.co/lmms-lab/llava-onevision-qwen2-7b-ov) | 32 | 64 | 56.3 | 58.4 |
26
+ | `llava-onevision-7b-matryoshka` | [LLaVA-OneVision-7B](https://huggingface.co/lmms-lab/llava-onevision-qwen2-7b-ov) | 32 | 32 / 64 / 128 / 320 / 640 | 54.7 / 55.8 / 57.0 / 57.4 / 57.5 | 58.4 |
27
+ | `llava-video-7b-128tok` | [LLaVA-Video-7B](https://huggingface.co/lmms-lab/LLaVA-Video-7B-Qwen2) | 64 | 128 | 58.1 | 60.7 |
28
+ | `qwen3.6-35b-a3b-64tok` | [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) | 64 | 64 | 64.8 | 70.6 |
29
+ | `qwen3.8-27b-64tok` | [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) | 64 | 64 | 58.7 | 64.1 |
30
+
31
+ Accuracy is the average over MVBench, EgoSchema, LongVideoBench and Video-MME (uncompressed: 6,272 vision tokens,
32
+ 10,816 for LLaVA-Video). The Matryoshka compactor serves all five budgets with one encoder (`TOKENS_PER_STEP` = 1 / 2 /
33
+ 4 / 10 / 20 tokens per frame) and transfers to images: POPE accuracy 81.11 / 83.89 / 85.63 / 86.12 and MME perception
34
+ 1081.4 / 1140.0 / 1346.9 / 1375.3 at 2 / 4 / 10 / 20 tokens per image crop. Qwen3.8 results restrict the first
35
+ generated token to A-E (`ANSWER_LETTERS=ABCDE`).
36
+
37
+ ## Usage
38
+
39
+ Install the [VCompactor code](https://github.com/DingYikang/longvlm/tree/release) (`requirements.txt` for LLaVA, `requirements-qwen.txt` for Qwen) and pass
40
+ `dingyikang/VCompactor/<subfolder>`; only that subfolder is downloaded.
41
+
42
+ ```python
43
+ from vcompactor import VCompactor
44
+
45
+ compactor = VCompactor.from_pretrained("dingyikang/VCompactor/llava-onevision-7b-64tok", device="cuda")
46
+ ```
47
+
48
+ ```bash
49
+ MODEL=<LLaVA-OneVision-7B directory> COMPACTOR=dingyikang/VCompactor/llava-onevision-7b-64tok bash scripts/eval.sh
50
+ ```
51
+
52
+ ## License
53
+
54
+ Apache-2.0. The compactors were trained with the frozen VLMs above (Apache-2.0) on videos and questions from
55
+ LLaVA-Video-178K; follow the terms of that dataset.