Image-Text-to-Text
Transformers
Safetensors
llava_onevision2_5
text-generation
multimodal
video-understanding
streaming
llava-onevision
conversational
custom_code
Instructions to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct
- SGLang
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with Docker Model Runner:
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct
Model card: redesigned layout with benchmark figure, no collapsed sections
Browse files- .gitattributes +1 -0
- README.md +102 -184
- README_CN.md +101 -183
- assets/video_benchmarks.png +3 -0
.gitattributes
CHANGED
|
@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
assets/video_benchmarks.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -9,77 +9,40 @@ tags:
|
|
| 9 |
- llava-onevision
|
| 10 |
---
|
| 11 |
|
| 12 |
-
|
| 13 |
|
| 14 |
# LLaVA-OneVision-2.5-4B-Instruct
|
| 15 |
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
LLaVA-OneVision-2.5-4B-Instruct is a 4B vision-language model for single-image, multi-image and video input. Alongside the usual offline tasks it is trained for **streaming** video, where frames arrive one at a time and the model decides at every step whether to stay silent or speak — the same weights and the same chat template serve both, switched by one keyword.
|
| 19 |
-
|
| 20 |
-
## Highlights
|
| 21 |
-
|
| 22 |
-
- **One model, two modes.** `apply_chat_template(..., streaming=True)` switches the default system prompt; eight protocol tokens carry the per-frame decision to stay silent, speak, or caption. Nothing else about the input changes between modes.
|
| 23 |
-
- **Two video front-ends.** Uniform frame sampling, or *codec mode*, which uses the encoder's bitcost to pack informative regions into a handful of canvases. On this checkpoint codec is ahead on **17 of the 21** video benchmarks where both were run.
|
| 24 |
-
- **Strong video and spatial results.** Against Qwen3.5-4B (thinking off), frame sampling leads on **17 of 21** comparable video benchmarks and codec on **16 of 18**, with the largest matched-setting margins on temporal grounding (TimeLens Charades +12.5) and spatial video (VSI-Bench +9.4).
|
| 25 |
-
- **Honest about images.** On still images the comparison is mixed — ahead on 18 of 31 comparable benchmarks, behind on OCR and document reading. The tables below are unfiltered.
|
| 26 |
-
|
| 27 |
-
## Model Overview
|
| 28 |
-
|
| 29 |
-
| Item | Value |
|
| 30 |
-
|---|---|
|
| 31 |
-
| **Type** | instruction-tuned vision-language model, LLaVA-OneVision-2 architecture |
|
| 32 |
-
| **Parameters** | 5.25B total (4B language model plus vision tower and adapter) |
|
| 33 |
-
| **Language side** | 32 layers, hidden size 2560, FFN 9216, 16 attention heads / 4 KV heads, linear attention with one full-attention layer every 4 |
|
| 34 |
-
| **Vision side** | 24 layers, hidden size 1024, `patch_size=16`, `spatial_merge_size=3`, attention within a frame |
|
| 35 |
-
| **Vocabulary** | 248320, including the 8 streaming protocol tokens |
|
| 36 |
-
| **Context length** | 262144 |
|
| 37 |
-
| **Weights** | bfloat16, 3 safetensors shards, about 9.9 GB |
|
| 38 |
|
| 39 |
-
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
-
|
| 42 |
|
| 43 |
-
|
| 44 |
|
| 45 |
-
|
| 46 |
|
| 47 |
-
|
| 48 |
-
|:---|---:|---:|:---|---:|---:|
|
| 49 |
-
| OCRBench v2 | **59.0** | 53.7 | RefSpatial-Bench \* | 29.1 | **62.1** |
|
| 50 |
-
| OCRBench | **85.2** | 79.8 | RefCOCO val (Acc@0.5) | 89.1 | **92.6** |
|
| 51 |
-
| PerceptionBench \* | **30.9** | 29.6 | RefCOCOg val (Acc@0.5) | 87.7 | **88.9** |
|
| 52 |
-
| MMStar | **66.9** | 63.9 | CrossPoint | 33.7 | **80.5** |
|
| 53 |
-
| MMBench-EN dev \* | **88.8** | 88.2 | SAT mcq circular \* | 68.7 | **74.0** |
|
| 54 |
-
| RealWorldQA | **74.4** | 73.9 | CV-Bench 2D | 80.7 | **83.3** |
|
| 55 |
-
| BLINK \* | 63.6 | **67.1** | CV-Bench 3D | 92.0 | **94.5** |
|
| 56 |
-
| V\*-Bench | 81.7 | **89.5** | EmbSpatial | 74.7 | **83.0** |
|
| 57 |
-
| DocVQA val \* | **95.2** | 92.9 | CRPE relation | 77.8 | **78.2** |
|
| 58 |
-
| ChartQA | **85.0** | 82.8 | ERQA | 43.5 | **46.5** |
|
| 59 |
-
| AI2D | **82.4** | 82.2 | MMSI-Bench | 31.4 | **33.3** |
|
| 60 |
-
| CountBench | **87.2** | 84.9 | MetaVQA | **61.8** | 61.0 |
|
| 61 |
-
| PixMo-Count | 63.1 | **69.9** | PointBench \* | 17.1 | **29.7** |
|
| 62 |
-
| VLMs are Blind | **80.4** | 77.2 | TraceSpatial 2D | **37.9** | 19.4 |
|
| 63 |
-
| VLMs are Biased | 25.0 | **25.3** | TraceSpatial 3D | 13.0 | **44.6** |
|
| 64 |
-
| MuirBench \* | 21.3 | **48.7** | | | |
|
| 65 |
|
| 66 |
-
|
| 67 |
-
|
|
|
|
|
|
|
| 68 |
|
| 69 |
-
|
| 70 |
-
- MuirBench: Qwen has only a rule-based score and is likely affected by the same first-letter parsing. The judge has not been run, so this row is not yet a fair comparison.
|
| 71 |
-
- MMBench-EN dev: per-row accuracy over 4329 rows; the official protocol is circular evaluation.
|
| 72 |
-
- DocVQA val: ANLS rule-based score. PerceptionBench: judged by gpt-oss-120b.
|
| 73 |
-
- PointBench: both sides emit 0–1000 coordinates while scoring is mostly in absolute pixels, so both numbers are low.
|
| 74 |
-
- RefSpatial-Bench: Qwen re-run greedy on 2026-11-17 under the current scoring script.
|
| 75 |
-
- SAT free-form and SAT mcq are omitted — all 150 correct options sit in first position, so answering "A" every time scores 100%. The row reported is the circular variant, which has a cyclic-consistency check.
|
| 76 |
|
| 77 |
-
<
|
|
|
|
|
|
|
| 78 |
|
| 79 |
### Video
|
| 80 |
|
| 81 |
| Benchmark | Qwen3.5-4B | OV2.5-4B (frames) | OV2.5-4B (codec) |
|
| 82 |
-
|:---|---:|---:|---:|
|
| 83 |
| **General video understanding** | | | |
|
| 84 |
| MotionBench | 60.1 | 62.4 | **64.2** |
|
| 85 |
| TOMATO \* | 34.9 | 33.6 | **38.6** |
|
|
@@ -87,10 +50,10 @@ Scores are rounded to one decimal; **bold** marks the best score in each row, an
|
|
| 87 |
| NextQA MC test | 80.7 | 86.1 | **86.4** |
|
| 88 |
| TempCompass MC | 73.4 | 75.8 | **79.7** |
|
| 89 |
| MMVU val | **52.6** | 44.1 | -- |
|
| 90 |
-
| Video-MME-v2
|
| 91 |
-
| EgoSchema subset
|
| 92 |
-
| Video-Index 512f
|
| 93 |
-
| MMOU test-mini
|
| 94 |
| **VideoMME** | | | |
|
| 95 |
| short, no subtitles | **81.7** | 79.7 | 81.1 |
|
| 96 |
| medium, no subtitles | 70.3 | 70.6 | **74.1** |
|
|
@@ -111,74 +74,44 @@ Scores are rounded to one decimal; **bold** marks the best score in each row, an
|
|
| 111 |
| TimeLens Charades | 48.1 | 60.6 | **61.3** |
|
| 112 |
| TimeLens QVHighlights | 67.5 | 68.4 | **70.3** |
|
| 113 |
|
| 114 |
-
|
| 115 |
-
<summary><b>Per-benchmark video settings (frames × pixels per frame, or canvases × pixels per canvas)</b></summary>
|
| 116 |
-
|
| 117 |
-
| Benchmark | Qwen3.5-4B | OV2.5-4B (frames) | OV2.5-4B (codec) |
|
| 118 |
-
|:---|:---|:---|:---|
|
| 119 |
-
| **General video understanding** | | | |
|
| 120 |
-
| MotionBench | 128f × 153,664 | 128f × 153,664 | TC128 × 313,600 |
|
| 121 |
-
| TOMATO | 16f × 153,664, fps=2 | model default | TC128 × 313,600 |
|
| 122 |
-
| MVBench | 64f × 153,664 | 64f × 153,664 | TC64 × 313,600 |
|
| 123 |
-
| NextQA MC test | 64f × 153,664 | 64f × 153,664 | TC64 × 313,600 |
|
| 124 |
-
| TempCompass MC | 64f × 153,664 | 64f × 153,664 | TC64 × 313,600 |
|
| 125 |
-
| MMVU val | 64f × 153,664 | 64f × 153,664 | -- |
|
| 126 |
-
| Video-MME-v2 | 64f × 330,000 | 64f × 330,000 | TC64 × 313,600 |
|
| 127 |
-
| EgoSchema subset | 64f × 153,664 | 64f × 153,664 | -- |
|
| 128 |
-
| Video-Index 512f | task frames, ≤262,144/f | task frames, ≤262,144/f | TC≤256 × 313,600 |
|
| 129 |
-
| MMOU test-mini | -- | model default | -- |
|
| 130 |
-
| **VideoMME** | | | |
|
| 131 |
-
| short, no subtitles | 128f × 321,489 | 128f × 321,489 | TC256 × 313,600 |
|
| 132 |
-
| medium, no subtitles | 256f × 136,900 | 256f × 136,900 | TC512 × 313,600 |
|
| 133 |
-
| long, no subtitles | 1450f × 120,000 | 750f × 250,000 | TC512 × 313,600 |
|
| 134 |
-
| short, subtitles | -- | 128f × 321,489 | TC256 × 313,600 |
|
| 135 |
-
| medium, subtitles | -- | 256f × 136,900 | TC256 × 313,600 |
|
| 136 |
-
| long, subtitles | -- | 750f × 250,000 | TC256 × 313,600 |
|
| 137 |
-
| **Video spatial** | | | |
|
| 138 |
-
| VSI-Bench | 128f × 153,664 | 128f × 153,664 | TC128 × 313,600 |
|
| 139 |
-
| ReVSI 64f | 64f × 153,664 | 64f × 153,664 | TC64 × 313,600 |
|
| 140 |
-
| **Long video** | | | |
|
| 141 |
-
| LVBench | 2048f × 120,000 | 950f × 200,000 | TC1024 × 156,800 |
|
| 142 |
-
| MLVU dev | 512f × 72,900 | 512f × 72,900 | -- |
|
| 143 |
-
| LongVideoBench val | 700f × 300,000 | 1280f × 80,000 | TC768 × 235,200 |
|
| 144 |
-
| VideoEval-Pro | 2048f × 100,000 | 950f × 200,000 | TC768 × 235,200 |
|
| 145 |
-
| **Temporal grounding (mIoU)** | | | |
|
| 146 |
-
| TimeLens ActivityNet | 2048f × 409,600, fps=4 | 2048f × 409,600, fps=4 | TC128 × 313,600 |
|
| 147 |
-
| TimeLens Charades | 2048f × 409,600, fps=4 | 2048f × 409,600, fps=4 | TC128 × 313,600 |
|
| 148 |
-
| TimeLens QVHighlights | 2048f × 409,600, fps=4 | 2048f × 409,600, fps=4 | TC128 × 313,600 |
|
| 149 |
-
|
| 150 |
-
</details>
|
| 151 |
-
|
| 152 |
-
<details>
|
| 153 |
-
<summary><b>Notes on video results</b></summary>
|
| 154 |
-
|
| 155 |
-
- TOMATO: Qwen ran at fps=2 / 16 frames; we passed no frame or pixel settings and used the model defaults.
|
| 156 |
-
- Video-MME-v2: subtitles are fed differently on the two sides.
|
| 157 |
-
- Video-Index 512f: frames are supplied by the task as images, so both sides take multi-image input. The codec number comes from the official-budget run (TC256 cap); the matching frame-sampling control on the video files scores 19.05.
|
| 158 |
-
- EgoSchema: the 500-question public subset. Answers to the full 5031 questions are not public.
|
| 159 |
-
- VideoMME with subtitles and MMOU: Qwen has no comparable result on the official path. Qwen's VideoMME long with subtitles run was invalid (judge recorded 0/900) and is left out rather than reported.
|
| 160 |
-
- Settings are the ones that produced each reported score. Most rows are matched; two are not: on **TOMATO** Qwen ran 16 frames at fps=2 while we used the model defaults, and on **MMOU** only we have a score.
|
| 161 |
-
- Where a score is the best over several settings, the search was uneven and mostly favoured Qwen: VideoMME long no-subtitles 28 settings vs our 6, LVBench 27 vs 16, LongVideoBench 28 vs 2, VideoEval-Pro 28 vs 2.
|
| 162 |
-
- The codec column is scored by rule, since the answers are option letters and the judge is equivalent there.
|
| 163 |
-
|
| 164 |
-
</details>
|
| 165 |
-
|
| 166 |
-
<details>
|
| 167 |
-
<summary><b>Evaluation setup</b></summary>
|
| 168 |
|
| 169 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 170 |
|
| 171 |
-
- **
|
| 172 |
-
- **Video.** Frames are sampled at fps=1 and cached offline, so evaluation decodes nothing and repeated runs see identical input. For each task both sides report the best score over every setting that produced one, so frame count and per-frame pixel budget vary by row and by model — on LVBench, Qwen's best came out of 27 settings and ours out of 16. The codec column uses the canvas budgets from the `codec` field of `preprocessor_config.json`, adjusted per task.
|
| 173 |
-
- **Scoring.** *judge* — both sides re-checked by Qwen3.8-27B at temperature 0, which sees the question, options, reference answer and raw output and answers YES or NO; the reported score is the greater of the rule-based and judged scores. *rule, official metric* — the benchmark's own metric. *rule, judge pending* — the judge should apply but has not run on at least one side: MuirBench, EmbSpatial, CRPE relation, ERQA, MetaVQA, NextQA MC test, TempCompass MC, EgoSchema subset, Video-Index 512f.
|
| 174 |
|
| 175 |
-
|
| 176 |
|
| 177 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 178 |
|
| 179 |
## Quickstart
|
| 180 |
|
| 181 |
-
|
| 182 |
|
| 183 |
```bash
|
| 184 |
pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
|
|
@@ -186,7 +119,7 @@ pip install flash-attn --no-build-isolation # optional, recommended
|
|
| 186 |
pip install decord # frame-sampling video path only
|
| 187 |
```
|
| 188 |
|
| 189 |
-
###
|
| 190 |
|
| 191 |
```python
|
| 192 |
import torch
|
|
@@ -215,10 +148,9 @@ with torch.inference_mode():
|
|
| 215 |
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 216 |
```
|
| 217 |
|
| 218 |
-
|
| 219 |
|
| 220 |
-
|
| 221 |
-
<summary><b>Video — frame sampling</b></summary>
|
| 222 |
|
| 223 |
```python
|
| 224 |
messages = [{"role": "user", "content": [
|
|
@@ -235,21 +167,33 @@ inputs = processor(
|
|
| 235 |
).to(model.device)
|
| 236 |
```
|
| 237 |
|
| 238 |
-
`target_fps` and `num_frames` are also accepted
|
| 239 |
-
|
| 240 |
-
Timestamps are always injected: the `<|video_pad|>` placeholder expands into one block per frame, prefixed with that frame's real sample time, so temporal questions work directly.
|
| 241 |
|
| 242 |
```
|
| 243 |
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 244 |
<1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 245 |
```
|
| 246 |
|
| 247 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 248 |
|
| 249 |
-
|
| 250 |
-
<summary><b>Streaming</b></summary>
|
| 251 |
|
| 252 |
-
|
|
|
|
|
|
|
| 253 |
|
| 254 |
| Mode | Call | Default system |
|
| 255 |
|---|---|---|
|
|
@@ -265,9 +209,7 @@ Frames arrive one per turn and each turn the model emits a state token. One keyw
|
|
| 265 |
| `<captioning>` | 248083 | this turn: speak, as event-level captioning |
|
| 266 |
| `<caption_begin>` | 248084 | user-side marker that this stream is a captioning task |
|
| 267 |
|
| 268 |
-
|
| 269 |
-
|
| 270 |
-
Strict user/assistant alternation, at most one frame per user turn; a turn with no new frame uses a bare text time prefix. Give each frame its `timestamp`, and the trailing newline as its own text item:
|
| 271 |
|
| 272 |
```python
|
| 273 |
messages = [{"role": "user", "content": [
|
|
@@ -288,55 +230,31 @@ text = processor.apply_chat_template(
|
|
| 288 |
<silence><|im_end|>
|
| 289 |
```
|
| 290 |
|
| 291 |
-
|
| 292 |
-
|
| 293 |
-
|
| 294 |
-
|
| 295 |
-
|
| 296 |
-
|
| 297 |
-
|
| 298 |
-
|
| 299 |
-
|
| 300 |
-
|
| 301 |
-
|
| 302 |
-
|
| 303 |
-
|
| 304 |
-
```
|
| 305 |
-
|
| 306 |
-
|
| 307 |
-
|
| 308 |
-
|
| 309 |
-
|
| 310 |
-
|
| 311 |
-
|
| 312 |
-
|
|
|
|
| 313 |
```
|
| 314 |
|
| 315 |
-
|
| 316 |
-
|
| 317 |
-
Defaults live in the `codec` field of `preprocessor_config.json` — 512 candidate frames, 64 canvases, patch 16, 3×3 blocks, 313600 pixels per canvas — overridden per call with `codec_config={"target_canvas": 32, ...}`. Other keys: `strategy` (`adaptive_pwindow`, `adaptive_rdo`, `readiness`), `num_sampled_frames`, `max_pixels`, `canvas_format`. Canvases cache in `~/.cache/codec_video_prep/canvases` (`CVP_CODEC_CACHE_ROOT` to relocate).
|
| 318 |
-
|
| 319 |
-
</details>
|
| 320 |
-
|
| 321 |
-
## Best Practices
|
| 322 |
-
|
| 323 |
-
<details>
|
| 324 |
-
<summary><b>Show the 8 practices</b></summary>
|
| 325 |
-
|
| 326 |
-
1. **Pass video paths, not pre-decoded frames.** Only a path lets the processor recover real sample times and source frame numbers. Given a bare frame list it falls back to frame ordinals as seconds, so a 100-second video looks N seconds long and temporal answers are wrong (it raises a `UserWarning`). If you must decode yourself, hand over the timing too: `videos=[{"frames": ..., "timestamps": ..., "frame_indices": ...}]`. Frames passed through `images=` get no timestamps at all.
|
| 327 |
-
|
| 328 |
-
2. **Keep `patch_positions` truthful.** In codec mode it comes back in the processor output and must be forwarded to `model.forward(...)` — do not drop it. In streaming, where each frame is handed over as an independent image, the layout is right but `t` is `0` for every frame; rewrite each frame's rows to `floor(seconds × fps)`, or every frame reads as the start of a new visual segment. Only the *rank* of `t` matters to the model, but real frame numbers keep the input auditable.
|
| 329 |
-
|
| 330 |
-
3. **Tune frames and per-frame pixels together.** Cost is roughly their product: 64–128 frames at 153664–321489 px for short video, 256 at 136900 for medium, 640–768 at 72900–108900 for long. For very long inputs set `OV_VISION_CHUNK_PATCHES=200000`, which encodes frames in groups split along frame boundaries — attention never crosses a frame, so the result is unchanged.
|
| 331 |
-
|
| 332 |
-
4. **Match codec geometry to the model.** `patch=16` and `block_size=3` must equal `patch_size` and `spatial_merge_size`; preprocessing raises rather than failing silently.
|
| 333 |
-
|
| 334 |
-
5. **Decode greedily, and give reasoning room.** `generation_config.json` sets only `eos_token_id=248044` and `use_cache=true`. Use `do_sample=False` for deterministic output and a generous `max_new_tokens` where the model reasons before answering — 128 cuts it off mid-reasoning. For batching, pass a list of texts with `padding=True`; padding is left-sided, so slice each output by its own `input_ids` length.
|
| 335 |
-
|
| 336 |
-
6. **Ask for the format you will parse.** On multiple-choice tasks the model may write a sentence rather than a bare letter; if a regex will read the output, request one: `Answer with the option's letter from the given choices directly.` For grounding, state the coordinate range in the prompt, e.g. integers 0–1000.
|
| 337 |
-
|
| 338 |
-
7. **Warm up remote code before going multi-process.** Modules compile into `$HF_HOME/modules/transformers_modules/<name>/` and concurrent first loads race there, so load the full model once in a single process first. That cache is keyed by leaf directory name only — give local copies distinct names.
|
| 339 |
-
|
| 340 |
-
8. **Strip the streaming tokens if you do not want them.** All eight survive `skip_special_tokens=True` by design.
|
| 341 |
|
| 342 |
-
|
|
|
|
| 9 |
- llava-onevision
|
| 10 |
---
|
| 11 |
|
| 12 |
+
<div align="center">
|
| 13 |
|
| 14 |
# LLaVA-OneVision-2.5-4B-Instruct
|
| 15 |
|
| 16 |
+
**One 4B model for images, long video and real-time streams**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
+
<a href="https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-2"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-LLaVA--OneVision-181717?logo=github"></a>
|
| 19 |
+
<a href="https://evolvinglmms-lab.github.io/LLaVA-OneVision-2/"><img alt="Project Page" src="https://img.shields.io/badge/Project-Page-2a78d6"></a>
|
| 20 |
+
<a href="https://arxiv.org/abs/2605.25979"><img alt="OV2 Report" src="https://img.shields.io/badge/arXiv-2605.25979-b31b1b"></a>
|
| 21 |
+
<img alt="License" src="https://img.shields.io/badge/License-Apache--2.0-1baf7a">
|
| 22 |
|
| 23 |
+
**English** | [中文](README_CN.md)
|
| 24 |
|
| 25 |
+
</div>
|
| 26 |
|
| 27 |
+
LLaVA-OneVision-2.5-4B-Instruct is a 4B vision-language model for single-image, multi-image and video input. Beyond the usual offline tasks, it is trained for **streaming** video: frames arrive one at a time, and at every step the model decides whether to stay silent or speak. The same weights and the same chat template serve both modes.
|
| 28 |
|
| 29 |
+
## Highlights
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
+
- **Offline and streaming in one model.** `apply_chat_template(..., streaming=True)` switches mode; eight protocol tokens carry the per-frame decision to stay silent, respond or caption. Nothing else about the input changes.
|
| 32 |
+
- **Codec-aligned video.** Besides uniform frame sampling, *codec mode* uses the encoder's bit-cost to pack the informative regions of a video into a handful of canvases. It beats frame sampling on **17 of 21** video benchmarks.
|
| 33 |
+
- **9× spatial compression.** A patch-16 vision tower with 3×3 merging gives one visual token per 48×48 pixels, so long videos fit comfortably in the 262k context.
|
| 34 |
+
- **Strong video, grounding and spatial results.** Against Qwen3.5-4B, frame sampling leads on **17 of 21** video benchmarks and codec on **16 of 18**, with the largest margins on temporal grounding (TimeLens Charades +12.5) and spatial video (VSI-Bench +9.4). On still images it leads on 18 of 31, mostly grounding and spatial tasks, and trails on OCR and document reading.
|
| 35 |
|
| 36 |
+
## Performance
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
|
| 38 |
+
<p align="center">
|
| 39 |
+
<img src="assets/video_benchmarks.png" width="92%" alt="Video benchmarks: LLaVA-OneVision-2.5-4B vs Qwen3.5-4B">
|
| 40 |
+
</p>
|
| 41 |
|
| 42 |
### Video
|
| 43 |
|
| 44 |
| Benchmark | Qwen3.5-4B | OV2.5-4B (frames) | OV2.5-4B (codec) |
|
| 45 |
+
|:---|:---:|:---:|:---:|
|
| 46 |
| **General video understanding** | | | |
|
| 47 |
| MotionBench | 60.1 | 62.4 | **64.2** |
|
| 48 |
| TOMATO \* | 34.9 | 33.6 | **38.6** |
|
|
|
|
| 50 |
| NextQA MC test | 80.7 | 86.1 | **86.4** |
|
| 51 |
| TempCompass MC | 73.4 | 75.8 | **79.7** |
|
| 52 |
| MMVU val | **52.6** | 44.1 | -- |
|
| 53 |
+
| Video-MME-v2 | 15.5 | 16.4 | **17.2** |
|
| 54 |
+
| EgoSchema subset | 68.2 | **80.0** | -- |
|
| 55 |
+
| Video-Index 512f | 19.6 | **20.0** | 19.8 |
|
| 56 |
+
| MMOU test-mini | -- | 46.0 | -- |
|
| 57 |
| **VideoMME** | | | |
|
| 58 |
| short, no subtitles | **81.7** | 79.7 | 81.1 |
|
| 59 |
| medium, no subtitles | 70.3 | 70.6 | **74.1** |
|
|
|
|
| 74 |
| TimeLens Charades | 48.1 | 60.6 | **61.3** |
|
| 75 |
| TimeLens QVHighlights | 67.5 | 68.4 | **70.3** |
|
| 76 |
|
| 77 |
+
### Image
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
|
| 79 |
+
| General perception and reading | Qwen3.5-4B | OV2.5-4B | Grounding and spatial | Qwen3.5-4B | OV2.5-4B |
|
| 80 |
+
|:---|:---:|:---:|:---|:---:|:---:|
|
| 81 |
+
| OCRBench v2 | **59.0** | 53.7 | RefSpatial-Bench | 29.1 | **62.1** |
|
| 82 |
+
| OCRBench | **85.2** | 79.8 | RefCOCO val (Acc@0.5) | 89.1 | **92.6** |
|
| 83 |
+
| PerceptionBench | **30.9** | 29.6 | RefCOCOg val (Acc@0.5) | 87.7 | **88.9** |
|
| 84 |
+
| MMStar | **66.9** | 63.9 | CrossPoint | 33.7 | **80.5** |
|
| 85 |
+
| MMBench-EN dev | **88.8** | 88.2 | SAT mcq circular | 68.7 | **74.0** |
|
| 86 |
+
| RealWorldQA | **74.4** | 73.9 | CV-Bench 2D | 80.7 | **83.3** |
|
| 87 |
+
| BLINK | 63.6 | **67.1** | CV-Bench 3D | 92.0 | **94.5** |
|
| 88 |
+
| V\*-Bench | 81.7 | **89.5** | EmbSpatial | 74.7 | **83.0** |
|
| 89 |
+
| DocVQA val | **95.2** | 92.9 | CRPE relation | 77.8 | **78.2** |
|
| 90 |
+
| ChartQA | **85.0** | 82.8 | ERQA | 43.5 | **46.5** |
|
| 91 |
+
| AI2D | **82.4** | 82.2 | MMSI-Bench | 31.4 | **33.3** |
|
| 92 |
+
| CountBench | **87.2** | 84.9 | MetaVQA | **61.8** | 61.0 |
|
| 93 |
+
| PixMo-Count | 63.1 | **69.9** | PointBench | 17.1 | **29.7** |
|
| 94 |
+
| VLMs are Blind | **80.4** | 77.2 | TraceSpatial 2D | **37.9** | 19.4 |
|
| 95 |
+
| VLMs are Biased | 25.0 | **25.3** | TraceSpatial 3D | 13.0 | **44.6** |
|
| 96 |
+
| MuirBench \* | 21.3 | **48.7** | | | |
|
| 97 |
|
| 98 |
+
<sub>Compared with Qwen3.5-4B, thinking disabled. **Bold** is the best score in each row; `--` means not available. Each side reports its best score over the settings tried, from single runs. `*` marks rows that are not matched comparisons: on TOMATO the two sides use different frame settings, and on MuirBench the Qwen score is rule-based only. Full settings and protocol will be given in the technical report.</sub>
|
|
|
|
|
|
|
| 99 |
|
| 100 |
+
## Model Overview
|
| 101 |
|
| 102 |
+
| | |
|
| 103 |
+
|:---|:---|
|
| 104 |
+
| **Type** | instruction-tuned vision-language model, LLaVA-OneVision-2 architecture |
|
| 105 |
+
| **Parameters** | 5.25B total (4B language model plus vision tower and adapter) |
|
| 106 |
+
| **Language side** | 32 layers, hidden size 2560, FFN 9216, 16 attention heads / 4 KV heads, linear attention with one full-attention layer every 4 |
|
| 107 |
+
| **Vision side** | 24 layers, hidden size 1024, `patch_size=16`, `spatial_merge_size=3`, attention within a frame |
|
| 108 |
+
| **Vocabulary** | 248320, including the 8 streaming protocol tokens |
|
| 109 |
+
| **Context length** | 262144 |
|
| 110 |
+
| **Weights** | bfloat16, 3 safetensors shards, about 9.9 GB |
|
| 111 |
|
| 112 |
## Quickstart
|
| 113 |
|
| 114 |
+
The modeling code ships with the weights, so `trust_remote_code=True` is required.
|
| 115 |
|
| 116 |
```bash
|
| 117 |
pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
|
|
|
|
| 119 |
pip install decord # frame-sampling video path only
|
| 120 |
```
|
| 121 |
|
| 122 |
+
### Image
|
| 123 |
|
| 124 |
```python
|
| 125 |
import torch
|
|
|
|
| 148 |
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 149 |
```
|
| 150 |
|
| 151 |
+
For several images, put one `{"type": "image"}` per image in `content` and pass `images=` in the same order. The per-image budget defaults to `min_pixels=3136` / `max_pixels=4000000`.
|
| 152 |
|
| 153 |
+
### Video: frame sampling
|
|
|
|
| 154 |
|
| 155 |
```python
|
| 156 |
messages = [{"role": "user", "content": [
|
|
|
|
| 167 |
).to(model.device)
|
| 168 |
```
|
| 169 |
|
| 170 |
+
`target_fps` and `num_frames` are also accepted, and `max_frames` caps at 768. Each frame is prefixed with its real sample time, so temporal questions work directly:
|
|
|
|
|
|
|
| 171 |
|
| 172 |
```
|
| 173 |
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 174 |
<1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 175 |
```
|
| 176 |
|
| 177 |
+
### Video: codec mode
|
| 178 |
+
|
| 179 |
+
Codec mode packs the informative regions of the video into a few canvases. Patches on one canvas may come from different source frames, and each keeps its own `(source frame, row, column)` position. Preprocessing comes from `codec-video-prep` (Linux x86_64, CPython 3.10/3.11; H.264, HEVC, VP9 and AV1 files):
|
| 180 |
+
|
| 181 |
+
```bash
|
| 182 |
+
pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple \
|
| 183 |
+
"codec-video-prep==0.3.0"
|
| 184 |
+
codec-video-prep-doctor
|
| 185 |
+
```
|
| 186 |
+
|
| 187 |
+
```python
|
| 188 |
+
inputs = processor(text=text, videos=["example.mp4"],
|
| 189 |
+
video_backend="codec", return_tensors="pt").to(model.device)
|
| 190 |
+
```
|
| 191 |
|
| 192 |
+
Defaults live in the `codec` field of `preprocessor_config.json` (512 candidate frames, 64 canvases, 313600 pixels per canvas) and can be overridden per call with `codec_config={"target_canvas": 32, ...}`.
|
|
|
|
| 193 |
|
| 194 |
+
### Streaming
|
| 195 |
+
|
| 196 |
+
Frames arrive one per user turn, and each turn the model emits a state token.
|
| 197 |
|
| 198 |
| Mode | Call | Default system |
|
| 199 |
|---|---|---|
|
|
|
|
| 209 |
| `<captioning>` | 248083 | this turn: speak, as event-level captioning |
|
| 210 |
| `<caption_begin>` | 248084 | user-side marker that this stream is a captioning task |
|
| 211 |
|
| 212 |
+
Turns strictly alternate between user and assistant, with at most one frame per user turn. Give each frame its `timestamp`:
|
|
|
|
|
|
|
| 213 |
|
| 214 |
```python
|
| 215 |
messages = [{"role": "user", "content": [
|
|
|
|
| 230 |
<silence><|im_end|>
|
| 231 |
```
|
| 232 |
|
| 233 |
+
The eight tokens are ordinary added tokens, so they survive `skip_special_tokens=True`: the state token is the output you read. The real-time loop around the model (frame scheduling, stopping, memory re-injection) is not part of this repository.
|
| 234 |
+
|
| 235 |
+
## Usage Tips
|
| 236 |
+
|
| 237 |
+
- **Pass video paths, not pre-decoded frames.** Only a path lets the processor recover real timestamps. If you decode yourself, pass `videos=[{"frames": ..., "timestamps": ..., "frame_indices": ...}]`.
|
| 238 |
+
- **Tune frames and pixels together.** Cost is roughly their product: 64–128 frames at 153664–321489 px for short video, 256 at 136900 for medium, 640–768 at 72900–108900 for long.
|
| 239 |
+
- **Forward `patch_positions` in codec mode.** It comes back in the processor output and must reach `model.forward(...)`.
|
| 240 |
+
- **Decode greedily and leave room.** Use `do_sample=False`, and a generous `max_new_tokens` on tasks where the model reasons before answering.
|
| 241 |
+
- **Ask for the format you will parse.** For multiple choice, add `Answer with the option's letter from the given choices directly.` For grounding, state the coordinate range, e.g. integers 0–1000.
|
| 242 |
+
- **Warm up before going multi-process.** Load the model once in a single process so the remote code is compiled before workers start.
|
| 243 |
+
|
| 244 |
+
## Citation
|
| 245 |
+
|
| 246 |
+
```bibtex
|
| 247 |
+
@misc{an2026llavaonevision2nextgenerationperceptualintelligence,
|
| 248 |
+
title = {LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence},
|
| 249 |
+
author = {Xiang An and Yin Xie and Feilong Tang and Yunyao Yan and Huajie Tan and Didi Zhu and Changrui Chen and Xiuwei Zhao and Bin Qin and Kaicheng Yang and Yifei Shen and Yuanhan Zhang and Kaichen Zhang and Wenkang Zhang and Zheng Cheng and Nansen Zhang and Chunsheng Wu and Chunjiang Ge and Zimin Ran and Dehua Song and Chunyuan Li and Shikun Feng and Ming Hu and Zhangquan Chen and Junbo Niu and Bo Li and Ziyong Feng and Ziwei Liu and Zongyuan Ge and Jiankang Deng},
|
| 250 |
+
year = {2026},
|
| 251 |
+
eprint = {2605.25979},
|
| 252 |
+
archivePrefix = {arXiv},
|
| 253 |
+
primaryClass = {cs.CV},
|
| 254 |
+
url = {https://arxiv.org/abs/2605.25979}
|
| 255 |
+
}
|
| 256 |
```
|
| 257 |
|
| 258 |
+
## License
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 259 |
|
| 260 |
+
Apache-2.0.
|
README_CN.md
CHANGED
|
@@ -9,77 +9,40 @@ tags:
|
|
| 9 |
- llava-onevision
|
| 10 |
---
|
| 11 |
|
| 12 |
-
|
| 13 |
|
| 14 |
# LLaVA-OneVision-2.5-4B-Instruct
|
| 15 |
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
LLaVA-OneVision-2.5-4B-Instruct 是一个 4B 视觉语言模型,支持单图、多图和视频输入。除常规离线任务外,它还针对**流式**视频做了训练:画面逐帧到达,模型在每一步决定是保持沉默还是发言——两者共用同一份权重和同一个对话模版,用一个关键字切换。
|
| 19 |
-
|
| 20 |
-
## 亮点
|
| 21 |
-
|
| 22 |
-
- **一个模型,两种模式。** `apply_chat_template(..., streaming=True)` 切换默认 system prompt;八个协议 token 承载每一帧"沉默 / 发言 / caption"的决策。除此之外输入格式在两种模式下完全相同。
|
| 23 |
-
- **两条视频前端。** 均匀抽帧,或 **codec 模式**——用编码器的 bitcost 把信息量大的区域打包进少量画布。在本 checkpoint 上,两边都跑出结果的 21 个视频 bench 中,codec **赢 17 个**。
|
| 24 |
-
- **视频与空间能力突出。** 对 Qwen3.5-4B(关闭 thinking),抽帧在 21 个可比视频 bench 中领先 **17 个**,codec 在 18 个中领先 **16 个**;在设置对齐的项上优势最大的是时间定位(TimeLens Charades +12.5)和视频空间(VSI-Bench +9.4)。
|
| 25 |
-
- **图像侧如实呈现。** 静态图像上互有胜负——31 个可比项中领先 18 个,OCR 与文档阅读落后。下面的表格未作筛选。
|
| 26 |
-
|
| 27 |
-
## 模型概览
|
| 28 |
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
| **语言侧** | 32 层,hidden size 2560,FFN 9216,16 个注意力头 / 4 个 KV 头,线性注意力每 4 层夹一个全注意力层 |
|
| 34 |
-
| **视觉侧** | 24 层,hidden size 1024,`patch_size=16`,`spatial_merge_size=3`,帧内注意力 |
|
| 35 |
-
| **词表** | 248320,含 8 个流式协议 token |
|
| 36 |
-
| **上下文长度** | 262144 |
|
| 37 |
-
| **权重** | bfloat16,3 个 safetensors 分片,约 9.9 GB |
|
| 38 |
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**。图像只有一列分数,因为 codec 路径只适用于视频。空格(`--`)表示尚未测出或不适用。
|
| 42 |
|
| 43 |
-
|
| 44 |
|
| 45 |
-
|
| 46 |
|
| 47 |
-
|
| 48 |
-
|:---|---:|---:|:---|---:|---:|
|
| 49 |
-
| OCRBench v2 | **59.0** | 53.7 | RefSpatial-Bench \* | 29.1 | **62.1** |
|
| 50 |
-
| OCRBench | **85.2** | 79.8 | RefCOCO val (Acc@0.5) | 89.1 | **92.6** |
|
| 51 |
-
| PerceptionBench \* | **30.9** | 29.6 | RefCOCOg val (Acc@0.5) | 87.7 | **88.9** |
|
| 52 |
-
| MMStar | **66.9** | 63.9 | CrossPoint | 33.7 | **80.5** |
|
| 53 |
-
| MMBench-EN dev \* | **88.8** | 88.2 | SAT mcq circular \* | 68.7 | **74.0** |
|
| 54 |
-
| RealWorldQA | **74.4** | 73.9 | CV-Bench 2D | 80.7 | **83.3** |
|
| 55 |
-
| BLINK \* | 63.6 | **67.1** | CV-Bench 3D | 92.0 | **94.5** |
|
| 56 |
-
| V\*-Bench | 81.7 | **89.5** | EmbSpatial | 74.7 | **83.0** |
|
| 57 |
-
| DocVQA val \* | **95.2** | 92.9 | CRPE relation | 77.8 | **78.2** |
|
| 58 |
-
| ChartQA | **85.0** | 82.8 | ERQA | 43.5 | **46.5** |
|
| 59 |
-
| AI2D | **82.4** | 82.2 | MMSI-Bench | 31.4 | **33.3** |
|
| 60 |
-
| CountBench | **87.2** | 84.9 | MetaVQA | **61.8** | 61.0 |
|
| 61 |
-
| PixMo-Count | 63.1 | **69.9** | PointBench \* | 17.1 | **29.7** |
|
| 62 |
-
| VLMs are Blind | **80.4** | 77.2 | TraceSpatial 2D | **37.9** | 19.4 |
|
| 63 |
-
| VLMs are Biased | 25.0 | **25.3** | TraceSpatial 3D | 13.0 | **44.6** |
|
| 64 |
-
| MuirBench \* | 21.3 | **48.7** | | | |
|
| 65 |
|
| 66 |
-
|
| 67 |
-
|
|
|
|
|
|
|
| 68 |
|
| 69 |
-
|
| 70 |
-
- MuirBench:Qwen 只有规则分,很可能受同样的首字母解析影响,judge 未出,这一行暂不宜作为公平比较。
|
| 71 |
-
- MMBench-EN dev:judge 为逐行准确率(4329 行);官方协议是循环评测。
|
| 72 |
-
- DocVQA val:按 ANLS 规则分。PerceptionBench:由 gpt-oss-120b 判分。
|
| 73 |
-
- PointBench:两边都输出 0~1000 坐标,而判分多按绝对像素算,因此两边绝对值都偏低。
|
| 74 |
-
- RefSpatial-Bench:Qwen 为 2026-11-17 按现行判分脚本贪心重跑的结果。
|
| 75 |
-
- SAT 自由作答与 SAT mcq 未列入——其 150 道题的正确选项全在第一位,一律答 A 即得 100%。表中报的是带循环一致性检查的 circular 变体。
|
| 76 |
|
| 77 |
-
<
|
|
|
|
|
|
|
| 78 |
|
| 79 |
### 视频
|
| 80 |
|
| 81 |
| Benchmark | Qwen3.5-4B | OV2.5-4B(抽帧) | OV2.5-4B(codec) |
|
| 82 |
-
|:---|---:|---:|---:|
|
| 83 |
| **通用视频理解** | | | |
|
| 84 |
| MotionBench | 60.1 | 62.4 | **64.2** |
|
| 85 |
| TOMATO \* | 34.9 | 33.6 | **38.6** |
|
|
@@ -87,10 +50,10 @@ OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**
|
|
| 87 |
| NextQA MC test | 80.7 | 86.1 | **86.4** |
|
| 88 |
| TempCompass MC | 73.4 | 75.8 | **79.7** |
|
| 89 |
| MMVU val | **52.6** | 44.1 | -- |
|
| 90 |
-
| Video-MME-v2
|
| 91 |
-
| EgoSchema subset
|
| 92 |
-
| Video-Index 512f
|
| 93 |
-
| MMOU test-mini
|
| 94 |
| **VideoMME** | | | |
|
| 95 |
| short,无字幕 | **81.7** | 79.7 | 81.1 |
|
| 96 |
| medium,无字幕 | 70.3 | 70.6 | **74.1** |
|
|
@@ -111,74 +74,44 @@ OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**
|
|
| 111 |
| TimeLens Charades | 48.1 | 60.6 | **61.3** |
|
| 112 |
| TimeLens QVHighlights | 67.5 | 68.4 | **70.3** |
|
| 113 |
|
| 114 |
-
|
| 115 |
-
<summary><b>视频逐项设置(帧数 × 每帧像素,或画布数 × 每画布像素)</b></summary>
|
| 116 |
-
|
| 117 |
-
| Benchmark | Qwen3.5-4B | OV2.5-4B(抽帧) | OV2.5-4B(codec) |
|
| 118 |
-
|:---|:---|:---|:---|
|
| 119 |
-
| **通用视频理解** | | | |
|
| 120 |
-
| MotionBench | 128f × 153,664 | 128f × 153,664 | TC128 × 313,600 |
|
| 121 |
-
| TOMATO | 16f × 153,664, fps=2 | model default | TC128 × 313,600 |
|
| 122 |
-
| MVBench | 64f × 153,664 | 64f × 153,664 | TC64 × 313,600 |
|
| 123 |
-
| NextQA MC test | 64f × 153,664 | 64f × 153,664 | TC64 × 313,600 |
|
| 124 |
-
| TempCompass MC | 64f × 153,664 | 64f × 153,664 | TC64 × 313,600 |
|
| 125 |
-
| MMVU val | 64f × 153,664 | 64f × 153,664 | -- |
|
| 126 |
-
| Video-MME-v2 | 64f × 330,000 | 64f × 330,000 | TC64 × 313,600 |
|
| 127 |
-
| EgoSchema subset | 64f × 153,664 | 64f × 153,664 | -- |
|
| 128 |
-
| Video-Index 512f | task frames, ≤262,144/f | task frames, ≤262,144/f | TC≤256 × 313,600 |
|
| 129 |
-
| MMOU test-mini | -- | model default | -- |
|
| 130 |
-
| **VideoMME** | | | |
|
| 131 |
-
| short,无字幕 | 128f × 321,489 | 128f × 321,489 | TC256 × 313,600 |
|
| 132 |
-
| medium,无字幕 | 256f × 136,900 | 256f × 136,900 | TC512 × 313,600 |
|
| 133 |
-
| long,无字幕 | 1450f × 120,000 | 750f × 250,000 | TC512 × 313,600 |
|
| 134 |
-
| short,有字幕 | -- | 128f × 321,489 | TC256 × 313,600 |
|
| 135 |
-
| medium,有字幕 | -- | 256f × 136,900 | TC256 × 313,600 |
|
| 136 |
-
| long,有字幕 | -- | 750f × 250,000 | TC256 × 313,600 |
|
| 137 |
-
| **视频空间** | | | |
|
| 138 |
-
| VSI-Bench | 128f × 153,664 | 128f × 153,664 | TC128 × 313,600 |
|
| 139 |
-
| ReVSI 64f | 64f × 153,664 | 64f × 153,664 | TC64 × 313,600 |
|
| 140 |
-
| **长视频** | | | |
|
| 141 |
-
| LVBench | 2048f × 120,000 | 950f × 200,000 | TC1024 × 156,800 |
|
| 142 |
-
| MLVU dev | 512f × 72,900 | 512f × 72,900 | -- |
|
| 143 |
-
| LongVideoBench val | 700f × 300,000 | 1280f × 80,000 | TC768 × 235,200 |
|
| 144 |
-
| VideoEval-Pro | 2048f × 100,000 | 950f × 200,000 | TC768 × 235,200 |
|
| 145 |
-
| **时间定位(mIoU)** | | | |
|
| 146 |
-
| TimeLens ActivityNet | 2048f × 409,600, fps=4 | 2048f × 409,600, fps=4 | TC128 × 313,600 |
|
| 147 |
-
| TimeLens Charades | 2048f × 409,600, fps=4 | 2048f × 409,600, fps=4 | TC128 × 313,600 |
|
| 148 |
-
| TimeLens QVHighlights | 2048f × 409,600, fps=4 | 2048f × 409,600, fps=4 | TC128 × 313,600 |
|
| 149 |
-
|
| 150 |
-
</details>
|
| 151 |
-
|
| 152 |
-
<details>
|
| 153 |
-
<summary><b>视频结果说明</b></summary>
|
| 154 |
-
|
| 155 |
-
- TOMATO:Qwen 为 fps=2 / 16 帧;我方未传帧数和像素参数,用的是模型默认值。
|
| 156 |
-
- Video-MME-v2:字幕送法两边不同。
|
| 157 |
-
- Video-Index 512f:帧由任务以图片给出,两边都是多图输入。codec 一列取自官方预算测试(TC256 上限);同一批的抽帧对照(视频文件输入)为 19.05。
|
| 158 |
-
- EgoSchema:公开答案的 500 题子集。全量 5031 题答案不公开。
|
| 159 |
-
- VideoMME 有字幕三项与 MMOU:Qwen 在官方路径下没有可比结果。Qwen 的 VideoMME long 有字幕那次运行无效(judge 记录为 0/900),故留空而非报出。
|
| 160 |
-
- 设置列是各自取得该分数时的实际设置。多数行两边一致,有两行不是:**TOMATO** Qwen 为 16 帧 fps=2 而我方用模型默认值;**MMOU** 只有我方有分。
|
| 161 |
-
- 对「取多设置最高分」的行,两边搜索量并不对等,且多数对 Qwen 更有利:VideoMME long 无字幕 28 个设置对我方 6 个,LVBench 27 对 16,LongVideoBench 28 对 2,VideoEval-Pro 28 对 2。
|
| 162 |
-
- codec 一列为规则分——回答均为选项字母,与 judge 等价。
|
| 163 |
-
|
| 164 |
-
</details>
|
| 165 |
-
|
| 166 |
-
<details>
|
| 167 |
-
<summary><b>评测口径</b></summary>
|
| 168 |
|
| 169 |
-
Qwen3.5-4B
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 170 |
|
| 171 |
-
|
| 172 |
-
- **视频。** 按 fps=1 抽帧并离线缓存,评测时不解码,重复跑输入完全一致。每个任务两边各取所有出分设置中的最高分,因此帧数和逐帧像素预算逐行、逐模型都不同——例如 LVBench,Qwen 的最高分出自 27 个设置,我们出自 16 个。codec 一列用 `preprocessor_config.json` 中 `codec` 字段的画布预算,按任务调整。
|
| 173 |
-
- **判分。** *judge*——两边都由 Qwen3.8-27B(温度 0)复判,它看到问题、选项、参考答案和原始输出,回答 YES 或 NO;最终分取规则分与 judge 分中的较大者。*规则分(官方指标)*——按 bench 自己的指标计算。*规则分(judge 未出)*——本应用 judge 但至少一边尚未判:MuirBench、EmbSpatial、CRPE relation、ERQA、MetaVQA、NextQA MC test、TempCompass MC、EgoSchema subset、Video-Index 512f。
|
| 174 |
|
| 175 |
-
|
| 176 |
|
| 177 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 178 |
|
| 179 |
## 快速开始
|
| 180 |
|
| 181 |
-
|
| 182 |
|
| 183 |
```bash
|
| 184 |
pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
|
|
@@ -215,10 +148,9 @@ with torch.inference_mode():
|
|
| 215 |
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 216 |
```
|
| 217 |
|
| 218 |
-
|
| 219 |
|
| 220 |
-
|
| 221 |
-
<summary><b>视频 — 抽帧</b></summary>
|
| 222 |
|
| 223 |
```python
|
| 224 |
messages = [{"role": "user", "content": [
|
|
@@ -235,21 +167,33 @@ inputs = processor(
|
|
| 235 |
).to(model.device)
|
| 236 |
```
|
| 237 |
|
| 238 |
-
也接受 `target_fps` 和 `num_frames`,
|
| 239 |
-
|
| 240 |
-
时间戳一定会注入:`<|video_pad|>` 占位符展开成逐帧的块,每块前面是该帧的真实采样时刻,因此可以直接问时间定位类问题。
|
| 241 |
|
| 242 |
```
|
| 243 |
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 244 |
<1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 245 |
```
|
| 246 |
|
| 247 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 248 |
|
| 249 |
-
|
| 250 |
-
<summary><b>流式</b></summary>
|
| 251 |
|
| 252 |
-
|
|
|
|
|
|
|
| 253 |
|
| 254 |
| 模式 | 调用 | 默认 system |
|
| 255 |
|---|---|---|
|
|
@@ -265,9 +209,7 @@ inputs = processor(
|
|
| 265 |
| `<captioning>` | 248083 | 当前轮发言,且是事件级 caption |
|
| 266 |
| `<caption_begin>` | 248084 | user 侧标记,说明这条流是 caption 任务 |
|
| 267 |
|
| 268 |
-
|
| 269 |
-
|
| 270 |
-
严格 user / assistant 交替,每个 user 轮最多一帧;没有新画面的轮次改用纯文本时间前缀。每帧给上 `timestamp`,帧后的换行作为单独的 text 项:
|
| 271 |
|
| 272 |
```python
|
| 273 |
messages = [{"role": "user", "content": [
|
|
@@ -288,55 +230,31 @@ text = processor.apply_chat_template(
|
|
| 288 |
<silence><|im_end|>
|
| 289 |
```
|
| 290 |
|
| 291 |
-
|
| 292 |
-
|
| 293 |
-
|
| 294 |
-
|
| 295 |
-
|
| 296 |
-
|
| 297 |
-
|
| 298 |
-
|
| 299 |
-
|
| 300 |
-
|
| 301 |
-
|
| 302 |
-
|
| 303 |
-
|
| 304 |
-
```
|
| 305 |
-
|
| 306 |
-
|
| 307 |
-
|
| 308 |
-
|
| 309 |
-
|
| 310 |
-
|
| 311 |
-
|
| 312 |
-
|
|
|
|
| 313 |
```
|
| 314 |
|
| 315 |
-
|
| 316 |
-
|
| 317 |
-
默认配方写在 `preprocessor_config.json` 的 `codec` 字段——512 候选帧、64 画布、patch 16、3×3 块、每画布 313600 像素——单次调用用 `codec_config={"target_canvas": 32, ...}` 覆盖。其余键:`strategy`(`adaptive_pwindow`、`adaptive_rdo`、`readiness`)、`num_sampled_frames`、`max_pixels`、`canvas_format`。画布缓存在 `~/.cache/codec_video_prep/canvases`(用 `CVP_CODEC_CACHE_ROOT` 换位置)。
|
| 318 |
-
|
| 319 |
-
</details>
|
| 320 |
-
|
| 321 |
-
## 最佳实践
|
| 322 |
-
|
| 323 |
-
<details>
|
| 324 |
-
<summary><b>展开 8 条实践</b></summary>
|
| 325 |
-
|
| 326 |
-
1. **传视频路径,不要传自己解码好的帧。** 只有路径才能让 processor 拿到真实采样时刻和源帧号。只给一串帧时它会退化成用帧序号当秒数,于是 100 秒的视频看起来只有 N 秒,时间类回答全错(此时会抛 `UserWarning`)。必须自己解码就把时间一并交出:`videos=[{"frames": ..., "timestamps": ..., "frame_indices": ...}]`。通过 `images=` 传帧则完全不注入时间戳。
|
| 327 |
-
|
| 328 |
-
2. **保持 `patch_positions` 为真。** codec 模式下它在 processor 输出里,必须转发给 `model.forward(...)`,不要丢掉。流式下每帧作为独立 image 交付,块布局是对的但 `t` 每帧都是 0,需要改成 `floor(秒数 × fps)`,否则每帧都会被当成一段新视觉的开头。模型只用 `t` 的**名次**,但写真实帧号能让输入可审计。
|
| 329 |
-
|
| 330 |
-
3. **帧数与逐帧像素一起调。** 开销约等于两者之积:短视频 64~128 帧 × 153664~321489 px,中等 256 帧 × 136900,长视频 640~768 帧 × 72900~108900。超长输入设 `OV_VISION_CHUNK_PATCHES=200000`,它按帧边界分组编码,注意力不跨帧,结果不变。
|
| 331 |
-
|
| 332 |
-
4. **codec 几何必须对齐模型。** `patch=16` 和 `block_size=3` 必须等于 `patch_size` 和 `spatial_merge_size`,不一致时预处理直接报错而非静默出错。
|
| 333 |
-
|
| 334 |
-
5. **贪心解码,并留足推理长度。** `generation_config.json` 只设了 `eos_token_id=248044` 和 `use_cache=true`。确定性输出用 `do_sample=False`;需要先推理再作答的任务要给足 `max_new_tokens`,128 会在中途截断。批量时传一组 text 并设 `padding=True`,padding 在左侧,按各自的 `input_ids` 长度切输出。
|
| 335 |
-
|
| 336 |
-
6. **要什么格式就在 prompt 里说。** 选择题上模型可能写整句而不是一个字母;如果后续用正则解析,就要求给字母:`Answer with the option's letter from the given choices directly.`。grounding 任务请写明坐标范围,例如 0~1000 的整数。
|
| 337 |
-
|
| 338 |
-
7. **多进程前先单进程预热 remote code。** 模块会编译到 `$HF_HOME/modules/transformers_modules/<名字>/`,多进程首次加载会在那里竞争,所以先单进程完整加载一次。该缓存只按叶子目录名索引——本地副本要取不同的目录名。
|
| 339 |
-
|
| 340 |
-
8. **不需要流式 token 就显式剥除。** 八个都按设计不会被 `skip_special_tokens=True` 过滤。
|
| 341 |
|
| 342 |
-
|
|
|
|
| 9 |
- llava-onevision
|
| 10 |
---
|
| 11 |
|
| 12 |
+
<div align="center">
|
| 13 |
|
| 14 |
# LLaVA-OneVision-2.5-4B-Instruct
|
| 15 |
|
| 16 |
+
**一个 4B 模型,同时处理图像、长视频和实时视频流**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
+
<a href="https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-2"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-LLaVA--OneVision-181717?logo=github"></a>
|
| 19 |
+
<a href="https://evolvinglmms-lab.github.io/LLaVA-OneVision-2/"><img alt="Project Page" src="https://img.shields.io/badge/Project-Page-2a78d6"></a>
|
| 20 |
+
<a href="https://arxiv.org/abs/2605.25979"><img alt="OV2 Report" src="https://img.shields.io/badge/arXiv-2605.25979-b31b1b"></a>
|
| 21 |
+
<img alt="License" src="https://img.shields.io/badge/License-Apache--2.0-1baf7a">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
|
| 23 |
+
[English](README.md) | **中文**
|
|
|
|
|
|
|
| 24 |
|
| 25 |
+
</div>
|
| 26 |
|
| 27 |
+
LLaVA-OneVision-2.5-4B-Instruct 是一个 4B 视觉语言模型,支持单图、多图和视频输入。除常规离线任务外,它还针对**流式**视频做了训练:画面逐帧到达,模型在每一步决定保持沉默还是发言。两种模式共用同一份权重和同一个对话模版。
|
| 28 |
|
| 29 |
+
## 亮点
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
+
- **离线与流式合一。** `apply_chat_template(..., streaming=True)` 即可切换模式;八个协议 token 承载每一帧"沉默 / 回答 / caption"的决策,输入格式的其余部分完全相同。
|
| 32 |
+
- **Codec 对齐的视频输入。** 除均匀抽帧外,**codec 模���**利用编码器的 bit-cost,把视频中信息量大的区域打包进少量画布。在 21 个视频 bench 中,它有 **17 个**优于抽帧。
|
| 33 |
+
- **9 倍空间压缩。** patch 16 的视觉塔加 3×3 合并,每 48×48 像素对应一个视觉 token,长视频可以从容放进 262k 上下文。
|
| 34 |
+
- **视频、定位与空间能力突出。** 对比 Qwen3.5-4B,抽帧在 21 个视频 bench 中领先 **17 个**,codec 在 18 个中领先 **16 个**,优势最大的是时间定位(TimeLens Charades +12.5)和视频空间(VSI-Bench +9.4)。静态图像上 31 项领先 18 项,主要是定位与空间任务;OCR 与文档阅读落后。
|
| 35 |
|
| 36 |
+
## 评测结果
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
|
| 38 |
+
<p align="center">
|
| 39 |
+
<img src="assets/video_benchmarks.png" width="92%" alt="视频评测:LLaVA-OneVision-2.5-4B 对比 Qwen3.5-4B">
|
| 40 |
+
</p>
|
| 41 |
|
| 42 |
### 视频
|
| 43 |
|
| 44 |
| Benchmark | Qwen3.5-4B | OV2.5-4B(抽帧) | OV2.5-4B(codec) |
|
| 45 |
+
|:---|:---:|:---:|:---:|
|
| 46 |
| **通用视频理解** | | | |
|
| 47 |
| MotionBench | 60.1 | 62.4 | **64.2** |
|
| 48 |
| TOMATO \* | 34.9 | 33.6 | **38.6** |
|
|
|
|
| 50 |
| NextQA MC test | 80.7 | 86.1 | **86.4** |
|
| 51 |
| TempCompass MC | 73.4 | 75.8 | **79.7** |
|
| 52 |
| MMVU val | **52.6** | 44.1 | -- |
|
| 53 |
+
| Video-MME-v2 | 15.5 | 16.4 | **17.2** |
|
| 54 |
+
| EgoSchema subset | 68.2 | **80.0** | -- |
|
| 55 |
+
| Video-Index 512f | 19.6 | **20.0** | 19.8 |
|
| 56 |
+
| MMOU test-mini | -- | 46.0 | -- |
|
| 57 |
| **VideoMME** | | | |
|
| 58 |
| short,无字幕 | **81.7** | 79.7 | 81.1 |
|
| 59 |
| medium,无字幕 | 70.3 | 70.6 | **74.1** |
|
|
|
|
| 74 |
| TimeLens Charades | 48.1 | 60.6 | **61.3** |
|
| 75 |
| TimeLens QVHighlights | 67.5 | 68.4 | **70.3** |
|
| 76 |
|
| 77 |
+
### 图像
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
|
| 79 |
+
| 通用感知与阅读 | Qwen3.5-4B | OV2.5-4B | 定位与空间 | Qwen3.5-4B | OV2.5-4B |
|
| 80 |
+
|:---|:---:|:---:|:---|:---:|:---:|
|
| 81 |
+
| OCRBench v2 | **59.0** | 53.7 | RefSpatial-Bench | 29.1 | **62.1** |
|
| 82 |
+
| OCRBench | **85.2** | 79.8 | RefCOCO val (Acc@0.5) | 89.1 | **92.6** |
|
| 83 |
+
| PerceptionBench | **30.9** | 29.6 | RefCOCOg val (Acc@0.5) | 87.7 | **88.9** |
|
| 84 |
+
| MMStar | **66.9** | 63.9 | CrossPoint | 33.7 | **80.5** |
|
| 85 |
+
| MMBench-EN dev | **88.8** | 88.2 | SAT mcq circular | 68.7 | **74.0** |
|
| 86 |
+
| RealWorldQA | **74.4** | 73.9 | CV-Bench 2D | 80.7 | **83.3** |
|
| 87 |
+
| BLINK | 63.6 | **67.1** | CV-Bench 3D | 92.0 | **94.5** |
|
| 88 |
+
| V\*-Bench | 81.7 | **89.5** | EmbSpatial | 74.7 | **83.0** |
|
| 89 |
+
| DocVQA val | **95.2** | 92.9 | CRPE relation | 77.8 | **78.2** |
|
| 90 |
+
| ChartQA | **85.0** | 82.8 | ERQA | 43.5 | **46.5** |
|
| 91 |
+
| AI2D | **82.4** | 82.2 | MMSI-Bench | 31.4 | **33.3** |
|
| 92 |
+
| CountBench | **87.2** | 84.9 | MetaVQA | **61.8** | 61.0 |
|
| 93 |
+
| PixMo-Count | 63.1 | **69.9** | PointBench | 17.1 | **29.7** |
|
| 94 |
+
| VLMs are Blind | **80.4** | 77.2 | TraceSpatial 2D | **37.9** | 19.4 |
|
| 95 |
+
| VLMs are Biased | 25.0 | **25.3** | TraceSpatial 3D | 13.0 | **44.6** |
|
| 96 |
+
| MuirBench \* | 21.3 | **48.7** | | | |
|
| 97 |
|
| 98 |
+
<sub>对比对象为关闭 thinking 的 Qwen3.5-4B。**加粗**为该行最高分,`--` 表示暂无结果。两边各取已测设置中的最高分,均为单次运行。带 `*` 的行两边设置不对齐:TOMATO 两边帧数设置不同,MuirBench 的 Qwen 分数仅为规则分。完整设置与评测口径见后续技术报告。</sub>
|
|
|
|
|
|
|
| 99 |
|
| 100 |
+
## 模型概览
|
| 101 |
|
| 102 |
+
| | |
|
| 103 |
+
|:---|:---|
|
| 104 |
+
| **类型** | 指令微调视觉语言模型,LLaVA-OneVision-2 架构 |
|
| 105 |
+
| **参数量** | 合计 5.25B(4B 语言模型 + 视觉塔与 adapter) |
|
| 106 |
+
| **语言侧** | 32 层,hidden size 2560,FFN 9216,16 个注意力头 / 4 个 KV 头,线性注意力每 4 层夹一个全注意力层 |
|
| 107 |
+
| **视觉侧** | 24 层,hidden size 1024,`patch_size=16`,`spatial_merge_size=3`,帧内注意力 |
|
| 108 |
+
| **词表** | 248320,含 8 个流式协议 token |
|
| 109 |
+
| **上下文长度** | 262144 |
|
| 110 |
+
| **权重** | bfloat16,3 个 safetensors 分片,约 9.9 GB |
|
| 111 |
|
| 112 |
## 快速开始
|
| 113 |
|
| 114 |
+
建模代码随权重分发,加载时需要 `trust_remote_code=True`。
|
| 115 |
|
| 116 |
```bash
|
| 117 |
pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
|
|
|
|
| 148 |
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 149 |
```
|
| 150 |
|
| 151 |
+
多图时每张图放一个 `{"type": "image"}`,`images=` 按同样顺序传入。单图像素预算默认 `min_pixels=3136` / `max_pixels=4000000`。
|
| 152 |
|
| 153 |
+
### 视频:抽帧
|
|
|
|
| 154 |
|
| 155 |
```python
|
| 156 |
messages = [{"role": "user", "content": [
|
|
|
|
| 167 |
).to(model.device)
|
| 168 |
```
|
| 169 |
|
| 170 |
+
也接受 `target_fps` 和 `num_frames`,`max_frames` 上限 768。每帧前面带该帧的真实采样时刻,可以直接问时间类问题:
|
|
|
|
|
|
|
| 171 |
|
| 172 |
```
|
| 173 |
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 174 |
<1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 175 |
```
|
| 176 |
|
| 177 |
+
### 视频:codec 模式
|
| 178 |
+
|
| 179 |
+
codec 模式把视频中信息量大的区域打包进少量画布。同一张画布上的 patch 可以来自不同源帧,各自保留 `(源帧, 行, 列)` 位置。预处理来自 `codec-video-prep`(Linux x86_64,CPython 3.10/3.11;支持 H.264、HEVC、VP9、AV1 文件):
|
| 180 |
+
|
| 181 |
+
```bash
|
| 182 |
+
pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple \
|
| 183 |
+
"codec-video-prep==0.3.0"
|
| 184 |
+
codec-video-prep-doctor
|
| 185 |
+
```
|
| 186 |
+
|
| 187 |
+
```python
|
| 188 |
+
inputs = processor(text=text, videos=["example.mp4"],
|
| 189 |
+
video_backend="codec", return_tensors="pt").to(model.device)
|
| 190 |
+
```
|
| 191 |
|
| 192 |
+
默认配置在 `preprocessor_config.json` 的 `codec` 字段(512 候选帧、64 画布、每画布 313600 像素),单次调用可用 `codec_config={"target_canvas": 32, ...}` 覆盖。
|
|
|
|
| 193 |
|
| 194 |
+
### 流式
|
| 195 |
+
|
| 196 |
+
画面每个 user 轮进来一帧,模型每轮输出一个状态 token。
|
| 197 |
|
| 198 |
| 模式 | 调用 | 默认 system |
|
| 199 |
|---|---|---|
|
|
|
|
| 209 |
| `<captioning>` | 248083 | 当前轮发言,且是事件级 caption |
|
| 210 |
| `<caption_begin>` | 248084 | user 侧标记,说明这条流是 caption 任务 |
|
| 211 |
|
| 212 |
+
user 与 assistant 严格交替,每个 user 轮最多一帧。每帧给上 `timestamp`:
|
|
|
|
|
|
|
| 213 |
|
| 214 |
```python
|
| 215 |
messages = [{"role": "user", "content": [
|
|
|
|
| 230 |
<silence><|im_end|>
|
| 231 |
```
|
| 232 |
|
| 233 |
+
八个 token 都是普通 added token,`skip_special_tokens=True` 不会过滤它们:状态 token 就是要读取的输出。围绕模型的实时循环(取帧调度、停止生成、记忆回插)不在本仓库内。
|
| 234 |
+
|
| 235 |
+
## 使用建议
|
| 236 |
+
|
| 237 |
+
- **传视频路径,不要传自己解码好的帧。** 只有路径才能让 processor 拿到真实时间戳。必须自己解码时,传 `videos=[{"frames": ..., "timestamps": ..., "frame_indices": ...}]`。
|
| 238 |
+
- **帧数与像素一起调。** 开销约等于两者之积:短视频 64~128 帧 × 153664~321489 px,中等 256 帧 × 136900,长视频 640~768 帧 × 72900~108900。
|
| 239 |
+
- **codec 模式下转发 `patch_positions`。** 它在 processor 输出里,必须传给 `model.forward(...)`。
|
| 240 |
+
- **贪心解码并留足长度。** 用 `do_sample=False`;需要先推理再作答的任务给足 `max_new_tokens`。
|
| 241 |
+
- **要什么格式就在 prompt 里说。** 选择题加上 `Answer with the option's letter from the given choices directly.`;grounding 任务写明坐标范围,例如 0~1000 的整数。
|
| 242 |
+
- **多进程前先预热。** 先在单进程里完整加载一次模型,让 remote code 编译完成后再启动多个 worker。
|
| 243 |
+
|
| 244 |
+
## 引用
|
| 245 |
+
|
| 246 |
+
```bibtex
|
| 247 |
+
@misc{an2026llavaonevision2nextgenerationperceptualintelligence,
|
| 248 |
+
title = {LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence},
|
| 249 |
+
author = {Xiang An and Yin Xie and Feilong Tang and Yunyao Yan and Huajie Tan and Didi Zhu and Changrui Chen and Xiuwei Zhao and Bin Qin and Kaicheng Yang and Yifei Shen and Yuanhan Zhang and Kaichen Zhang and Wenkang Zhang and Zheng Cheng and Nansen Zhang and Chunsheng Wu and Chunjiang Ge and Zimin Ran and Dehua Song and Chunyuan Li and Shikun Feng and Ming Hu and Zhangquan Chen and Junbo Niu and Bo Li and Ziyong Feng and Ziwei Liu and Zongyuan Ge and Jiankang Deng},
|
| 250 |
+
year = {2026},
|
| 251 |
+
eprint = {2605.25979},
|
| 252 |
+
archivePrefix = {arXiv},
|
| 253 |
+
primaryClass = {cs.CV},
|
| 254 |
+
url = {https://arxiv.org/abs/2605.25979}
|
| 255 |
+
}
|
| 256 |
```
|
| 257 |
|
| 258 |
+
## 许可
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 259 |
|
| 260 |
+
Apache-2.0。
|
assets/video_benchmarks.png
ADDED
|
Git LFS Details
|