yiyexy commited on
Commit
9a8d128
·
verified ·
1 Parent(s): 4826424

Model card: redesigned layout with benchmark figure, no collapsed sections

Browse files
Files changed (4) hide show
  1. .gitattributes +1 -0
  2. README.md +102 -184
  3. README_CN.md +101 -183
  4. assets/video_benchmarks.png +3 -0
.gitattributes CHANGED
@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
 
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
37
+ assets/video_benchmarks.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -9,77 +9,40 @@ tags:
9
  - llava-onevision
10
  ---
11
 
12
- **English** | [中文](README_CN.md)
13
 
14
  # LLaVA-OneVision-2.5-4B-Instruct
15
 
16
- > This repository contains the instruction-tuned weights, the remote modeling code and both chat-template modes.
17
-
18
- LLaVA-OneVision-2.5-4B-Instruct is a 4B vision-language model for single-image, multi-image and video input. Alongside the usual offline tasks it is trained for **streaming** video, where frames arrive one at a time and the model decides at every step whether to stay silent or speak — the same weights and the same chat template serve both, switched by one keyword.
19
-
20
- ## Highlights
21
-
22
- - **One model, two modes.** `apply_chat_template(..., streaming=True)` switches the default system prompt; eight protocol tokens carry the per-frame decision to stay silent, speak, or caption. Nothing else about the input changes between modes.
23
- - **Two video front-ends.** Uniform frame sampling, or *codec mode*, which uses the encoder's bitcost to pack informative regions into a handful of canvases. On this checkpoint codec is ahead on **17 of the 21** video benchmarks where both were run.
24
- - **Strong video and spatial results.** Against Qwen3.5-4B (thinking off), frame sampling leads on **17 of 21** comparable video benchmarks and codec on **16 of 18**, with the largest matched-setting margins on temporal grounding (TimeLens Charades +12.5) and spatial video (VSI-Bench +9.4).
25
- - **Honest about images.** On still images the comparison is mixed — ahead on 18 of 31 comparable benchmarks, behind on OCR and document reading. The tables below are unfiltered.
26
-
27
- ## Model Overview
28
-
29
- | Item | Value |
30
- |---|---|
31
- | **Type** | instruction-tuned vision-language model, LLaVA-OneVision-2 architecture |
32
- | **Parameters** | 5.25B total (4B language model plus vision tower and adapter) |
33
- | **Language side** | 32 layers, hidden size 2560, FFN 9216, 16 attention heads / 4 KV heads, linear attention with one full-attention layer every 4 |
34
- | **Vision side** | 24 layers, hidden size 1024, `patch_size=16`, `spatial_merge_size=3`, attention within a frame |
35
- | **Vocabulary** | 248320, including the 8 streaming protocol tokens |
36
- | **Context length** | 262144 |
37
- | **Weights** | bfloat16, 3 safetensors shards, about 9.9 GB |
38
 
39
- ## Benchmark Results
 
 
 
40
 
41
- OV2.5-4B (stage 4, iteration 11750) against **Qwen3.5-4B with thinking disabled**. Images are scored once: the codec path applies to video only. Empty cells (`--`) mean the result is not yet available or not applicable.
42
 
43
- Scores are rounded to one decimal; **bold** marks the best score in each row, and `*` marks a row with a caveat listed in the notes. Per-benchmark settings, notes and the evaluation protocol are in the collapsed sections under each table.
44
 
45
- ### Image
46
 
47
- | General perception and reading | Qwen3.5-4B | OV2.5-4B | Grounding and spatial | Qwen3.5-4B | OV2.5-4B |
48
- |:---|---:|---:|:---|---:|---:|
49
- | OCRBench v2 | **59.0** | 53.7 | RefSpatial-Bench \* | 29.1 | **62.1** |
50
- | OCRBench | **85.2** | 79.8 | RefCOCO val (Acc@0.5) | 89.1 | **92.6** |
51
- | PerceptionBench \* | **30.9** | 29.6 | RefCOCOg val (Acc@0.5) | 87.7 | **88.9** |
52
- | MMStar | **66.9** | 63.9 | CrossPoint | 33.7 | **80.5** |
53
- | MMBench-EN dev \* | **88.8** | 88.2 | SAT mcq circular \* | 68.7 | **74.0** |
54
- | RealWorldQA | **74.4** | 73.9 | CV-Bench 2D | 80.7 | **83.3** |
55
- | BLINK \* | 63.6 | **67.1** | CV-Bench 3D | 92.0 | **94.5** |
56
- | V\*-Bench | 81.7 | **89.5** | EmbSpatial | 74.7 | **83.0** |
57
- | DocVQA val \* | **95.2** | 92.9 | CRPE relation | 77.8 | **78.2** |
58
- | ChartQA | **85.0** | 82.8 | ERQA | 43.5 | **46.5** |
59
- | AI2D | **82.4** | 82.2 | MMSI-Bench | 31.4 | **33.3** |
60
- | CountBench | **87.2** | 84.9 | MetaVQA | **61.8** | 61.0 |
61
- | PixMo-Count | 63.1 | **69.9** | PointBench \* | 17.1 | **29.7** |
62
- | VLMs are Blind | **80.4** | 77.2 | TraceSpatial 2D | **37.9** | 19.4 |
63
- | VLMs are Biased | 25.0 | **25.3** | TraceSpatial 3D | 13.0 | **44.6** |
64
- | MuirBench \* | 21.3 | **48.7** | | | |
65
 
66
- <details>
67
- <summary><b>Notes on image results</b></summary>
 
 
68
 
69
- - BLINK: 63.56 is Qwen's judged score. Its rule-based score was 54.97 — the extraction reads the first letter, and about 15% of its answers reason before giving the option.
70
- - MuirBench: Qwen has only a rule-based score and is likely affected by the same first-letter parsing. The judge has not been run, so this row is not yet a fair comparison.
71
- - MMBench-EN dev: per-row accuracy over 4329 rows; the official protocol is circular evaluation.
72
- - DocVQA val: ANLS rule-based score. PerceptionBench: judged by gpt-oss-120b.
73
- - PointBench: both sides emit 0–1000 coordinates while scoring is mostly in absolute pixels, so both numbers are low.
74
- - RefSpatial-Bench: Qwen re-run greedy on 2026-11-17 under the current scoring script.
75
- - SAT free-form and SAT mcq are omitted — all 150 correct options sit in first position, so answering "A" every time scores 100%. The row reported is the circular variant, which has a cyclic-consistency check.
76
 
77
- </details>
 
 
78
 
79
  ### Video
80
 
81
  | Benchmark | Qwen3.5-4B | OV2.5-4B (frames) | OV2.5-4B (codec) |
82
- |:---|---:|---:|---:|
83
  | **General video understanding** | | | |
84
  | MotionBench | 60.1 | 62.4 | **64.2** |
85
  | TOMATO \* | 34.9 | 33.6 | **38.6** |
@@ -87,10 +50,10 @@ Scores are rounded to one decimal; **bold** marks the best score in each row, an
87
  | NextQA MC test | 80.7 | 86.1 | **86.4** |
88
  | TempCompass MC | 73.4 | 75.8 | **79.7** |
89
  | MMVU val | **52.6** | 44.1 | -- |
90
- | Video-MME-v2 \* | 15.5 | 16.4 | **17.2** |
91
- | EgoSchema subset \* | 68.2 | **80.0** | -- |
92
- | Video-Index 512f \* | 19.6 | **20.0** | 19.8 |
93
- | MMOU test-mini \* | -- | 46.0 | -- |
94
  | **VideoMME** | | | |
95
  | short, no subtitles | **81.7** | 79.7 | 81.1 |
96
  | medium, no subtitles | 70.3 | 70.6 | **74.1** |
@@ -111,74 +74,44 @@ Scores are rounded to one decimal; **bold** marks the best score in each row, an
111
  | TimeLens Charades | 48.1 | 60.6 | **61.3** |
112
  | TimeLens QVHighlights | 67.5 | 68.4 | **70.3** |
113
 
114
- <details>
115
- <summary><b>Per-benchmark video settings (frames × pixels per frame, or canvases × pixels per canvas)</b></summary>
116
-
117
- | Benchmark | Qwen3.5-4B | OV2.5-4B (frames) | OV2.5-4B (codec) |
118
- |:---|:---|:---|:---|
119
- | **General video understanding** | | | |
120
- | MotionBench | 128f × 153,664 | 128f × 153,664 | TC128 × 313,600 |
121
- | TOMATO | 16f × 153,664, fps=2 | model default | TC128 × 313,600 |
122
- | MVBench | 64f × 153,664 | 64f × 153,664 | TC64 × 313,600 |
123
- | NextQA MC test | 64f × 153,664 | 64f × 153,664 | TC64 × 313,600 |
124
- | TempCompass MC | 64f × 153,664 | 64f × 153,664 | TC64 × 313,600 |
125
- | MMVU val | 64f × 153,664 | 64f × 153,664 | -- |
126
- | Video-MME-v2 | 64f × 330,000 | 64f × 330,000 | TC64 × 313,600 |
127
- | EgoSchema subset | 64f × 153,664 | 64f × 153,664 | -- |
128
- | Video-Index 512f | task frames, ≤262,144/f | task frames, ≤262,144/f | TC≤256 × 313,600 |
129
- | MMOU test-mini | -- | model default | -- |
130
- | **VideoMME** | | | |
131
- | short, no subtitles | 128f × 321,489 | 128f × 321,489 | TC256 × 313,600 |
132
- | medium, no subtitles | 256f × 136,900 | 256f × 136,900 | TC512 × 313,600 |
133
- | long, no subtitles | 1450f × 120,000 | 750f × 250,000 | TC512 × 313,600 |
134
- | short, subtitles | -- | 128f × 321,489 | TC256 × 313,600 |
135
- | medium, subtitles | -- | 256f × 136,900 | TC256 × 313,600 |
136
- | long, subtitles | -- | 750f × 250,000 | TC256 × 313,600 |
137
- | **Video spatial** | | | |
138
- | VSI-Bench | 128f × 153,664 | 128f × 153,664 | TC128 × 313,600 |
139
- | ReVSI 64f | 64f × 153,664 | 64f × 153,664 | TC64 × 313,600 |
140
- | **Long video** | | | |
141
- | LVBench | 2048f × 120,000 | 950f × 200,000 | TC1024 × 156,800 |
142
- | MLVU dev | 512f × 72,900 | 512f × 72,900 | -- |
143
- | LongVideoBench val | 700f × 300,000 | 1280f × 80,000 | TC768 × 235,200 |
144
- | VideoEval-Pro | 2048f × 100,000 | 950f × 200,000 | TC768 × 235,200 |
145
- | **Temporal grounding (mIoU)** | | | |
146
- | TimeLens ActivityNet | 2048f × 409,600, fps=4 | 2048f × 409,600, fps=4 | TC128 × 313,600 |
147
- | TimeLens Charades | 2048f × 409,600, fps=4 | 2048f × 409,600, fps=4 | TC128 × 313,600 |
148
- | TimeLens QVHighlights | 2048f × 409,600, fps=4 | 2048f × 409,600, fps=4 | TC128 × 313,600 |
149
-
150
- </details>
151
-
152
- <details>
153
- <summary><b>Notes on video results</b></summary>
154
-
155
- - TOMATO: Qwen ran at fps=2 / 16 frames; we passed no frame or pixel settings and used the model defaults.
156
- - Video-MME-v2: subtitles are fed differently on the two sides.
157
- - Video-Index 512f: frames are supplied by the task as images, so both sides take multi-image input. The codec number comes from the official-budget run (TC256 cap); the matching frame-sampling control on the video files scores 19.05.
158
- - EgoSchema: the 500-question public subset. Answers to the full 5031 questions are not public.
159
- - VideoMME with subtitles and MMOU: Qwen has no comparable result on the official path. Qwen's VideoMME long with subtitles run was invalid (judge recorded 0/900) and is left out rather than reported.
160
- - Settings are the ones that produced each reported score. Most rows are matched; two are not: on **TOMATO** Qwen ran 16 frames at fps=2 while we used the model defaults, and on **MMOU** only we have a score.
161
- - Where a score is the best over several settings, the search was uneven and mostly favoured Qwen: VideoMME long no-subtitles 28 settings vs our 6, LVBench 27 vs 16, LongVideoBench 28 vs 2, VideoEval-Pro 28 vs 2.
162
- - The codec column is scored by rule, since the answers are option letters and the judge is equivalent there.
163
-
164
- </details>
165
-
166
- <details>
167
- <summary><b>Evaluation setup</b></summary>
168
 
169
- Qwen3.5-4B runs with thinking disabled. Some of its results came from temperature-0.7 sampling rather than greedy decoding; rows since re-run greedy use the re-run. Every number is a single run, with no variance estimate.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
170
 
171
- - **Images.** OV2.5 uses the processor's default pixel budget; Qwen uses 65,536–16,777,216 pixels per image.
172
- - **Video.** Frames are sampled at fps=1 and cached offline, so evaluation decodes nothing and repeated runs see identical input. For each task both sides report the best score over every setting that produced one, so frame count and per-frame pixel budget vary by row and by model — on LVBench, Qwen's best came out of 27 settings and ours out of 16. The codec column uses the canvas budgets from the `codec` field of `preprocessor_config.json`, adjusted per task.
173
- - **Scoring.** *judge* — both sides re-checked by Qwen3.8-27B at temperature 0, which sees the question, options, reference answer and raw output and answers YES or NO; the reported score is the greater of the rule-based and judged scores. *rule, official metric* — the benchmark's own metric. *rule, judge pending* — the judge should apply but has not run on at least one side: MuirBench, EmbSpatial, CRPE relation, ERQA, MetaVQA, NextQA MC test, TempCompass MC, EgoSchema subset, Video-Index 512f.
174
 
175
- All generation is greedy with `batch_size=1`.
176
 
177
- </details>
 
 
 
 
 
 
 
 
178
 
179
  ## Quickstart
180
 
181
- > The modeling code ships with the weights, so `trust_remote_code=True` is required everywhere.
182
 
183
  ```bash
184
  pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
@@ -186,7 +119,7 @@ pip install flash-attn --no-build-isolation # optional, recommended
186
  pip install decord # frame-sampling video path only
187
  ```
188
 
189
- ### Images
190
 
191
  ```python
192
  import torch
@@ -215,10 +148,9 @@ with torch.inference_mode():
215
  print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
216
  ```
217
 
218
- Images go through `smart_resize`, keeping aspect ratio and aligning to the patch grid; the budget defaults to `min_pixels=3136` / `max_pixels=4000000`. For several images, put one `{"type": "image"}` per image in `content` and pass `images=` in the same order.
219
 
220
- <details>
221
- <summary><b>Video — frame sampling</b></summary>
222
 
223
  ```python
224
  messages = [{"role": "user", "content": [
@@ -235,21 +167,33 @@ inputs = processor(
235
  ).to(model.device)
236
  ```
237
 
238
- `target_fps` and `num_frames` are also accepted; precedence is in `choose_target_frames` in `video_processing_llava_onevision2_5.py`. `max_frames` caps at 768.
239
-
240
- Timestamps are always injected: the `<|video_pad|>` placeholder expands into one block per frame, prefixed with that frame's real sample time, so temporal questions work directly.
241
 
242
  ```
243
  <0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
244
  <1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
245
  ```
246
 
247
- </details>
 
 
 
 
 
 
 
 
 
 
 
 
 
248
 
249
- <details>
250
- <summary><b>Streaming</b></summary>
251
 
252
- Frames arrive one per turn and each turn the model emits a state token. One keyword switches the default system prompt and changes nothing else; an explicit `system` message wins in both modes.
 
 
253
 
254
  | Mode | Call | Default system |
255
  |---|---|---|
@@ -265,9 +209,7 @@ Frames arrive one per turn and each turn the model emits a state token. One keyw
265
  | `<captioning>` | 248083 | this turn: speak, as event-level captioning |
266
  | `<caption_begin>` | 248084 | user-side marker that this stream is a captioning task |
267
 
268
- All eight are non-special added tokens, so `skip_special_tokens=True` still returns them — the state token is the output you read. The memory tokens wrap a JSON object, e.g. `{"timestep": "00:00:06", "content": "..."}`.
269
-
270
- Strict user/assistant alternation, at most one frame per user turn; a turn with no new frame uses a bare text time prefix. Give each frame its `timestamp`, and the trailing newline as its own text item:
271
 
272
  ```python
273
  messages = [{"role": "user", "content": [
@@ -288,55 +230,31 @@ text = processor.apply_chat_template(
288
  <silence><|im_end|>
289
  ```
290
 
291
- A plain string `content` is emitted verbatim, for harnesses that render the markers themselves.
292
-
293
- > **Not in this repository:** the real-time loop around the model — frame scheduling, deciding when to stop generating, re-injecting question and answer memory as it falls due, and carrying context across decision points.
294
-
295
- </details>
296
-
297
- <details>
298
- <summary><b>Video — codec mode</b></summary>
299
-
300
- Codec mode uses encoder bitcost to pack informative regions into a few *canvases*. Patches on one canvas may come from different source frames, each carrying its own `(source frame, row, column)` for 3-D RoPE.
301
-
302
- Preprocessing comes from `codec-video-prep` (bundles patched FFmpeg and dav1d; Linux x86_64, CPython 3.10/3.11, `numpy<2`, system `ffprobe`; H.264/HEVC/VP9/AV1 only; offline files only, since budgeting needs the whole video):
303
-
304
- ```bash
305
- pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple \
306
- "codec-video-prep==0.3.0"
307
- codec-video-prep-doctor
308
- ```
309
-
310
- ```python
311
- inputs = processor(text=text, videos=["example.mp4"],
312
- video_backend="codec", return_tensors="pt").to(model.device)
 
313
  ```
314
 
315
- Each canvas gets one time marker: a point `<6.5 seconds>` when all its patches come from one source frame, a range `<0.0 to 1.8 seconds>` when it spans several (schema `canvas_patch_span_point_range_v1`).
316
-
317
- Defaults live in the `codec` field of `preprocessor_config.json` — 512 candidate frames, 64 canvases, patch 16, 3×3 blocks, 313600 pixels per canvas — overridden per call with `codec_config={"target_canvas": 32, ...}`. Other keys: `strategy` (`adaptive_pwindow`, `adaptive_rdo`, `readiness`), `num_sampled_frames`, `max_pixels`, `canvas_format`. Canvases cache in `~/.cache/codec_video_prep/canvases` (`CVP_CODEC_CACHE_ROOT` to relocate).
318
-
319
- </details>
320
-
321
- ## Best Practices
322
-
323
- <details>
324
- <summary><b>Show the 8 practices</b></summary>
325
-
326
- 1. **Pass video paths, not pre-decoded frames.** Only a path lets the processor recover real sample times and source frame numbers. Given a bare frame list it falls back to frame ordinals as seconds, so a 100-second video looks N seconds long and temporal answers are wrong (it raises a `UserWarning`). If you must decode yourself, hand over the timing too: `videos=[{"frames": ..., "timestamps": ..., "frame_indices": ...}]`. Frames passed through `images=` get no timestamps at all.
327
-
328
- 2. **Keep `patch_positions` truthful.** In codec mode it comes back in the processor output and must be forwarded to `model.forward(...)` — do not drop it. In streaming, where each frame is handed over as an independent image, the layout is right but `t` is `0` for every frame; rewrite each frame's rows to `floor(seconds × fps)`, or every frame reads as the start of a new visual segment. Only the *rank* of `t` matters to the model, but real frame numbers keep the input auditable.
329
-
330
- 3. **Tune frames and per-frame pixels together.** Cost is roughly their product: 64–128 frames at 153664–321489 px for short video, 256 at 136900 for medium, 640–768 at 72900–108900 for long. For very long inputs set `OV_VISION_CHUNK_PATCHES=200000`, which encodes frames in groups split along frame boundaries — attention never crosses a frame, so the result is unchanged.
331
-
332
- 4. **Match codec geometry to the model.** `patch=16` and `block_size=3` must equal `patch_size` and `spatial_merge_size`; preprocessing raises rather than failing silently.
333
-
334
- 5. **Decode greedily, and give reasoning room.** `generation_config.json` sets only `eos_token_id=248044` and `use_cache=true`. Use `do_sample=False` for deterministic output and a generous `max_new_tokens` where the model reasons before answering — 128 cuts it off mid-reasoning. For batching, pass a list of texts with `padding=True`; padding is left-sided, so slice each output by its own `input_ids` length.
335
-
336
- 6. **Ask for the format you will parse.** On multiple-choice tasks the model may write a sentence rather than a bare letter; if a regex will read the output, request one: `Answer with the option's letter from the given choices directly.` For grounding, state the coordinate range in the prompt, e.g. integers 0–1000.
337
-
338
- 7. **Warm up remote code before going multi-process.** Modules compile into `$HF_HOME/modules/transformers_modules/<name>/` and concurrent first loads race there, so load the full model once in a single process first. That cache is keyed by leaf directory name only — give local copies distinct names.
339
-
340
- 8. **Strip the streaming tokens if you do not want them.** All eight survive `skip_special_tokens=True` by design.
341
 
342
- </details>
 
9
  - llava-onevision
10
  ---
11
 
12
+ <div align="center">
13
 
14
  # LLaVA-OneVision-2.5-4B-Instruct
15
 
16
+ **One 4B model for images, long video and real-time streams**
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
17
 
18
+ <a href="https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-2"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-LLaVA--OneVision-181717?logo=github"></a>
19
+ <a href="https://evolvinglmms-lab.github.io/LLaVA-OneVision-2/"><img alt="Project Page" src="https://img.shields.io/badge/Project-Page-2a78d6"></a>
20
+ <a href="https://arxiv.org/abs/2605.25979"><img alt="OV2 Report" src="https://img.shields.io/badge/arXiv-2605.25979-b31b1b"></a>
21
+ <img alt="License" src="https://img.shields.io/badge/License-Apache--2.0-1baf7a">
22
 
23
+ **English** | [中文](README_CN.md)
24
 
25
+ </div>
26
 
27
+ LLaVA-OneVision-2.5-4B-Instruct is a 4B vision-language model for single-image, multi-image and video input. Beyond the usual offline tasks, it is trained for **streaming** video: frames arrive one at a time, and at every step the model decides whether to stay silent or speak. The same weights and the same chat template serve both modes.
28
 
29
+ ## Highlights
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
30
 
31
+ - **Offline and streaming in one model.** `apply_chat_template(..., streaming=True)` switches mode; eight protocol tokens carry the per-frame decision to stay silent, respond or caption. Nothing else about the input changes.
32
+ - **Codec-aligned video.** Besides uniform frame sampling, *codec mode* uses the encoder's bit-cost to pack the informative regions of a video into a handful of canvases. It beats frame sampling on **17 of 21** video benchmarks.
33
+ - **9× spatial compression.** A patch-16 vision tower with 3×3 merging gives one visual token per 48×48 pixels, so long videos fit comfortably in the 262k context.
34
+ - **Strong video, grounding and spatial results.** Against Qwen3.5-4B, frame sampling leads on **17 of 21** video benchmarks and codec on **16 of 18**, with the largest margins on temporal grounding (TimeLens Charades +12.5) and spatial video (VSI-Bench +9.4). On still images it leads on 18 of 31, mostly grounding and spatial tasks, and trails on OCR and document reading.
35
 
36
+ ## Performance
 
 
 
 
 
 
37
 
38
+ <p align="center">
39
+ <img src="assets/video_benchmarks.png" width="92%" alt="Video benchmarks: LLaVA-OneVision-2.5-4B vs Qwen3.5-4B">
40
+ </p>
41
 
42
  ### Video
43
 
44
  | Benchmark | Qwen3.5-4B | OV2.5-4B (frames) | OV2.5-4B (codec) |
45
+ |:---|:---:|:---:|:---:|
46
  | **General video understanding** | | | |
47
  | MotionBench | 60.1 | 62.4 | **64.2** |
48
  | TOMATO \* | 34.9 | 33.6 | **38.6** |
 
50
  | NextQA MC test | 80.7 | 86.1 | **86.4** |
51
  | TempCompass MC | 73.4 | 75.8 | **79.7** |
52
  | MMVU val | **52.6** | 44.1 | -- |
53
+ | Video-MME-v2 | 15.5 | 16.4 | **17.2** |
54
+ | EgoSchema subset | 68.2 | **80.0** | -- |
55
+ | Video-Index 512f | 19.6 | **20.0** | 19.8 |
56
+ | MMOU test-mini | -- | 46.0 | -- |
57
  | **VideoMME** | | | |
58
  | short, no subtitles | **81.7** | 79.7 | 81.1 |
59
  | medium, no subtitles | 70.3 | 70.6 | **74.1** |
 
74
  | TimeLens Charades | 48.1 | 60.6 | **61.3** |
75
  | TimeLens QVHighlights | 67.5 | 68.4 | **70.3** |
76
 
77
+ ### Image
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
78
 
79
+ | General perception and reading | Qwen3.5-4B | OV2.5-4B | Grounding and spatial | Qwen3.5-4B | OV2.5-4B |
80
+ |:---|:---:|:---:|:---|:---:|:---:|
81
+ | OCRBench v2 | **59.0** | 53.7 | RefSpatial-Bench | 29.1 | **62.1** |
82
+ | OCRBench | **85.2** | 79.8 | RefCOCO val (Acc@0.5) | 89.1 | **92.6** |
83
+ | PerceptionBench | **30.9** | 29.6 | RefCOCOg val (Acc@0.5) | 87.7 | **88.9** |
84
+ | MMStar | **66.9** | 63.9 | CrossPoint | 33.7 | **80.5** |
85
+ | MMBench-EN dev | **88.8** | 88.2 | SAT mcq circular | 68.7 | **74.0** |
86
+ | RealWorldQA | **74.4** | 73.9 | CV-Bench 2D | 80.7 | **83.3** |
87
+ | BLINK | 63.6 | **67.1** | CV-Bench 3D | 92.0 | **94.5** |
88
+ | V\*-Bench | 81.7 | **89.5** | EmbSpatial | 74.7 | **83.0** |
89
+ | DocVQA val | **95.2** | 92.9 | CRPE relation | 77.8 | **78.2** |
90
+ | ChartQA | **85.0** | 82.8 | ERQA | 43.5 | **46.5** |
91
+ | AI2D | **82.4** | 82.2 | MMSI-Bench | 31.4 | **33.3** |
92
+ | CountBench | **87.2** | 84.9 | MetaVQA | **61.8** | 61.0 |
93
+ | PixMo-Count | 63.1 | **69.9** | PointBench | 17.1 | **29.7** |
94
+ | VLMs are Blind | **80.4** | 77.2 | TraceSpatial 2D | **37.9** | 19.4 |
95
+ | VLMs are Biased | 25.0 | **25.3** | TraceSpatial 3D | 13.0 | **44.6** |
96
+ | MuirBench \* | 21.3 | **48.7** | | | |
97
 
98
+ <sub>Compared with Qwen3.5-4B, thinking disabled. **Bold** is the best score in each row; `--` means not available. Each side reports its best score over the settings tried, from single runs. `*` marks rows that are not matched comparisons: on TOMATO the two sides use different frame settings, and on MuirBench the Qwen score is rule-based only. Full settings and protocol will be given in the technical report.</sub>
 
 
99
 
100
+ ## Model Overview
101
 
102
+ | | |
103
+ |:---|:---|
104
+ | **Type** | instruction-tuned vision-language model, LLaVA-OneVision-2 architecture |
105
+ | **Parameters** | 5.25B total (4B language model plus vision tower and adapter) |
106
+ | **Language side** | 32 layers, hidden size 2560, FFN 9216, 16 attention heads / 4 KV heads, linear attention with one full-attention layer every 4 |
107
+ | **Vision side** | 24 layers, hidden size 1024, `patch_size=16`, `spatial_merge_size=3`, attention within a frame |
108
+ | **Vocabulary** | 248320, including the 8 streaming protocol tokens |
109
+ | **Context length** | 262144 |
110
+ | **Weights** | bfloat16, 3 safetensors shards, about 9.9 GB |
111
 
112
  ## Quickstart
113
 
114
+ The modeling code ships with the weights, so `trust_remote_code=True` is required.
115
 
116
  ```bash
117
  pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
 
119
  pip install decord # frame-sampling video path only
120
  ```
121
 
122
+ ### Image
123
 
124
  ```python
125
  import torch
 
148
  print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
149
  ```
150
 
151
+ For several images, put one `{"type": "image"}` per image in `content` and pass `images=` in the same order. The per-image budget defaults to `min_pixels=3136` / `max_pixels=4000000`.
152
 
153
+ ### Video: frame sampling
 
154
 
155
  ```python
156
  messages = [{"role": "user", "content": [
 
167
  ).to(model.device)
168
  ```
169
 
170
+ `target_fps` and `num_frames` are also accepted, and `max_frames` caps at 768. Each frame is prefixed with its real sample time, so temporal questions work directly:
 
 
171
 
172
  ```
173
  <0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
174
  <1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
175
  ```
176
 
177
+ ### Video: codec mode
178
+
179
+ Codec mode packs the informative regions of the video into a few canvases. Patches on one canvas may come from different source frames, and each keeps its own `(source frame, row, column)` position. Preprocessing comes from `codec-video-prep` (Linux x86_64, CPython 3.10/3.11; H.264, HEVC, VP9 and AV1 files):
180
+
181
+ ```bash
182
+ pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple \
183
+ "codec-video-prep==0.3.0"
184
+ codec-video-prep-doctor
185
+ ```
186
+
187
+ ```python
188
+ inputs = processor(text=text, videos=["example.mp4"],
189
+ video_backend="codec", return_tensors="pt").to(model.device)
190
+ ```
191
 
192
+ Defaults live in the `codec` field of `preprocessor_config.json` (512 candidate frames, 64 canvases, 313600 pixels per canvas) and can be overridden per call with `codec_config={"target_canvas": 32, ...}`.
 
193
 
194
+ ### Streaming
195
+
196
+ Frames arrive one per user turn, and each turn the model emits a state token.
197
 
198
  | Mode | Call | Default system |
199
  |---|---|---|
 
209
  | `<captioning>` | 248083 | this turn: speak, as event-level captioning |
210
  | `<caption_begin>` | 248084 | user-side marker that this stream is a captioning task |
211
 
212
+ Turns strictly alternate between user and assistant, with at most one frame per user turn. Give each frame its `timestamp`:
 
 
213
 
214
  ```python
215
  messages = [{"role": "user", "content": [
 
230
  <silence><|im_end|>
231
  ```
232
 
233
+ The eight tokens are ordinary added tokens, so they survive `skip_special_tokens=True`: the state token is the output you read. The real-time loop around the model (frame scheduling, stopping, memory re-injection) is not part of this repository.
234
+
235
+ ## Usage Tips
236
+
237
+ - **Pass video paths, not pre-decoded frames.** Only a path lets the processor recover real timestamps. If you decode yourself, pass `videos=[{"frames": ..., "timestamps": ..., "frame_indices": ...}]`.
238
+ - **Tune frames and pixels together.** Cost is roughly their product: 64–128 frames at 153664–321489 px for short video, 256 at 136900 for medium, 640–768 at 72900–108900 for long.
239
+ - **Forward `patch_positions` in codec mode.** It comes back in the processor output and must reach `model.forward(...)`.
240
+ - **Decode greedily and leave room.** Use `do_sample=False`, and a generous `max_new_tokens` on tasks where the model reasons before answering.
241
+ - **Ask for the format you will parse.** For multiple choice, add `Answer with the option's letter from the given choices directly.` For grounding, state the coordinate range, e.g. integers 0–1000.
242
+ - **Warm up before going multi-process.** Load the model once in a single process so the remote code is compiled before workers start.
243
+
244
+ ## Citation
245
+
246
+ ```bibtex
247
+ @misc{an2026llavaonevision2nextgenerationperceptualintelligence,
248
+ title = {LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence},
249
+ author = {Xiang An and Yin Xie and Feilong Tang and Yunyao Yan and Huajie Tan and Didi Zhu and Changrui Chen and Xiuwei Zhao and Bin Qin and Kaicheng Yang and Yifei Shen and Yuanhan Zhang and Kaichen Zhang and Wenkang Zhang and Zheng Cheng and Nansen Zhang and Chunsheng Wu and Chunjiang Ge and Zimin Ran and Dehua Song and Chunyuan Li and Shikun Feng and Ming Hu and Zhangquan Chen and Junbo Niu and Bo Li and Ziyong Feng and Ziwei Liu and Zongyuan Ge and Jiankang Deng},
250
+ year = {2026},
251
+ eprint = {2605.25979},
252
+ archivePrefix = {arXiv},
253
+ primaryClass = {cs.CV},
254
+ url = {https://arxiv.org/abs/2605.25979}
255
+ }
256
  ```
257
 
258
+ ## License
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
259
 
260
+ Apache-2.0.
README_CN.md CHANGED
@@ -9,77 +9,40 @@ tags:
9
  - llava-onevision
10
  ---
11
 
12
- [English](README.md) | **中文**
13
 
14
  # LLaVA-OneVision-2.5-4B-Instruct
15
 
16
- > 本仓库包含指令微调后的权重、随权重分发的建模代码,以及离线 / 流式两套对话模版。
17
-
18
- LLaVA-OneVision-2.5-4B-Instruct 是一个 4B 视觉语言模型,支持单图、多图和视频输入。除常规离线任务外,它还针对**流式**视频做了训练:画面逐帧到达,模型在每一步决定是保持沉默还是发言——两者共用同一份权重和同一个对话模版,用一个关键字切换。
19
-
20
- ## 亮点
21
-
22
- - **一个模型,两种模式。** `apply_chat_template(..., streaming=True)` 切换默认 system prompt;八个协议 token 承载每一帧"沉默 / 发言 / caption"的决策。除此之外输入格式在两种模式下完全相同。
23
- - **两条视频前端。** 均匀抽帧,或 **codec 模式**——用编码器的 bitcost 把信息量大的区域打包进少量画布。在本 checkpoint 上,两边都跑出结果的 21 个视频 bench 中,codec **赢 17 个**。
24
- - **视频与空间能力突出。** 对 Qwen3.5-4B(关闭 thinking),抽帧在 21 个可比视频 bench 中领先 **17 个**,codec 在 18 个中领先 **16 个**;在设置对齐的项上优势最大的是时间定位(TimeLens Charades +12.5)和视频空间(VSI-Bench +9.4)。
25
- - **图像侧如实呈现。** 静态图像上互有胜负——31 个可比项中领先 18 个,OCR 与文档阅读落后。下面的表格未作筛选。
26
-
27
- ## 模型概览
28
 
29
- | 项目 | 取值 |
30
- |---|---|
31
- | **类型** | 指令微调视觉语言模型,LLaVA-OneVision-2 架构 |
32
- | **参数量** | 合计 5.25B(4B 语言模型 + 视觉塔与 adapter) |
33
- | **语言侧** | 32 层,hidden size 2560,FFN 9216,16 个注意力头 / 4 个 KV 头,线性注意力每 4 层夹一个全注意力层 |
34
- | **视觉侧** | 24 层,hidden size 1024,`patch_size=16`,`spatial_merge_size=3`,帧内注意力 |
35
- | **词表** | 248320,含 8 个流式协议 token |
36
- | **上下文长度** | 262144 |
37
- | **权重** | bfloat16,3 个 safetensors 分片,约 9.9 GB |
38
 
39
- ## 评测结果
40
-
41
- OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**。图像只有一列分数,因为 codec 路径只适用于视频。空格(`--`)表示尚未测出或不适用。
42
 
43
- 分数统一保留一位小数;**加粗**为该行最高分,`*` 表示该行有注意事项,见表后的说明。逐项设置、说明和评测口径都收在各表下方的折叠区里。
44
 
45
- ### 图像
46
 
47
- | 通用感知与阅读 | Qwen3.5-4B | OV2.5-4B | 定位与空间 | Qwen3.5-4B | OV2.5-4B |
48
- |:---|---:|---:|:---|---:|---:|
49
- | OCRBench v2 | **59.0** | 53.7 | RefSpatial-Bench \* | 29.1 | **62.1** |
50
- | OCRBench | **85.2** | 79.8 | RefCOCO val (Acc@0.5) | 89.1 | **92.6** |
51
- | PerceptionBench \* | **30.9** | 29.6 | RefCOCOg val (Acc@0.5) | 87.7 | **88.9** |
52
- | MMStar | **66.9** | 63.9 | CrossPoint | 33.7 | **80.5** |
53
- | MMBench-EN dev \* | **88.8** | 88.2 | SAT mcq circular \* | 68.7 | **74.0** |
54
- | RealWorldQA | **74.4** | 73.9 | CV-Bench 2D | 80.7 | **83.3** |
55
- | BLINK \* | 63.6 | **67.1** | CV-Bench 3D | 92.0 | **94.5** |
56
- | V\*-Bench | 81.7 | **89.5** | EmbSpatial | 74.7 | **83.0** |
57
- | DocVQA val \* | **95.2** | 92.9 | CRPE relation | 77.8 | **78.2** |
58
- | ChartQA | **85.0** | 82.8 | ERQA | 43.5 | **46.5** |
59
- | AI2D | **82.4** | 82.2 | MMSI-Bench | 31.4 | **33.3** |
60
- | CountBench | **87.2** | 84.9 | MetaVQA | **61.8** | 61.0 |
61
- | PixMo-Count | 63.1 | **69.9** | PointBench \* | 17.1 | **29.7** |
62
- | VLMs are Blind | **80.4** | 77.2 | TraceSpatial 2D | **37.9** | 19.4 |
63
- | VLMs are Biased | 25.0 | **25.3** | TraceSpatial 3D | 13.0 | **44.6** |
64
- | MuirBench \* | 21.3 | **48.7** | | | |
65
 
66
- <details>
67
- <summary><b>图像结果说明</b></summary>
 
 
68
 
69
- - BLINK:63.56 是 Qwen 的 judge 分。其规则分为 54.97——抽取取回答首字母,而它约 15% 的回答先分析后给选项。
70
- - MuirBench:Qwen 只有规则分,很可能受同样的首字母解析影响,judge 未出,这一行暂不宜作为公平比较。
71
- - MMBench-EN dev:judge 为逐行准确率(4329 行);官方协议是循环评测。
72
- - DocVQA val:按 ANLS 规则分。PerceptionBench:由 gpt-oss-120b 判分。
73
- - PointBench:两边都输出 0~1000 坐标,而判分多按绝对像素算,因此两边绝对值都偏低。
74
- - RefSpatial-Bench:Qwen 为 2026-11-17 按现行判分脚本贪心重跑的结果。
75
- - SAT 自由作答与 SAT mcq 未列入——其 150 道题的正确选项全在第一位,一律答 A 即得 100%。表中报的是带循环一致性检查的 circular 变体。
76
 
77
- </details>
 
 
78
 
79
  ### 视频
80
 
81
  | Benchmark | Qwen3.5-4B | OV2.5-4B(抽帧) | OV2.5-4B(codec) |
82
- |:---|---:|---:|---:|
83
  | **通用视频理解** | | | |
84
  | MotionBench | 60.1 | 62.4 | **64.2** |
85
  | TOMATO \* | 34.9 | 33.6 | **38.6** |
@@ -87,10 +50,10 @@ OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**
87
  | NextQA MC test | 80.7 | 86.1 | **86.4** |
88
  | TempCompass MC | 73.4 | 75.8 | **79.7** |
89
  | MMVU val | **52.6** | 44.1 | -- |
90
- | Video-MME-v2 \* | 15.5 | 16.4 | **17.2** |
91
- | EgoSchema subset \* | 68.2 | **80.0** | -- |
92
- | Video-Index 512f \* | 19.6 | **20.0** | 19.8 |
93
- | MMOU test-mini \* | -- | 46.0 | -- |
94
  | **VideoMME** | | | |
95
  | short,无字幕 | **81.7** | 79.7 | 81.1 |
96
  | medium,无字幕 | 70.3 | 70.6 | **74.1** |
@@ -111,74 +74,44 @@ OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**
111
  | TimeLens Charades | 48.1 | 60.6 | **61.3** |
112
  | TimeLens QVHighlights | 67.5 | 68.4 | **70.3** |
113
 
114
- <details>
115
- <summary><b>视频逐项设置(帧数 × 每帧像素,或画布数 × 每画布像素)</b></summary>
116
-
117
- | Benchmark | Qwen3.5-4B | OV2.5-4B(抽帧) | OV2.5-4B(codec) |
118
- |:---|:---|:---|:---|
119
- | **通用视频理解** | | | |
120
- | MotionBench | 128f × 153,664 | 128f × 153,664 | TC128 × 313,600 |
121
- | TOMATO | 16f × 153,664, fps=2 | model default | TC128 × 313,600 |
122
- | MVBench | 64f × 153,664 | 64f × 153,664 | TC64 × 313,600 |
123
- | NextQA MC test | 64f × 153,664 | 64f × 153,664 | TC64 × 313,600 |
124
- | TempCompass MC | 64f × 153,664 | 64f × 153,664 | TC64 × 313,600 |
125
- | MMVU val | 64f × 153,664 | 64f × 153,664 | -- |
126
- | Video-MME-v2 | 64f × 330,000 | 64f × 330,000 | TC64 × 313,600 |
127
- | EgoSchema subset | 64f × 153,664 | 64f × 153,664 | -- |
128
- | Video-Index 512f | task frames, ≤262,144/f | task frames, ≤262,144/f | TC≤256 × 313,600 |
129
- | MMOU test-mini | -- | model default | -- |
130
- | **VideoMME** | | | |
131
- | short,无字幕 | 128f × 321,489 | 128f × 321,489 | TC256 × 313,600 |
132
- | medium,无字幕 | 256f × 136,900 | 256f × 136,900 | TC512 × 313,600 |
133
- | long,无字幕 | 1450f × 120,000 | 750f × 250,000 | TC512 × 313,600 |
134
- | short,有字幕 | -- | 128f × 321,489 | TC256 × 313,600 |
135
- | medium,有字幕 | -- | 256f × 136,900 | TC256 × 313,600 |
136
- | long,有字幕 | -- | 750f × 250,000 | TC256 × 313,600 |
137
- | **视频空间** | | | |
138
- | VSI-Bench | 128f × 153,664 | 128f × 153,664 | TC128 × 313,600 |
139
- | ReVSI 64f | 64f × 153,664 | 64f × 153,664 | TC64 × 313,600 |
140
- | **长视频** | | | |
141
- | LVBench | 2048f × 120,000 | 950f × 200,000 | TC1024 × 156,800 |
142
- | MLVU dev | 512f × 72,900 | 512f × 72,900 | -- |
143
- | LongVideoBench val | 700f × 300,000 | 1280f × 80,000 | TC768 × 235,200 |
144
- | VideoEval-Pro | 2048f × 100,000 | 950f × 200,000 | TC768 × 235,200 |
145
- | **时间定位(mIoU)** | | | |
146
- | TimeLens ActivityNet | 2048f × 409,600, fps=4 | 2048f × 409,600, fps=4 | TC128 × 313,600 |
147
- | TimeLens Charades | 2048f × 409,600, fps=4 | 2048f × 409,600, fps=4 | TC128 × 313,600 |
148
- | TimeLens QVHighlights | 2048f × 409,600, fps=4 | 2048f × 409,600, fps=4 | TC128 × 313,600 |
149
-
150
- </details>
151
-
152
- <details>
153
- <summary><b>视频结果说明</b></summary>
154
-
155
- - TOMATO:Qwen 为 fps=2 / 16 帧;我方未传帧数和像素参数,用的是模型默认值。
156
- - Video-MME-v2:字幕送法两边不同。
157
- - Video-Index 512f:帧由任务以图片给出,两边都是多图输入。codec 一列取自官方预算测试(TC256 上限);同一批的抽帧对照(视频文件输入)为 19.05。
158
- - EgoSchema:公开答案的 500 题子集。全量 5031 题答案不公开。
159
- - VideoMME 有字幕三项与 MMOU:Qwen 在官方路径下没有可比结果。Qwen 的 VideoMME long 有字幕那次运行无效(judge 记录为 0/900),故留空而非报出。
160
- - 设置列是各自取得该分数时的实际设置。多数行两边一致,有两行不是:**TOMATO** Qwen 为 16 帧 fps=2 而我方用模型默认值;**MMOU** 只有我方有分。
161
- - 对「取多设置最高分」的行,两边搜索量并不对等,且多数对 Qwen 更有利:VideoMME long 无字幕 28 个设置对我方 6 个,LVBench 27 对 16,LongVideoBench 28 对 2,VideoEval-Pro 28 对 2。
162
- - codec 一列为规则分——回答均为选项字母,与 judge 等价。
163
-
164
- </details>
165
-
166
- <details>
167
- <summary><b>评测口径</b></summary>
168
 
169
- Qwen3.5-4B 关闭 thinking。它有一部分结果实际来自 temperature=0.7 采样而非贪心解码;已做过贪心重跑的行用重跑结果。所有数字都是单次运行,没有方差估计。
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
170
 
171
- - **图像。** OV2.5 用 processor 的默认像素预算;Qwen 为每图 65,536~16,777,216 px。
172
- - **视频。** 按 fps=1 抽帧并离线缓存,评测时不解码,重复跑输入完全一致。每个任务两边各取所有出分设置中的最高分,因此帧数和逐帧像素预算逐行、逐模型都不同——例如 LVBench,Qwen 的最高分出自 27 个设置,我们出自 16 个。codec 一列用 `preprocessor_config.json` 中 `codec` 字段的画布预算,按任务调整。
173
- - **判分。** *judge*——两边都由 Qwen3.8-27B(温度 0)复判,它看到问题、选项、参考答案和原始输出,回答 YES 或 NO;最终分取规则分与 judge 分中的较大者。*规则分(官方指标)*——按 bench 自己的指标计算。*规则分(judge 未出)*——本应用 judge 但至少一边尚未判:MuirBench、EmbSpatial、CRPE relation、ERQA、MetaVQA、NextQA MC test、TempCompass MC、EgoSchema subset、Video-Index 512f。
174
 
175
- 所有生成都是贪心,`batch_size=1`。
176
 
177
- </details>
 
 
 
 
 
 
 
 
178
 
179
  ## 快速开始
180
 
181
- > 建模代码随权重分发,所有入口都必须 `trust_remote_code=True`。
182
 
183
  ```bash
184
  pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
@@ -215,10 +148,9 @@ with torch.inference_mode():
215
  print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
216
  ```
217
 
218
- 图片走 `smart_resize`,保持长宽比并对齐 patch 网格,预算默认 `min_pixels=3136` / `max_pixels=4000000`。多图时每张图放���个 `{"type": "image"}`,`images=` 按同样顺序传入。
219
 
220
- <details>
221
- <summary><b>视频 — 抽帧</b></summary>
222
 
223
  ```python
224
  messages = [{"role": "user", "content": [
@@ -235,21 +167,33 @@ inputs = processor(
235
  ).to(model.device)
236
  ```
237
 
238
- 也接受 `target_fps` 和 `num_frames`,优先级见 `video_processing_llava_onevision2_5.py` 的 `choose_target_frames`。`max_frames` 封顶 768。
239
-
240
- 时间戳一定会注入:`<|video_pad|>` 占位符展开成逐帧的块,每块前面是该帧的真实采样时刻,因此可以直接问时间定位类问题。
241
 
242
  ```
243
  <0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
244
  <1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
245
  ```
246
 
247
- </details>
 
 
 
 
 
 
 
 
 
 
 
 
 
248
 
249
- <details>
250
- <summary><b>流式</b></summary>
251
 
252
- 画面每轮进来一帧,模型每轮输出一个状态 token。一个关键字切换默认 system,其余什么都不改;显式传入的 `system` 在两种模式下都优先。
 
 
253
 
254
  | 模式 | 调用 | 默认 system |
255
  |---|---|---|
@@ -265,9 +209,7 @@ inputs = processor(
265
  | `<captioning>` | 248083 | 当前轮发言,且是事件级 caption |
266
  | `<caption_begin>` | 248084 | user 侧标记,说明这条流是 caption 任务 |
267
 
268
- 八个都是非 special 的 added token,所以 `skip_special_tokens=True` 仍会解出来——状态 token 正是你要读的输出。两个 memory token 内部包 JSON,例如 `{"timestep": "00:00:06", "content": "..."}`。
269
-
270
- 严格 user / assistant 交替,每个 user 轮最多一帧;没有新画面的轮次改用纯文本时间前缀。每帧给上 `timestamp`,帧后的换行作为单独的 text 项:
271
 
272
  ```python
273
  messages = [{"role": "user", "content": [
@@ -288,55 +230,31 @@ text = processor.apply_chat_template(
288
  <silence><|im_end|>
289
  ```
290
 
291
- content 传字符串则原样输出,适合自己已经拼好标记的 harness。
292
-
293
- > **本仓库不包含:** 围绕模型的实时循环——按时钟取帧、决定何时停止生成、到期回插 question / answer memory、在决策点之间维持上下文。
294
-
295
- </details>
296
-
297
- <details>
298
- <summary><b>视频 — codec 模式</b></summary>
299
-
300
- codec 模式用编码器 bitcost 把信息量大的区域打包进少量**画布**。同一张画布上的 patch 可以来自不同源帧,各自带 `(源帧, 行, 列)` 供 3-D RoPE 使用。
301
-
302
- 预处理来自 `codec-video-prep`(自带打过补丁的 FFmpeg 和 dav1d;限 Linux x86_64、CPython 3.10/3.11、`numpy<2`、系统 `ffprobe`;只支持 H.264/HEVC/VP9/AV1;只适用于离线文件,因为预算计算需要整段视频):
303
-
304
- ```bash
305
- pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple \
306
- "codec-video-prep==0.3.0"
307
- codec-video-prep-doctor
308
- ```
309
-
310
- ```python
311
- inputs = processor(text=text, videos=["example.mp4"],
312
- video_backend="codec", return_tensors="pt").to(model.device)
 
313
  ```
314
 
315
- 每张画布只有一个时间标记:所有 patch 来自同一源帧时是点 `<6.5 seconds>`,跨多帧时是区间 `<0.0 to 1.8 seconds>`(schema `canvas_patch_span_point_range_v1`)。
316
-
317
- 默认配方写在 `preprocessor_config.json` 的 `codec` 字段——512 候选帧、64 画布、patch 16、3×3 块、每画布 313600 像素——单次调用用 `codec_config={"target_canvas": 32, ...}` 覆盖。其余键:`strategy`(`adaptive_pwindow`、`adaptive_rdo`、`readiness`)、`num_sampled_frames`、`max_pixels`、`canvas_format`。画布缓存在 `~/.cache/codec_video_prep/canvases`(用 `CVP_CODEC_CACHE_ROOT` 换位置)。
318
-
319
- </details>
320
-
321
- ## 最佳实践
322
-
323
- <details>
324
- <summary><b>展开 8 条实践</b></summary>
325
-
326
- 1. **传视频路径,不要传自己解码好的帧。** 只有路径才能让 processor 拿到真实采样时刻和源帧号。只给一串帧时它会退化成用帧序号当秒数,于是 100 秒的视频看起来只有 N 秒,时间类回答全错(此时会抛 `UserWarning`)。必须自己解码就把时间一并交出:`videos=[{"frames": ..., "timestamps": ..., "frame_indices": ...}]`。通过 `images=` 传帧则完全不注入时间戳。
327
-
328
- 2. **保持 `patch_positions` 为真。** codec 模式下它在 processor 输出里,必须转发给 `model.forward(...)`,不要丢掉。流式下每帧作为独立 image 交付,块布局是对的但 `t` 每帧都是 0,需要改成 `floor(秒数 × fps)`,否则每帧都会被当成一段新视觉的开头。模型只用 `t` 的**名次**,但写真实帧号能让输入可审计。
329
-
330
- 3. **帧数与逐帧像素一起调。** 开销约等于两者之积:短视频 64~128 帧 × 153664~321489 px,中等 256 帧 × 136900,长视频 640~768 帧 × 72900~108900。超长输入设 `OV_VISION_CHUNK_PATCHES=200000`,它按帧边界分组编码,注意力不跨帧,结果不变。
331
-
332
- 4. **codec 几何必须对齐模型。** `patch=16` 和 `block_size=3` 必须等于 `patch_size` 和 `spatial_merge_size`,不一致时预处理直接报错而非静默出错。
333
-
334
- 5. **贪心解码,并留足推理长度。** `generation_config.json` 只设了 `eos_token_id=248044` 和 `use_cache=true`。确定性输出用 `do_sample=False`;需要先推理再作答的任务要给足 `max_new_tokens`,128 会在中途截断。批量时传一组 text 并设 `padding=True`,padding 在左侧,按各自的 `input_ids` 长度切输出。
335
-
336
- 6. **要什么格式就在 prompt 里说。** 选择题上模型可能写整句而不是一个字母;如果后续用正则解析,就要求给字母:`Answer with the option's letter from the given choices directly.`。grounding 任务请写明坐标范围,例如 0~1000 的整数。
337
-
338
- 7. **多进程前先单进程预热 remote code。** 模块会编译到 `$HF_HOME/modules/transformers_modules/<名字>/`,多进程首次加载会在那里竞争,所以先单进程完整加载一次。该缓存只按叶子目录名索引——本地副本要取不同的目录名。
339
-
340
- 8. **不需要流式 token 就显式剥除。** 八个都按设计不会被 `skip_special_tokens=True` 过滤。
341
 
342
- </details>
 
9
  - llava-onevision
10
  ---
11
 
12
+ <div align="center">
13
 
14
  # LLaVA-OneVision-2.5-4B-Instruct
15
 
16
+ **一个 4B 模型,同时处理图像、长视频和实时视频流**
 
 
 
 
 
 
 
 
 
 
 
17
 
18
+ <a href="https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-2"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-LLaVA--OneVision-181717?logo=github"></a>
19
+ <a href="https://evolvinglmms-lab.github.io/LLaVA-OneVision-2/"><img alt="Project Page" src="https://img.shields.io/badge/Project-Page-2a78d6"></a>
20
+ <a href="https://arxiv.org/abs/2605.25979"><img alt="OV2 Report" src="https://img.shields.io/badge/arXiv-2605.25979-b31b1b"></a>
21
+ <img alt="License" src="https://img.shields.io/badge/License-Apache--2.0-1baf7a">
 
 
 
 
 
22
 
23
+ [English](README.md) | **中文**
 
 
24
 
25
+ </div>
26
 
27
+ LLaVA-OneVision-2.5-4B-Instruct 是一个 4B 视觉语言模型,支持单图、多图和视频输入。除常规离线任务外,它还针对**流式**视频做了训练:画面逐帧到达,模型在每一步决定保持沉默还是发言。两种模式共用同一份权重和同一个对话模版。
28
 
29
+ ## 亮点
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
30
 
31
+ - **离线与流式合一。** `apply_chat_template(..., streaming=True)` 即可切换模式;八个协议 token 承载每一帧"沉默 / 回答 / caption"的决策,输入格式的其余部分完全相同。
32
+ - **Codec 对齐的视频输入。** 除均匀抽帧外,**codec 模���**利用编码器的 bit-cost,把视频中信息量大的区域打包进少量画布。在 21 个视频 bench 中,它有 **17 个**优于抽帧。
33
+ - **9 倍空间压缩。** patch 16 的视觉塔加 3×3 合并,每 48×48 像素对应一个视觉 token,长视频可以从容放进 262k 上下文。
34
+ - **视频、定位与空间能力突出。** 对比 Qwen3.5-4B,抽帧在 21 个视频 bench 中领先 **17 个**,codec 在 18 个中领先 **16 个**,优势最大的是时间定位(TimeLens Charades +12.5)和视频空间(VSI-Bench +9.4)。静态图像上 31 项领先 18 项,主要是定位与空间任务;OCR 与文档阅读落后。
35
 
36
+ ## 评测结果
 
 
 
 
 
 
37
 
38
+ <p align="center">
39
+ <img src="assets/video_benchmarks.png" width="92%" alt="视频评测:LLaVA-OneVision-2.5-4B 对比 Qwen3.5-4B">
40
+ </p>
41
 
42
  ### 视频
43
 
44
  | Benchmark | Qwen3.5-4B | OV2.5-4B(抽帧) | OV2.5-4B(codec) |
45
+ |:---|:---:|:---:|:---:|
46
  | **通用视频理解** | | | |
47
  | MotionBench | 60.1 | 62.4 | **64.2** |
48
  | TOMATO \* | 34.9 | 33.6 | **38.6** |
 
50
  | NextQA MC test | 80.7 | 86.1 | **86.4** |
51
  | TempCompass MC | 73.4 | 75.8 | **79.7** |
52
  | MMVU val | **52.6** | 44.1 | -- |
53
+ | Video-MME-v2 | 15.5 | 16.4 | **17.2** |
54
+ | EgoSchema subset | 68.2 | **80.0** | -- |
55
+ | Video-Index 512f | 19.6 | **20.0** | 19.8 |
56
+ | MMOU test-mini | -- | 46.0 | -- |
57
  | **VideoMME** | | | |
58
  | short,无字幕 | **81.7** | 79.7 | 81.1 |
59
  | medium,无字幕 | 70.3 | 70.6 | **74.1** |
 
74
  | TimeLens Charades | 48.1 | 60.6 | **61.3** |
75
  | TimeLens QVHighlights | 67.5 | 68.4 | **70.3** |
76
 
77
+ ### 图像
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
78
 
79
+ | 通用感知与阅读 | Qwen3.5-4B | OV2.5-4B | 定位与空间 | Qwen3.5-4B | OV2.5-4B |
80
+ |:---|:---:|:---:|:---|:---:|:---:|
81
+ | OCRBench v2 | **59.0** | 53.7 | RefSpatial-Bench | 29.1 | **62.1** |
82
+ | OCRBench | **85.2** | 79.8 | RefCOCO val (Acc@0.5) | 89.1 | **92.6** |
83
+ | PerceptionBench | **30.9** | 29.6 | RefCOCOg val (Acc@0.5) | 87.7 | **88.9** |
84
+ | MMStar | **66.9** | 63.9 | CrossPoint | 33.7 | **80.5** |
85
+ | MMBench-EN dev | **88.8** | 88.2 | SAT mcq circular | 68.7 | **74.0** |
86
+ | RealWorldQA | **74.4** | 73.9 | CV-Bench 2D | 80.7 | **83.3** |
87
+ | BLINK | 63.6 | **67.1** | CV-Bench 3D | 92.0 | **94.5** |
88
+ | V\*-Bench | 81.7 | **89.5** | EmbSpatial | 74.7 | **83.0** |
89
+ | DocVQA val | **95.2** | 92.9 | CRPE relation | 77.8 | **78.2** |
90
+ | ChartQA | **85.0** | 82.8 | ERQA | 43.5 | **46.5** |
91
+ | AI2D | **82.4** | 82.2 | MMSI-Bench | 31.4 | **33.3** |
92
+ | CountBench | **87.2** | 84.9 | MetaVQA | **61.8** | 61.0 |
93
+ | PixMo-Count | 63.1 | **69.9** | PointBench | 17.1 | **29.7** |
94
+ | VLMs are Blind | **80.4** | 77.2 | TraceSpatial 2D | **37.9** | 19.4 |
95
+ | VLMs are Biased | 25.0 | **25.3** | TraceSpatial 3D | 13.0 | **44.6** |
96
+ | MuirBench \* | 21.3 | **48.7** | | | |
97
 
98
+ <sub>对比对象为关闭 thinking 的 Qwen3.5-4B。**加粗**为该行最高分,`--` 表示暂无结果。两边各取已测设置中的最高分,均为单次运行。带 `*` 的行两边设置不对齐:TOMATO 两边帧数设置不同,MuirBench 的 Qwen 分数仅为规则分。完整设置与评测口径见后续技术报告。</sub>
 
 
99
 
100
+ ## 模型概览
101
 
102
+ | | |
103
+ |:---|:---|
104
+ | **类型** | 指令微调视觉语言模型,LLaVA-OneVision-2 架构 |
105
+ | **参数量** | 合计 5.25B(4B 语言模型 + 视觉塔与 adapter) |
106
+ | **语言侧** | 32 层,hidden size 2560,FFN 9216,16 个注意力头 / 4 个 KV 头,线性注意力每 4 层夹一个全注意力层 |
107
+ | **视觉侧** | 24 层,hidden size 1024,`patch_size=16`,`spatial_merge_size=3`,帧内注意力 |
108
+ | **词表** | 248320,含 8 个流式协议 token |
109
+ | **上下文长度** | 262144 |
110
+ | **权重** | bfloat16,3 个 safetensors 分片,约 9.9 GB |
111
 
112
  ## 快速开始
113
 
114
+ 建模代码随权重分发,加载时需要 `trust_remote_code=True`。
115
 
116
  ```bash
117
  pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
 
148
  print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
149
  ```
150
 
151
+ 多图时每张图放一个 `{"type": "image"}`,`images=` 按同样顺序传入。单图像素预算默认 `min_pixels=3136` / `max_pixels=4000000`。
152
 
153
+ ### 视频:抽帧
 
154
 
155
  ```python
156
  messages = [{"role": "user", "content": [
 
167
  ).to(model.device)
168
  ```
169
 
170
+ 也接受 `target_fps` 和 `num_frames`,`max_frames` 上限 768。每帧前面带该帧的真实采样时刻,可以直接问时间类问题:
 
 
171
 
172
  ```
173
  <0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
174
  <1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
175
  ```
176
 
177
+ ### 视频:codec 模式
178
+
179
+ codec 模式把视频中信息量大的区域打包进少量画布。同一张画布上的 patch 可以来自不同源帧,各自保留 `(源帧, 行, 列)` 位置。预处理来自 `codec-video-prep`(Linux x86_64,CPython 3.10/3.11;支持 H.264、HEVC、VP9、AV1 文件):
180
+
181
+ ```bash
182
+ pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple \
183
+ "codec-video-prep==0.3.0"
184
+ codec-video-prep-doctor
185
+ ```
186
+
187
+ ```python
188
+ inputs = processor(text=text, videos=["example.mp4"],
189
+ video_backend="codec", return_tensors="pt").to(model.device)
190
+ ```
191
 
192
+ 默认配置在 `preprocessor_config.json` 的 `codec` 字段(512 候选帧、64 画布、每画布 313600 像素),单次调用可用 `codec_config={"target_canvas": 32, ...}` 覆盖。
 
193
 
194
+ ### 流式
195
+
196
+ 画面每个 user 轮进来一帧,模型每轮输出一个状态 token。
197
 
198
  | 模式 | 调用 | 默认 system |
199
  |---|---|---|
 
209
  | `<captioning>` | 248083 | 当前轮发言,且是事件级 caption |
210
  | `<caption_begin>` | 248084 | user 侧标记,说明这条流是 caption 任务 |
211
 
212
+ user 与 assistant 严格交替,每个 user 轮最多一帧。每帧给上 `timestamp`:
 
 
213
 
214
  ```python
215
  messages = [{"role": "user", "content": [
 
230
  <silence><|im_end|>
231
  ```
232
 
233
+ 八个 token 都是普通 added token,`skip_special_tokens=True` 不会过滤它们:状态 token 就是要读取的输出。围绕模型的实时循环(取帧调度、停止生成、记忆回插)不在本仓库内。
234
+
235
+ ## 使用建议
236
+
237
+ - **传视频路径,不要传自己解码好的帧。** 只有路径才能让 processor 拿到真实时间戳。必须自己解码时,传 `videos=[{"frames": ..., "timestamps": ..., "frame_indices": ...}]`。
238
+ - **帧数与像素一起调。** 开销约等于两者之积:短视频 64~128 帧 × 153664~321489 px,中等 256 帧 × 136900,长视频 640~768 帧 × 72900~108900。
239
+ - **codec 模式下转发 `patch_positions`。** 它在 processor 输出里,必须传给 `model.forward(...)`。
240
+ - **贪心解码并留足长度。** 用 `do_sample=False`;需要先推理再作答的任务给足 `max_new_tokens`。
241
+ - **要什么格式就在 prompt 里说。** 选择题加上 `Answer with the option's letter from the given choices directly.`;grounding 任务写明坐标范围,例如 0~1000 的整数。
242
+ - **多进程前先预热。** 先在单进程里完整加载一次模型,让 remote code 编译完成后再启动多个 worker。
243
+
244
+ ## 引用
245
+
246
+ ```bibtex
247
+ @misc{an2026llavaonevision2nextgenerationperceptualintelligence,
248
+ title = {LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence},
249
+ author = {Xiang An and Yin Xie and Feilong Tang and Yunyao Yan and Huajie Tan and Didi Zhu and Changrui Chen and Xiuwei Zhao and Bin Qin and Kaicheng Yang and Yifei Shen and Yuanhan Zhang and Kaichen Zhang and Wenkang Zhang and Zheng Cheng and Nansen Zhang and Chunsheng Wu and Chunjiang Ge and Zimin Ran and Dehua Song and Chunyuan Li and Shikun Feng and Ming Hu and Zhangquan Chen and Junbo Niu and Bo Li and Ziyong Feng and Ziwei Liu and Zongyuan Ge and Jiankang Deng},
250
+ year = {2026},
251
+ eprint = {2605.25979},
252
+ archivePrefix = {arXiv},
253
+ primaryClass = {cs.CV},
254
+ url = {https://arxiv.org/abs/2605.25979}
255
+ }
256
  ```
257
 
258
+ ## 许可
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
259
 
260
+ Apache-2.0。
assets/video_benchmarks.png ADDED

Git LFS Details

  • SHA256: 58a514fbc94a524df77c369e8c65990cbe50de944baf586bca1ed9b3826c35f6
  • Pointer size: 131 Bytes
  • Size of remote file: 219 kB