Image-Text-to-Text
Transformers
Safetensors
llava_onevision2_5
text-generation
multimodal
video-understanding
streaming
llava-onevision
conversational
custom_code
Instructions to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct
- SGLang
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with Docker Model Runner:
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct
Tighten model cards (-47% lines)
Browse files- README.md +70 -283
- README_CN.md +66 -279
README.md
CHANGED
|
@@ -13,25 +13,19 @@ tags:
|
|
| 13 |
|
| 14 |
# LLaVA-OneVision-2.5-4B-Instruct
|
| 15 |
|
| 16 |
-
|
| 17 |
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
The modeling code ships with the weights, so **`trust_remote_code=True` is required**.
|
| 21 |
-
|
| 22 |
-
---
|
| 23 |
|
| 24 |
## Install
|
| 25 |
|
| 26 |
```bash
|
| 27 |
pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
|
| 28 |
-
pip install flash-attn --no-build-isolation # optional,
|
| 29 |
-
pip install decord #
|
| 30 |
```
|
| 31 |
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
## Single image
|
| 35 |
|
| 36 |
```python
|
| 37 |
import torch
|
|
@@ -42,57 +36,30 @@ MODEL = "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct"
|
|
| 42 |
|
| 43 |
processor = AutoProcessor.from_pretrained(MODEL, trust_remote_code=True)
|
| 44 |
model = AutoModelForImageTextToText.from_pretrained(
|
| 45 |
-
MODEL,
|
| 46 |
-
|
| 47 |
-
dtype=torch.bfloat16,
|
| 48 |
-
attn_implementation="flash_attention_2", # use "sdpa" without flash-attn
|
| 49 |
device_map="auto",
|
| 50 |
).eval()
|
| 51 |
|
| 52 |
-
image = Image.open("example.jpg").convert("RGB")
|
| 53 |
messages = [{"role": "user", "content": [
|
| 54 |
{"type": "image"},
|
| 55 |
{"type": "text", "text": "Describe this image in detail."},
|
| 56 |
]}]
|
| 57 |
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 58 |
-
inputs = processor(text=text, images=[
|
|
|
|
| 59 |
|
| 60 |
with torch.inference_mode():
|
| 61 |
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
|
| 62 |
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 63 |
```
|
| 64 |
|
| 65 |
-
Images go through `smart_resize`,
|
| 66 |
-
|
| 67 |
-
```python
|
| 68 |
-
processor.image_processor.min_pixels = 256 * 28 * 28
|
| 69 |
-
processor.image_processor.max_pixels = 1280 * 28 * 28
|
| 70 |
-
```
|
| 71 |
-
|
| 72 |
-
---
|
| 73 |
-
|
| 74 |
-
## Multiple images
|
| 75 |
-
|
| 76 |
-
Put one `{"type": "image"}` per image in `content`, and pass `images` in the same order:
|
| 77 |
-
|
| 78 |
-
```python
|
| 79 |
-
messages = [{"role": "user", "content": [
|
| 80 |
-
{"type": "image"},
|
| 81 |
-
{"type": "image"},
|
| 82 |
-
{"type": "text", "text": "What changed between these two images?"},
|
| 83 |
-
]}]
|
| 84 |
-
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 85 |
-
inputs = processor(text=text, images=[img_a, img_b], return_tensors="pt").to(model.device)
|
| 86 |
-
```
|
| 87 |
-
|
| 88 |
-
Each placeholder expands to `<|vision_start|><|image_pad|><|vision_end|>`, matched to the `images` list by position.
|
| 89 |
|
| 90 |
-
|
| 91 |
|
| 92 |
## Video — frame sampling
|
| 93 |
|
| 94 |
-
Pass the **video path**, not pre-decoded frames:
|
| 95 |
-
|
| 96 |
```python
|
| 97 |
messages = [{"role": "user", "content": [
|
| 98 |
{"type": "video"},
|
|
@@ -101,24 +68,16 @@ messages = [{"role": "user", "content": [
|
|
| 101 |
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 102 |
|
| 103 |
inputs = processor(
|
| 104 |
-
text=text,
|
| 105 |
-
videos=["example.mp4"],
|
| 106 |
max_frames=64,
|
| 107 |
-
min_pixels=153664,
|
| 108 |
-
max_pixels=153664, # for video, set min and max equal to fix the per-frame budget
|
| 109 |
return_tensors="pt",
|
| 110 |
).to(model.device)
|
| 111 |
-
|
| 112 |
-
with torch.inference_mode():
|
| 113 |
-
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
|
| 114 |
-
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 115 |
```
|
| 116 |
|
| 117 |
-
`
|
| 118 |
-
|
| 119 |
-
### Frames and per-frame pixels
|
| 120 |
|
| 121 |
-
Cost is roughly
|
| 122 |
|
| 123 |
| Case | Frames | Pixels per frame |
|
| 124 |
|---|---|---|
|
|
@@ -126,47 +85,33 @@ Cost is roughly **frames × pixels per frame**, so tune the two together. The pr
|
|
| 126 |
| Medium | 256 | 136900 |
|
| 127 |
| Long video, global understanding | 640 – 768 | 72900 – 108900 |
|
| 128 |
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
```bash
|
| 132 |
-
export OV_VISION_CHUNK_PATCHES=200000 # 0 (default) = off
|
| 133 |
-
```
|
| 134 |
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
### Timestamps are injected automatically
|
| 138 |
-
|
| 139 |
-
Video input always carries timestamps; this cannot be turned off. The `<|vision_start|><|video_pad|><|vision_end|>` placeholder is rewritten into one block per frame, each prefixed with its time:
|
| 140 |
|
| 141 |
```
|
| 142 |
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 143 |
<1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 144 |
```
|
| 145 |
|
| 146 |
-
|
| 147 |
|
| 148 |
-
|
| 149 |
-
|
| 150 |
-
|
| 151 |
-
> ```python
|
| 152 |
-
> videos=[{"frames": frames, "timestamps": seconds_list, "frame_indices": index_list}]
|
| 153 |
-
> ```
|
| 154 |
-
> Passing frames through `images=` injects no timestamps at all.
|
| 155 |
-
|
| 156 |
-
---
|
| 157 |
|
| 158 |
-
|
| 159 |
|
| 160 |
-
|
| 161 |
|
| 162 |
-
|
| 163 |
|
| 164 |
-
| Mode | Call | Default system
|
| 165 |
|---|---|---|
|
| 166 |
| Offline (default) | `apply_chat_template(messages, ...)` | `You are a helpful assistant.` |
|
| 167 |
| Streaming | `apply_chat_template(messages, ..., streaming=True)` | `You are a real-time streaming video assistant.` |
|
| 168 |
|
| 169 |
-
|
| 170 |
|
| 171 |
### Protocol tokens
|
| 172 |
|
|
@@ -174,36 +119,16 @@ That default system turn is the **only** thing the switch changes �� everything
|
|
| 174 |
|---|---|---|
|
| 175 |
| `<silence>` | 248077 | this turn: say nothing |
|
| 176 |
| `<response>` | 248078 | this turn: speak |
|
| 177 |
-
| `<question_memory>` / `</question_memory>` | 248079 / 248080 | a new question
|
| 178 |
-
| `<answer_memory>` / `</answer_memory>` | 248081 / 248082 | an earlier answer re-injected
|
| 179 |
-
| `<captioning>` | 248083 | this turn: speak, as event-level captioning
|
| 180 |
| `<caption_begin>` | 248084 | user-side marker that this stream is a captioning task |
|
| 181 |
|
| 182 |
-
All eight are
|
| 183 |
-
|
| 184 |
-
`<question_memory>` and `<answer_memory>` wrap a JSON object, for example `{"timestep": "00:00:06", "content": "..."}`, where `timestep` is the original wall-clock moment in `HH:MM:SS`.
|
| 185 |
-
|
| 186 |
-
### Turn layout
|
| 187 |
-
|
| 188 |
-
Strict user/assistant alternation. Each user turn carries at most one frame; a turn with no new frame uses a bare text time prefix instead.
|
| 189 |
|
| 190 |
-
|
| 191 |
-
<|im_start|>system
|
| 192 |
-
You are a real-time streaming video assistant.<|im_end|>
|
| 193 |
-
<|im_start|>user
|
| 194 |
-
<question_memory>{"timestep": "00:00:00", "content": "..."}</question_memory>
|
| 195 |
-
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 196 |
-
<|im_end|>
|
| 197 |
-
<|im_start|>assistant
|
| 198 |
-
<silence><|im_end|>
|
| 199 |
-
<|im_start|>user
|
| 200 |
-
<1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 201 |
-
<|im_end|>
|
| 202 |
-
<|im_start|>assistant
|
| 203 |
-
<response><|im_end|>
|
| 204 |
-
```
|
| 205 |
|
| 206 |
-
|
| 207 |
|
| 208 |
```python
|
| 209 |
messages = [{"role": "user", "content": [
|
|
@@ -215,195 +140,76 @@ text = processor.apply_chat_template(
|
|
| 215 |
messages, tokenize=False, add_generation_prompt=True, streaming=True)
|
| 216 |
```
|
| 217 |
|
| 218 |
-
|
| 219 |
-
|
| 220 |
-
### Two values the caller must compute
|
| 221 |
-
|
| 222 |
-
The template guarantees the **format**. Two **numbers** depend on the video and cannot be recovered from a bare frame, so you have to supply them:
|
| 223 |
-
|
| 224 |
-
1. **The text timestamp.** Pass `timestamp = source_frame_index / fps` for each frame, using the video's real frame rate.
|
| 225 |
-
2. **The `t` column of `patch_positions`.** If each frame is handed to the processor as an independent image, the returned `patch_positions` are in the right block layout but `t` is **0 for every frame**. Rewrite each frame's rows to its real source frame index, `floor(seconds × fps)`. Leave it at 0 and every frame reads as the start of a new visual segment, which is not what the model was trained on.
|
| 226 |
-
|
| 227 |
-
The model uses `t` only through its **rank** within the visual segment (0, 1, 2, …), never its absolute value, and the vision tower attends within a frame only — so what matters is that the order is right, not the exact numbers. Keeping them equal to the true frame numbers simply makes the input auditable.
|
| 228 |
-
|
| 229 |
-
The automatic per-frame timestamp expansion described above belongs to the offline `<|video_pad|>` branch. Streaming feeds frames one at a time and does not go through it.
|
| 230 |
-
|
| 231 |
-
> **What is not in this repository.** These weights give you the trained behavior, the template and the tokens. The real-time loop around them — grabbing frames on a clock, deciding when to stop generating, re-injecting question and answer memory as it falls due, and carrying context across decision points — is serving-side work and is not shipped here.
|
| 232 |
-
|
| 233 |
-
---
|
| 234 |
-
|
| 235 |
-
## Video — codec mode
|
| 236 |
-
|
| 237 |
-
Instead of sampling whole frames, codec mode uses the encoder's bitcost to find the informative regions and packs them into a small number of **canvases**.
|
| 238 |
-
|
| 239 |
-
The key difference: **patches on one canvas can come from several different source frames.** Every patch carries its own source position `(source frame, source patch row, source patch column)`, and the vision tower uses those for 3-D RoPE. Feeding canvases through `images=` throws that away.
|
| 240 |
-
|
| 241 |
-
### Install the preprocessing package
|
| 242 |
-
|
| 243 |
-
Codec preprocessing comes from `codec-video-prep`, which bundles patched FFmpeg and dav1d for bitcost extraction — nothing to compile:
|
| 244 |
-
|
| 245 |
-
```bash
|
| 246 |
-
pip install -i https://test.pypi.org/simple/ \
|
| 247 |
-
--extra-index-url https://pypi.org/simple \
|
| 248 |
-
"codec-video-prep==0.3.0"
|
| 249 |
-
```
|
| 250 |
-
|
| 251 |
-
`--extra-index-url` is required: the package is on TestPyPI, its dependencies (numpy, opencv-python-headless, Pillow, PyYAML) are on PyPI proper.
|
| 252 |
-
|
| 253 |
-
Check the install:
|
| 254 |
-
|
| 255 |
-
```bash
|
| 256 |
-
codec-video-prep-doctor
|
| 257 |
-
```
|
| 258 |
-
|
| 259 |
-
Requirements:
|
| 260 |
-
|
| 261 |
-
- **Linux x86_64, CPython 3.10 or 3.11** (manylinux_2_17 wheels). Not macOS, ARM or Alpine musl.
|
| 262 |
-
- **NumPy < 2** (`numpy>=1.23,<2`). If your inference environment is on NumPy 2, run preprocessing in a separate environment and hand over the saved output.
|
| 263 |
-
- System `ffprobe` (`apt-get install ffmpeg` or equivalent).
|
| 264 |
-
- Bitcost support covers **H.264, HEVC, VP9 and AV1**. Other codecs (MPEG-4, MPEG-2) have no implementation, and **the file extension does not tell you the codec**:
|
| 265 |
-
|
| 266 |
-
```bash
|
| 267 |
-
ffprobe -v error -select_streams v:0 -show_entries stream=codec_name \
|
| 268 |
-
-of default=noprint_wrappers=1 input.mp4
|
| 269 |
-
```
|
| 270 |
-
|
| 271 |
-
- Sampling and budgeting need the whole video up front, so this is for **offline files**, not live streams.
|
| 272 |
-
|
| 273 |
-
### Usage
|
| 274 |
-
|
| 275 |
-
Set `video_backend="codec"` and pass a video path:
|
| 276 |
-
|
| 277 |
-
```python
|
| 278 |
-
inputs = processor(
|
| 279 |
-
text=text,
|
| 280 |
-
videos=["example.mp4"],
|
| 281 |
-
video_backend="codec",
|
| 282 |
-
return_tensors="pt",
|
| 283 |
-
).to(model.device)
|
| 284 |
-
|
| 285 |
-
with torch.inference_mode():
|
| 286 |
-
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
|
| 287 |
-
```
|
| 288 |
-
|
| 289 |
-
Alongside `input_ids` and `attention_mask`, `inputs` carries `pixel_values`, `image_grid_thw` (one row per canvas) and **`patch_positions`** (`[total_patches, 3]`, block layout). `patch_positions` is forwarded to `model.forward(...)` for 3-D RoPE — **do not drop it**.
|
| 290 |
-
|
| 291 |
-
### One time marker per canvas
|
| 292 |
-
|
| 293 |
-
Each canvas gets a single marker, and its form depends on how many source frames that canvas draws from:
|
| 294 |
-
|
| 295 |
-
| Canvas | Marker | Example |
|
| 296 |
-
|---|---|---|
|
| 297 |
-
| All patches from one source frame | a point | `<6.5 seconds>` |
|
| 298 |
-
| Patches spanning several frames | a range | `<0.0 to 1.8 seconds>` |
|
| 299 |
|
| 300 |
```
|
|
|
|
|
|
|
| 301 |
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 302 |
-
<
|
| 303 |
-
<
|
|
|
|
| 304 |
```
|
| 305 |
|
| 306 |
-
|
| 307 |
-
|
| 308 |
-
### Budget
|
| 309 |
-
|
| 310 |
-
The default recipe — 512 candidate frames, 64 canvases, patch 16, 3×3 blocks, 313600 pixels per canvas — lives in the `codec` field of `preprocessor_config.json`. Override per call with `codec_config=`:
|
| 311 |
-
|
| 312 |
-
```python
|
| 313 |
-
inputs = processor(
|
| 314 |
-
text=text, videos=["example.mp4"], video_backend="codec",
|
| 315 |
-
codec_config={"target_canvas": 32, "num_sampled_frames": 256},
|
| 316 |
-
return_tensors="pt",
|
| 317 |
-
)
|
| 318 |
-
```
|
| 319 |
|
| 320 |
-
|
| 321 |
-
|---|---|---|
|
| 322 |
-
| `strategy` | `adaptive_pwindow` | also `adaptive_rdo`, `readiness` |
|
| 323 |
-
| `target_canvas` | 64 | canvas budget; `readiness` does not bound the count, read the actual output |
|
| 324 |
-
| `num_sampled_frames` | 512 | candidate frames |
|
| 325 |
-
| `patch` / `block_size` | 16 / 3 | must match the vision tower's patch size and spatial merge |
|
| 326 |
-
| `max_pixels` | 313600 | pixels per canvas |
|
| 327 |
-
| `canvas_format` | `png` | `png`, `jpg` or `npy` |
|
| 328 |
|
| 329 |
-
|
| 330 |
|
| 331 |
-
|
|
|
|
| 332 |
|
| 333 |
-
|
| 334 |
|
| 335 |
-
|
| 336 |
|
| 337 |
-
|
| 338 |
-
texts = [processor.apply_chat_template(m, tokenize=False, add_generation_prompt=True)
|
| 339 |
-
for m in batch_messages]
|
| 340 |
-
inputs = processor(text=texts, images=batch_images, padding=True,
|
| 341 |
-
return_tensors="pt").to(model.device)
|
| 342 |
-
```
|
| 343 |
|
| 344 |
-
|
| 345 |
|
| 346 |
-
-
|
| 347 |
|
| 348 |
-
|
| 349 |
|
| 350 |
```bash
|
| 351 |
-
|
| 352 |
-
|
|
|
|
| 353 |
```
|
| 354 |
|
| 355 |
```python
|
| 356 |
-
|
| 357 |
-
|
| 358 |
```
|
| 359 |
|
| 360 |
-
|
| 361 |
|
| 362 |
-
|
| 363 |
|
| 364 |
-
|
| 365 |
|
| 366 |
-
##
|
| 367 |
|
| 368 |
-
|
| 369 |
|
| 370 |
-
```
|
| 371 |
-
model.generate(**inputs, max_new_tokens=512, do_sample=False)
|
| 372 |
-
```
|
| 373 |
-
|
| 374 |
-
Tasks where the model reasons before answering need a generous `max_new_tokens`; small values such as 128 cut it off mid-reasoning.
|
| 375 |
|
| 376 |
-
|
| 377 |
-
|
| 378 |
-
---
|
| 379 |
|
| 380 |
-
|
| 381 |
-
|
| 382 |
-
**Answer format affects regex-based scoring.** On some multiple-choice tasks the model writes "The correct answer is ..." or reasons first, rather than emitting a bare letter. Scripts that extract the first letter will mark those wrong. To force a letter, ask for one: `Answer with the option's letter from the given choices directly.`
|
| 383 |
-
|
| 384 |
-
**Coordinate convention.** For grounding, state the range in the prompt, e.g. `Answer with the bounding box as [x1, y1, x2, y2], each coordinate an integer between 0 and 1000 (1000 = full image width/height).` Without it the convention varies by task.
|
| 385 |
-
|
| 386 |
-
**State tokens survive `skip_special_tokens`.** The eight streaming tokens are non-special by design. If you are post-processing offline output and want them gone, strip them explicitly.
|
| 387 |
-
|
| 388 |
-
---
|
| 389 |
|
| 390 |
## Benchmarks
|
| 391 |
|
| 392 |
-
<!-- TODO before publishing: fill from
|
| 393 |
-
Do NOT reuse the preview base model's table — those numbers belong to a
|
| 394 |
-
checkpoint and would misreport this one. -->
|
| 395 |
|
| 396 |
-
_To be filled in from this checkpoint's evaluation run._
|
| 397 |
-
|
| 398 |
-
The protocol below is the one these numbers must be produced under.
|
| 399 |
-
|
| 400 |
-
---
|
| 401 |
|
| 402 |
## Evaluation protocol
|
| 403 |
|
| 404 |
-
|
| 405 |
-
|
| 406 |
-
Video tasks use `fps=1` with a fixed frame cap F and per-frame pixel budget MP (`min_pixels` and `max_pixels` set equal). Frames are cached offline, so evaluation decodes nothing at run time and repeated runs see identical input.
|
| 407 |
|
| 408 |
| Task | F | MP |
|
| 409 |
|---|---|---|
|
|
@@ -418,36 +224,17 @@ Video tasks use `fps=1` with a fixed frame cap F and per-frame pixel budget MP (
|
|
| 418 |
| VSTAT | 32 | 131072 |
|
| 419 |
| MVBench, NextQA, TempCompass, MMVU, ReVSI | 64 | 153664 |
|
| 420 |
|
| 421 |
-
|
| 422 |
-
|
| 423 |
-
### Scoring
|
| 424 |
-
|
| 425 |
-
Most tasks are re-checked by a judge model. Tasks that score by pure geometry (IoU, point-in-mask) or that otherwise never extract a letter from text skip it, because judging adds nothing; those rows equal their rule-based score.
|
| 426 |
-
|
| 427 |
-
| Judge | Value |
|
| 428 |
-
|---|---|
|
| 429 |
-
| Model | Qwen3.8-27B |
|
| 430 |
-
| Serving | vLLM 0.19, `--tensor-parallel-size 4`, `--gpu-memory-utilization 0.72`, `--max-model-len 16384`, `--enforce-eager` |
|
| 431 |
-
| Temperature | 0 |
|
| 432 |
-
| Max tokens | 6 |
|
| 433 |
-
| Thinking | off (`enable_thinking=False`) |
|
| 434 |
-
|
| 435 |
-
The judge receives the question, options, reference answer and the model's raw output, and answers YES or NO. This corrects answers that are right but do not match the extraction regex. The final score is the greater of the rule-based and judged scores.
|
| 436 |
-
|
| 437 |
-
On reproducibility: temperature 0 is greedy and therefore deterministic in principle, but no seed is fixed, and vLLM's continuous batching changes float accumulation order with batch composition, so borderline samples can still flip. Strict reproduction needs the judge weights, vLLM version and tensor-parallel size all pinned.
|
| 438 |
-
|
| 439 |
-
---
|
| 440 |
|
| 441 |
## Architecture
|
| 442 |
|
| 443 |
| Item | Value |
|
| 444 |
|---|---|
|
| 445 |
-
| Language layers | 32 (linear attention
|
| 446 |
| hidden_size | 2560 |
|
| 447 |
| Attention heads / KV heads | 16 / 4 |
|
| 448 |
| Vocabulary | 248320 |
|
| 449 |
-
| Vision layers | 24 |
|
| 450 |
-
| Vision hidden_size | 1024 |
|
| 451 |
| patch_size / spatial_merge_size | 16 / 3 |
|
| 452 |
| max_position_embeddings | 262144 |
|
| 453 |
| Weights | bfloat16, 3 shards, about 9.9 GB |
|
|
|
|
| 13 |
|
| 14 |
# LLaVA-OneVision-2.5-4B-Instruct
|
| 15 |
|
| 16 |
+
Instruction-tuned 4B multimodal model on the LLaVA-OneVision-2 architecture, for single-image, multi-image and video input. It also handles streaming video, where frames arrive one at a time and the model decides at each step whether to speak — same weights, same chat template, switched by one keyword. See [Streaming](#streaming).
|
| 17 |
|
| 18 |
+
Vision encoder: 24 layers, `patch_size=16`, `spatial_merge_size=3`. Language side: 32 layers, `hidden_size=2560`, linear attention with one full-attention layer every 4, `max_position_embeddings=262144`. The modeling code ships with the weights, so `trust_remote_code=True` is required.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
## Install
|
| 21 |
|
| 22 |
```bash
|
| 23 |
pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
|
| 24 |
+
pip install flash-attn --no-build-isolation # optional, recommended
|
| 25 |
+
pip install decord # frame-sampling video path only
|
| 26 |
```
|
| 27 |
|
| 28 |
+
## Images
|
|
|
|
|
|
|
| 29 |
|
| 30 |
```python
|
| 31 |
import torch
|
|
|
|
| 36 |
|
| 37 |
processor = AutoProcessor.from_pretrained(MODEL, trust_remote_code=True)
|
| 38 |
model = AutoModelForImageTextToText.from_pretrained(
|
| 39 |
+
MODEL, trust_remote_code=True, dtype=torch.bfloat16,
|
| 40 |
+
attn_implementation="flash_attention_2", # "sdpa" without flash-attn
|
|
|
|
|
|
|
| 41 |
device_map="auto",
|
| 42 |
).eval()
|
| 43 |
|
|
|
|
| 44 |
messages = [{"role": "user", "content": [
|
| 45 |
{"type": "image"},
|
| 46 |
{"type": "text", "text": "Describe this image in detail."},
|
| 47 |
]}]
|
| 48 |
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 49 |
+
inputs = processor(text=text, images=[Image.open("example.jpg").convert("RGB")],
|
| 50 |
+
return_tensors="pt").to(model.device)
|
| 51 |
|
| 52 |
with torch.inference_mode():
|
| 53 |
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
|
| 54 |
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 55 |
```
|
| 56 |
|
| 57 |
+
Images go through `smart_resize`, keeping aspect ratio and aligning to the patch grid. The pixel budget defaults to `min_pixels=3136` / `max_pixels=4000000`; override it on `processor.image_processor`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
|
| 59 |
+
For several images, put one `{"type": "image"}` per image in `content` and pass `images=` in the same order. Each placeholder expands to `<|vision_start|><|image_pad|><|vision_end|>`, matched by position.
|
| 60 |
|
| 61 |
## Video — frame sampling
|
| 62 |
|
|
|
|
|
|
|
| 63 |
```python
|
| 64 |
messages = [{"role": "user", "content": [
|
| 65 |
{"type": "video"},
|
|
|
|
| 68 |
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 69 |
|
| 70 |
inputs = processor(
|
| 71 |
+
text=text, videos=["example.mp4"],
|
|
|
|
| 72 |
max_frames=64,
|
| 73 |
+
min_pixels=153664, max_pixels=153664, # set equal to fix the per-frame budget
|
|
|
|
| 74 |
return_tensors="pt",
|
| 75 |
).to(model.device)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
```
|
| 77 |
|
| 78 |
+
`target_fps` and `num_frames` are also accepted; precedence is in `choose_target_frames` in `video_processing_llava_onevision2_5.py`. `max_frames` caps at 768.
|
|
|
|
|
|
|
| 79 |
|
| 80 |
+
Cost is roughly frames × pixels per frame, so tune the two together:
|
| 81 |
|
| 82 |
| Case | Frames | Pixels per frame |
|
| 83 |
|---|---|---|
|
|
|
|
| 85 |
| Medium | 256 | 136900 |
|
| 86 |
| Long video, global understanding | 640 – 768 | 72900 – 108900 |
|
| 87 |
|
| 88 |
+
All frames are encoded in one vision-tower forward pass, which can exhaust memory on very long inputs. `OV_VISION_CHUNK_PATCHES=200000` encodes them in groups instead, split along frame boundaries — attention never crosses a frame, so the result is unchanged.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 89 |
|
| 90 |
+
**Timestamps.** Video input always carries them. The `<|video_pad|>` placeholder expands into one block per frame, each prefixed with its time, taken from the decoder's real sample positions:
|
|
|
|
|
|
|
|
|
|
|
|
|
| 91 |
|
| 92 |
```
|
| 93 |
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 94 |
<1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 95 |
```
|
| 96 |
|
| 97 |
+
Pass a path, not pre-decoded frames. Given a bare frame list the processor has no real sample times and falls back to frame ordinals as seconds, so a 100-second video looks N seconds long and temporal answers are wrong (it raises a `UserWarning`). If you must decode yourself, hand over the timing as well:
|
| 98 |
|
| 99 |
+
```python
|
| 100 |
+
videos=[{"frames": frames, "timestamps": seconds_list, "frame_indices": index_list}]
|
| 101 |
+
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 102 |
|
| 103 |
+
Frames passed through `images=` get no timestamps at all.
|
| 104 |
|
| 105 |
+
## Streaming
|
| 106 |
|
| 107 |
+
Frames arrive one per turn, and each turn the model emits a state token: stay silent, or speak. One keyword switches the default system prompt, and changes nothing else:
|
| 108 |
|
| 109 |
+
| Mode | Call | Default system |
|
| 110 |
|---|---|---|
|
| 111 |
| Offline (default) | `apply_chat_template(messages, ...)` | `You are a helpful assistant.` |
|
| 112 |
| Streaming | `apply_chat_template(messages, ..., streaming=True)` | `You are a real-time streaming video assistant.` |
|
| 113 |
|
| 114 |
+
An explicit `system` message wins in both modes. Offline is the default so existing evaluation harnesses work unchanged.
|
| 115 |
|
| 116 |
### Protocol tokens
|
| 117 |
|
|
|
|
| 119 |
|---|---|---|
|
| 120 |
| `<silence>` | 248077 | this turn: say nothing |
|
| 121 |
| `<response>` | 248078 | this turn: speak |
|
| 122 |
+
| `<question_memory>` / `</question_memory>` | 248079 / 248080 | a new question, written once at the moment it is asked |
|
| 123 |
+
| `<answer_memory>` / `</answer_memory>` | 248081 / 248082 | an earlier answer, re-injected once it is due |
|
| 124 |
+
| `<captioning>` | 248083 | this turn: speak, as event-level captioning |
|
| 125 |
| `<caption_begin>` | 248084 | user-side marker that this stream is a captioning task |
|
| 126 |
|
| 127 |
+
All eight are non-special added tokens, so `skip_special_tokens=True` still returns them — the state token is the output you read. They fit the existing 248320-row embedding, and no previously published token id moved. The memory tokens wrap a JSON object, e.g. `{"timestep": "00:00:06", "content": "..."}`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 128 |
|
| 129 |
+
### Building a turn
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 130 |
|
| 131 |
+
Strict user/assistant alternation, at most one frame per user turn; a turn with no new frame uses a bare text time prefix instead. Give each frame its `timestamp`, and the trailing newline as its own text item:
|
| 132 |
|
| 133 |
```python
|
| 134 |
messages = [{"role": "user", "content": [
|
|
|
|
| 140 |
messages, tokenize=False, add_generation_prompt=True, streaming=True)
|
| 141 |
```
|
| 142 |
|
| 143 |
+
which renders as:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 144 |
|
| 145 |
```
|
| 146 |
+
<|im_start|>user
|
| 147 |
+
<question_memory>{"timestep": "00:00:00", "content": "..."}</question_memory>
|
| 148 |
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 149 |
+
<|im_end|>
|
| 150 |
+
<|im_start|>assistant
|
| 151 |
+
<silence><|im_end|>
|
| 152 |
```
|
| 153 |
|
| 154 |
+
A plain string `content` is emitted verbatim, for harnesses that render the markers themselves.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 155 |
|
| 156 |
+
### Two values you have to compute
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 157 |
|
| 158 |
+
The template fixes the format; two numbers come from the video and cannot be recovered from a bare frame:
|
| 159 |
|
| 160 |
+
1. **`timestamp`** — the source frame index divided by the video's real fps.
|
| 161 |
+
2. **The `t` column of `patch_positions`** — handed independent images, the processor returns the right block layout but `t=0` for every frame. Rewrite each frame's rows to `floor(seconds × fps)`, or every frame reads as the start of a new visual segment.
|
| 162 |
|
| 163 |
+
Only the *rank* of `t` within the segment matters to the model, and the vision tower attends within a frame only; keeping real frame numbers just makes the input auditable.
|
| 164 |
|
| 165 |
+
The automatic timestamp expansion described above belongs to the offline `<|video_pad|>` branch. Streaming feeds frames one at a time and does not use it.
|
| 166 |
|
| 167 |
+
Not included here: the real-time loop around the model — frame scheduling, deciding when to stop generating, re-injecting question and answer memory as it falls due, and carrying context across decision points.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 168 |
|
| 169 |
+
## Video — codec mode
|
| 170 |
|
| 171 |
+
Instead of sampling whole frames, codec mode uses encoder bitcost to find informative regions and packs them into a few *canvases*. Patches on one canvas may come from different source frames, each carrying its own `(source frame, row, column)` for 3-D RoPE — so canvases must not be fed through `images=`.
|
| 172 |
|
| 173 |
+
Preprocessing comes from `codec-video-prep` (bundles patched FFmpeg and dav1d; Linux x86_64, CPython 3.10/3.11, `numpy<2`, system `ffprobe`; H.264/HEVC/VP9/AV1 only; offline files only, since budgeting needs the whole video):
|
| 174 |
|
| 175 |
```bash
|
| 176 |
+
pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple \
|
| 177 |
+
"codec-video-prep==0.3.0"
|
| 178 |
+
codec-video-prep-doctor
|
| 179 |
```
|
| 180 |
|
| 181 |
```python
|
| 182 |
+
inputs = processor(text=text, videos=["example.mp4"],
|
| 183 |
+
video_backend="codec", return_tensors="pt").to(model.device)
|
| 184 |
```
|
| 185 |
|
| 186 |
+
The output carries `patch_positions` (`[total_patches, 3]`) alongside `pixel_values` and `image_grid_thw`. It is forwarded to `model.forward(...)` for 3-D RoPE — do not drop it.
|
| 187 |
|
| 188 |
+
Each canvas gets one time marker: a point `<6.5 seconds>` when all its patches come from one source frame, a range `<0.0 to 1.8 seconds>` when it spans several (schema `canvas_patch_span_point_range_v1`). Temporal questions work as on the frame path.
|
| 189 |
|
| 190 |
+
Defaults live in the `codec` field of `preprocessor_config.json` — 512 candidate frames, 64 canvases, patch 16, 3×3 blocks, 313600 pixels per canvas — and are overridden per call with `codec_config={"target_canvas": 32, ...}`. Other keys: `strategy` (`adaptive_pwindow`, `adaptive_rdo`, `readiness`), `num_sampled_frames`, `max_pixels`, `canvas_format`. `patch=16` and `block_size=3` must match the model's `patch_size` and `spatial_merge_size`; preprocessing raises rather than failing silently. Canvases cache in `~/.cache/codec_video_prep/canvases` (`CVP_CODEC_CACHE_ROOT` to relocate).
|
| 191 |
|
| 192 |
+
## Notes
|
| 193 |
|
| 194 |
+
**Batching.** Pass a list of texts and the matching images with `padding=True`. Padding is left-sided, so slice each output by its own `input_ids` length.
|
| 195 |
|
| 196 |
+
**Offline.** Set `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1`. Load the full model once in a single process before starting more: remote-code modules compile into `$HF_HOME/modules/transformers_modules/<name>/`, and concurrent first loads race there. That cache is keyed by leaf directory name only, so give local copies distinct names.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 197 |
|
| 198 |
+
**Generation.** `generation_config.json` sets only `eos_token_id=248044` and `use_cache=true`. Use `do_sample=False` for deterministic output, and a generous `max_new_tokens` on tasks where the model reasons before answering — 128 cuts it off mid-reasoning.
|
|
|
|
|
|
|
| 199 |
|
| 200 |
+
**Answer format.** On multiple-choice tasks the model may write a sentence rather than a bare letter. If a regex will parse the output, ask for a letter: `Answer with the option's letter from the given choices directly.` For grounding, state the coordinate range in the prompt, e.g. integers 0–1000. The eight streaming tokens survive `skip_special_tokens`; strip them explicitly if you do not want them offline.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 201 |
|
| 202 |
## Benchmarks
|
| 203 |
|
| 204 |
+
<!-- TODO before publishing: fill from this checkpoint's own evaluation run.
|
| 205 |
+
Do NOT reuse the preview base model's table — those numbers belong to a
|
| 206 |
+
different checkpoint and would misreport this one. -->
|
| 207 |
|
| 208 |
+
_To be filled in from this checkpoint's evaluation run, under the protocol below._
|
|
|
|
|
|
|
|
|
|
|
|
|
| 209 |
|
| 210 |
## Evaluation protocol
|
| 211 |
|
| 212 |
+
Video tasks run at `fps=1` with a fixed frame cap F and per-frame pixel budget MP (`min_pixels` = `max_pixels`). Frames are cached offline, so runs decode nothing and repeat on identical input. Image tasks use the default pixel budget. All generation is greedy, `max_new_tokens=4096`, `batch_size=1`.
|
|
|
|
|
|
|
| 213 |
|
| 214 |
| Task | F | MP |
|
| 215 |
|---|---|---|
|
|
|
|
| 224 |
| VSTAT | 32 | 131072 |
|
| 225 |
| MVBench, NextQA, TempCompass, MMVU, ReVSI | 64 | 153664 |
|
| 226 |
|
| 227 |
+
Most tasks are re-checked by a judge model (Qwen3.8-27B, temperature 0) that sees the question, options, reference answer and raw output and answers YES or NO, recovering answers that are correct but miss the extraction regex; the reported score is the greater of the rule-based and judged scores. Tasks scored by geometry (IoU, point-in-mask) skip the judge and use their rule-based score.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 228 |
|
| 229 |
## Architecture
|
| 230 |
|
| 231 |
| Item | Value |
|
| 232 |
|---|---|
|
| 233 |
+
| Language layers | 32 (linear attention, full attention every 4) |
|
| 234 |
| hidden_size | 2560 |
|
| 235 |
| Attention heads / KV heads | 16 / 4 |
|
| 236 |
| Vocabulary | 248320 |
|
| 237 |
+
| Vision layers / hidden_size | 24 / 1024 |
|
|
|
|
| 238 |
| patch_size / spatial_merge_size | 16 / 3 |
|
| 239 |
| max_position_embeddings | 262144 |
|
| 240 |
| Weights | bfloat16, 3 shards, about 9.9 GB |
|
README_CN.md
CHANGED
|
@@ -13,25 +13,19 @@ tags:
|
|
| 13 |
|
| 14 |
# LLaVA-OneVision-2.5-4B-Instruct
|
| 15 |
|
| 16 |
-
基于 LLaVA-OneVision-2 架构的 4B 指令微调多模态模型,支持单图、多图和视频输入。视
|
| 17 |
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
建模代码随权重一起分发,因此**必须 `trust_remote_code=True`**。
|
| 21 |
-
|
| 22 |
-
---
|
| 23 |
|
| 24 |
## 安装
|
| 25 |
|
| 26 |
```bash
|
| 27 |
pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
|
| 28 |
-
pip install flash-attn --no-build-isolation # 可选,
|
| 29 |
pip install decord # 仅抽帧视频路径需要
|
| 30 |
```
|
| 31 |
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
## 单图推理
|
| 35 |
|
| 36 |
```python
|
| 37 |
import torch
|
|
@@ -42,56 +36,29 @@ MODEL = "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct"
|
|
| 42 |
|
| 43 |
processor = AutoProcessor.from_pretrained(MODEL, trust_remote_code=True)
|
| 44 |
model = AutoModelForImageTextToText.from_pretrained(
|
| 45 |
-
MODEL,
|
| 46 |
-
trust_remote_code=True,
|
| 47 |
-
dtype=torch.bfloat16,
|
| 48 |
attn_implementation="flash_attention_2", # 没装 flash-attn 用 "sdpa"
|
| 49 |
device_map="auto",
|
| 50 |
).eval()
|
| 51 |
|
| 52 |
-
image = Image.open("example.jpg").convert("RGB")
|
| 53 |
messages = [{"role": "user", "content": [
|
| 54 |
{"type": "image"},
|
| 55 |
{"type": "text", "text": "Describe this image in detail."},
|
| 56 |
]}]
|
| 57 |
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 58 |
-
inputs = processor(text=text, images=[
|
|
|
|
| 59 |
|
| 60 |
with torch.inference_mode():
|
| 61 |
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
|
| 62 |
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 63 |
```
|
| 64 |
|
| 65 |
-
图片走 `smart_resize`,保持长宽比并对齐 patch 网格。像素预算默认 `min_pixels=3136` / `max_pixels=4000000`,可在 processor 上覆盖
|
| 66 |
-
|
| 67 |
-
```python
|
| 68 |
-
processor.image_processor.min_pixels = 256 * 28 * 28
|
| 69 |
-
processor.image_processor.max_pixels = 1280 * 28 * 28
|
| 70 |
-
```
|
| 71 |
-
|
| 72 |
-
---
|
| 73 |
|
| 74 |
-
|
| 75 |
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
```python
|
| 79 |
-
messages = [{"role": "user", "content": [
|
| 80 |
-
{"type": "image"},
|
| 81 |
-
{"type": "image"},
|
| 82 |
-
{"type": "text", "text": "What changed between these two images?"},
|
| 83 |
-
]}]
|
| 84 |
-
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 85 |
-
inputs = processor(text=text, images=[img_a, img_b], return_tensors="pt").to(model.device)
|
| 86 |
-
```
|
| 87 |
-
|
| 88 |
-
每个占位符展开为 `<|vision_start|><|image_pad|><|vision_end|>`,按位置与 `images` 列表一一对应。
|
| 89 |
-
|
| 90 |
-
---
|
| 91 |
-
|
| 92 |
-
## 视频推理 — 抽帧
|
| 93 |
-
|
| 94 |
-
传**视频路径**,不要传自己解码好的帧:
|
| 95 |
|
| 96 |
```python
|
| 97 |
messages = [{"role": "user", "content": [
|
|
@@ -101,24 +68,16 @@ messages = [{"role": "user", "content": [
|
|
| 101 |
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 102 |
|
| 103 |
inputs = processor(
|
| 104 |
-
text=text,
|
| 105 |
-
videos=["example.mp4"],
|
| 106 |
max_frames=64,
|
| 107 |
-
min_pixels=153664,
|
| 108 |
-
max_pixels=153664, # 视频场景把 min 和 max 设成相等,固定逐帧预算
|
| 109 |
return_tensors="pt",
|
| 110 |
).to(model.device)
|
| 111 |
-
|
| 112 |
-
with torch.inference_mode():
|
| 113 |
-
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
|
| 114 |
-
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 115 |
```
|
| 116 |
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
### 帧数与逐帧像素
|
| 120 |
|
| 121 |
-
开销
|
| 122 |
|
| 123 |
| 场景 | 帧数 | 逐帧像素 |
|
| 124 |
|---|---|---|
|
|
@@ -126,47 +85,33 @@ print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_token
|
|
| 126 |
| 中等 | 256 | 136900 |
|
| 127 |
| 长视频、整体理解 | 640 – 768 | 72900 – 108900 |
|
| 128 |
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
```bash
|
| 132 |
-
export OV_VISION_CHUNK_PATCHES=200000 # 0(默认)= 关闭
|
| 133 |
-
```
|
| 134 |
-
|
| 135 |
-
分组沿逐帧的序列边界切分,注意力不跨帧,因此结果与一次性前向在数学上等价,只是用一点速度换显存。
|
| 136 |
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
视频输入一定带时间戳,且无法关闭。`<|vision_start|><|video_pad|><|vision_end|>` 占位符会被改写成逐帧的块,每块前面是该帧的时间:
|
| 140 |
|
| 141 |
```
|
| 142 |
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 143 |
<1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 144 |
```
|
| 145 |
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
> **传路径,不要传自己的帧。** `videos=` 两者都收,但并不等价。只有传路径,processor 才能拿到真实采样时刻和真实源帧号。只给一串帧时它两样都没有,会退化成拿帧序号当秒数——第 0、1、2 帧变成 0.0、1.0、2.0 秒——于是 100 秒的视频看起来只有 N 秒,时间类回答全错。这种情况下 processor 会抛 `UserWarning`。
|
| 149 |
-
>
|
| 150 |
-
> 如果必须自己解码,请把时间信息一并交出:
|
| 151 |
-
> ```python
|
| 152 |
-
> videos=[{"frames": frames, "timestamps": seconds_list, "frame_indices": index_list}]
|
| 153 |
-
> ```
|
| 154 |
-
> 通过 `images=` 传帧则完全不注入时间戳。
|
| 155 |
|
| 156 |
-
|
|
|
|
|
|
|
| 157 |
|
| 158 |
-
|
| 159 |
|
| 160 |
-
流式
|
| 161 |
|
| 162 |
-
|
| 163 |
|
| 164 |
| 模式 | 调用 | 默认 system |
|
| 165 |
|---|---|---|
|
| 166 |
| 离线(默认) | `apply_chat_template(messages, ...)` | `You are a helpful assistant.` |
|
| 167 |
| 流式 | `apply_chat_template(messages, ..., streaming=True)` | `You are a real-time streaming video assistant.` |
|
| 168 |
|
| 169 |
-
|
| 170 |
|
| 171 |
### 协议 token
|
| 172 |
|
|
@@ -174,36 +119,16 @@ export OV_VISION_CHUNK_PATCHES=200000 # 0(默认)= 关闭
|
|
| 174 |
|---|---|---|
|
| 175 |
| `<silence>` | 248077 | 当前轮不发言 |
|
| 176 |
| `<response>` | 248078 | 当前轮发言 |
|
| 177 |
-
| `<question_memory>` / `</question_memory>` | 248079 / 248080 | 新问题
|
| 178 |
-
| `<answer_memory>` / `</answer_memory>` | 248081 / 248082 | 此前回答到期后回插
|
| 179 |
-
| `<captioning>` | 248083 | 当前轮发言,且是事件级 caption
|
| 180 |
| `<caption_begin>` | 248084 | user 侧标记,说明这条流是 caption 任务 |
|
| 181 |
|
| 182 |
-
八个都是
|
| 183 |
-
|
| 184 |
-
`<question_memory>` 和 `<answer_memory>` 内部包一个 JSON 对象,例如 `{"timestep": "00:00:06", "content": "..."}`,其中 `timestep` 是原始提问/回答时刻,格式 `HH:MM:SS`。
|
| 185 |
-
|
| 186 |
-
### 轮次结构
|
| 187 |
|
| 188 |
-
|
| 189 |
-
|
| 190 |
-
```
|
| 191 |
-
<|im_start|>system
|
| 192 |
-
You are a real-time streaming video assistant.<|im_end|>
|
| 193 |
-
<|im_start|>user
|
| 194 |
-
<question_memory>{"timestep": "00:00:00", "content": "..."}</question_memory>
|
| 195 |
-
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 196 |
-
<|im_end|>
|
| 197 |
-
<|im_start|>assistant
|
| 198 |
-
<silence><|im_end|>
|
| 199 |
-
<|im_start|>user
|
| 200 |
-
<1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 201 |
-
<|im_end|>
|
| 202 |
-
<|im_start|>assistant
|
| 203 |
-
<response><|im_end|>
|
| 204 |
-
```
|
| 205 |
|
| 206 |
-
|
| 207 |
|
| 208 |
```python
|
| 209 |
messages = [{"role": "user", "content": [
|
|
@@ -215,195 +140,76 @@ text = processor.apply_chat_template(
|
|
| 215 |
messages, tokenize=False, add_generation_prompt=True, streaming=True)
|
| 216 |
```
|
| 217 |
|
| 218 |
-
|
| 219 |
-
|
| 220 |
-
### 调用方必须自己算的两个值
|
| 221 |
-
|
| 222 |
-
模版保证的是**格式**,有两个**数值**取决于视频本身,无法从一张裸帧里恢复,需要你提供:
|
| 223 |
-
|
| 224 |
-
1. **文本时间戳**:每帧传 `timestamp = 源帧号 / fps`,fps 用视频真实帧率。
|
| 225 |
-
2. **`patch_positions` 的 `t` 列**:如果把每帧当作独立 image 交给 processor,返回的 `patch_positions` 块布局是对的,但 `t` **每帧都是 0**。需要把每帧对应的行改成它真实的源帧号 `floor(秒数 × fps)`。不改的话,每帧都会被当成一段新视觉的开头,与训练时不一致。
|
| 226 |
-
|
| 227 |
-
模型只通过 `t` 在本段视觉内的**名次**(0, 1, 2, …)使用它,不用绝对值;视觉塔也只做帧内注意力。所以真正重要的是先后顺序对,而不是数值本身;让它等于真实帧号只是为了输入可审计。
|
| 228 |
-
|
| 229 |
-
上一节说的逐帧时间戳自动展开属于离线的 `<|video_pad|>` 分支。流式是逐帧输入,不走那条分支。
|
| 230 |
-
|
| 231 |
-
> **本仓库不包含什么。** 这份权重给你的是训练出来的行为、模版和 token。围绕它的实时循环——按时钟取帧、决定何时停止生成、在到期时回插 question / answer memory、在决策点之间维持上下文——属于服务端的工作,不在这里分发。
|
| 232 |
-
|
| 233 |
-
---
|
| 234 |
-
|
| 235 |
-
## 视频推理 — codec 模式
|
| 236 |
-
|
| 237 |
-
codec 模式不按整帧采样,而是用编码器的 bitcost 找出信息量大的区域,打包进少量**画布(canvas)**。
|
| 238 |
-
|
| 239 |
-
关键区别:**同一张画布上的 patch 可以来自多个不同的源帧。**每个 patch 带自己的源位置 `(源帧, 源 patch 行, 源 patch 列)`,视觉塔用它做 3-D RoPE。把画布当普通图片通过 `images=` 传入会丢掉这些信息。
|
| 240 |
-
|
| 241 |
-
### 安装预处理包
|
| 242 |
-
|
| 243 |
-
codec 预处理来自 `codec-video-prep`,它自带打过补丁的 FFmpeg 和 dav1d 用于提取 bitcost,无需编译:
|
| 244 |
-
|
| 245 |
-
```bash
|
| 246 |
-
pip install -i https://test.pypi.org/simple/ \
|
| 247 |
-
--extra-index-url https://pypi.org/simple \
|
| 248 |
-
"codec-video-prep==0.3.0"
|
| 249 |
-
```
|
| 250 |
-
|
| 251 |
-
`--extra-index-url` 是必需的:包本身在 TestPyPI,依赖(numpy、opencv-python-headless、Pillow、PyYAML)在正式 PyPI。
|
| 252 |
-
|
| 253 |
-
检查安装:
|
| 254 |
-
|
| 255 |
-
```bash
|
| 256 |
-
codec-video-prep-doctor
|
| 257 |
-
```
|
| 258 |
-
|
| 259 |
-
要求:
|
| 260 |
-
|
| 261 |
-
- **Linux x86_64、CPython 3.10 或 3.11**(manylinux_2_17 轮子)。不支持 macOS、ARM、Alpine musl。
|
| 262 |
-
- **NumPy < 2**(`numpy>=1.23,<2`)。如果推理环境是 NumPy 2,请在单独环境做预处理再交付产物。
|
| 263 |
-
- 系统 `ffprobe`(`apt-get install ffmpeg` 或等价)。
|
| 264 |
-
- bitcost 支持 **H.264、HEVC、VP9、AV1**。其他编码(MPEG-4、MPEG-2)没有实现,而且**文件后缀说明不了编码格式**:
|
| 265 |
-
|
| 266 |
-
```bash
|
| 267 |
-
ffprobe -v error -select_streams v:0 -show_entries stream=codec_name \
|
| 268 |
-
-of default=noprint_wrappers=1 input.mp4
|
| 269 |
-
```
|
| 270 |
-
|
| 271 |
-
- 采样和预算需要先拿到整段视频,所以这是给**离线文件**用的,不适用于实时流。
|
| 272 |
-
|
| 273 |
-
### 调用
|
| 274 |
-
|
| 275 |
-
设 `video_backend="codec"` 并传视频路径:
|
| 276 |
-
|
| 277 |
-
```python
|
| 278 |
-
inputs = processor(
|
| 279 |
-
text=text,
|
| 280 |
-
videos=["example.mp4"],
|
| 281 |
-
video_backend="codec",
|
| 282 |
-
return_tensors="pt",
|
| 283 |
-
).to(model.device)
|
| 284 |
-
|
| 285 |
-
with torch.inference_mode():
|
| 286 |
-
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
|
| 287 |
-
```
|
| 288 |
-
|
| 289 |
-
除 `input_ids` 和 `attention_mask` 外,`inputs` 还带 `pixel_values`、`image_grid_thw`(每张画布一行)和 **`patch_positions`**(`[总 patch 数, 3]`,块布局)。`patch_positions` 会转发给 `model.forward(...)` 做 3-D RoPE,**不要丢掉**。
|
| 290 |
-
|
| 291 |
-
### 每张画布一个时间标记
|
| 292 |
-
|
| 293 |
-
每张画布只有一个标记,形式取决于它取自几个源帧:
|
| 294 |
-
|
| 295 |
-
| 画布 | 标记 | 例子 |
|
| 296 |
-
|---|---|---|
|
| 297 |
-
| 所有 patch 来自同一源帧 | 点 | `<6.5 seconds>` |
|
| 298 |
-
| patch 跨多帧 | 区间 | `<0.0 to 1.8 seconds>` |
|
| 299 |
|
| 300 |
```
|
|
|
|
|
|
|
| 301 |
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 302 |
-
<
|
| 303 |
-
<
|
|
|
|
| 304 |
```
|
| 305 |
|
| 306 |
-
|
| 307 |
|
| 308 |
-
###
|
| 309 |
|
| 310 |
-
|
| 311 |
|
| 312 |
-
``
|
| 313 |
-
|
| 314 |
-
text=text, videos=["example.mp4"], video_backend="codec",
|
| 315 |
-
codec_config={"target_canvas": 32, "num_sampled_frames": 256},
|
| 316 |
-
return_tensors="pt",
|
| 317 |
-
)
|
| 318 |
-
```
|
| 319 |
|
| 320 |
-
|
| 321 |
-
|---|---|---|
|
| 322 |
-
| `strategy` | `adaptive_pwindow` | 还有 `adaptive_rdo`、`readiness` |
|
| 323 |
-
| `target_canvas` | 64 | 画布预算;`readiness` 不约束数量,以实际输出为准 |
|
| 324 |
-
| `num_sampled_frames` | 512 | 候选帧数 |
|
| 325 |
-
| `patch` / `block_size` | 16 / 3 | 必须与视觉塔的 patch size 和 spatial merge 一致 |
|
| 326 |
-
| `max_pixels` | 313600 | 每画布像素 |
|
| 327 |
-
| `canvas_format` | `png` | `png`、`jpg` 或 `npy` |
|
| 328 |
-
|
| 329 |
-
> `patch=16`、`block_size=3` 对应本模型的 `patch_size=16` 和 `spatial_merge_size=3`。改动会让画布网格与 merger 错位,预处理会直接报错而不是静默出错。
|
| 330 |
|
| 331 |
-
|
| 332 |
|
| 333 |
-
|
| 334 |
|
| 335 |
-
##
|
| 336 |
|
| 337 |
-
```
|
| 338 |
-
texts = [processor.apply_chat_template(m, tokenize=False, add_generation_prompt=True)
|
| 339 |
-
for m in batch_messages]
|
| 340 |
-
inputs = processor(text=texts, images=batch_images, padding=True,
|
| 341 |
-
return_tensors="pt").to(model.device)
|
| 342 |
-
```
|
| 343 |
-
|
| 344 |
-
padding 在左侧;按各自的 `input_ids` 长度切输出。
|
| 345 |
-
|
| 346 |
-
---
|
| 347 |
|
| 348 |
-
|
| 349 |
|
| 350 |
```bash
|
| 351 |
-
|
| 352 |
-
|
| 353 |
-
|
| 354 |
-
|
| 355 |
-
```python
|
| 356 |
-
processor = AutoProcessor.from_pretrained("/local/path", trust_remote_code=True)
|
| 357 |
-
model = AutoModelForImageTextToText.from_pretrained("/local/path", trust_remote_code=True, ...)
|
| 358 |
```
|
| 359 |
|
| 360 |
-
**多进程启动前先单进程预热一次。** `trust_remote_code` 模块会被编译到 `$HF_HOME/modules/transformers_modules/<名字>/`。多个进程首次同时加载会在这个目录上竞争,部分进程报 `AttributeError: module '...modeling_llava_onevision2_5' has no attribute 'LlavaOnevision2ForConditionalGeneration'`。先用单进程完整加载一次模型——光 `AutoConfig` 不会生成 `modeling_*.py` 的缓存——再启动其余进程。
|
| 361 |
-
|
| 362 |
-
缓存目录只按**叶子目录名**索引。两个都叫 `model_hf` 的本地 checkpoint 会共用同一条缓存、互相覆盖;本地副本请取不同的目录名。
|
| 363 |
-
|
| 364 |
-
---
|
| 365 |
-
|
| 366 |
-
## 生成参数
|
| 367 |
-
|
| 368 |
-
`generation_config.json` 只设了 `eos_token_id=248044` 和 `use_cache=true`,没有采样默认值。要确定性输出就用贪心解码:
|
| 369 |
-
|
| 370 |
```python
|
| 371 |
-
|
|
|
|
| 372 |
```
|
| 373 |
|
| 374 |
-
|
| 375 |
|
| 376 |
-
|
| 377 |
|
| 378 |
-
|
| 379 |
|
| 380 |
-
##
|
| 381 |
|
| 382 |
-
**
|
| 383 |
|
| 384 |
-
**
|
| 385 |
|
| 386 |
-
**
|
| 387 |
|
| 388 |
-
|
| 389 |
|
| 390 |
## 评测结果
|
| 391 |
|
| 392 |
-
<!-- 上传前待补:用本
|
| 393 |
不要沿用 preview base 模型的那张表——那些数字属于另一个 checkpoint,
|
| 394 |
放在这里会误报本模型的能力。 -->
|
| 395 |
|
| 396 |
-
_待用本 checkpoint 的评测结果补充。_
|
| 397 |
-
|
| 398 |
-
下面的口径就是这些数字必须依据的口径。
|
| 399 |
-
|
| 400 |
-
---
|
| 401 |
|
| 402 |
## 评测口径
|
| 403 |
|
| 404 |
-
|
| 405 |
-
|
| 406 |
-
视频任务用 `fps=1`,固定帧数上限 F 和逐帧像素预算 MP(`min_pixels` 与 `max_pixels` 设为相等)。帧是离线缓存好的,所以评测时不解码,重复跑看到的输入完全一致。
|
| 407 |
|
| 408 |
| 任务 | F | MP |
|
| 409 |
|---|---|---|
|
|
@@ -418,36 +224,17 @@ _待用本 checkpoint 的评测结果补充。_
|
|
| 418 |
| VSTAT | 32 | 131072 |
|
| 419 |
| MVBench、NextQA、TempCompass、MMVU、ReVSI | 64 | 153664 |
|
| 420 |
|
| 421 |
-
|
| 422 |
-
|
| 423 |
-
### 判分
|
| 424 |
-
|
| 425 |
-
多数任务会再经过一个 judge 模型复核。纯几何判分(IoU、点落在掩码内)或本来就不从文本里抽字母的任务跳过 judge,因为判也没有意义;这些行等于它们的规则分。
|
| 426 |
-
|
| 427 |
-
| Judge | 值 |
|
| 428 |
-
|---|---|
|
| 429 |
-
| 模型 | Qwen3.8-27B |
|
| 430 |
-
| 服务 | vLLM 0.19,`--tensor-parallel-size 4`、`--gpu-memory-utilization 0.72`、`--max-model-len 16384`、`--enforce-eager` |
|
| 431 |
-
| 温度 | 0 |
|
| 432 |
-
| 最大 token | 6 |
|
| 433 |
-
| Thinking | 关(`enable_thinking=False`) |
|
| 434 |
-
|
| 435 |
-
judge 拿到问题、选项、参考答案和模型原始输出,回答 YES 或 NO。这样能纠正"答对了但不匹配抽取正则"的情况。最终分取规则分与 judge 分中的较大者。
|
| 436 |
-
|
| 437 |
-
关于可复现性:温度 0 是贪心,原则上确定,但没有固定随机种子,且 vLLM 的连续批处理会随 batch 组成改变浮点累加顺序,所以边界样本仍可能翻转。严格复现需要同时锁定 judge 权重、vLLM 版本和 tensor-parallel 大小。
|
| 438 |
-
|
| 439 |
-
---
|
| 440 |
|
| 441 |
## 结构参数
|
| 442 |
|
| 443 |
| 项 | 值 |
|
| 444 |
|---|---|
|
| 445 |
-
| 语言层数 | 32(线性注意力
|
| 446 |
| hidden_size | 2560 |
|
| 447 |
| 注意力头 / KV 头 | 16 / 4 |
|
| 448 |
| 词表 | 248320 |
|
| 449 |
-
| 视觉层数 | 24 |
|
| 450 |
-
| 视觉 hidden_size | 1024 |
|
| 451 |
| patch_size / spatial_merge_size | 16 / 3 |
|
| 452 |
| max_position_embeddings | 262144 |
|
| 453 |
| 权重 | bfloat16,3 分片,约 9.9 GB |
|
|
|
|
| 13 |
|
| 14 |
# LLaVA-OneVision-2.5-4B-Instruct
|
| 15 |
|
| 16 |
+
基于 LLaVA-OneVision-2 架构的 4B 指令微调多模态模型,支持单图、多图和视频输入。也支持流式视频:画面逐帧到达,模型每一步决定是否发言——同一份权重、同一个对话模版,用一个关键字切换,见[流式](#流式)。
|
| 17 |
|
| 18 |
+
视觉编码器 24 层,`patch_size=16`,`spatial_merge_size=3`。语言侧 32 层,`hidden_size=2560`,线性注意力每 4 层夹一个全注意力层,`max_position_embeddings=262144`。建模代码随权重分发,必须 `trust_remote_code=True`。
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
## 安装
|
| 21 |
|
| 22 |
```bash
|
| 23 |
pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
|
| 24 |
+
pip install flash-attn --no-build-isolation # 可选,建议装
|
| 25 |
pip install decord # 仅抽帧视频路径需要
|
| 26 |
```
|
| 27 |
|
| 28 |
+
## 图片
|
|
|
|
|
|
|
| 29 |
|
| 30 |
```python
|
| 31 |
import torch
|
|
|
|
| 36 |
|
| 37 |
processor = AutoProcessor.from_pretrained(MODEL, trust_remote_code=True)
|
| 38 |
model = AutoModelForImageTextToText.from_pretrained(
|
| 39 |
+
MODEL, trust_remote_code=True, dtype=torch.bfloat16,
|
|
|
|
|
|
|
| 40 |
attn_implementation="flash_attention_2", # 没装 flash-attn 用 "sdpa"
|
| 41 |
device_map="auto",
|
| 42 |
).eval()
|
| 43 |
|
|
|
|
| 44 |
messages = [{"role": "user", "content": [
|
| 45 |
{"type": "image"},
|
| 46 |
{"type": "text", "text": "Describe this image in detail."},
|
| 47 |
]}]
|
| 48 |
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 49 |
+
inputs = processor(text=text, images=[Image.open("example.jpg").convert("RGB")],
|
| 50 |
+
return_tensors="pt").to(model.device)
|
| 51 |
|
| 52 |
with torch.inference_mode():
|
| 53 |
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
|
| 54 |
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 55 |
```
|
| 56 |
|
| 57 |
+
图片走 `smart_resize`,保持长宽比并对齐 patch 网格。像素预算默认 `min_pixels=3136` / `max_pixels=4000000`,可在 `processor.image_processor` 上覆盖。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
|
| 59 |
+
多图时每张图放一个 `{"type": "image"}`,`images=` 按同样顺序传入。每个占位符展开成 `<|vision_start|><|image_pad|><|vision_end|>`,按位置对应。
|
| 60 |
|
| 61 |
+
## 视频 — 抽帧
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 62 |
|
| 63 |
```python
|
| 64 |
messages = [{"role": "user", "content": [
|
|
|
|
| 68 |
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 69 |
|
| 70 |
inputs = processor(
|
| 71 |
+
text=text, videos=["example.mp4"],
|
|
|
|
| 72 |
max_frames=64,
|
| 73 |
+
min_pixels=153664, max_pixels=153664, # 设成相等以固定逐帧预算
|
|
|
|
| 74 |
return_tensors="pt",
|
| 75 |
).to(model.device)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
```
|
| 77 |
|
| 78 |
+
也接受 `target_fps` 和 `num_frames`,优先级见 `video_processing_llava_onevision2_5.py` 的 `choose_target_frames`。`max_frames` 封顶 768。
|
|
|
|
|
|
|
| 79 |
|
| 80 |
+
开销约等于帧数 × 逐帧像素,两者要一起调:
|
| 81 |
|
| 82 |
| 场景 | 帧数 | 逐帧像素 |
|
| 83 |
|---|---|---|
|
|
|
|
| 85 |
| 中等 | 256 | 136900 |
|
| 86 |
| 长视频、整体理解 | 640 – 768 | 72900 – 108900 |
|
| 87 |
|
| 88 |
+
所有帧是在视觉塔里一次前向编码的,超长输入可能打爆显存。设 `OV_VISION_CHUNK_PATCHES=200000` 可改为分组编码,按帧边界切分,注意力不跨帧,结果不变。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 89 |
|
| 90 |
+
**时间戳**一定会注入。`<|video_pad|>` 占位符展开成逐帧的块,每块前面是该帧的时间,取自解码器的真实采样位置:
|
|
|
|
|
|
|
| 91 |
|
| 92 |
```
|
| 93 |
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 94 |
<1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 95 |
```
|
| 96 |
|
| 97 |
+
要传路径,不要传自己解码好的帧。只给一串帧时 processor 拿不到真实采样时刻,会退化成用帧序号当秒数,于是 100 秒的视频看起来只有 N 秒,时间类回答全错(此时会抛 `UserWarning`)。必须自己解码的话,把时间信息一并交出:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 98 |
|
| 99 |
+
```python
|
| 100 |
+
videos=[{"frames": frames, "timestamps": seconds_list, "frame_indices": index_list}]
|
| 101 |
+
```
|
| 102 |
|
| 103 |
+
通过 `images=` 传帧则完全不注入时间戳。
|
| 104 |
|
| 105 |
+
## 流式
|
| 106 |
|
| 107 |
+
画面每轮进来一帧,模型每轮输出一个状态 token:沉默,或发言。一个关键字切换默认 system,其余什么都不改:
|
| 108 |
|
| 109 |
| 模式 | 调用 | 默认 system |
|
| 110 |
|---|---|---|
|
| 111 |
| 离线(默认) | `apply_chat_template(messages, ...)` | `You are a helpful assistant.` |
|
| 112 |
| 流式 | `apply_chat_template(messages, ..., streaming=True)` | `You are a real-time streaming video assistant.` |
|
| 113 |
|
| 114 |
+
显式传入的 `system` 在两种模式下都优先。默认走离线,是为了让现有离线评测不改代码也能用。
|
| 115 |
|
| 116 |
### 协议 token
|
| 117 |
|
|
|
|
| 119 |
|---|---|---|
|
| 120 |
| `<silence>` | 248077 | 当前轮不发言 |
|
| 121 |
| `<response>` | 248078 | 当前轮发言 |
|
| 122 |
+
| `<question_memory>` / `</question_memory>` | 248079 / 248080 | 新问题,在提问时刻写入一次 |
|
| 123 |
+
| `<answer_memory>` / `</answer_memory>` | 248081 / 248082 | 此前回答到期后回插一次 |
|
| 124 |
+
| `<captioning>` | 248083 | 当前轮发言,且是事件级 caption |
|
| 125 |
| `<caption_begin>` | 248084 | user 侧标记,说明这条流是 caption 任务 |
|
| 126 |
|
| 127 |
+
八个都是非 special 的 added token,所以 `skip_special_tokens=True` 仍会解出来——状态 token 正是你要读的输出。它们落在现有 248320 行 embedding 内,且已发布的 token id 一个都没动。两个 memory token 内部包 JSON,例如 `{"timestep": "00:00:06", "content": "..."}`。
|
|
|
|
|
|
|
|
|
|
|
|
|
| 128 |
|
| 129 |
+
### 怎么构造一轮
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 130 |
|
| 131 |
+
严格 user / assistant 交替,每个 user 轮最多一帧;没有新画面的轮次改用纯文本时间前缀。每帧给上 `timestamp`,帧后的换行作为单独的 text 项:
|
| 132 |
|
| 133 |
```python
|
| 134 |
messages = [{"role": "user", "content": [
|
|
|
|
| 140 |
messages, tokenize=False, add_generation_prompt=True, streaming=True)
|
| 141 |
```
|
| 142 |
|
| 143 |
+
渲染结果:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 144 |
|
| 145 |
```
|
| 146 |
+
<|im_start|>user
|
| 147 |
+
<question_memory>{"timestep": "00:00:00", "content": "..."}</question_memory>
|
| 148 |
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 149 |
+
<|im_end|>
|
| 150 |
+
<|im_start|>assistant
|
| 151 |
+
<silence><|im_end|>
|
| 152 |
```
|
| 153 |
|
| 154 |
+
content 传字符串则原样输出,适合自己已经拼好标记的 harness。
|
| 155 |
|
| 156 |
+
### 两个要你自己算的值
|
| 157 |
|
| 158 |
+
模版固定的是格式,有两个数值来自视频本身,从一张裸帧里恢复不出来:
|
| 159 |
|
| 160 |
+
1. **`timestamp`**——源帧号除以视频真实 fps。
|
| 161 |
+
2. **`patch_positions` 的 `t` 列**——把每帧当独立 image 交给 processor 时,返回的块布局是对的,但 `t` 每帧都是 0。需要改成 `floor(秒数 × fps)`,否则每帧都会被当成一段新视觉的开头。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 162 |
|
| 163 |
+
模型只用 `t` 在本段内的**名次**,视觉塔也只做帧内注意力;写真实帧号只是为了输入可审计。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 164 |
|
| 165 |
+
上面说的逐帧时间戳自动展开属于离线的 `<|video_pad|>` 分支,流式逐帧输入用不到。
|
| 166 |
|
| 167 |
+
本仓库不包含围绕模型的实时循环:按时钟取帧、决定何时停止生成、到期回插 question / answer memory、在决策点之间维持上下文。
|
| 168 |
|
| 169 |
+
## 视频 — codec 模式
|
| 170 |
|
| 171 |
+
codec 模式不按整帧采样,而是用编码器 bitcost 找出信息量大的区域,打包进少量**画布**。同一张画布上的 patch 可以来自不同源帧,各自带 `(源帧, 行, 列)` 供 3-D RoPE 使用——所以画布不能通过 `images=` 传。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 172 |
|
| 173 |
+
预处理来自 `codec-video-prep`(自带打过补丁的 FFmpeg 和 dav1d;限 Linux x86_64、CPython 3.10/3.11、`numpy<2`、系统 `ffprobe`;只支持 H.264/HEVC/VP9/AV1;只适用于离线文件,因为预算计算需要整段视频):
|
| 174 |
|
| 175 |
```bash
|
| 176 |
+
pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple \
|
| 177 |
+
"codec-video-prep==0.3.0"
|
| 178 |
+
codec-video-prep-doctor
|
|
|
|
|
|
|
|
|
|
|
|
|
| 179 |
```
|
| 180 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 181 |
```python
|
| 182 |
+
inputs = processor(text=text, videos=["example.mp4"],
|
| 183 |
+
video_backend="codec", return_tensors="pt").to(model.device)
|
| 184 |
```
|
| 185 |
|
| 186 |
+
输出除 `pixel_values` 和 `image_grid_thw` 外还带 `patch_positions`(`[总 patch 数, 3]`),会转发给 `model.forward(...)` 做 3-D RoPE,不要丢掉。
|
| 187 |
|
| 188 |
+
每张画布只有一个时间标记:所有 patch 来自同一源帧时是点 `<6.5 seconds>`,跨多帧时是区间 `<0.0 to 1.8 seconds>`(schema `canvas_patch_span_point_range_v1`)。时间类问题与抽帧路径表现一致。
|
| 189 |
|
| 190 |
+
默认配方写在 `preprocessor_config.json` 的 `codec` 字段——512 候选帧、64 画布、patch 16、3×3 块、每画布 313600 像素——单次调用用 `codec_config={"target_canvas": 32, ...}` 覆盖。其余键:`strategy`(`adaptive_pwindow`、`adaptive_rdo`、`readiness`)、`num_sampled_frames`、`max_pixels`、`canvas_format`。`patch=16` 和 `block_size=3` 必须与模型的 `patch_size`、`spatial_merge_size` 一致,不一致时预处理直接报错而非静默出错。画布缓存在 `~/.cache/codec_video_prep/canvases`(用 `CVP_CODEC_CACHE_ROOT` 换位置)。
|
| 191 |
|
| 192 |
+
## 杂项
|
| 193 |
|
| 194 |
+
**批量。** 传一组 text 和对应的图片,`padding=True`。padding 在左侧,按各自的 `input_ids` 长度切输出。
|
| 195 |
|
| 196 |
+
**离线。** 设 `HF_HUB_OFFLINE=1` 和 `TRANSFORMERS_OFFLINE=1`。多进程启动前先单进程完整加载一次:remote-code 模块会编译到 `$HF_HOME/modules/transformers_modules/<名字>/`,多进程首次加载会在那里竞争。该缓存只按叶子目录名索引,所以本地副本要取不同的目录名。
|
| 197 |
|
| 198 |
+
**生成。** `generation_config.json` 只设了 `eos_token_id=248044` 和 `use_cache=true`。确定性输出用 `do_sample=False`;需要先推理再作答的任务要给足 `max_new_tokens`,128 会在推理中途截断。
|
| 199 |
|
| 200 |
+
**回答格式。** 选择题上模型可能写整句而不是一个字母。如果后续要用正则解析,就在 prompt 里要求给字母:`Answer with the option's letter from the given choices directly.`。grounding 任务请在 prompt 里写明坐标范围,例如 0–1000 的整数。八个流式 token 不会被 `skip_special_tokens` 过滤,离线场景不想要就显式剥除。
|
| 201 |
|
| 202 |
## 评测结果
|
| 203 |
|
| 204 |
+
<!-- 上传前待补:用本 checkpoint 自己的评测跑结果填写。
|
| 205 |
不要沿用 preview base 模型的那张表——那些数字属于另一个 checkpoint,
|
| 206 |
放在这里会误报本模型的能力。 -->
|
| 207 |
|
| 208 |
+
_待用本 checkpoint 的评测结果补充,口径见下。_
|
|
|
|
|
|
|
|
|
|
|
|
|
| 209 |
|
| 210 |
## 评测口径
|
| 211 |
|
| 212 |
+
视频任务用 `fps=1`,固定帧数上限 F 和逐帧像素预算 MP(`min_pixels` = `max_pixels`)。帧离线缓存好,评测时不解码,重复跑输入完全一致。图像任务用默认像素预算。全部贪心生成,`max_new_tokens=4096`,`batch_size=1`。
|
|
|
|
|
|
|
| 213 |
|
| 214 |
| 任务 | F | MP |
|
| 215 |
|---|---|---|
|
|
|
|
| 224 |
| VSTAT | 32 | 131072 |
|
| 225 |
| MVBench、NextQA、TempCompass、MMVU、ReVSI | 64 | 153664 |
|
| 226 |
|
| 227 |
+
多数任务会再经一个 judge 模型(Qwen3.8-27B,温度 0)复核:它看到问题、选项、参考答案和原始输出,回答 YES 或 NO,用于捞回"答对了但不匹配抽取正则"的情况;最终分取规则分与 judge 分中的较大者。纯几何判分的任务(IoU、点落在掩码内)跳过 judge,直接用规则分。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 228 |
|
| 229 |
## 结构参数
|
| 230 |
|
| 231 |
| 项 | 值 |
|
| 232 |
|---|---|
|
| 233 |
+
| 语言层数 | 32(线性注意力,每 4 层一个全注意力) |
|
| 234 |
| hidden_size | 2560 |
|
| 235 |
| 注意力头 / KV 头 | 16 / 4 |
|
| 236 |
| 词表 | 248320 |
|
| 237 |
+
| 视觉层数 / hidden_size | 24 / 1024 |
|
|
|
|
| 238 |
| patch_size / spatial_merge_size | 16 / 3 |
|
| 239 |
| max_position_embeddings | 262144 |
|
| 240 |
| 权重 | bfloat16,3 分片,约 9.9 GB |
|