yiyexy commited on
Commit
89a83c4
·
verified ·
1 Parent(s): 165351a

Tighten model cards (-47% lines)

Browse files
Files changed (2) hide show
  1. README.md +70 -283
  2. README_CN.md +66 -279
README.md CHANGED
@@ -13,25 +13,19 @@ tags:
13
 
14
  # LLaVA-OneVision-2.5-4B-Instruct
15
 
16
- An instruction-tuned 4B multimodal model on the LLaVA-OneVision-2 architecture, for single-image, multi-image and video input. Vision encoder: 24 layers, `spatial_merge_size=3`. Language side: 32 layers, `hidden_size=2560`, linear attention interleaved with full attention (one full-attention layer every 4), `max_position_embeddings=262144`.
17
 
18
- Beyond the offline tasks, this release is also trained for **streaming** video: frames arrive one at a time and the model decides at every step whether to stay silent or speak. Both behaviors live in one set of weights and one chat template; see [Streaming mode](#streaming-mode).
19
-
20
- The modeling code ships with the weights, so **`trust_remote_code=True` is required**.
21
-
22
- ---
23
 
24
  ## Install
25
 
26
  ```bash
27
  pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
28
- pip install flash-attn --no-build-isolation # optional, strongly recommended
29
- pip install decord # only for the frame-sampling video path
30
  ```
31
 
32
- ---
33
-
34
- ## Single image
35
 
36
  ```python
37
  import torch
@@ -42,57 +36,30 @@ MODEL = "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct"
42
 
43
  processor = AutoProcessor.from_pretrained(MODEL, trust_remote_code=True)
44
  model = AutoModelForImageTextToText.from_pretrained(
45
- MODEL,
46
- trust_remote_code=True,
47
- dtype=torch.bfloat16,
48
- attn_implementation="flash_attention_2", # use "sdpa" without flash-attn
49
  device_map="auto",
50
  ).eval()
51
 
52
- image = Image.open("example.jpg").convert("RGB")
53
  messages = [{"role": "user", "content": [
54
  {"type": "image"},
55
  {"type": "text", "text": "Describe this image in detail."},
56
  ]}]
57
  text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
58
- inputs = processor(text=text, images=[image], return_tensors="pt").to(model.device)
 
59
 
60
  with torch.inference_mode():
61
  out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
62
  print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
63
  ```
64
 
65
- Images go through `smart_resize`, which keeps the aspect ratio and aligns to the patch grid. The pixel budget defaults to `min_pixels=3136` / `max_pixels=4000000` and can be overridden on the processor:
66
-
67
- ```python
68
- processor.image_processor.min_pixels = 256 * 28 * 28
69
- processor.image_processor.max_pixels = 1280 * 28 * 28
70
- ```
71
-
72
- ---
73
-
74
- ## Multiple images
75
-
76
- Put one `{"type": "image"}` per image in `content`, and pass `images` in the same order:
77
-
78
- ```python
79
- messages = [{"role": "user", "content": [
80
- {"type": "image"},
81
- {"type": "image"},
82
- {"type": "text", "text": "What changed between these two images?"},
83
- ]}]
84
- text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
85
- inputs = processor(text=text, images=[img_a, img_b], return_tensors="pt").to(model.device)
86
- ```
87
-
88
- Each placeholder expands to `<|vision_start|><|image_pad|><|vision_end|>`, matched to the `images` list by position.
89
 
90
- ---
91
 
92
  ## Video — frame sampling
93
 
94
- Pass the **video path**, not pre-decoded frames:
95
-
96
  ```python
97
  messages = [{"role": "user", "content": [
98
  {"type": "video"},
@@ -101,24 +68,16 @@ messages = [{"role": "user", "content": [
101
  text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
102
 
103
  inputs = processor(
104
- text=text,
105
- videos=["example.mp4"],
106
  max_frames=64,
107
- min_pixels=153664,
108
- max_pixels=153664, # for video, set min and max equal to fix the per-frame budget
109
  return_tensors="pt",
110
  ).to(model.device)
111
-
112
- with torch.inference_mode():
113
- out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
114
- print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
115
  ```
116
 
117
- `processor.__call__` also accepts `target_fps` (sample at a fixed rate, capped by `max_frames`) and `num_frames` (a fixed count). Precedence is in `choose_target_frames` in `video_processing_llava_onevision2_5.py`.
118
-
119
- ### Frames and per-frame pixels
120
 
121
- Cost is roughly **frames × pixels per frame**, so tune the two together. The processor caps `max_frames` at 768; raising the frame count without lowering the pixel budget runs out of memory quickly.
122
 
123
  | Case | Frames | Pixels per frame |
124
  |---|---|---|
@@ -126,47 +85,33 @@ Cost is roughly **frames × pixels per frame**, so tune the two together. The pr
126
  | Medium | 256 | 136900 |
127
  | Long video, global understanding | 640 – 768 | 72900 – 108900 |
128
 
129
- Very long inputs can still exhaust memory inside the vision tower, because all frames are encoded in one forward pass. Set `OV_VISION_CHUNK_PATCHES` to encode them in groups instead:
130
-
131
- ```bash
132
- export OV_VISION_CHUNK_PATCHES=200000 # 0 (default) = off
133
- ```
134
 
135
- Grouping follows the per-frame sequence boundaries, so attention never crosses a frame and the result is mathematically the same as the single pass. It only trades a little speed for memory.
136
-
137
- ### Timestamps are injected automatically
138
-
139
- Video input always carries timestamps; this cannot be turned off. The `<|vision_start|><|video_pad|><|vision_end|>` placeholder is rewritten into one block per frame, each prefixed with its time:
140
 
141
  ```
142
  <0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
143
  <1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
144
  ```
145
 
146
- The seconds come from the decoder's real sample positions, not from an assumed uniform spacing, so you can ask for temporal localization directly.
147
 
148
- > **Pass a path, not your own frames.** `videos=` accepts both, but they are not equivalent. Only the path lets the processor recover real sample times and real source frame numbers. Given a bare frame list it has neither, and falls back to frame ordinals as seconds — frames 0, 1, 2 become 0.0, 1.0, 2.0 seconds — so a 100-second video looks N seconds long and temporal answers are wrong. The processor raises a `UserWarning` in that case.
149
- >
150
- > If you must decode yourself, hand over the timing as well:
151
- > ```python
152
- > videos=[{"frames": frames, "timestamps": seconds_list, "frame_indices": index_list}]
153
- > ```
154
- > Passing frames through `images=` injects no timestamps at all.
155
-
156
- ---
157
 
158
- ## Streaming mode
159
 
160
- In streaming use the video is not handed over as a whole. Frames arrive one per turn, and at every turn the model emits a **state token**: either it stays silent, or it speaks.
161
 
162
- The same weights and the same `chat_template.jinja` serve both modes. One keyword switches them:
163
 
164
- | Mode | Call | Default system prompt |
165
  |---|---|---|
166
  | Offline (default) | `apply_chat_template(messages, ...)` | `You are a helpful assistant.` |
167
  | Streaming | `apply_chat_template(messages, ..., streaming=True)` | `You are a real-time streaming video assistant.` |
168
 
169
- That default system turn is the **only** thing the switch changes — everything else is decided by the content you pass. An explicit `system` message always wins, in both modes. Offline is the default so that existing evaluation harnesses keep working without passing anything.
170
 
171
  ### Protocol tokens
172
 
@@ -174,36 +119,16 @@ That default system turn is the **only** thing the switch changes �� everything
174
  |---|---|---|
175
  | `<silence>` | 248077 | this turn: say nothing |
176
  | `<response>` | 248078 | this turn: speak |
177
- | `<question_memory>` / `</question_memory>` | 248079 / 248080 | a new question or task instruction, written once at the moment it is asked |
178
- | `<answer_memory>` / `</answer_memory>` | 248081 / 248082 | an earlier answer re-injected into the context after it is due |
179
- | `<captioning>` | 248083 | this turn: speak, as event-level captioning rather than as a reply |
180
  | `<caption_begin>` | 248084 | user-side marker that this stream is a captioning task |
181
 
182
- All eight are **non-special added tokens**. `decode(..., skip_special_tokens=True)` therefore still returns them, which is deliberate: the state token is the output you need to read. They fit inside the existing 248320-row embedding, so no vocabulary resize was needed, and no pre-existing token id moved.
183
-
184
- `<question_memory>` and `<answer_memory>` wrap a JSON object, for example `{"timestep": "00:00:06", "content": "..."}`, where `timestep` is the original wall-clock moment in `HH:MM:SS`.
185
-
186
- ### Turn layout
187
-
188
- Strict user/assistant alternation. Each user turn carries at most one frame; a turn with no new frame uses a bare text time prefix instead.
189
 
190
- ```
191
- <|im_start|>system
192
- You are a real-time streaming video assistant.<|im_end|>
193
- <|im_start|>user
194
- <question_memory>{"timestep": "00:00:00", "content": "..."}</question_memory>
195
- <0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
196
- <|im_end|>
197
- <|im_start|>assistant
198
- <silence><|im_end|>
199
- <|im_start|>user
200
- <1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
201
- <|im_end|>
202
- <|im_start|>assistant
203
- <response><|im_end|>
204
- ```
205
 
206
- Build it with one `{"type": "image"}` per frame, carrying its `timestamp` in seconds, and the trailing newline as its own text item:
207
 
208
  ```python
209
  messages = [{"role": "user", "content": [
@@ -215,195 +140,76 @@ text = processor.apply_chat_template(
215
  messages, tokenize=False, add_generation_prompt=True, streaming=True)
216
  ```
217
 
218
- Content may also be passed as a plain string, which is emitted verbatim — useful when your harness already renders `<T seconds><|vision_start|>…` itself.
219
-
220
- ### Two values the caller must compute
221
-
222
- The template guarantees the **format**. Two **numbers** depend on the video and cannot be recovered from a bare frame, so you have to supply them:
223
-
224
- 1. **The text timestamp.** Pass `timestamp = source_frame_index / fps` for each frame, using the video's real frame rate.
225
- 2. **The `t` column of `patch_positions`.** If each frame is handed to the processor as an independent image, the returned `patch_positions` are in the right block layout but `t` is **0 for every frame**. Rewrite each frame's rows to its real source frame index, `floor(seconds × fps)`. Leave it at 0 and every frame reads as the start of a new visual segment, which is not what the model was trained on.
226
-
227
- The model uses `t` only through its **rank** within the visual segment (0, 1, 2, …), never its absolute value, and the vision tower attends within a frame only — so what matters is that the order is right, not the exact numbers. Keeping them equal to the true frame numbers simply makes the input auditable.
228
-
229
- The automatic per-frame timestamp expansion described above belongs to the offline `<|video_pad|>` branch. Streaming feeds frames one at a time and does not go through it.
230
-
231
- > **What is not in this repository.** These weights give you the trained behavior, the template and the tokens. The real-time loop around them — grabbing frames on a clock, deciding when to stop generating, re-injecting question and answer memory as it falls due, and carrying context across decision points — is serving-side work and is not shipped here.
232
-
233
- ---
234
-
235
- ## Video — codec mode
236
-
237
- Instead of sampling whole frames, codec mode uses the encoder's bitcost to find the informative regions and packs them into a small number of **canvases**.
238
-
239
- The key difference: **patches on one canvas can come from several different source frames.** Every patch carries its own source position `(source frame, source patch row, source patch column)`, and the vision tower uses those for 3-D RoPE. Feeding canvases through `images=` throws that away.
240
-
241
- ### Install the preprocessing package
242
-
243
- Codec preprocessing comes from `codec-video-prep`, which bundles patched FFmpeg and dav1d for bitcost extraction — nothing to compile:
244
-
245
- ```bash
246
- pip install -i https://test.pypi.org/simple/ \
247
- --extra-index-url https://pypi.org/simple \
248
- "codec-video-prep==0.3.0"
249
- ```
250
-
251
- `--extra-index-url` is required: the package is on TestPyPI, its dependencies (numpy, opencv-python-headless, Pillow, PyYAML) are on PyPI proper.
252
-
253
- Check the install:
254
-
255
- ```bash
256
- codec-video-prep-doctor
257
- ```
258
-
259
- Requirements:
260
-
261
- - **Linux x86_64, CPython 3.10 or 3.11** (manylinux_2_17 wheels). Not macOS, ARM or Alpine musl.
262
- - **NumPy < 2** (`numpy>=1.23,<2`). If your inference environment is on NumPy 2, run preprocessing in a separate environment and hand over the saved output.
263
- - System `ffprobe` (`apt-get install ffmpeg` or equivalent).
264
- - Bitcost support covers **H.264, HEVC, VP9 and AV1**. Other codecs (MPEG-4, MPEG-2) have no implementation, and **the file extension does not tell you the codec**:
265
-
266
- ```bash
267
- ffprobe -v error -select_streams v:0 -show_entries stream=codec_name \
268
- -of default=noprint_wrappers=1 input.mp4
269
- ```
270
-
271
- - Sampling and budgeting need the whole video up front, so this is for **offline files**, not live streams.
272
-
273
- ### Usage
274
-
275
- Set `video_backend="codec"` and pass a video path:
276
-
277
- ```python
278
- inputs = processor(
279
- text=text,
280
- videos=["example.mp4"],
281
- video_backend="codec",
282
- return_tensors="pt",
283
- ).to(model.device)
284
-
285
- with torch.inference_mode():
286
- out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
287
- ```
288
-
289
- Alongside `input_ids` and `attention_mask`, `inputs` carries `pixel_values`, `image_grid_thw` (one row per canvas) and **`patch_positions`** (`[total_patches, 3]`, block layout). `patch_positions` is forwarded to `model.forward(...)` for 3-D RoPE — **do not drop it**.
290
-
291
- ### One time marker per canvas
292
-
293
- Each canvas gets a single marker, and its form depends on how many source frames that canvas draws from:
294
-
295
- | Canvas | Marker | Example |
296
- |---|---|---|
297
- | All patches from one source frame | a point | `<6.5 seconds>` |
298
- | Patches spanning several frames | a range | `<0.0 to 1.8 seconds>` |
299
 
300
  ```
 
 
301
  <0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
302
- <0.0 to 1.8 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
303
- <2.0 to 4.1 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
 
304
  ```
305
 
306
- This is the serialization used in training (schema `canvas_patch_span_point_range_v1`), decided purely by each canvas's actual source-frame span. Temporal questions work the same as on the frame path.
307
-
308
- ### Budget
309
-
310
- The default recipe — 512 candidate frames, 64 canvases, patch 16, 3×3 blocks, 313600 pixels per canvas — lives in the `codec` field of `preprocessor_config.json`. Override per call with `codec_config=`:
311
-
312
- ```python
313
- inputs = processor(
314
- text=text, videos=["example.mp4"], video_backend="codec",
315
- codec_config={"target_canvas": 32, "num_sampled_frames": 256},
316
- return_tensors="pt",
317
- )
318
- ```
319
 
320
- | Option | Default | Meaning |
321
- |---|---|---|
322
- | `strategy` | `adaptive_pwindow` | also `adaptive_rdo`, `readiness` |
323
- | `target_canvas` | 64 | canvas budget; `readiness` does not bound the count, read the actual output |
324
- | `num_sampled_frames` | 512 | candidate frames |
325
- | `patch` / `block_size` | 16 / 3 | must match the vision tower's patch size and spatial merge |
326
- | `max_pixels` | 313600 | pixels per canvas |
327
- | `canvas_format` | `png` | `png`, `jpg` or `npy` |
328
 
329
- > `patch=16` and `block_size=3` correspond to this model's `patch_size=16` and `spatial_merge_size=3`. Changing them misaligns the canvas grid with the merger, and preprocessing raises rather than failing silently.
330
 
331
- Canvases are cached in `~/.cache/codec_video_prep/canvases` (`CVP_CODEC_CACHE_ROOT` to relocate), so several questions about one video decode it once.
 
332
 
333
- ---
334
 
335
- ## Batching
336
 
337
- ```python
338
- texts = [processor.apply_chat_template(m, tokenize=False, add_generation_prompt=True)
339
- for m in batch_messages]
340
- inputs = processor(text=texts, images=batch_images, padding=True,
341
- return_tensors="pt").to(model.device)
342
- ```
343
 
344
- Padding is left-sided; slice each output by its own `input_ids` length.
345
 
346
- ---
347
 
348
- ## Offline use
349
 
350
  ```bash
351
- export HF_HUB_OFFLINE=1
352
- export TRANSFORMERS_OFFLINE=1
 
353
  ```
354
 
355
  ```python
356
- processor = AutoProcessor.from_pretrained("/local/path", trust_remote_code=True)
357
- model = AutoModelForImageTextToText.from_pretrained("/local/path", trust_remote_code=True, ...)
358
  ```
359
 
360
- **Warm up once before launching multiple processes.** `trust_remote_code` modules are compiled into `$HF_HOME/modules/transformers_modules/<name>/`. Processes loading for the first time simultaneously race on that directory, and some fail with `AttributeError: module '...modeling_llava_onevision2_5' has no attribute 'LlavaOnevision2ForConditionalGeneration'`. Load the full model once in a single process first — `AutoConfig` alone does not generate the `modeling_*.py` cache — then start the rest.
361
 
362
- The cache directory is keyed by the **leaf directory name** only. Two local checkpoints both called `model_hf` share one cache entry and overwrite each other's modules; give local copies distinct directory names.
363
 
364
- ---
365
 
366
- ## Generation
367
 
368
- `generation_config.json` sets only `eos_token_id=248044` and `use_cache=true`; no sampling defaults. For deterministic output use greedy decoding:
369
 
370
- ```python
371
- model.generate(**inputs, max_new_tokens=512, do_sample=False)
372
- ```
373
-
374
- Tasks where the model reasons before answering need a generous `max_new_tokens`; small values such as 128 cut it off mid-reasoning.
375
 
376
- In streaming mode the useful output is usually the first state token, so a small `max_new_tokens` with an early stop is appropriate there instead.
377
-
378
- ---
379
 
380
- ## Known behaviors
381
-
382
- **Answer format affects regex-based scoring.** On some multiple-choice tasks the model writes "The correct answer is ..." or reasons first, rather than emitting a bare letter. Scripts that extract the first letter will mark those wrong. To force a letter, ask for one: `Answer with the option's letter from the given choices directly.`
383
-
384
- **Coordinate convention.** For grounding, state the range in the prompt, e.g. `Answer with the bounding box as [x1, y1, x2, y2], each coordinate an integer between 0 and 1000 (1000 = full image width/height).` Without it the convention varies by task.
385
-
386
- **State tokens survive `skip_special_tokens`.** The eight streaming tokens are non-special by design. If you are post-processing offline output and want them gone, strip them explicitly.
387
-
388
- ---
389
 
390
  ## Benchmarks
391
 
392
- <!-- TODO before publishing: fill from the instruct checkpoint's own evaluation run.
393
- Do NOT reuse the preview base model's table — those numbers belong to a different
394
- checkpoint and would misreport this one. -->
395
 
396
- _To be filled in from this checkpoint's evaluation run._
397
-
398
- The protocol below is the one these numbers must be produced under.
399
-
400
- ---
401
 
402
  ## Evaluation protocol
403
 
404
- ### Frame settings
405
-
406
- Video tasks use `fps=1` with a fixed frame cap F and per-frame pixel budget MP (`min_pixels` and `max_pixels` set equal). Frames are cached offline, so evaluation decodes nothing at run time and repeated runs see identical input.
407
 
408
  | Task | F | MP |
409
  |---|---|---|
@@ -418,36 +224,17 @@ Video tasks use `fps=1` with a fixed frame cap F and per-frame pixel budget MP (
418
  | VSTAT | 32 | 131072 |
419
  | MVBench, NextQA, TempCompass, MMVU, ReVSI | 64 | 153664 |
420
 
421
- Image tasks use the preprocessor's default pixel budget. All generation is greedy with `max_new_tokens=4096` and `batch_size=1`.
422
-
423
- ### Scoring
424
-
425
- Most tasks are re-checked by a judge model. Tasks that score by pure geometry (IoU, point-in-mask) or that otherwise never extract a letter from text skip it, because judging adds nothing; those rows equal their rule-based score.
426
-
427
- | Judge | Value |
428
- |---|---|
429
- | Model | Qwen3.8-27B |
430
- | Serving | vLLM 0.19, `--tensor-parallel-size 4`, `--gpu-memory-utilization 0.72`, `--max-model-len 16384`, `--enforce-eager` |
431
- | Temperature | 0 |
432
- | Max tokens | 6 |
433
- | Thinking | off (`enable_thinking=False`) |
434
-
435
- The judge receives the question, options, reference answer and the model's raw output, and answers YES or NO. This corrects answers that are right but do not match the extraction regex. The final score is the greater of the rule-based and judged scores.
436
-
437
- On reproducibility: temperature 0 is greedy and therefore deterministic in principle, but no seed is fixed, and vLLM's continuous batching changes float accumulation order with batch composition, so borderline samples can still flip. Strict reproduction needs the judge weights, vLLM version and tensor-parallel size all pinned.
438
-
439
- ---
440
 
441
  ## Architecture
442
 
443
  | Item | Value |
444
  |---|---|
445
- | Language layers | 32 (linear attention interleaved with full attention, every 4) |
446
  | hidden_size | 2560 |
447
  | Attention heads / KV heads | 16 / 4 |
448
  | Vocabulary | 248320 |
449
- | Vision layers | 24 |
450
- | Vision hidden_size | 1024 |
451
  | patch_size / spatial_merge_size | 16 / 3 |
452
  | max_position_embeddings | 262144 |
453
  | Weights | bfloat16, 3 shards, about 9.9 GB |
 
13
 
14
  # LLaVA-OneVision-2.5-4B-Instruct
15
 
16
+ Instruction-tuned 4B multimodal model on the LLaVA-OneVision-2 architecture, for single-image, multi-image and video input. It also handles streaming video, where frames arrive one at a time and the model decides at each step whether to speak — same weights, same chat template, switched by one keyword. See [Streaming](#streaming).
17
 
18
+ Vision encoder: 24 layers, `patch_size=16`, `spatial_merge_size=3`. Language side: 32 layers, `hidden_size=2560`, linear attention with one full-attention layer every 4, `max_position_embeddings=262144`. The modeling code ships with the weights, so `trust_remote_code=True` is required.
 
 
 
 
19
 
20
  ## Install
21
 
22
  ```bash
23
  pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
24
+ pip install flash-attn --no-build-isolation # optional, recommended
25
+ pip install decord # frame-sampling video path only
26
  ```
27
 
28
+ ## Images
 
 
29
 
30
  ```python
31
  import torch
 
36
 
37
  processor = AutoProcessor.from_pretrained(MODEL, trust_remote_code=True)
38
  model = AutoModelForImageTextToText.from_pretrained(
39
+ MODEL, trust_remote_code=True, dtype=torch.bfloat16,
40
+ attn_implementation="flash_attention_2", # "sdpa" without flash-attn
 
 
41
  device_map="auto",
42
  ).eval()
43
 
 
44
  messages = [{"role": "user", "content": [
45
  {"type": "image"},
46
  {"type": "text", "text": "Describe this image in detail."},
47
  ]}]
48
  text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
49
+ inputs = processor(text=text, images=[Image.open("example.jpg").convert("RGB")],
50
+ return_tensors="pt").to(model.device)
51
 
52
  with torch.inference_mode():
53
  out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
54
  print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
55
  ```
56
 
57
+ Images go through `smart_resize`, keeping aspect ratio and aligning to the patch grid. The pixel budget defaults to `min_pixels=3136` / `max_pixels=4000000`; override it on `processor.image_processor`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
58
 
59
+ For several images, put one `{"type": "image"}` per image in `content` and pass `images=` in the same order. Each placeholder expands to `<|vision_start|><|image_pad|><|vision_end|>`, matched by position.
60
 
61
  ## Video — frame sampling
62
 
 
 
63
  ```python
64
  messages = [{"role": "user", "content": [
65
  {"type": "video"},
 
68
  text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
69
 
70
  inputs = processor(
71
+ text=text, videos=["example.mp4"],
 
72
  max_frames=64,
73
+ min_pixels=153664, max_pixels=153664, # set equal to fix the per-frame budget
 
74
  return_tensors="pt",
75
  ).to(model.device)
 
 
 
 
76
  ```
77
 
78
+ `target_fps` and `num_frames` are also accepted; precedence is in `choose_target_frames` in `video_processing_llava_onevision2_5.py`. `max_frames` caps at 768.
 
 
79
 
80
+ Cost is roughly frames × pixels per frame, so tune the two together:
81
 
82
  | Case | Frames | Pixels per frame |
83
  |---|---|---|
 
85
  | Medium | 256 | 136900 |
86
  | Long video, global understanding | 640 – 768 | 72900 – 108900 |
87
 
88
+ All frames are encoded in one vision-tower forward pass, which can exhaust memory on very long inputs. `OV_VISION_CHUNK_PATCHES=200000` encodes them in groups instead, split along frame boundaries — attention never crosses a frame, so the result is unchanged.
 
 
 
 
89
 
90
+ **Timestamps.** Video input always carries them. The `<|video_pad|>` placeholder expands into one block per frame, each prefixed with its time, taken from the decoder's real sample positions:
 
 
 
 
91
 
92
  ```
93
  <0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
94
  <1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
95
  ```
96
 
97
+ Pass a path, not pre-decoded frames. Given a bare frame list the processor has no real sample times and falls back to frame ordinals as seconds, so a 100-second video looks N seconds long and temporal answers are wrong (it raises a `UserWarning`). If you must decode yourself, hand over the timing as well:
98
 
99
+ ```python
100
+ videos=[{"frames": frames, "timestamps": seconds_list, "frame_indices": index_list}]
101
+ ```
 
 
 
 
 
 
102
 
103
+ Frames passed through `images=` get no timestamps at all.
104
 
105
+ ## Streaming
106
 
107
+ Frames arrive one per turn, and each turn the model emits a state token: stay silent, or speak. One keyword switches the default system prompt, and changes nothing else:
108
 
109
+ | Mode | Call | Default system |
110
  |---|---|---|
111
  | Offline (default) | `apply_chat_template(messages, ...)` | `You are a helpful assistant.` |
112
  | Streaming | `apply_chat_template(messages, ..., streaming=True)` | `You are a real-time streaming video assistant.` |
113
 
114
+ An explicit `system` message wins in both modes. Offline is the default so existing evaluation harnesses work unchanged.
115
 
116
  ### Protocol tokens
117
 
 
119
  |---|---|---|
120
  | `<silence>` | 248077 | this turn: say nothing |
121
  | `<response>` | 248078 | this turn: speak |
122
+ | `<question_memory>` / `</question_memory>` | 248079 / 248080 | a new question, written once at the moment it is asked |
123
+ | `<answer_memory>` / `</answer_memory>` | 248081 / 248082 | an earlier answer, re-injected once it is due |
124
+ | `<captioning>` | 248083 | this turn: speak, as event-level captioning |
125
  | `<caption_begin>` | 248084 | user-side marker that this stream is a captioning task |
126
 
127
+ All eight are non-special added tokens, so `skip_special_tokens=True` still returns them — the state token is the output you read. They fit the existing 248320-row embedding, and no previously published token id moved. The memory tokens wrap a JSON object, e.g. `{"timestep": "00:00:06", "content": "..."}`.
 
 
 
 
 
 
128
 
129
+ ### Building a turn
 
 
 
 
 
 
 
 
 
 
 
 
 
 
130
 
131
+ Strict user/assistant alternation, at most one frame per user turn; a turn with no new frame uses a bare text time prefix instead. Give each frame its `timestamp`, and the trailing newline as its own text item:
132
 
133
  ```python
134
  messages = [{"role": "user", "content": [
 
140
  messages, tokenize=False, add_generation_prompt=True, streaming=True)
141
  ```
142
 
143
+ which renders as:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
144
 
145
  ```
146
+ <|im_start|>user
147
+ <question_memory>{"timestep": "00:00:00", "content": "..."}</question_memory>
148
  <0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
149
+ <|im_end|>
150
+ <|im_start|>assistant
151
+ <silence><|im_end|>
152
  ```
153
 
154
+ A plain string `content` is emitted verbatim, for harnesses that render the markers themselves.
 
 
 
 
 
 
 
 
 
 
 
 
155
 
156
+ ### Two values you have to compute
 
 
 
 
 
 
 
157
 
158
+ The template fixes the format; two numbers come from the video and cannot be recovered from a bare frame:
159
 
160
+ 1. **`timestamp`** — the source frame index divided by the video's real fps.
161
+ 2. **The `t` column of `patch_positions`** — handed independent images, the processor returns the right block layout but `t=0` for every frame. Rewrite each frame's rows to `floor(seconds × fps)`, or every frame reads as the start of a new visual segment.
162
 
163
+ Only the *rank* of `t` within the segment matters to the model, and the vision tower attends within a frame only; keeping real frame numbers just makes the input auditable.
164
 
165
+ The automatic timestamp expansion described above belongs to the offline `<|video_pad|>` branch. Streaming feeds frames one at a time and does not use it.
166
 
167
+ Not included here: the real-time loop around the model — frame scheduling, deciding when to stop generating, re-injecting question and answer memory as it falls due, and carrying context across decision points.
 
 
 
 
 
168
 
169
+ ## Video — codec mode
170
 
171
+ Instead of sampling whole frames, codec mode uses encoder bitcost to find informative regions and packs them into a few *canvases*. Patches on one canvas may come from different source frames, each carrying its own `(source frame, row, column)` for 3-D RoPE — so canvases must not be fed through `images=`.
172
 
173
+ Preprocessing comes from `codec-video-prep` (bundles patched FFmpeg and dav1d; Linux x86_64, CPython 3.10/3.11, `numpy<2`, system `ffprobe`; H.264/HEVC/VP9/AV1 only; offline files only, since budgeting needs the whole video):
174
 
175
  ```bash
176
+ pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple \
177
+ "codec-video-prep==0.3.0"
178
+ codec-video-prep-doctor
179
  ```
180
 
181
  ```python
182
+ inputs = processor(text=text, videos=["example.mp4"],
183
+ video_backend="codec", return_tensors="pt").to(model.device)
184
  ```
185
 
186
+ The output carries `patch_positions` (`[total_patches, 3]`) alongside `pixel_values` and `image_grid_thw`. It is forwarded to `model.forward(...)` for 3-D RoPE — do not drop it.
187
 
188
+ Each canvas gets one time marker: a point `<6.5 seconds>` when all its patches come from one source frame, a range `<0.0 to 1.8 seconds>` when it spans several (schema `canvas_patch_span_point_range_v1`). Temporal questions work as on the frame path.
189
 
190
+ Defaults live in the `codec` field of `preprocessor_config.json` — 512 candidate frames, 64 canvases, patch 16, 3×3 blocks, 313600 pixels per canvas — and are overridden per call with `codec_config={"target_canvas": 32, ...}`. Other keys: `strategy` (`adaptive_pwindow`, `adaptive_rdo`, `readiness`), `num_sampled_frames`, `max_pixels`, `canvas_format`. `patch=16` and `block_size=3` must match the model's `patch_size` and `spatial_merge_size`; preprocessing raises rather than failing silently. Canvases cache in `~/.cache/codec_video_prep/canvases` (`CVP_CODEC_CACHE_ROOT` to relocate).
191
 
192
+ ## Notes
193
 
194
+ **Batching.** Pass a list of texts and the matching images with `padding=True`. Padding is left-sided, so slice each output by its own `input_ids` length.
195
 
196
+ **Offline.** Set `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1`. Load the full model once in a single process before starting more: remote-code modules compile into `$HF_HOME/modules/transformers_modules/<name>/`, and concurrent first loads race there. That cache is keyed by leaf directory name only, so give local copies distinct names.
 
 
 
 
197
 
198
+ **Generation.** `generation_config.json` sets only `eos_token_id=248044` and `use_cache=true`. Use `do_sample=False` for deterministic output, and a generous `max_new_tokens` on tasks where the model reasons before answering — 128 cuts it off mid-reasoning.
 
 
199
 
200
+ **Answer format.** On multiple-choice tasks the model may write a sentence rather than a bare letter. If a regex will parse the output, ask for a letter: `Answer with the option's letter from the given choices directly.` For grounding, state the coordinate range in the prompt, e.g. integers 0–1000. The eight streaming tokens survive `skip_special_tokens`; strip them explicitly if you do not want them offline.
 
 
 
 
 
 
 
 
201
 
202
  ## Benchmarks
203
 
204
+ <!-- TODO before publishing: fill from this checkpoint's own evaluation run.
205
+ Do NOT reuse the preview base model's table — those numbers belong to a
206
+ different checkpoint and would misreport this one. -->
207
 
208
+ _To be filled in from this checkpoint's evaluation run, under the protocol below._
 
 
 
 
209
 
210
  ## Evaluation protocol
211
 
212
+ Video tasks run at `fps=1` with a fixed frame cap F and per-frame pixel budget MP (`min_pixels` = `max_pixels`). Frames are cached offline, so runs decode nothing and repeat on identical input. Image tasks use the default pixel budget. All generation is greedy, `max_new_tokens=4096`, `batch_size=1`.
 
 
213
 
214
  | Task | F | MP |
215
  |---|---|---|
 
224
  | VSTAT | 32 | 131072 |
225
  | MVBench, NextQA, TempCompass, MMVU, ReVSI | 64 | 153664 |
226
 
227
+ Most tasks are re-checked by a judge model (Qwen3.8-27B, temperature 0) that sees the question, options, reference answer and raw output and answers YES or NO, recovering answers that are correct but miss the extraction regex; the reported score is the greater of the rule-based and judged scores. Tasks scored by geometry (IoU, point-in-mask) skip the judge and use their rule-based score.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
228
 
229
  ## Architecture
230
 
231
  | Item | Value |
232
  |---|---|
233
+ | Language layers | 32 (linear attention, full attention every 4) |
234
  | hidden_size | 2560 |
235
  | Attention heads / KV heads | 16 / 4 |
236
  | Vocabulary | 248320 |
237
+ | Vision layers / hidden_size | 24 / 1024 |
 
238
  | patch_size / spatial_merge_size | 16 / 3 |
239
  | max_position_embeddings | 262144 |
240
  | Weights | bfloat16, 3 shards, about 9.9 GB |
README_CN.md CHANGED
@@ -13,25 +13,19 @@ tags:
13
 
14
  # LLaVA-OneVision-2.5-4B-Instruct
15
 
16
- 基于 LLaVA-OneVision-2 架构的 4B 指令微调多模态模型,支持单图、多图和视频输入。视觉编码器 24 层,`spatial_merge_size=3`;语言侧 32 层,`hidden_size=2560`,线性注意力与全注意力交替(每 4 层一个全注意力层),`max_position_embeddings=262144`。
17
 
18
- 除离线任务外,本次发布还针对**流式(streaming)**视频做了训练:画面逐帧到达,模型在每一步决定是保持沉默还是发言。两种行为共用同一份权重和同一个对话模版,详见[流式模式](#流式模式)。
19
-
20
- 建模代码随权重一起分发,因此**必须 `trust_remote_code=True`**。
21
-
22
- ---
23
 
24
  ## 安装
25
 
26
  ```bash
27
  pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
28
- pip install flash-attn --no-build-isolation # 可选,强烈建议
29
  pip install decord # 仅抽帧视频路径需要
30
  ```
31
 
32
- ---
33
-
34
- ## 单图推理
35
 
36
  ```python
37
  import torch
@@ -42,56 +36,29 @@ MODEL = "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct"
42
 
43
  processor = AutoProcessor.from_pretrained(MODEL, trust_remote_code=True)
44
  model = AutoModelForImageTextToText.from_pretrained(
45
- MODEL,
46
- trust_remote_code=True,
47
- dtype=torch.bfloat16,
48
  attn_implementation="flash_attention_2", # 没装 flash-attn 用 "sdpa"
49
  device_map="auto",
50
  ).eval()
51
 
52
- image = Image.open("example.jpg").convert("RGB")
53
  messages = [{"role": "user", "content": [
54
  {"type": "image"},
55
  {"type": "text", "text": "Describe this image in detail."},
56
  ]}]
57
  text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
58
- inputs = processor(text=text, images=[image], return_tensors="pt").to(model.device)
 
59
 
60
  with torch.inference_mode():
61
  out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
62
  print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
63
  ```
64
 
65
- 图片走 `smart_resize`,保持长宽比并对齐 patch 网格。像素预算默认 `min_pixels=3136` / `max_pixels=4000000`,可在 processor 上覆盖:
66
-
67
- ```python
68
- processor.image_processor.min_pixels = 256 * 28 * 28
69
- processor.image_processor.max_pixels = 1280 * 28 * 28
70
- ```
71
-
72
- ---
73
 
74
- ## 多图推理
75
 
76
- `content` 里每张图放一个 `{"type": "image"}`,`images` 按同样顺序传入:
77
-
78
- ```python
79
- messages = [{"role": "user", "content": [
80
- {"type": "image"},
81
- {"type": "image"},
82
- {"type": "text", "text": "What changed between these two images?"},
83
- ]}]
84
- text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
85
- inputs = processor(text=text, images=[img_a, img_b], return_tensors="pt").to(model.device)
86
- ```
87
-
88
- 每个占位符展开为 `<|vision_start|><|image_pad|><|vision_end|>`,按位置与 `images` 列表一一对应。
89
-
90
- ---
91
-
92
- ## 视频推理 — 抽帧
93
-
94
- 传**视频路径**,不要传自己解码好的帧:
95
 
96
  ```python
97
  messages = [{"role": "user", "content": [
@@ -101,24 +68,16 @@ messages = [{"role": "user", "content": [
101
  text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
102
 
103
  inputs = processor(
104
- text=text,
105
- videos=["example.mp4"],
106
  max_frames=64,
107
- min_pixels=153664,
108
- max_pixels=153664, # 视频场景把 min 和 max 设成相等,固定逐帧预算
109
  return_tensors="pt",
110
  ).to(model.device)
111
-
112
- with torch.inference_mode():
113
- out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
114
- print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
115
  ```
116
 
117
- `processor.__call__` 还接受 `target_fps`(按固定帧率采样,上限为 `max_frames`)和 `num_frames`(固定帧数)。优先级见 `video_processing_llava_onevision2_5.py` 的 `choose_target_frames`。
118
-
119
- ### 帧数与逐帧像素
120
 
121
- 开销大致是**帧数 × 逐帧像素**,两者要一起调。processor 把 `max_frames` 封顶在 768;只加帧数不降像素预算会很快打爆显存。
122
 
123
  | 场景 | 帧数 | 逐帧像素 |
124
  |---|---|---|
@@ -126,47 +85,33 @@ print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_token
126
  | 中等 | 256 | 136900 |
127
  | 长视频、整体理解 | 640 – 768 | 72900 – 108900 |
128
 
129
- 超长输入仍可能在视觉塔内部打爆显存,因为所有帧是一次前向编码的。设置 `OV_VISION_CHUNK_PATCHES` 可改为分组编码:
130
-
131
- ```bash
132
- export OV_VISION_CHUNK_PATCHES=200000 # 0(默认)= 关闭
133
- ```
134
-
135
- 分组沿逐帧的序列边界切分,注意力不跨帧,因此结果与一次性前向在数学上等价,只是用一点速度换显存。
136
 
137
- ### 时间戳自动注入
138
-
139
- 视频输入一定带时间戳,且无法关闭。`<|vision_start|><|video_pad|><|vision_end|>` 占位符会被改写成逐帧的块,每块前面是该帧的时间:
140
 
141
  ```
142
  <0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
143
  <1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
144
  ```
145
 
146
- 秒数来自解码器的真实采样位置,不是假定的均匀间隔,所以可以直接问时间定位类问题。
147
-
148
- > **传路径,不要传自己的帧。** `videos=` 两者都收,但并不等价。只有传路径,processor 才能拿到真实采样时刻和真实源帧号。只给一串帧时它两样都没有,会退化成拿帧序号当秒数——第 0、1、2 帧变成 0.0、1.0、2.0 秒——于是 100 秒的视频看起来只有 N 秒,时间类回答全错。这种情况下 processor 会抛 `UserWarning`。
149
- >
150
- > 如果必须自己解码,请把时间信息一并交出:
151
- > ```python
152
- > videos=[{"frames": frames, "timestamps": seconds_list, "frame_indices": index_list}]
153
- > ```
154
- > 通过 `images=` 传帧则完全不注入时间戳。
155
 
156
- ---
 
 
157
 
158
- ## 流式模式
159
 
160
- 流式场景下视频不是整段交付的。画面每轮进来一帧,模型在每一轮输出一个**状态 token**:要么沉默,要么发言。
161
 
162
- 同一份权重、同一个 `chat_template.jinja` 同时服务两种模式,用一个关键字切换:
163
 
164
  | 模式 | 调用 | 默认 system |
165
  |---|---|---|
166
  | 离线(默认) | `apply_chat_template(messages, ...)` | `You are a helpful assistant.` |
167
  | 流式 | `apply_chat_template(messages, ..., streaming=True)` | `You are a real-time streaming video assistant.` |
168
 
169
- 这个开关**只**改默认 system 这一处,其余全部由传入内容决定。显式传入的 `system` 消息在两种模式下都优先。默认是离线模式,这样现有的离线评测不传参数也能照常工作。
170
 
171
  ### 协议 token
172
 
@@ -174,36 +119,16 @@ export OV_VISION_CHUNK_PATCHES=200000 # 0(默认)= 关闭
174
  |---|---|---|
175
  | `<silence>` | 248077 | 当前轮不发言 |
176
  | `<response>` | 248078 | 当前轮发言 |
177
- | `<question_memory>` / `</question_memory>` | 248079 / 248080 | 新问题或任务指令,在提问时刻写入一次 |
178
- | `<answer_memory>` / `</answer_memory>` | 248081 / 248082 | 此前回答到期后回插到上下文 |
179
- | `<captioning>` | 248083 | 当前轮发言,且是事件级 caption 而非回答 |
180
  | `<caption_begin>` | 248084 | user 侧标记,说明这条流是 caption 任务 |
181
 
182
- 八个都是**非 special 的 added token**。因此 `decode(..., skip_special_tokens=True)` 仍会把它们解出来——这是刻意的,状态 token 正是你要读的输出。它们落在现有 248320 行 embedding 之内,不需要扩词表,且没有任何已有 token id 发生移动。
183
-
184
- `<question_memory>` 和 `<answer_memory>` 内部包一个 JSON 对象,例如 `{"timestep": "00:00:06", "content": "..."}`,其中 `timestep` 是原始提问/回答时刻,格式 `HH:MM:SS`。
185
-
186
- ### 轮次结构
187
 
188
- 严格 user / assistant 交替。每个 user 轮最多带一帧画面;没有新画面的轮次改用纯文本时间前缀。
189
-
190
- ```
191
- <|im_start|>system
192
- You are a real-time streaming video assistant.<|im_end|>
193
- <|im_start|>user
194
- <question_memory>{"timestep": "00:00:00", "content": "..."}</question_memory>
195
- <0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
196
- <|im_end|>
197
- <|im_start|>assistant
198
- <silence><|im_end|>
199
- <|im_start|>user
200
- <1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
201
- <|im_end|>
202
- <|im_start|>assistant
203
- <response><|im_end|>
204
- ```
205
 
206
- 构造方式:每帧一个 `{"type": "image"}` 并带上以秒为单位的 `timestamp`,帧后的换行作为单独的 text 项:
207
 
208
  ```python
209
  messages = [{"role": "user", "content": [
@@ -215,195 +140,76 @@ text = processor.apply_chat_template(
215
  messages, tokenize=False, add_generation_prompt=True, streaming=True)
216
  ```
217
 
218
- content 也可以直接传字符串,会原样输出——适合调用方自己已经拼好 `<T seconds><|vision_start|>…` 的情况。
219
-
220
- ### 调用方必须自己算的两个值
221
-
222
- 模版保证的是**格式**,有两个**数值**取决于视频本身,无法从一张裸帧里恢复,需要你提供:
223
-
224
- 1. **文本时间戳**:每帧传 `timestamp = 源帧号 / fps`,fps 用视频真实帧率。
225
- 2. **`patch_positions` 的 `t` 列**:如果把每帧当作独立 image 交给 processor,返回的 `patch_positions` 块布局是对的,但 `t` **每帧都是 0**。需要把每帧对应的行改成它真实的源帧号 `floor(秒数 × fps)`。不改的话,每帧都会被当成一段新视觉的开头,与训练时不一致。
226
-
227
- 模型只通过 `t` 在本段视觉内的**名次**(0, 1, 2, …)使用它,不用绝对值;视觉塔也只做帧内注意力。所以真正重要的是先后顺序对,而不是数值本身;让它等于真实帧号只是为了输入可审计。
228
-
229
- 上一节说的逐帧时间戳自动展开属于离线的 `<|video_pad|>` 分支。流式是逐帧输入,不走那条分支。
230
-
231
- > **本仓库不包含什么。** 这份权重给你的是训练出来的行为、模版和 token。围绕它的实时循环——按时钟取帧、决定何时停止生成、在到期时回插 question / answer memory、在决策点之间维持上下文——属于服务端的工作,不在这里分发。
232
-
233
- ---
234
-
235
- ## 视频推理 — codec 模式
236
-
237
- codec 模式不按整帧采样,而是用编码器的 bitcost 找出信息量大的区域,打包进少量**画布(canvas)**。
238
-
239
- 关键区别:**同一张画布上的 patch 可以来自多个不同的源帧。**每个 patch 带自己的源位置 `(源帧, 源 patch 行, 源 patch 列)`,视觉塔用它做 3-D RoPE。把画布当普通图片通过 `images=` 传入会丢掉这些信息。
240
-
241
- ### 安装预处理包
242
-
243
- codec 预处理来自 `codec-video-prep`,它自带打过补丁的 FFmpeg 和 dav1d 用于提取 bitcost,无需编译:
244
-
245
- ```bash
246
- pip install -i https://test.pypi.org/simple/ \
247
- --extra-index-url https://pypi.org/simple \
248
- "codec-video-prep==0.3.0"
249
- ```
250
-
251
- `--extra-index-url` 是必需的:包本身在 TestPyPI,依赖(numpy、opencv-python-headless、Pillow、PyYAML)在正式 PyPI。
252
-
253
- 检查安装:
254
-
255
- ```bash
256
- codec-video-prep-doctor
257
- ```
258
-
259
- 要求:
260
-
261
- - **Linux x86_64、CPython 3.10 或 3.11**(manylinux_2_17 轮子)。不支持 macOS、ARM、Alpine musl。
262
- - **NumPy < 2**(`numpy>=1.23,<2`)。如果推理环境是 NumPy 2,请在单独环境做预处理再交付产物。
263
- - 系统 `ffprobe`(`apt-get install ffmpeg` 或等价)。
264
- - bitcost 支持 **H.264、HEVC、VP9、AV1**。其他编码(MPEG-4、MPEG-2)没有实现,而且**文件后缀说明不了编码格式**:
265
-
266
- ```bash
267
- ffprobe -v error -select_streams v:0 -show_entries stream=codec_name \
268
- -of default=noprint_wrappers=1 input.mp4
269
- ```
270
-
271
- - 采样和预算需要先拿到整段视频,所以这是给**离线文件**用的,不适用于实时流。
272
-
273
- ### 调用
274
-
275
- 设 `video_backend="codec"` 并传视频路径:
276
-
277
- ```python
278
- inputs = processor(
279
- text=text,
280
- videos=["example.mp4"],
281
- video_backend="codec",
282
- return_tensors="pt",
283
- ).to(model.device)
284
-
285
- with torch.inference_mode():
286
- out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
287
- ```
288
-
289
- 除 `input_ids` 和 `attention_mask` 外,`inputs` 还带 `pixel_values`、`image_grid_thw`(每张画布一行)和 **`patch_positions`**(`[总 patch 数, 3]`,块布局)。`patch_positions` 会转发给 `model.forward(...)` 做 3-D RoPE,**不要丢掉**。
290
-
291
- ### 每张画布一个时间标记
292
-
293
- 每张画布只有一个标记,形式取决于它取自几个源帧:
294
-
295
- | 画布 | 标记 | 例子 |
296
- |---|---|---|
297
- | 所有 patch 来自同一源帧 | 点 | `<6.5 seconds>` |
298
- | patch 跨多帧 | 区间 | `<0.0 to 1.8 seconds>` |
299
 
300
  ```
 
 
301
  <0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
302
- <0.0 to 1.8 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
303
- <2.0 to 4.1 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
 
304
  ```
305
 
306
- 这就是训练用的序列化方式(schema `canvas_patch_span_point_range_v1`),完全由每张画布真实的源帧跨度决定。时间类问题与抽帧路径表现一致。
307
 
308
- ### 调整预算
309
 
310
- 默认配方——512 候选帧、64 画布、patch 16、3×3 块、每画布 313600 像素——写在 `preprocessor_config.json` 的 `codec` 字段里。单次调用可用 `codec_config=` 覆盖:
311
 
312
- ```python
313
- inputs = processor(
314
- text=text, videos=["example.mp4"], video_backend="codec",
315
- codec_config={"target_canvas": 32, "num_sampled_frames": 256},
316
- return_tensors="pt",
317
- )
318
- ```
319
 
320
- | 选项 | 默认 | 含义 |
321
- |---|---|---|
322
- | `strategy` | `adaptive_pwindow` | 还有 `adaptive_rdo`、`readiness` |
323
- | `target_canvas` | 64 | 画布预算;`readiness` 不约束数量,以实际输出为准 |
324
- | `num_sampled_frames` | 512 | 候选帧数 |
325
- | `patch` / `block_size` | 16 / 3 | 必须与视觉塔的 patch size 和 spatial merge 一致 |
326
- | `max_pixels` | 313600 | 每画布像素 |
327
- | `canvas_format` | `png` | `png`、`jpg` 或 `npy` |
328
-
329
- > `patch=16`、`block_size=3` 对应本模型的 `patch_size=16` 和 `spatial_merge_size=3`。改动会让画布网格与 merger 错位,预处理会直接报错而不是静默出错。
330
 
331
- 画布缓存在 `~/.cache/codec_video_prep/canvases`(用 `CVP_CODEC_CACHE_ROOT` 换位置),所以对同一视频问多个问题只解码一次。
332
 
333
- ---
334
 
335
- ## 批量推理
336
 
337
- ```python
338
- texts = [processor.apply_chat_template(m, tokenize=False, add_generation_prompt=True)
339
- for m in batch_messages]
340
- inputs = processor(text=texts, images=batch_images, padding=True,
341
- return_tensors="pt").to(model.device)
342
- ```
343
-
344
- padding 在左侧;按各自的 `input_ids` 长度切输出。
345
-
346
- ---
347
 
348
- ## 离线使用
349
 
350
  ```bash
351
- export HF_HUB_OFFLINE=1
352
- export TRANSFORMERS_OFFLINE=1
353
- ```
354
-
355
- ```python
356
- processor = AutoProcessor.from_pretrained("/local/path", trust_remote_code=True)
357
- model = AutoModelForImageTextToText.from_pretrained("/local/path", trust_remote_code=True, ...)
358
  ```
359
 
360
- **多进程启动前先单进程预热一次。** `trust_remote_code` 模块会被编译到 `$HF_HOME/modules/transformers_modules/<名字>/`。多个进程首次同时加载会在这个目录上竞争,部分进程报 `AttributeError: module '...modeling_llava_onevision2_5' has no attribute 'LlavaOnevision2ForConditionalGeneration'`。先用单进程完整加载一次模型——光 `AutoConfig` 不会生成 `modeling_*.py` 的缓存——再启动其余进程。
361
-
362
- 缓存目录只按**叶子目录名**索引。两个都叫 `model_hf` 的本地 checkpoint 会共用同一条缓存、互相覆盖;本地副本请取不同的目录名。
363
-
364
- ---
365
-
366
- ## 生成参数
367
-
368
- `generation_config.json` 只设了 `eos_token_id=248044` 和 `use_cache=true`,没有采样默认值。要确定性输出就用贪心解码:
369
-
370
  ```python
371
- model.generate(**inputs, max_new_tokens=512, do_sample=False)
 
372
  ```
373
 
374
- 需要先推理再作答的任务要给足 `max_new_tokens`;128 这类小值会在推理中途被截断。
375
 
376
- 流式模式下通常只需要第一个状态 token,那里适合用较小的 `max_new_tokens` 加提前停止。
377
 
378
- ---
379
 
380
- ## 已知行为
381
 
382
- **回答格式会影响正则判分。** 有些选择题上模型会写"The correct answer is ..."或先推理再作答,而不是直接给一个字母。按首字母抽取的脚本会把这些判错。要强制给字母就在 prompt 里要求:`Answer with the option's letter from the given choices directly.`
383
 
384
- **坐标约定。** grounding 任务请在 prompt 里写明范围,例如 `Answer with the bounding box as [x1, y1, x2, y2], each coordinate an integer between 0 and 1000 (1000 = full image width/height).`,否则不同任务的约定不一致。
385
 
386
- **状态 token 不会被 `skip_special_tokens` 过滤。** 八个流式 token 按设计是非 special 的。如果在离线输出里不想要它们,需要显式剥除。
387
 
388
- ---
389
 
390
  ## 评测结果
391
 
392
- <!-- 上传前待补:用本 instruct checkpoint 自己的评测跑结果填写。
393
  不要沿用 preview base 模型的那张表——那些数字属于另一个 checkpoint,
394
  放在这里会误报本模型的能力。 -->
395
 
396
- _待用本 checkpoint 的评测结果补充。_
397
-
398
- 下面的口径就是这些数字必须依据的口径。
399
-
400
- ---
401
 
402
  ## 评测口径
403
 
404
- ### 帧设置
405
-
406
- 视频任务用 `fps=1`,固定帧数上限 F 和逐帧像素预算 MP(`min_pixels` 与 `max_pixels` 设为相等)。帧是离线缓存好的,所以评测时不解码,重复跑看到的输入完全一致。
407
 
408
  | 任务 | F | MP |
409
  |---|---|---|
@@ -418,36 +224,17 @@ _待用本 checkpoint 的评测结果补充。_
418
  | VSTAT | 32 | 131072 |
419
  | MVBench、NextQA、TempCompass、MMVU、ReVSI | 64 | 153664 |
420
 
421
- 图像任务用 preprocessor 的���认像素预算。所有生成都是贪心,`max_new_tokens=4096`,`batch_size=1`。
422
-
423
- ### 判分
424
-
425
- 多数任务会再经过一个 judge 模型复核。纯几何判分(IoU、点落在掩码内)或本来就不从文本里抽字母的任务跳过 judge,因为判也没有意义;这些行等于它们的规则分。
426
-
427
- | Judge | 值 |
428
- |---|---|
429
- | 模型 | Qwen3.8-27B |
430
- | 服务 | vLLM 0.19,`--tensor-parallel-size 4`、`--gpu-memory-utilization 0.72`、`--max-model-len 16384`、`--enforce-eager` |
431
- | 温度 | 0 |
432
- | 最大 token | 6 |
433
- | Thinking | 关(`enable_thinking=False`) |
434
-
435
- judge 拿到问题、选项、参考答案和模型原始输出,回答 YES 或 NO。这样能纠正"答对了但不匹配抽取正则"的情况。最终分取规则分与 judge 分中的较大者。
436
-
437
- 关于可复现性:温度 0 是贪心,原则上确定,但没有固定随机种子,且 vLLM 的连续批处理会随 batch 组成改变浮点累加顺序,所以边界样本仍可能翻转。严格复现需要同时锁定 judge 权重、vLLM 版本和 tensor-parallel 大小。
438
-
439
- ---
440
 
441
  ## 结构参数
442
 
443
  | 项 | 值 |
444
  |---|---|
445
- | 语言层数 | 32(线性注意力与全注意力交替,每 4 层一个) |
446
  | hidden_size | 2560 |
447
  | 注意力头 / KV 头 | 16 / 4 |
448
  | 词表 | 248320 |
449
- | 视觉层数 | 24 |
450
- | 视觉 hidden_size | 1024 |
451
  | patch_size / spatial_merge_size | 16 / 3 |
452
  | max_position_embeddings | 262144 |
453
  | 权重 | bfloat16,3 分片,约 9.9 GB |
 
13
 
14
  # LLaVA-OneVision-2.5-4B-Instruct
15
 
16
+ 基于 LLaVA-OneVision-2 架构的 4B 指令微调多模态模型,支持单图、多图和视频输入。也支持流式视频:画面逐帧到达,模型每一步决定是否发言——同一份权重、同一个对话模版,用一个关键字切换,见[流式](#流式)。
17
 
18
+ 视觉编码器 24 层,`patch_size=16`,`spatial_merge_size=3`。语言侧 32 层,`hidden_size=2560`,线性注意力每 4 层夹一个全注意力层,`max_position_embeddings=262144`。建模代码随权重分发,必须 `trust_remote_code=True`。
 
 
 
 
19
 
20
  ## 安装
21
 
22
  ```bash
23
  pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
24
+ pip install flash-attn --no-build-isolation # 可选,建议装
25
  pip install decord # 仅抽帧视频路径需要
26
  ```
27
 
28
+ ## 图片
 
 
29
 
30
  ```python
31
  import torch
 
36
 
37
  processor = AutoProcessor.from_pretrained(MODEL, trust_remote_code=True)
38
  model = AutoModelForImageTextToText.from_pretrained(
39
+ MODEL, trust_remote_code=True, dtype=torch.bfloat16,
 
 
40
  attn_implementation="flash_attention_2", # 没装 flash-attn 用 "sdpa"
41
  device_map="auto",
42
  ).eval()
43
 
 
44
  messages = [{"role": "user", "content": [
45
  {"type": "image"},
46
  {"type": "text", "text": "Describe this image in detail."},
47
  ]}]
48
  text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
49
+ inputs = processor(text=text, images=[Image.open("example.jpg").convert("RGB")],
50
+ return_tensors="pt").to(model.device)
51
 
52
  with torch.inference_mode():
53
  out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
54
  print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
55
  ```
56
 
57
+ 图片走 `smart_resize`,保持长宽比并对齐 patch 网格。像素预算默认 `min_pixels=3136` / `max_pixels=4000000`,可在 `processor.image_processor` 上覆盖。
 
 
 
 
 
 
 
58
 
59
+ 多图时每张图放一个 `{"type": "image"}`,`images=` 按同样顺序传入。每个占位符展开成 `<|vision_start|><|image_pad|><|vision_end|>`,按位置对应。
60
 
61
+ ## 视频 — 抽帧
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
62
 
63
  ```python
64
  messages = [{"role": "user", "content": [
 
68
  text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
69
 
70
  inputs = processor(
71
+ text=text, videos=["example.mp4"],
 
72
  max_frames=64,
73
+ min_pixels=153664, max_pixels=153664, # 设成相等以固定逐帧预算
 
74
  return_tensors="pt",
75
  ).to(model.device)
 
 
 
 
76
  ```
77
 
78
+ 也接受 `target_fps` 和 `num_frames`,优先级见 `video_processing_llava_onevision2_5.py` 的 `choose_target_frames`。`max_frames` 封顶 768。
 
 
79
 
80
+ 开销约等于帧数 × 逐帧像素,两者要一起调:
81
 
82
  | 场景 | 帧数 | 逐帧像素 |
83
  |---|---|---|
 
85
  | 中等 | 256 | 136900 |
86
  | 长视频、整体理解 | 640 – 768 | 72900 – 108900 |
87
 
88
+ 所有帧是在视觉塔里一次前向编码的,超长输入可能打爆显存。设 `OV_VISION_CHUNK_PATCHES=200000` 可改为分组编码,按帧边界切分,注意力不跨帧,结果不变。
 
 
 
 
 
 
89
 
90
+ **时间戳**一定会注入。`<|video_pad|>` 占位符展开成逐帧的块,每块前面是该帧的时间,取自解码器的真实采样位置:
 
 
91
 
92
  ```
93
  <0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
94
  <1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
95
  ```
96
 
97
+ 要传路径,不要传自己解码好的帧。只给一串帧时 processor 拿不到真实采样时刻,会退化成用帧序号当秒数,于是 100 秒的视频看起来只有 N 秒,时间类回答全错(此时会抛 `UserWarning`)。必须自己解码的话,把时间信息一并交出:
 
 
 
 
 
 
 
 
98
 
99
+ ```python
100
+ videos=[{"frames": frames, "timestamps": seconds_list, "frame_indices": index_list}]
101
+ ```
102
 
103
+ 通过 `images=` 传帧则完全不注入时间戳。
104
 
105
+ ## 流式
106
 
107
+ 画面每轮进来一帧,模型每轮输出一个状态 token:沉默,或发言。一个关键字切换默认 system,其余什么都不改:
108
 
109
  | 模式 | 调用 | 默认 system |
110
  |---|---|---|
111
  | 离线(默认) | `apply_chat_template(messages, ...)` | `You are a helpful assistant.` |
112
  | 流式 | `apply_chat_template(messages, ..., streaming=True)` | `You are a real-time streaming video assistant.` |
113
 
114
+ 显式传入的 `system` 在两种模式下都优先。默认走离线,是为了让现有离线评测不改代码也能用。
115
 
116
  ### 协议 token
117
 
 
119
  |---|---|---|
120
  | `<silence>` | 248077 | 当前轮不发言 |
121
  | `<response>` | 248078 | 当前轮发言 |
122
+ | `<question_memory>` / `</question_memory>` | 248079 / 248080 | 新问题,在提问时刻写入一次 |
123
+ | `<answer_memory>` / `</answer_memory>` | 248081 / 248082 | 此前回答到期后回插一次 |
124
+ | `<captioning>` | 248083 | 当前轮发言,且是事件级 caption |
125
  | `<caption_begin>` | 248084 | user 侧标记,说明这条流是 caption 任务 |
126
 
127
+ 八个都是非 special 的 added token,所以 `skip_special_tokens=True` 仍会解出来——状态 token 正是你要读的输出。它们落在现有 248320 行 embedding 内,且已发布的 token id 一个都没动。两个 memory token 内部包 JSON,例如 `{"timestep": "00:00:06", "content": "..."}`。
 
 
 
 
128
 
129
+ ### 怎么构造一轮
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
130
 
131
+ 严格 user / assistant 交替,每个 user 轮最多一帧;没有新画面的轮次改用纯文本时间前缀。每帧给上 `timestamp`,帧后的换行作为单独的 text 项:
132
 
133
  ```python
134
  messages = [{"role": "user", "content": [
 
140
  messages, tokenize=False, add_generation_prompt=True, streaming=True)
141
  ```
142
 
143
+ 渲染结果:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
144
 
145
  ```
146
+ <|im_start|>user
147
+ <question_memory>{"timestep": "00:00:00", "content": "..."}</question_memory>
148
  <0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
149
+ <|im_end|>
150
+ <|im_start|>assistant
151
+ <silence><|im_end|>
152
  ```
153
 
154
+ content 传字符串则原样输出,适合自己已经拼好标记的 harness。
155
 
156
+ ### 两个要你自己算的值
157
 
158
+ 模版固定的是格式,有两个数值来自视频本身,从一张裸帧里恢复不出来:
159
 
160
+ 1. **`timestamp`**——源帧号除以视频真实 fps。
161
+ 2. **`patch_positions` 的 `t` 列**——把每帧当独立 image 交给 processor 时,返回的块布局是对的,但 `t` 每帧都是 0。需要改成 `floor(秒数 × fps)`,否则每帧都会被当成一段新视觉的开头。
 
 
 
 
 
162
 
163
+ 模型只用 `t` 在本段内的**名次**,视觉塔也只做帧内注意力;写真实帧号只是为了输入可审计。
 
 
 
 
 
 
 
 
 
164
 
165
+ 上面说的逐帧时间戳自动展开属于离线的 `<|video_pad|>` 分支,流式逐帧输入用不到。
166
 
167
+ 本仓库不包含围绕模型的实时循环:按时钟取帧、决定何时停止生成、到期回插 question / answer memory、在决策点之间维持上下文。
168
 
169
+ ## 视频 — codec 模式
170
 
171
+ codec 模式不按整帧采样,而是用编码器 bitcost 找出信息量大的区域,打包进少量**画布**。同一张画布上的 patch 可以来自不同源帧,各自带 `(源帧, 行, 列)` 供 3-D RoPE 使用——所以画布不能通过 `images=` 传。
 
 
 
 
 
 
 
 
 
172
 
173
+ 预处理来自 `codec-video-prep`(自带打过补丁的 FFmpeg 和 dav1d;限 Linux x86_64、CPython 3.10/3.11、`numpy<2`、系统 `ffprobe`;只支持 H.264/HEVC/VP9/AV1;只适用于离线文件,因为预算计算需要整段视频):
174
 
175
  ```bash
176
+ pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple \
177
+ "codec-video-prep==0.3.0"
178
+ codec-video-prep-doctor
 
 
 
 
179
  ```
180
 
 
 
 
 
 
 
 
 
 
 
181
  ```python
182
+ inputs = processor(text=text, videos=["example.mp4"],
183
+ video_backend="codec", return_tensors="pt").to(model.device)
184
  ```
185
 
186
+ 输出除 `pixel_values` 和 `image_grid_thw` 外还带 `patch_positions`(`[总 patch 数, 3]`),会转发给 `model.forward(...)` 做 3-D RoPE,不要丢掉。
187
 
188
+ 每张画布只有一个时间标记:所有 patch 来自同一源帧时是点 `<6.5 seconds>`,跨多帧时是区间 `<0.0 to 1.8 seconds>`(schema `canvas_patch_span_point_range_v1`)。时间类问题与抽帧路径表现一致。
189
 
190
+ 默认配方写在 `preprocessor_config.json` 的 `codec` 字段——512 候选帧、64 画布、patch 16、3×3 块、每画布 313600 像素——单次调用用 `codec_config={"target_canvas": 32, ...}` 覆盖。其余键:`strategy`(`adaptive_pwindow`、`adaptive_rdo`、`readiness`)、`num_sampled_frames`、`max_pixels`、`canvas_format`。`patch=16` 和 `block_size=3` 必须与模型的 `patch_size`、`spatial_merge_size` 一致,不一致时预处理直接报错而非静默出错。画布缓存在 `~/.cache/codec_video_prep/canvases`(用 `CVP_CODEC_CACHE_ROOT` 换位置)。
191
 
192
+ ## 杂项
193
 
194
+ **批量。** 传一组 text 和对应的图片,`padding=True`。padding 在左侧,按各自的 `input_ids` 长度切输出。
195
 
196
+ **离线。** 设 `HF_HUB_OFFLINE=1` 和 `TRANSFORMERS_OFFLINE=1`。多进程启动前先单进程完整加载一次:remote-code 模块会编译到 `$HF_HOME/modules/transformers_modules/<名字>/`,多进程首次加载会在那里竞争。该缓存只按叶子目录名索引,所以本地副本要取不同的目录名。
197
 
198
+ **生成。** `generation_config.json` 只设了 `eos_token_id=248044` 和 `use_cache=true`。确定性输出用 `do_sample=False`;需要先推理再作答的任务要给足 `max_new_tokens`,128 会在推理中途截断。
199
 
200
+ **回答格式。** 选择题上模型可能写整句而不是一个字母。如果后续要用正则解析,就在 prompt 里要求给字母:`Answer with the option's letter from the given choices directly.`。grounding 任务请在 prompt 里写明坐标范围,例如 0–1000 的整数。八个流式 token 不会被 `skip_special_tokens` 过滤,离线场景不想要就显式剥除。
201
 
202
  ## 评测结果
203
 
204
+ <!-- 上传前待补:用本 checkpoint 自己的评测跑结果填写。
205
  不要沿用 preview base 模型的那张表——那些数字属于另一个 checkpoint,
206
  放在这里会误报本模型的能力。 -->
207
 
208
+ _待用本 checkpoint 的评测结果补充,口径见下。_
 
 
 
 
209
 
210
  ## 评测口径
211
 
212
+ 视频任务用 `fps=1`,固定帧数上限 F 和逐帧像素预算 MP(`min_pixels` = `max_pixels`)。帧离线缓存好,评测时不解码,重复跑输入完全一致。图像任务用默认像素预算。全部贪心生成,`max_new_tokens=4096`,`batch_size=1`。
 
 
213
 
214
  | 任务 | F | MP |
215
  |---|---|---|
 
224
  | VSTAT | 32 | 131072 |
225
  | MVBench、NextQA、TempCompass、MMVU、ReVSI | 64 | 153664 |
226
 
227
+ 多数任务会再经一个 judge 模型(Qwen3.8-27B,温度 0)复核:它看到问题、选项、参考答案和原始输出,回答 YES 或 NO,用于捞回"答对了但不匹配抽取正则"的情况;最终分取规则分与 judge 分中的较大者。纯几何判分的任务(IoU、点落在掩码内)跳过 judge,直接用规则分。
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
228
 
229
  ## 结构参数
230
 
231
  | 项 | 值 |
232
  |---|---|
233
+ | 语言层数 | 32(线性注意力,每 4 层一个全注意力) |
234
  | hidden_size | 2560 |
235
  | 注意力头 / KV 头 | 16 / 4 |
236
  | 词表 | 248320 |
237
+ | 视觉层数 / hidden_size | 24 / 1024 |
 
238
  | patch_size / spatial_merge_size | 16 / 3 |
239
  | max_position_embeddings | 262144 |
240
  | 权重 | bfloat16,3 分片,约 9.9 GB |