yiyexy commited on
Commit
d13683f
·
verified ·
1 Parent(s): 7df09b2

Clean stale comments and docstrings (no code change)

Browse files
README.md CHANGED
@@ -124,7 +124,7 @@ LLaVA-OneVision-2.5-4B-Instruct is a 4B vision-language model for single-image,
124
 
125
  <table style="width:100%; display:table;">
126
  <tbody>
127
- <tr><td align="left" style="width:28%"><b>Type</b></td><td align="left">instruction-tuned vision-language model, LLaVA-OneVision-2 architecture</td></tr>
128
  <tr><td align="left" style="width:28%"><b>Parameters</b></td><td align="left">5.25B total (4B language model plus vision tower and adapter)</td></tr>
129
  <tr><td align="left" style="width:28%"><b>Language side</b></td><td align="left">32 layers, hidden size 2560, FFN 9216, 16 attention heads / 4 KV heads, linear attention with one full-attention layer every 4</td></tr>
130
  <tr><td align="left" style="width:28%"><b>Vision side</b></td><td align="left">24 layers, hidden size 1024, <code>patch_size=16</code>, <code>spatial_merge_size=3</code>, attention within a frame</td></tr>
 
124
 
125
  <table style="width:100%; display:table;">
126
  <tbody>
127
+ <tr><td align="left" style="width:28%"><b>Type</b></td><td align="left">instruction-tuned vision-language model, LLaVA-OneVision-2.5 architecture</td></tr>
128
  <tr><td align="left" style="width:28%"><b>Parameters</b></td><td align="left">5.25B total (4B language model plus vision tower and adapter)</td></tr>
129
  <tr><td align="left" style="width:28%"><b>Language side</b></td><td align="left">32 layers, hidden size 2560, FFN 9216, 16 attention heads / 4 KV heads, linear attention with one full-attention layer every 4</td></tr>
130
  <tr><td align="left" style="width:28%"><b>Vision side</b></td><td align="left">24 layers, hidden size 1024, <code>patch_size=16</code>, <code>spatial_merge_size=3</code>, attention within a frame</td></tr>
README_CN.md CHANGED
@@ -124,7 +124,7 @@ LLaVA-OneVision-2.5-4B-Instruct 是一个 4B 视觉语言模型,支持单图
124
 
125
  <table style="width:100%; display:table;">
126
  <tbody>
127
- <tr><td align="left" style="width:28%"><b>类型</b></td><td align="left">指令微调视觉语言模型,LLaVA-OneVision-2 架构</td></tr>
128
  <tr><td align="left" style="width:28%"><b>参数量</b></td><td align="left">合计 5.25B(4B 语言模型 + 视觉塔与 adapter)</td></tr>
129
  <tr><td align="left" style="width:28%"><b>语言侧</b></td><td align="left">32 层,hidden size 2560,FFN 9216,16 个注意力头 / 4 个 KV 头,线性注意力每 4 层夹一个全注意力层</td></tr>
130
  <tr><td align="left" style="width:28%"><b>视觉侧</b></td><td align="left">24 层,hidden size 1024,<code>patch_size=16</code>,<code>spatial_merge_size=3</code>,帧内注意力</td></tr>
 
124
 
125
  <table style="width:100%; display:table;">
126
  <tbody>
127
+ <tr><td align="left" style="width:28%"><b>类型</b></td><td align="left">指令微调视觉语言模型,LLaVA-OneVision-2.5 架构</td></tr>
128
  <tr><td align="left" style="width:28%"><b>参数量</b></td><td align="left">合计 5.25B(4B 语言模型 + 视觉塔与 adapter)</td></tr>
129
  <tr><td align="left" style="width:28%"><b>语言侧</b></td><td align="left">32 层,hidden size 2560,FFN 9216,16 个注意力头 / 4 个 KV 头,线性注意力每 4 层夹一个全注意力层</td></tr>
130
  <tr><td align="left" style="width:28%"><b>视觉侧</b></td><td align="left">24 层,hidden size 1024,<code>patch_size=16</code>,<code>spatial_merge_size=3</code>,帧内注意力</td></tr>
chat_template.jinja CHANGED
@@ -1,11 +1,10 @@
1
- {#- OV2.5 Qwen3.5 chat template with two modes, aligned with the training renderer (aiak `qwen2-vl`
2
  template + canvas timestamps). Pick the mode per call:
3
  processor.apply_chat_template(messages, ...) -> offline (default)
4
  processor.apply_chat_template(messages, ..., streaming=True) -> streaming
5
  The modes differ only in the default system turn, exactly as in training: offline samples get
6
  the qwen2-vl default system, data_type=streaming samples get the streaming prompt. An explicit
7
- system message always wins. (Both placeholders are filled from the training code at export
8
- time by finalize_streaming_export.py, so they cannot drift.)
9
  Layout: <|im_start|>system\n{system}<|im_end|>\n then per turn <|im_start|>{role}\n{content}<|im_end|>\n.
10
  - String content is emitted verbatim (callers that pre-render "<T seconds><|vision_start|>...").
11
  - An image item may carry "timestamp" (seconds = frame_index / fps); it is rendered as the trainer
 
1
+ {#- OV2.5 Qwen3.5 chat template with two modes, aligned with the training renderer (`qwen2-vl`
2
  template + canvas timestamps). Pick the mode per call:
3
  processor.apply_chat_template(messages, ...) -> offline (default)
4
  processor.apply_chat_template(messages, ..., streaming=True) -> streaming
5
  The modes differ only in the default system turn, exactly as in training: offline samples get
6
  the qwen2-vl default system, data_type=streaming samples get the streaming prompt. An explicit
7
+ system message always wins.
 
8
  Layout: <|im_start|>system\n{system}<|im_end|>\n then per turn <|im_start|>{role}\n{content}<|im_end|>\n.
9
  - String content is emitted verbatim (callers that pre-render "<T seconds><|vision_start|>...").
10
  - An image item may carry "timestamp" (seconds = frame_index / fps); it is rendered as the trainer
configuration_llava_onevision2_5.py CHANGED
@@ -34,9 +34,7 @@ class LlavaOnevision2VisionConfig(PreTrainedConfig):
34
  class LlavaOnevision2Config(PreTrainedConfig):
35
  r"""
36
  This is the configuration class to store the configuration of a [`LlavaOnevision2Model`]. It is used to instantiate a
37
- LlavaOnevision2Model model according to the specified arguments, defining the model architecture. Instantiating a configuration
38
- with the defaults will yield a similar configuration to that of
39
- Llava-Onevision 1.5 [lmms-lab/LLaVA-OneVision-1.5-8B-Instruct](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-8B-Instruct).
40
 
41
  Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the
42
  documentation from [`PreTrainedConfig`] for more information.
 
34
  class LlavaOnevision2Config(PreTrainedConfig):
35
  r"""
36
  This is the configuration class to store the configuration of a [`LlavaOnevision2Model`]. It is used to instantiate a
37
+ LlavaOnevision2Model model according to the specified arguments, defining the model architecture.
 
 
38
 
39
  Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the
40
  documentation from [`PreTrainedConfig`] for more information.
modeling_llava_onevision2_5.py CHANGED
@@ -61,8 +61,7 @@ def _mrope_positions_from_patch_positions(
61
  ):
62
  """Train-parity 3D M-RoPE positions.
63
 
64
- Verbatim port of the TRAINING-side ``_rope_from_patch_positions`` (aiak
65
- llava_onevision2_model.py) so eval positions match what the model saw in
66
  training exactly — including its convention that the text position cursor
67
  does NOT skip past a visual's span (text after an image resumes at the
68
  pre-image counter). Do NOT "fix" that here; train/infer parity wins.
@@ -773,7 +772,7 @@ class Siglip2MultiheadAttentionPoolingHead(nn.Module):
773
 
774
  class LlavaOnevision2VisionPretrainedModel(LlavaOnevision2PreTrainedModel):
775
  """
776
- LLaVA-OneVision 2.0 Vision Model.
777
 
778
  This vision model is designed to work with Qwen2VL-style image processing:
779
  - Receives pre-processed patches in 3x3 block spatial order
@@ -1085,18 +1084,18 @@ class LlavaOnevision2VisionPretrainedModel(LlavaOnevision2PreTrainedModel):
1085
  grid_thw: Optional[torch.Tensor] = None,
1086
  patch_positions: Optional[torch.Tensor] = None,
1087
  ) -> dict:
1088
- """Stage-by-stage forward used by HF<->Megatron consistency tests.
1089
 
1090
  Mirrors the production ``forward`` exactly EXCEPT that ``patch_positions``
1091
  passed in are assumed to be in row-major (t, h, w) order — matching the
1092
- Megatron-side ``onevision_encoder_model.forward_debug`` test contract,
1093
  which also receives row-major positions and applies a block-layout
1094
  reorder. Production ``forward`` skips that reorder because production
1095
  ``patch_positions`` come pre-arranged in block layout from the
1096
  Qwen2VL processor (`build_patch_positions`).
1097
 
1098
- Layer-wise outputs match the schema produced by Megatron-side
1099
- ``onevision_encoder_model.forward_debug``.
1100
  """
1101
  output: dict[str, torch.Tensor] = {}
1102
 
@@ -1581,7 +1580,7 @@ class LlavaOnevision2ForConditionalGeneration(LlavaOnevision2PreTrainedModel, Ge
1581
  `position_ids`, this tensor is not affected by padding.
1582
 
1583
  Note (P0 native-video alias):
1584
- The companion ``Llava_Onevision2Processor.__call__(videos=...)`` does NOT
1585
  pass ``pixel_values_videos`` / ``video_grid_thw`` / ``second_per_grid_ts``
1586
  to this forward. Instead it aliases the video patch tensor as
1587
  ``pixel_values=`` and ``image_grid_thw=``, so video inputs share the
@@ -1590,8 +1589,7 @@ class LlavaOnevision2ForConditionalGeneration(LlavaOnevision2PreTrainedModel, Ge
1590
  text tags emitted by the processor). The ``*_videos`` and
1591
  ``second_per_grid_ts`` kwargs are kept declared here only for API
1592
  completeness and future use (e.g. 3D mRoPE / ``get_rope_index``); they
1593
- are NOT consumed by the current OneVision encoder. See
1594
- NATIVE_VIDEO_PLAN.md (stage 3.1) for the full rationale.
1595
 
1596
  Example:
1597
 
@@ -1600,8 +1598,8 @@ class LlavaOnevision2ForConditionalGeneration(LlavaOnevision2PreTrainedModel, Ge
1600
  >>> import requests
1601
  >>> from transformers import AutoProcessor, LlavaOnevision2ForConditionalGeneration
1602
 
1603
- >>> model = LlavaOnevision2ForConditionalGeneration.from_pretrained("Deep-VLM/LLaVA-OneVision-1.5-8B-Instruct-hf", trust_remote_code=True)
1604
- >>> processor = AutoProcessor.from_pretrained("Deep-VLM/LLaVA-OneVision-1.5-8B-Instruct-hf", trust_remote_code=True)
1605
 
1606
  >>> messages = [
1607
  {
 
61
  ):
62
  """Train-parity 3D M-RoPE positions.
63
 
64
+ Verbatim port of the TRAINING-side ``_rope_from_patch_positions`` (training code) so eval positions match what the model saw in
 
65
  training exactly — including its convention that the text position cursor
66
  does NOT skip past a visual's span (text after an image resumes at the
67
  pre-image counter). Do NOT "fix" that here; train/infer parity wins.
 
772
 
773
  class LlavaOnevision2VisionPretrainedModel(LlavaOnevision2PreTrainedModel):
774
  """
775
+ LLaVA-OneVision-2.5 Vision Model.
776
 
777
  This vision model is designed to work with Qwen2VL-style image processing:
778
  - Receives pre-processed patches in 3x3 block spatial order
 
1084
  grid_thw: Optional[torch.Tensor] = None,
1085
  patch_positions: Optional[torch.Tensor] = None,
1086
  ) -> dict:
1087
+ """Stage-by-stage forward used by consistency tests against the training implementation.
1088
 
1089
  Mirrors the production ``forward`` exactly EXCEPT that ``patch_positions``
1090
  passed in are assumed to be in row-major (t, h, w) order — matching the
1091
+ training-side ``forward_debug`` test contract,
1092
  which also receives row-major positions and applies a block-layout
1093
  reorder. Production ``forward`` skips that reorder because production
1094
  ``patch_positions`` come pre-arranged in block layout from the
1095
  Qwen2VL processor (`build_patch_positions`).
1096
 
1097
+ Layer-wise outputs match the schema produced by the training-side
1098
+ ``forward_debug``.
1099
  """
1100
  output: dict[str, torch.Tensor] = {}
1101
 
 
1580
  `position_ids`, this tensor is not affected by padding.
1581
 
1582
  Note (P0 native-video alias):
1583
+ The companion ``LlavaOnevision2Processor.__call__(videos=...)`` does NOT
1584
  pass ``pixel_values_videos`` / ``video_grid_thw`` / ``second_per_grid_ts``
1585
  to this forward. Instead it aliases the video patch tensor as
1586
  ``pixel_values=`` and ``image_grid_thw=``, so video inputs share the
 
1589
  text tags emitted by the processor). The ``*_videos`` and
1590
  ``second_per_grid_ts`` kwargs are kept declared here only for API
1591
  completeness and future use (e.g. 3D mRoPE / ``get_rope_index``); they
1592
+ are NOT consumed by the current OneVision encoder.
 
1593
 
1594
  Example:
1595
 
 
1598
  >>> import requests
1599
  >>> from transformers import AutoProcessor, LlavaOnevision2ForConditionalGeneration
1600
 
1601
+ >>> model = LlavaOnevision2ForConditionalGeneration.from_pretrained("lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True)
1602
+ >>> processor = AutoProcessor.from_pretrained("lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True)
1603
 
1604
  >>> messages = [
1605
  {