webbrain-one commited on
Commit
e228eff
·
verified ·
1 Parent(s): 9527855

Upload webbrain-one/webbrain-compass-vld-v6-2b (Compass VLD v6)

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: lfm1.0
4
+ base_model: LiquidAI/LFM3-VL-2B
5
+ pipeline_tag: image-text-to-text
6
+ library_name: transformers
7
+ tags:
8
+ - webbrain
9
+ - compass-v6
10
+ - liquid
11
+ - lfm3
12
+ - lfm3-vl
13
+ - browser-agent
14
+ - decision-model
15
+ - webbrain-vld
16
+ - custom_code
17
+ ---
18
+
19
+ # WebBrain Compass VLD v6 (2B)
20
+
21
+ **WebBrain Compass VLD v6 2B** is WebBrain's default multimodal WebGPU in-browser browser agent model, succeeding Compass VLD Tiny v5.
22
+ It fine-tunes `webbrain-one/lfm3-vld-2b-multipass` (derived from `LiquidAI/LFM3-VL-2B`) on a license-clean 70k multimodal browser agent training mix, operating entirely without tool calls in the browser across three core roles:
23
+
24
+ 1. **Tiny Ask & Standalone Chat**: Single-turn and multi-turn Q&A from page text/screenshot and offline Wikipedia / Emergency Box passages with byte-exact citation tokens (`[WB-W-XXXXXX]`). Politely refuses Act/Dev requests in user's language.
25
+ 2. **Vision Description**: WebBrain standard six-section structured descriptions of browser viewports (Page purpose, Visible text, Inputs, State signals, Blockers, Unknowns).
26
+ 3. **Decisions via Verify Protocol**: Fast verify scoring ('Proposed answer: ... Is the proposed answer correct? Answer Yes or No.') evaluated via Yes-minus-No logit margins and linear probe on final hidden state (`decision_hidden`).
27
+
28
+ ## Companion Repositories
29
+
30
+ - [`webbrain-one/webbrain-compass-vld-v6-2b-onnx`](https://huggingface.co/webbrain-one/webbrain-compass-vld-v6-2b-onnx): WebGPU ONNX bundle (`decoder_q8f16.onnx`, `decoder_q4f16.onnx` with `decision_hidden` exposed, `vision_encoder_fp16.onnx`, and trained linear decision probes).
31
+ - [`webbrain-one/webbrain-compass-vld-v6-2b-gguf`](https://huggingface.co/webbrain-one/webbrain-compass-vld-v6-2b-gguf): GGUF exports (`BF16`, `Q8_0`, `Q4_K_M`, and `mmproj-F16`) for `llama.cpp` (`/v1/chat/completions` + `/v1/systemone`).
32
+ - [`webbrain-one/webbrain-compass-vld-v6-2b-lora`](https://huggingface.co/webbrain-one/webbrain-compass-vld-v6-2b-lora): LoRA adapter (`r=32, alpha=64`) and projector weights.
33
+
34
+ ## Evaluation Results & Gates
35
+
36
+ | Evaluation Gate | Target | Achieved Result | Status |
37
+ | :--- | :--- | :--- | :--- |
38
+ | **Vision Suite (Q4_K_M) (E1)** | >= 44/100 strict & > 58 (v5 baseline) | **62/100 strict** (mean rubric: 0.8930, 0 errors, 1.13s latency) | **PASS** |
39
+ | **Vision Suite (Q8_0) (E1)** | Reference verification | **66/100 strict** (mean rubric: 0.8994, 0 errors, 1.50s latency) | **PASS** |
40
+ | **Six-Section Validity (E5)** | >= 99.0% | **99.00%** (495/500 held-out VL-2 val cases valid) | **PASS** |
41
+ | **d1 final_test Raw Verify (E3)** | >= 120/126 verify, 24/24 comp, 0/48 false comp, >= 23/24 unknown | **122/126 verify**, **24/24 comp**, **0/48 false comp**, **24/24 unknown** | **PASS** |
42
+ | **In-Browser Chat Held-Out (E4)** | Lang >= 95%, Act >= 95%, Citations 100%, 0 leaks | Lang: **95.95%**, Act: **100.0%**, Citations: **100.0%**, Leaks: **0** | **PASS** |
43
+ | **Routing Replay (E2)** | Structured >= 95/100, loose > 32/89, 0 length hits | Structured: **96/100**, Loose: **33/89**, Length hits: **0** | **PASS** |
44
+
45
+ ## License & Provenance Audit
46
+
47
+ - **Base Model**: `LiquidAI/LFM3-VL-2B` (LFM Open License v1.0).
48
+ - **Training Mix**: 70,717 train rows, 1,631 val rows across 5 license-audited components:
49
+ - `vision_vl2` (46,879): CC-BY-4.0 (WebSight, ScreenParse, MiniWoB++, synthetic web renders).
50
+ - `vision_curated` (5,880): Permissive (Orchard MIT, markov-ai Apache-2.0, look-before-you-click MIT).
51
+ - `decision_d1` (7,236): Synthetic / Internal d1 balanced-V3 browser decision tasks.
52
+ - `routing_compass` (6,950): Commercial manifest (WebBrain traces, Orchard MIT, Reduced-A11y CC BY 4.0, markov-ai Apache-2.0, Mind2Web CC BY 4.0).
53
+ - `chat_inbrowser` (3,822): CC-BY-4.0 / CC-BY-SA (Multilingual page QA, Turkish >= 20%, Act refusals, offline RAG, query translations).
54
+ - **Strict Exclusions**: WebChain (strictly excluded), WebLINX (strictly excluded), Compass v3 (strictly excluded).
chat_template.jinja ADDED
@@ -0,0 +1,280 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {{- bos_token -}}
2
+ {%- set preserve_thinking = preserve_thinking | default(false) -%}
3
+ {%- set enable_thinking = enable_thinking | default(true) -%}
4
+
5
+ {%- macro render_value(v) -%}
6
+ {%- if v is string -%}
7
+ {{- v -}}
8
+ {%- elif v is boolean -%}
9
+ {{- "true" if v else "false" -}}
10
+ {%- elif v is none -%}
11
+ {{- "null" -}}
12
+ {%- elif v is mapping or (v is sequence and v is not string) -%}
13
+ {{- v | tojson -}}
14
+ {%- else -%}
15
+ {{- v -}}
16
+ {%- endif -%}
17
+ {%- endmacro -%}
18
+
19
+ {%- macro parse_content(content) -%}
20
+ {%- if content is string -%}
21
+ {{- content -}}
22
+ {%- elif content is mapping and content.get("type") == "image" -%}
23
+ {{- "<|vision_start|><|image|><|vision_end|>" -}}
24
+ {%- elif content is mapping and content.get("type") == "video" -%}
25
+ {{- "<|vision_start|><|video|><|vision_end|>" -}}
26
+ {%- elif content is mapping and content.get("type") == "text" -%}
27
+ {{- (content.get("text") or "") | string -}}
28
+ {%- elif content is mapping -%}
29
+ {{- content | tojson -}}
30
+ {%- elif content is iterable -%}
31
+ {%- set _ns = namespace(result="") -%}
32
+ {%- for item in content -%}
33
+ {%- if item is string -%}
34
+ {%- set _ns.result = _ns.result + item -%}
35
+ {%- elif item is mapping and item.get("type") == "image" -%}
36
+ {%- set _ns.result = _ns.result + "<|vision_start|><|image|><|vision_end|>" -%}
37
+ {%- elif item is mapping and item.get("type") == "video" -%}
38
+ {%- set _ns.result = _ns.result + "<|vision_start|><|video|><|vision_end|>" -%}
39
+ {%- elif item is mapping and item.get("type") == "text" -%}
40
+ {%- set _ns.result = _ns.result + ((item.get("text") or "") | string) -%}
41
+ {%- else -%}
42
+ {%- set _ns.result = _ns.result + (item | tojson) -%}
43
+ {%- endif -%}
44
+ {%- endfor -%}
45
+ {{- _ns.result -}}
46
+ {%- elif content is not none -%}
47
+ {{- content | tojson -}}
48
+ {%- endif -%}
49
+ {%- endmacro -%}
50
+
51
+ {%- macro render_tool_calls(tool_calls) -%}
52
+ {%- for tool_call in tool_calls -%}
53
+ {%- if not loop.first -%}{{- "\n\n" -}}{%- endif -%}
54
+ {%- set func = tool_call["function"] if "function" in tool_call else tool_call -%}
55
+ {%- set func_args = func.get("arguments") -%}
56
+ {%- if func_args is string and (func_args | trim) not in ["", "{}", "null"] -%}
57
+ {{- raise_exception("Tool call arguments must be a mapping, got a JSON-encoded string: parse arguments with json.loads() before applying the chat template") -}}
58
+ {%- endif -%}
59
+ {{- "<tool_call>\n<function=" + func["name"] + ">\n" -}}
60
+ {%- if func_args is mapping -%}
61
+ {%- for arg_name, arg_value in func_args.items() -%}
62
+ {{- "<parameter=" + arg_name + ">\n" + render_value(arg_value) + "\n</parameter>\n" -}}
63
+ {%- endfor -%}
64
+ {%- endif -%}
65
+ {{- "</function>\n</tool_call>" -}}
66
+ {%- endfor -%}
67
+ {%- endmacro -%}
68
+
69
+ {%- macro tool_to_json(tool) -%}
70
+ {%- set fn = tool.function if tool.function is defined else tool -%}
71
+ {%- set ns_t = namespace(first=true) -%}
72
+ {{- "{" -}}
73
+ {%- for k, v in fn.items() -%}
74
+ {%- if k != "defer_loading" -%}
75
+ {%- if not ns_t.first -%}{{- ", " -}}{%- endif -%}
76
+ {%- set ns_t.first = false -%}
77
+ {{- k | tojson -}}{{- ": " -}}{{- v | tojson -}}
78
+ {%- endif -%}
79
+ {%- endfor -%}
80
+ {{- "}" -}}
81
+ {%- endmacro -%}
82
+
83
+ {%- macro is_tool_reference(item) -%}
84
+ {{- "1" if item is mapping and item.get("type") == "tool_reference" else "" -}}
85
+ {%- endmacro -%}
86
+
87
+ {%- macro referenced_tool_to_json(name) -%}
88
+ {%- set _ns = namespace(found=none) -%}
89
+ {%- for tool in tools or [] -%}
90
+ {%- set fn = tool.function if tool.function is defined else tool -%}
91
+ {%- if fn.name == name -%}
92
+ {%- set _ns.found = fn -%}
93
+ {%- endif -%}
94
+ {%- endfor -%}
95
+ {%- if _ns.found is none -%}
96
+ {{- raise_exception("Tool reference '" + name | string + "' not found in available tools. Every tool referenced by a tool_reference must be in `tools` on every request.") -}}
97
+ {%- endif -%}
98
+ {{- tool_to_json(_ns.found) -}}
99
+ {%- endmacro -%}
100
+
101
+ {%- macro render_tool_response(content) -%}
102
+ {%- if content is string and "<tool_response>" in content -%}
103
+ {{- content -}}
104
+ {%- else -%}
105
+ {{- "<tool_response>\n" -}}
106
+ {%- if content is string -%}
107
+ {{- content -}}
108
+ {%- elif content is mapping -%}
109
+ {{- parse_content(content) -}}
110
+ {%- elif content is iterable -%}
111
+ {%- for item in content -%}
112
+ {%- if not loop.first -%}{{- "\n" -}}{%- endif -%}
113
+ {%- if is_tool_reference(item) -%}
114
+ {%- if loop.first or not is_tool_reference(loop.previtem) -%}
115
+ {{- "<additional_tools>\n" -}}
116
+ {%- endif -%}
117
+ {{- referenced_tool_to_json(item.name) -}}
118
+ {%- if loop.last or not is_tool_reference(loop.nextitem) -%}
119
+ {{- "\n</additional_tools>" -}}
120
+ {%- endif -%}
121
+ {%- elif item is mapping and item.result is defined and item.result is mapping and item.result.tools is defined -%}
122
+ {{- "<additional_tools>\n" -}}
123
+ {%- set _tl = item.result.tools -%}
124
+ {%- if _tl is string -%}
125
+ {{- _tl -}}
126
+ {%- elif _tl is mapping -%}
127
+ {{- tool_to_json(_tl) -}}
128
+ {%- else -%}
129
+ {%- for t in _tl -%}
130
+ {%- if not loop.first -%}{{- "\n" -}}{%- endif -%}
131
+ {{- tool_to_json(t) -}}
132
+ {%- endfor -%}
133
+ {%- endif -%}
134
+ {{- "\n</additional_tools>" -}}
135
+ {%- else -%}
136
+ {{- parse_content(item) -}}
137
+ {%- endif -%}
138
+ {%- endfor -%}
139
+ {%- elif content is not none -%}
140
+ {{- content | tojson -}}
141
+ {%- endif -%}
142
+ {{- "\n</tool_response>" -}}
143
+ {%- endif -%}
144
+ {%- endmacro -%}
145
+
146
+ {%- set ns = namespace(system_prompt="", last_user_index=-1) -%}
147
+ {%- if messages and messages[0]["role"] == "system" -%}
148
+ {%- if messages[0].get("content") -%}
149
+ {%- set ns.system_prompt = parse_content(messages[0]["content"]) -%}
150
+ {%- endif -%}
151
+ {%- set messages = messages[1:] -%}
152
+ {%- endif -%}
153
+ {%- if tools -%}
154
+ {%- set tools_ns = namespace(lines=[]) -%}
155
+ {%- for tool in tools -%}
156
+ {%- set fn = tool.function if tool.function is defined else tool -%}
157
+ {%- set _dyn = (fn.defer_loading is defined and fn.defer_loading) or (tool.defer_loading is defined and tool.defer_loading) -%}
158
+ {%- if not _dyn -%}
159
+ {%- set tools_ns.lines = tools_ns.lines + [tool_to_json(fn)] -%}
160
+ {%- endif -%}
161
+ {%- endfor -%}
162
+ {%- if tools_ns.lines -%}
163
+ {%- set ns.system_prompt = ns.system_prompt + ("\n\n" if ns.system_prompt else "") + "<tools>\n" + (tools_ns.lines | join("\n")) + "\n</tools>" -%}
164
+ {%- endif -%}
165
+ {%- endif -%}
166
+ {%- if ns.system_prompt -%}
167
+ {{- "<|im_start|>system\n" + ns.system_prompt + "<|im_end|>\n" -}}
168
+ {%- endif -%}
169
+ {%- for message in messages -%}
170
+ {%- if message["role"] == "user" -%}
171
+ {%- set ns.last_user_index = loop.index0 -%}
172
+ {%- endif -%}
173
+ {%- endfor -%}
174
+ {%- set carry = namespace(pending=[]) -%}
175
+ {%- for message in messages -%}
176
+ {%- set _next = messages[loop.index0 + 1] if not loop.last else none -%}
177
+ {%- set _merge = message.role == "tool" and not message.get("additional_tools")
178
+ and _next is not none and _next.role == "tool" and not _next.get("additional_tools")
179
+ and message.get("tool_call_id") and _next.get("tool_call_id") == message.get("tool_call_id") -%}
180
+ {%- set _items = [] -%}
181
+ {%- if message.role == "tool" and not message.get("additional_tools") -%}
182
+ {%- set _c = message.get("content") -%}
183
+ {%- if _c is string -%}
184
+ {%- set _items = [{"type": "text", "text": _c}] if _c else [] -%}
185
+ {%- elif _c is mapping -%}
186
+ {%- set _items = [_c] -%}
187
+ {%- elif _c is iterable -%}
188
+ {%- set _items = _c | list -%}
189
+ {%- elif _c is not none -%}
190
+ {%- set _items = [_c] -%}
191
+ {%- endif -%}
192
+ {%- endif -%}
193
+ {%- if _merge -%}
194
+ {%- set carry.pending = carry.pending + _items -%}
195
+ {%- else -%}
196
+ {{- "<|im_start|>" + message.role + "\n" -}}
197
+ {%- if message.role == "assistant" -%}
198
+ {%- generation -%}
199
+ {%- set thinking = message.thinking if message.thinking is defined else message.reasoning if message.reasoning is defined else message.reasoning_content -%}
200
+ {%- set keep_thinking = preserve_thinking or loop.index0 > ns.last_user_index -%}
201
+ {%- if thinking is defined and keep_thinking -%}
202
+ {{- "<think>" + (thinking or "") + "</think>" -}}
203
+ {%- elif keep_thinking and "</think>" not in (message.get("content") if message.get("content") is string else "") -%}
204
+ {{- "<think></think>" -}}
205
+ {%- endif -%}
206
+ {%- set _cfm_tag = "CONTINUE_FINAL_MESSAGE_TAG " -%}
207
+ {%- set _has_cfm = false -%}
208
+ {%- set content = "" -%}
209
+ {%- if message.get("content") -%}
210
+ {%- set content = parse_content(message.content) -%}
211
+ {%- endif -%}
212
+ {%- if not keep_thinking -%}
213
+ {%- if "</think>" in content -%}
214
+ {%- set content = content.split("</think>")[-1] | trim -%}
215
+ {%- endif -%}
216
+ {%- set content = "<think></think>" + content -%}
217
+ {%- endif -%}
218
+ {%- if content.endswith(_cfm_tag) -%}
219
+ {%- set _has_cfm = true -%}
220
+ {%- set _trunc_len = (content | length) - (_cfm_tag | length) -%}
221
+ {%- set content = content[:_trunc_len] -%}
222
+ {%- endif -%}
223
+ {{- content -}}
224
+ {%- if message.tool_calls -%}
225
+ {{- render_tool_calls(message.tool_calls) -}}
226
+ {%- endif -%}
227
+ {%- if _has_cfm -%}
228
+ {{- _cfm_tag -}}
229
+ {%- endif -%}
230
+ {{- "<|im_end|>\n" -}}
231
+ {%- endgeneration -%}
232
+ {%- elif message.role == "tool" -%}
233
+ {%- if message.get("additional_tools") -%}
234
+ {%- set _at = message["additional_tools"] -%}
235
+ {%- if _at is mapping or (_at is sequence and _at is not string) -%}
236
+ {%- set _src = _at -%}
237
+ {%- else -%}
238
+ {%- set _src = message.get("content") -%}
239
+ {%- endif -%}
240
+ {%- if _src is string and "<tool_response>" in _src -%}
241
+ {{- _src -}}
242
+ {%- elif _src is string and "<additional_tools>" in _src -%}
243
+ {{- "<tool_response>\n" + _src + "\n</tool_response>" -}}
244
+ {%- elif _src -%}
245
+ {{- "<tool_response>\n<additional_tools>\n" -}}
246
+ {%- if _src is string -%}
247
+ {{- _src -}}
248
+ {%- elif _src is mapping -%}
249
+ {{- tool_to_json(_src) -}}
250
+ {%- else -%}
251
+ {%- for t in _src -%}
252
+ {%- if not loop.first -%}{{- "\n" -}}{%- endif -%}
253
+ {{- tool_to_json(t) -}}
254
+ {%- endfor -%}
255
+ {%- endif -%}
256
+ {{- "\n</additional_tools>\n</tool_response>" -}}
257
+ {%- endif -%}
258
+ {%- elif carry.pending -%}
259
+ {{- render_tool_response(carry.pending + _items) -}}
260
+ {%- set carry.pending = [] -%}
261
+ {%- else -%}
262
+ {{- render_tool_response("" if message.get("content") is none else message.get("content")) -}}
263
+ {%- endif -%}
264
+ {{- "<|im_end|>\n" -}}
265
+ {%- else %}
266
+ {%- if message.get("content") -%}
267
+ {{- parse_content(message["content"]) -}}
268
+ {%- endif -%}
269
+ {{- "<|im_end|>\n" -}}
270
+ {%- endif %}
271
+ {%- endif -%}
272
+ {%- endfor -%}
273
+ {%- if add_generation_prompt -%}
274
+ {{- "<|im_start|>assistant\n" -}}
275
+ {%- if enable_thinking -%}
276
+ {{- "<think>" -}}
277
+ {%- else -%}
278
+ {{- "<think></think>" -}}
279
+ {%- endif -%}
280
+ {%- endif -%}
config.json ADDED
@@ -0,0 +1,155 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Lfm3VlForConditionalGeneration"
4
+ ],
5
+ "auto_map": {
6
+ "AutoConfig": "configuration_lfm3_vl.Lfm3VlConfig",
7
+ "AutoModel": "modeling_lfm3_vl.Lfm3VlModel",
8
+ "AutoModelForImageTextToText": "modeling_lfm3_vl.Lfm3VlForConditionalGeneration"
9
+ },
10
+ "bos_token_id": 124894,
11
+ "downsample_factor": 2,
12
+ "dtype": "bfloat16",
13
+ "eos_token_id": 124900,
14
+ "image_token_id": 125030,
15
+ "model_type": "lfm3_vl",
16
+ "pad_token_id": 124893,
17
+ "projector_bias": true,
18
+ "projector_hidden_act": "gelu",
19
+ "projector_hidden_size": 4096,
20
+ "projector_layer_norm_eps": 1e-06,
21
+ "text_config": {
22
+ "_name_or_path": "",
23
+ "architectures": null,
24
+ "bos_token_id": 124894,
25
+ "chunk_size_feed_forward": 0,
26
+ "conv_L_cache": 7,
27
+ "conv_bias": false,
28
+ "dtype": "float32",
29
+ "eos_token_id": 124900,
30
+ "head_dim": 128,
31
+ "hidden_size": 2048,
32
+ "id2label": {
33
+ "0": "LABEL_0",
34
+ "1": "LABEL_1"
35
+ },
36
+ "initializer_range": 0.02,
37
+ "intermediate_size": 7680,
38
+ "is_encoder_decoder": false,
39
+ "label2id": {
40
+ "LABEL_0": 0,
41
+ "LABEL_1": 1
42
+ },
43
+ "layer_types": [
44
+ "conv",
45
+ "conv",
46
+ "full_attention",
47
+ "conv",
48
+ "sliding_attention",
49
+ "conv",
50
+ "full_attention",
51
+ "conv",
52
+ "sliding_attention",
53
+ "conv",
54
+ "full_attention",
55
+ "conv",
56
+ "sliding_attention",
57
+ "conv",
58
+ "full_attention",
59
+ "conv",
60
+ "sliding_attention",
61
+ "conv",
62
+ "full_attention",
63
+ "conv",
64
+ "sliding_attention",
65
+ "conv",
66
+ "full_attention",
67
+ "conv"
68
+ ],
69
+ "max_position_embeddings": 128000,
70
+ "model_type": "lfm3",
71
+ "norm_eps": 1e-05,
72
+ "num_attention_heads": 16,
73
+ "num_hidden_layers": 24,
74
+ "num_key_value_heads": 4,
75
+ "output_attentions": false,
76
+ "output_hidden_states": false,
77
+ "pad_token_id": 124893,
78
+ "problem_type": null,
79
+ "return_dict": true,
80
+ "rope_parameters": {
81
+ "full_attention": {
82
+ "partial_rotary_factor": 0.5,
83
+ "rope_theta": 1000000.0,
84
+ "rope_type": "default"
85
+ },
86
+ "sliding_attention": {
87
+ "rope_theta": 10000,
88
+ "rope_type": "default"
89
+ }
90
+ },
91
+ "sliding_window": 256,
92
+ "tie_word_embeddings": true,
93
+ "use_cache": true,
94
+ "use_sandwich_norm": true,
95
+ "vocab_size": 128000
96
+ },
97
+ "tie_word_embeddings": true,
98
+ "transformers_version": "5.21.0",
99
+ "video_token_id": 125029,
100
+ "vision_config": {
101
+ "_name_or_path": "",
102
+ "architectures": null,
103
+ "chunk_size_feed_forward": 0,
104
+ "conv_kernel_size": 3,
105
+ "dtype": "float32",
106
+ "hidden_size": 1024,
107
+ "id2label": {
108
+ "0": "LABEL_0",
109
+ "1": "LABEL_1"
110
+ },
111
+ "in_channels": 3,
112
+ "initializer_range": 0.02,
113
+ "intermediate_size": 2560,
114
+ "is_encoder_decoder": false,
115
+ "label2id": {
116
+ "LABEL_0": 0,
117
+ "LABEL_1": 1
118
+ },
119
+ "layer_types": [
120
+ "conv",
121
+ "conv",
122
+ "full_attention",
123
+ "conv",
124
+ "full_attention",
125
+ "conv",
126
+ "full_attention",
127
+ "conv",
128
+ "full_attention",
129
+ "conv",
130
+ "full_attention",
131
+ "conv",
132
+ "full_attention",
133
+ "conv"
134
+ ],
135
+ "model_type": "lfm3_vl_vision",
136
+ "norm_eps": 1e-05,
137
+ "num_attention_heads": 16,
138
+ "num_hidden_layers": 14,
139
+ "num_key_value_heads": 8,
140
+ "output_attentions": false,
141
+ "output_hidden_states": false,
142
+ "patch_size": 16,
143
+ "problem_type": null,
144
+ "return_dict": true,
145
+ "rope_parameters": {
146
+ "rope_theta": 1000000.0,
147
+ "rope_type": "axial"
148
+ },
149
+ "spatial_merge_size": 2,
150
+ "temporal_patch_size": 2
151
+ },
152
+ "vision_end_token_id": 125010,
153
+ "vision_start_token_id": 125009,
154
+ "webbrain_vld": true
155
+ }
configuration_lfm3.py ADDED
@@ -0,0 +1,132 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2026 The HuggingFace Team. All rights reserved.
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+ from huggingface_hub.dataclasses import strict
15
+
16
+ from transformers.configuration_utils import PreTrainedConfig
17
+ from transformers.utils import auto_docstring
18
+
19
+
20
+ @auto_docstring(checkpoint="LiquidAI/LFM3-1.7B")
21
+ @strict
22
+ class Lfm3Config(PreTrainedConfig):
23
+ r"""
24
+ rope_parameters (`dict[str, RopeParameters]`, *optional*):
25
+ RoPE parameters keyed by attention layer type, for example `{"full_attention": {"rope_type": "default",
26
+ "rope_theta": 1000000.0, "partial_rotary_factor": 0.5}, "sliding_attention": {"rope_type": "default",
27
+ "rope_theta": 10000.0}}`. `conv` layers do not use RoPE and have no entry. Full-attention layers rotate the
28
+ first `partial_rotary_factor * head_dim` dimensions and leave the rest unrotated.
29
+ conv_bias (`bool`, *optional*, defaults to `False`):
30
+ Whether to use bias in the conv layers.
31
+ conv_L_cache (`int`, *optional*, defaults to 7):
32
+ L_cache dim in the conv layers.
33
+ layer_types (`list[str]`, *optional*):
34
+ Per-layer operator type, each one of `"full_attention"`, `"sliding_attention"` or `"conv"`. Defaults to the
35
+ LFM3 pattern of a full-attention layer every fourth layer starting from the third, a sliding-attention layer
36
+ in between, and gated short convolutions elsewhere.
37
+ sliding_window (`int`, *optional*, defaults to 256):
38
+ Sliding-window size of the `sliding_attention` layers.
39
+ use_sandwich_norm (`bool`, *optional*, defaults to `True`):
40
+ Whether to apply an RMSNorm to the operator and feed-forward outputs before they enter the residual stream.
41
+
42
+ ```python
43
+ >>> from transformers import Lfm3Model, Lfm3Config
44
+
45
+ >>> # Initializing a LFM3 model
46
+ >>> configuration = Lfm3Config()
47
+
48
+ >>> # Initializing a model from the LFM3-1.7B style configuration
49
+ >>> model = Lfm3Model(configuration)
50
+
51
+ >>> # Accessing the model configuration
52
+ >>> configuration = model.config
53
+ ```
54
+ """
55
+
56
+ model_type = "lfm3"
57
+ keys_to_ignore_at_inference = ["past_key_values"]
58
+ default_theta = 1000000.0
59
+
60
+ vocab_size: int = 128_000
61
+ hidden_size: int = 2048
62
+ intermediate_size: int = 7680
63
+ num_hidden_layers: int = 24
64
+ num_attention_heads: int = 16
65
+ num_key_value_heads: int = 4
66
+ max_position_embeddings: int = 128_000
67
+ initializer_range: float = 0.02
68
+ norm_eps: float = 0.00001
69
+ use_cache: bool = True
70
+ pad_token_id: int | None = 124_893
71
+ bos_token_id: int | None = 124_894
72
+ eos_token_id: int | list[int] | None = 124_900
73
+ tie_word_embeddings: bool = True
74
+ rope_parameters: dict | None = None
75
+ conv_bias: bool = False
76
+ conv_L_cache: int = 7
77
+ layer_types: list[str] | None = None
78
+ _attention_layer_types = ("full_attention", "sliding_attention")
79
+ head_dim: int | None = None
80
+ sliding_window: int | None = 256
81
+ use_sandwich_norm: bool = True
82
+
83
+ def __post_init__(self, **kwargs):
84
+ if self.head_dim is None:
85
+ self.head_dim = self.hidden_size // self.num_attention_heads
86
+ if self.rope_parameters is None:
87
+ self.rope_parameters = {
88
+ "full_attention": {"rope_type": "default", "rope_theta": 1_000_000.0, "partial_rotary_factor": 0.5},
89
+ "sliding_attention": {"rope_type": "default", "rope_theta": 10_000.0},
90
+ }
91
+ if self.layer_types is None:
92
+ self.layer_types = [
93
+ "full_attention" if i % 4 == 2 else "sliding_attention" if i % 4 == 0 and i > 0 else "conv"
94
+ for i in range(self.num_hidden_layers)
95
+ ]
96
+
97
+ self.tie_word_embeddings = kwargs.pop("tie_embedding", self.tie_word_embeddings)
98
+ super().__post_init__(**kwargs)
99
+
100
+ def validate_architecture(self):
101
+ """Part of ``@strict``-powered validation."""
102
+ if len(self.layer_types) != self.num_hidden_layers:
103
+ raise ValueError(
104
+ f"`layer_types` has {len(self.layer_types)} entries but `num_hidden_layers` is {self.num_hidden_layers}."
105
+ )
106
+ allowed_layer_types = (*self._attention_layer_types, "conv")
107
+ invalid_layer_types = [layer_type for layer_type in self.layer_types if layer_type not in allowed_layer_types]
108
+ if invalid_layer_types:
109
+ raise ValueError(f"`layer_types` entries must be one of {allowed_layer_types}; got {invalid_layer_types}.")
110
+ if "sliding_attention" in self.layer_types and self.sliding_window is None:
111
+ raise ValueError("`layer_types` has `sliding_attention` layers but `sliding_window` is None.")
112
+ missing_rope = [
113
+ layer_type
114
+ for layer_type in self._attention_layer_types
115
+ if layer_type in self.layer_types and layer_type not in self.rope_parameters
116
+ ]
117
+ if missing_rope:
118
+ raise ValueError(f"`rope_parameters` has no entry for the {missing_rope} layers in `layer_types`.")
119
+ for layer_type, rope_params in self.rope_parameters.items():
120
+ if not isinstance(rope_params, dict):
121
+ continue
122
+ if (
123
+ rope_params.get("rope_type", "default") in ("dynamic", "longrope")
124
+ and rope_params.get("partial_rotary_factor", 1.0) < 1
125
+ ):
126
+ raise ValueError(
127
+ f"`{rope_params['rope_type']}` RoPE recomputes its frequencies during the forward pass over the "
128
+ f"rotated dimension only and cannot keep LFM3's partial rotation on `{layer_type}` layers."
129
+ )
130
+
131
+
132
+ __all__ = ["Lfm3Config"]
configuration_lfm3_vl.py ADDED
@@ -0,0 +1,211 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2026 The HuggingFace Team. All rights reserved.
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+ from huggingface_hub.dataclasses import strict
15
+
16
+ from transformers.configuration_utils import PreTrainedConfig
17
+ from transformers.models.auto import CONFIG_MAPPING, AutoConfig
18
+ from transformers.utils import auto_docstring
19
+
20
+ from .configuration_lfm3 import Lfm3Config
21
+
22
+
23
+ def _config_class(model_type):
24
+ # the LFM3 configs ship with the checkpoint instead of the transformers registry
25
+ remote = {"lfm3": Lfm3Config, "lfm3_vl_vision": Lfm3VlVisionConfig}
26
+ return remote.get(model_type) or CONFIG_MAPPING[model_type]
27
+
28
+
29
+ @auto_docstring(checkpoint="LiquidAI/LFM3-VL-2B")
30
+ @strict
31
+ class Lfm3VlVisionConfig(PreTrainedConfig):
32
+ r"""
33
+ layer_types (`list[str]`, *optional*):
34
+ Per-layer operator type, each one of `"full_attention"` or `"conv"`. Defaults to two gated convolution
35
+ layers followed by alternating attention and convolution layers.
36
+ patch_size (`int`, *optional*, defaults to 16):
37
+ The spatial size of each image patch.
38
+ temporal_patch_size (`int`, *optional*, defaults to 2):
39
+ The number of video frames packed into one patch.
40
+ spatial_merge_size (`int`, *optional*, defaults to 2):
41
+ The side of the square block of patches merged into one language token.
42
+ conv_kernel_size (`int`, *optional*, defaults to 3):
43
+ The kernel size of the depthwise 2D convolution layers.
44
+ norm_eps (`float`, *optional*, defaults to 1e-05):
45
+ The epsilon used by the RMS normalization layers.
46
+ """
47
+
48
+ model_type = "lfm3_vl_vision"
49
+ base_config_key = "vision_config"
50
+ default_theta = 1_000_000.0
51
+ default_rope_type = "axial"
52
+ _attention_layer_types = ("full_attention",)
53
+
54
+ hidden_size: int = 1024
55
+ intermediate_size: int = 2560
56
+ num_hidden_layers: int = 14
57
+ num_attention_heads: int = 16
58
+ num_key_value_heads: int = 8
59
+ layer_types: list[str] | None = None
60
+ patch_size: int = 16
61
+ temporal_patch_size: int = 2
62
+ in_channels: int = 3
63
+ spatial_merge_size: int = 2
64
+ conv_kernel_size: int = 3
65
+ norm_eps: float = 1e-5
66
+ initializer_range: float = 0.02
67
+ rope_parameters: dict | None = None
68
+
69
+ def __post_init__(self, **kwargs):
70
+ if self.layer_types is None:
71
+ self.layer_types = ["conv" if i < 2 or i % 2 else "full_attention" for i in range(self.num_hidden_layers)]
72
+ super().__post_init__(**kwargs)
73
+
74
+ def validate_architecture(self):
75
+ """Part of ``@strict``-powered validation."""
76
+ if len(self.layer_types) != self.num_hidden_layers:
77
+ raise ValueError(
78
+ f"`layer_types` has {len(self.layer_types)} entries but `num_hidden_layers` is {self.num_hidden_layers}."
79
+ )
80
+ allowed_layer_types = (*self._attention_layer_types, "conv")
81
+ invalid_layer_types = [layer_type for layer_type in self.layer_types if layer_type not in allowed_layer_types]
82
+ if invalid_layer_types:
83
+ raise ValueError(f"`layer_types` entries must be one of {allowed_layer_types}; got {invalid_layer_types}.")
84
+ if self.hidden_size % self.num_attention_heads:
85
+ raise ValueError("`hidden_size` must be a multiple of `num_attention_heads`.")
86
+
87
+
88
+ # The first checkpoints spell every text and `vision_*` field at the top level of the config
89
+ _LEGACY_TEXT_KEYS = (
90
+ "vocab_size",
91
+ "hidden_size",
92
+ "intermediate_size",
93
+ "num_hidden_layers",
94
+ "num_attention_heads",
95
+ "num_key_value_heads",
96
+ "head_dim",
97
+ "layer_types",
98
+ "conv_bias",
99
+ "conv_L_cache",
100
+ "sliding_window",
101
+ "max_position_embeddings",
102
+ "use_sandwich_norm",
103
+ "use_cache",
104
+ )
105
+ _LEGACY_VISION_KEYS = {
106
+ "vision_hidden_size": "hidden_size",
107
+ "vision_intermediate_size": "intermediate_size",
108
+ "vision_num_hidden_layers": "num_hidden_layers",
109
+ "vision_num_attention_heads": "num_attention_heads",
110
+ "vision_num_key_value_heads": "num_key_value_heads",
111
+ "vision_layer_types": "layer_types",
112
+ "vision_patch_size": "patch_size",
113
+ "vision_temporal_patch_size": "temporal_patch_size",
114
+ "vision_num_channels": "in_channels",
115
+ "vision_spatial_merge_size": "spatial_merge_size",
116
+ }
117
+ _LEGACY_DROPPED_KEYS = ("vision_patch_dim", "vision_max_position_embeddings")
118
+
119
+
120
+ @auto_docstring(checkpoint="LiquidAI/LFM3-VL-2B")
121
+ @strict
122
+ class Lfm3VlConfig(PreTrainedConfig):
123
+ r"""
124
+ video_token_id (`int`, *optional*, defaults to 125029):
125
+ The id of the placeholder token for video patches.
126
+ vision_start_token_id (`int`, *optional*, defaults to 125009):
127
+ The id of the token opening an image or a video frame.
128
+ vision_end_token_id (`int`, *optional*, defaults to 125010):
129
+ The id of the token closing an image or a video frame.
130
+ projector_layer_norm_eps (`float`, *optional*, defaults to 1e-06):
131
+ The epsilon of the layer normalization applied to the vision features before projection.
132
+ downsample_factor (`int`, *optional*, defaults to 2):
133
+ The side of the square block of vision tokens merged by the projector, equal to the vision encoder's
134
+ `spatial_merge_size`.
135
+ """
136
+
137
+ model_type = "lfm3_vl"
138
+ sub_configs = {"text_config": AutoConfig, "vision_config": Lfm3VlVisionConfig}
139
+
140
+ text_config: dict | PreTrainedConfig | None = None
141
+ vision_config: dict | PreTrainedConfig | None = None
142
+ image_token_id: int = 125_030
143
+ video_token_id: int = 125_029
144
+ vision_start_token_id: int = 125_009
145
+ vision_end_token_id: int = 125_010
146
+ projector_hidden_act: str = "gelu"
147
+ projector_hidden_size: int = 4096
148
+ projector_bias: bool = True
149
+ projector_layer_norm_eps: float = 1e-6
150
+ downsample_factor: int = 2
151
+ tie_word_embeddings: bool = True
152
+
153
+ def __post_init__(self, **kwargs):
154
+ if "vision_hidden_size" in kwargs:
155
+ kwargs = self._convert_legacy_flat_config(kwargs)
156
+
157
+ if isinstance(self.vision_config, dict):
158
+ self.vision_config = self.sub_configs["vision_config"](**self.vision_config)
159
+ elif self.vision_config is None:
160
+ self.vision_config = self.sub_configs["vision_config"]()
161
+
162
+ if isinstance(self.text_config, dict):
163
+ self.text_config = {"model_type": "lfm3", **self.text_config}
164
+ self.text_config = _config_class(self.text_config["model_type"])(**self.text_config)
165
+ elif self.text_config is None:
166
+ self.text_config = _config_class("lfm3")()
167
+ self.validate_vision_fields()
168
+
169
+ super().__post_init__(**kwargs)
170
+
171
+ def validate_vision_fields(self):
172
+ """Part of `@strict`-powered validation, so `validate()` and `save_pretrained` refuse later edits too."""
173
+ if self.downsample_factor != self.vision_config.spatial_merge_size:
174
+ raise ValueError("downsample_factor must match vision_config.spatial_merge_size")
175
+ # an `inputs_embeds` forward embeds both placeholder ids to find them
176
+ vocab_size = self.text_config.vocab_size
177
+ for name in ("image_token_id", "video_token_id"):
178
+ if not 0 <= getattr(self, name) < vocab_size:
179
+ raise ValueError(f"{name}={getattr(self, name)} must lie in [0, text_config.vocab_size={vocab_size})")
180
+
181
+ def _convert_legacy_flat_config(self, kwargs: dict) -> dict:
182
+ text_config = {key: kwargs.pop(key) for key in _LEGACY_TEXT_KEYS if key in kwargs}
183
+ text_config["norm_eps"] = kwargs.pop("rms_norm_eps", 1e-5)
184
+ text_config["rope_parameters"] = {
185
+ "full_attention": {
186
+ "rope_type": "default",
187
+ "rope_theta": kwargs.pop("full_attention_rope_theta", 1_000_000.0),
188
+ "partial_rotary_factor": kwargs.pop("partial_rotary_factor", 0.5),
189
+ },
190
+ "sliding_attention": {
191
+ "rope_type": "default",
192
+ "rope_theta": kwargs.pop("sliding_attention_rope_theta", 10_000.0),
193
+ },
194
+ }
195
+ for key in ("pad_token_id", "bos_token_id", "eos_token_id"):
196
+ if kwargs.get(key) is not None:
197
+ text_config[key] = kwargs[key]
198
+ text_config["tie_word_embeddings"] = self.tie_word_embeddings
199
+
200
+ vision_config = {new: kwargs.pop(old) for old, new in _LEGACY_VISION_KEYS.items() if old in kwargs}
201
+ vision_config["norm_eps"] = text_config["norm_eps"]
202
+ vision_config["rope_parameters"] = {"rope_type": "axial", "rope_theta": kwargs.pop("vision_rope_theta", 1e6)}
203
+ for key in _LEGACY_DROPPED_KEYS:
204
+ kwargs.pop(key, None)
205
+
206
+ self.text_config = text_config
207
+ self.vision_config = vision_config
208
+ return kwargs
209
+
210
+
211
+ __all__ = ["Lfm3VlConfig", "Lfm3VlVisionConfig"]
generation_config.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 124894,
4
+ "do_sample": true,
5
+ "eos_token_id": 124900,
6
+ "output_attentions": false,
7
+ "output_hidden_states": false,
8
+ "pad_token_id": 124893,
9
+ "repetition_penalty": 1.1,
10
+ "temperature": 0.1,
11
+ "top_k": 50,
12
+ "transformers_version": "5.21.0",
13
+ "use_cache": true
14
+ }
image_processing_lfm3_vl.py ADDED
@@ -0,0 +1,293 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2026 The HuggingFace Team. All rights reserved.
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+ import math
15
+ import warnings
16
+ from collections.abc import Iterable
17
+
18
+ import torch
19
+ from torchvision.transforms.v2 import functional as tvF
20
+
21
+ from transformers.image_processing_backends import TorchvisionBackend
22
+ from transformers.image_processing_utils import BatchFeature
23
+ from transformers.image_transforms import group_images_by_shape, reorder_images
24
+ from transformers.image_utils import ImageInput, PILImageResampling, SizeDict
25
+ from transformers.processing_utils import ImagesKwargs, Unpack
26
+ from transformers.utils import TensorType, auto_docstring
27
+
28
+
29
+ class Lfm3VlImageProcessorKwargs(ImagesKwargs, total=False):
30
+ r"""
31
+ min_pixels (`int`, *optional*, defaults to `56 * 56`):
32
+ The min pixels of the image to resize the image.
33
+ max_pixels (`int`, *optional*, defaults to `28 * 28 * 1280`):
34
+ The max pixels of the image to resize the image.
35
+ patch_size (`int`, *optional*, defaults to 14):
36
+ The spatial patch size of the vision encoder.
37
+ temporal_patch_size (`int`, *optional*, defaults to 2):
38
+ The temporal patch size of the vision encoder.
39
+ merge_size (`int`, *optional*, defaults to 2):
40
+ The merge size of the vision encoder to llm encoder.
41
+ """
42
+
43
+ min_pixels: int
44
+ max_pixels: int
45
+ patch_size: int
46
+ temporal_patch_size: int
47
+ merge_size: int
48
+
49
+
50
+ def smart_resize(
51
+ height: int, width: int, factor: int = 28, min_pixels: int = 56 * 56, max_pixels: int = 14 * 14 * 4 * 1280
52
+ ):
53
+ """Rescales the image so that the following conditions are met:
54
+
55
+ 1. Both dimensions (height and width) are divisible by 'factor'.
56
+
57
+ 2. The total number of pixels is within the range ['min_pixels', 'max_pixels'].
58
+
59
+ 3. The aspect ratio of the image is maintained as closely as possible.
60
+
61
+ """
62
+ if max(height, width) / min(height, width) > 200:
63
+ raise ValueError(
64
+ f"absolute aspect ratio must be smaller than 200, got {max(height, width) / min(height, width)}"
65
+ )
66
+ h_bar = round(height / factor) * factor
67
+ w_bar = round(width / factor) * factor
68
+ if h_bar * w_bar > max_pixels:
69
+ beta = math.sqrt((height * width) / max_pixels)
70
+ h_bar = max(factor, math.floor(height / beta / factor) * factor)
71
+ w_bar = max(factor, math.floor(width / beta / factor) * factor)
72
+ elif h_bar * w_bar < min_pixels:
73
+ beta = math.sqrt(min_pixels / (height * width))
74
+ h_bar = math.ceil(height * beta / factor) * factor
75
+ w_bar = math.ceil(width * beta / factor) * factor
76
+ return h_bar, w_bar
77
+
78
+
79
+ @auto_docstring
80
+ class Lfm3VlImageProcessor(TorchvisionBackend):
81
+ do_resize = True
82
+ resample = PILImageResampling.BICUBIC
83
+ size = {"shortest_edge": 65_536, "longest_edge": 16_777_216}
84
+ default_to_square = False
85
+ do_rescale = True
86
+ do_normalize = True
87
+ image_mean = [0.5, 0.5, 0.5]
88
+ image_std = [0.5, 0.5, 0.5]
89
+ do_convert_rgb = True
90
+ patch_size = 16
91
+ temporal_patch_size = 2
92
+ merge_size = 2
93
+ valid_kwargs = Lfm3VlImageProcessorKwargs
94
+ model_input_names = ["pixel_values", "image_grid_thw"]
95
+
96
+ def __init__(self, **kwargs: Unpack[Lfm3VlImageProcessorKwargs]):
97
+ # backward compatibility: override size with min_pixels and max_pixels if they are provided
98
+ size = kwargs.pop("size", None)
99
+ size = dict(self.size) if size is None else size
100
+ if (min_pixels := kwargs.pop("min_pixels", None)) is not None:
101
+ size["shortest_edge"] = min_pixels
102
+ size.pop("min_pixels", None)
103
+ if (max_pixels := kwargs.pop("max_pixels", None)) is not None:
104
+ size["longest_edge"] = max_pixels
105
+ size.pop("max_pixels", None)
106
+ super().__init__(size=size, **kwargs)
107
+
108
+ def _standardize_kwargs(
109
+ self,
110
+ size: int | Iterable[int] | dict[str, int] | SizeDict | None = None,
111
+ min_pixels: int | None = None,
112
+ max_pixels: int | None = None,
113
+ **kwargs,
114
+ ) -> dict:
115
+ if min_pixels is not None or max_pixels is not None:
116
+ warnings.warn(
117
+ "Passing `min_pixels` and `max_pixels` to a processor call is deprecated and will be removed in v5.23. "
118
+ "Pass in `size={'longest_edge': xxx, 'shortest_edge': xxx} to override the target size.`",
119
+ FutureWarning,
120
+ )
121
+
122
+ size_dict = dict(size) if isinstance(size, (dict, SizeDict)) else {}
123
+ if min_pixels is not None:
124
+ size_dict["shortest_edge"] = min_pixels
125
+ if max_pixels is not None:
126
+ size_dict["longest_edge"] = max_pixels
127
+ size = SizeDict(**size_dict)
128
+ return super()._standardize_kwargs(size=size, **kwargs)
129
+
130
+ @auto_docstring
131
+ def preprocess(
132
+ self,
133
+ images: ImageInput,
134
+ **kwargs: Unpack[Lfm3VlImageProcessorKwargs],
135
+ ) -> BatchFeature:
136
+ return super().preprocess(images, **kwargs)
137
+
138
+ def resize(
139
+ self,
140
+ images: "torch.Tensor",
141
+ size: SizeDict,
142
+ resample: "PILImageResampling | tvF.InterpolationMode | int | None",
143
+ factor: int,
144
+ **kwargs,
145
+ ) -> "torch.Tensor":
146
+ """Resize dynamically based on input image aspect ratio."""
147
+ if not size.shortest_edge or not size.longest_edge:
148
+ raise ValueError(f"`size` dict must contain 'shortest_edge' and 'longest_edge' keys but got {size}.")
149
+
150
+ height, width = images.shape[-2:]
151
+ resized_height, resized_width = smart_resize(
152
+ height,
153
+ width,
154
+ factor=factor,
155
+ min_pixels=size.shortest_edge,
156
+ max_pixels=size.longest_edge,
157
+ )
158
+ return super().resize(
159
+ image=images,
160
+ size=SizeDict(height=resized_height, width=resized_width),
161
+ resample=resample,
162
+ )
163
+
164
+ def patchify(
165
+ self,
166
+ images: "torch.Tensor",
167
+ patch_size: int,
168
+ merge_size: int,
169
+ temporal_patch_size: int,
170
+ ) -> tuple["torch.Tensor", int, int]:
171
+ """Patchifies each image into flat layout of shape (`seq_len`, `patch_dim`) so we can concat dynamically shaped pixels."""
172
+ batch_size, channel, resized_height, resized_width = images.shape
173
+ grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
174
+ patches = images.reshape(
175
+ batch_size,
176
+ channel,
177
+ grid_h // merge_size,
178
+ merge_size,
179
+ patch_size,
180
+ grid_w // merge_size,
181
+ merge_size,
182
+ patch_size,
183
+ )
184
+ # Reorder dimensions to group grid and patch information for subsequent flattening.
185
+ # [batch, grid_h/merge, grid_w/merge, merge, merge, channel, patch, patch]
186
+ patches = patches.permute(0, 2, 5, 3, 6, 1, 4, 7)
187
+
188
+ flatten_patches = (
189
+ patches.unsqueeze(6)
190
+ .expand(-1, -1, -1, -1, -1, -1, temporal_patch_size, -1, -1)
191
+ .reshape(
192
+ batch_size,
193
+ grid_h * grid_w,
194
+ channel * temporal_patch_size * patch_size * patch_size,
195
+ )
196
+ )
197
+ return flatten_patches, grid_h, grid_w
198
+
199
+ def _preprocess(
200
+ self,
201
+ images: list["torch.Tensor"],
202
+ do_resize: bool,
203
+ size: SizeDict,
204
+ resample: "PILImageResampling | tvF.InterpolationMode | int | None",
205
+ do_rescale: bool,
206
+ rescale_factor: float,
207
+ do_normalize: bool,
208
+ image_mean: float | list[float] | None,
209
+ image_std: float | list[float] | None,
210
+ patch_size: int,
211
+ temporal_patch_size: int,
212
+ merge_size: int,
213
+ disable_grouping: bool | None,
214
+ return_tensors: str | TensorType | None,
215
+ **kwargs,
216
+ ) -> BatchFeature:
217
+ grouped_images, grouped_images_index = group_images_by_shape(images, disable_grouping=disable_grouping)
218
+ resized_images_grouped = {}
219
+ for shape, stacked_images in grouped_images.items():
220
+ if do_resize:
221
+ stacked_images = self.resize(
222
+ images=stacked_images,
223
+ size=size,
224
+ resample=resample,
225
+ factor=patch_size * merge_size,
226
+ )
227
+ resized_images_grouped[shape] = stacked_images
228
+ resized_images = reorder_images(resized_images_grouped, grouped_images_index)
229
+
230
+ grouped_images, grouped_images_index = group_images_by_shape(resized_images, disable_grouping=disable_grouping)
231
+ processed_images_grouped = {}
232
+ processed_grids = {}
233
+ for shape, stacked_images in grouped_images.items():
234
+ stacked_images = self.rescale_and_normalize(
235
+ stacked_images, do_rescale, rescale_factor, do_normalize, image_mean, image_std
236
+ )
237
+ patches, grid_h, grid_w = self.patchify(
238
+ stacked_images,
239
+ patch_size=patch_size,
240
+ merge_size=merge_size,
241
+ temporal_patch_size=temporal_patch_size,
242
+ )
243
+
244
+ processed_images_grouped[shape] = patches
245
+ processed_grids[shape] = [[1, grid_h, grid_w]] * len(stacked_images)
246
+
247
+ processed_images = reorder_images(processed_images_grouped, grouped_images_index)
248
+ processed_grids_ordered = reorder_images(processed_grids, grouped_images_index)
249
+ pixel_values = processed_images[0] if len(processed_images) == 1 else torch.cat(processed_images, dim=0)
250
+ image_grid_thw = torch.tensor(processed_grids_ordered, dtype=torch.long)
251
+
252
+ return BatchFeature(
253
+ data={"pixel_values": pixel_values, "image_grid_thw": image_grid_thw}, tensor_type=return_tensors
254
+ )
255
+
256
+ def get_number_of_image_patches(self, height: int, width: int, images_kwargs: dict | None = None) -> int:
257
+ """
258
+ A utility that returns number of image patches for a given image size.
259
+
260
+ Note: Do not remove this method! It is used by vLLM to infer the number of patches and placeholders
261
+ without an image input.
262
+
263
+ Args:
264
+ height (`int`):
265
+ Height of the input image.
266
+ width (`int`):
267
+ Width of the input image.
268
+ images_kwargs (`dict`, *optional*)
269
+ Any kwargs to override defaults of the image processor.
270
+ Returns:
271
+ `int`: Number of image patches per image.
272
+ """
273
+ # `size` is the supported override (`min_pixels` / `max_pixels` are deprecated): count with the budget the
274
+ # preprocessing applies
275
+ images_kwargs = dict(images_kwargs or {})
276
+ size = images_kwargs.pop("size", None) or self.size
277
+ images_kwargs.setdefault("min_pixels", size["shortest_edge"])
278
+ images_kwargs.setdefault("max_pixels", size["longest_edge"])
279
+ images_kwargs = images_kwargs or {}
280
+ min_pixels = images_kwargs["min_pixels"] if "min_pixels" in images_kwargs else self.size["shortest_edge"]
281
+ max_pixels = images_kwargs["max_pixels"] if "max_pixels" in images_kwargs else self.size["longest_edge"]
282
+ patch_size = images_kwargs.get("patch_size", self.patch_size)
283
+ merge_size = images_kwargs.get("merge_size", self.merge_size)
284
+
285
+ factor = patch_size * merge_size
286
+ resized_height, resized_width = smart_resize(
287
+ height, width, factor, min_pixels=min_pixels, max_pixels=max_pixels
288
+ )
289
+ grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
290
+ return grid_h * grid_w
291
+
292
+
293
+ __all__ = ["Lfm3VlImageProcessor"]
lfm3_conv.py ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2026 The HuggingFace Team. All rights reserved.
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+ """Inference kernels for LFM3's width-7 depthwise convolution."""
15
+
16
+ import torch
17
+
18
+ from transformers.utils import is_triton_available
19
+
20
+
21
+ if is_triton_available():
22
+ import triton
23
+ import triton.language as tl
24
+
25
+ # the length and strides are runtime arguments, or every prompt length would compile its own kernel
26
+ @triton.jit(do_not_specialize=["T", "X0", "X1", "X2", "S0", "S1", "ST"])
27
+ def _prefill(
28
+ X,
29
+ W,
30
+ BIAS,
31
+ SEQ,
32
+ Y,
33
+ T,
34
+ D: tl.constexpr,
35
+ X0,
36
+ X1,
37
+ X2,
38
+ S0,
39
+ S1,
40
+ ST,
41
+ HAS_BIAS: tl.constexpr,
42
+ PACKED: tl.constexpr,
43
+ BLOCK: tl.constexpr,
44
+ ):
45
+ # int64 offsets: batch * channels * length can exceed 2**31 elements
46
+ b, d = tl.program_id(0).to(tl.int64), tl.program_id(1).to(tl.int64)
47
+ t = tl.program_id(2).to(tl.int64) * BLOCK + tl.arange(0, BLOCK)
48
+ acc = tl.full((BLOCK,), 0, tl.float32)
49
+ if PACKED:
50
+ doc = tl.load(SEQ + b * S0 + (t - T + ST) * S1, (t < T) & (t >= T - ST), other=-1)
51
+ for k in tl.static_range(7):
52
+ pos = t + k - 6
53
+ valid = (t < T) & (pos >= 0)
54
+ if PACKED:
55
+ prev = tl.load(SEQ + b * S0 + (pos - T + ST) * S1, valid & (pos >= T - ST), other=-1)
56
+ valid = valid & (doc == prev)
57
+ x = tl.load(X + b * X0 + d * X1 + pos * X2, valid, other=0).to(tl.float32)
58
+ w = tl.load(W + d * 7 + k).to(tl.float32)
59
+ acc += x * w
60
+ if HAS_BIAS:
61
+ acc += tl.load(BIAS + d).to(tl.float32)
62
+ tl.store(Y + (b * D + d) * T + t, acc, t < T)
63
+
64
+ @triton.jit
65
+ def _decode(
66
+ X,
67
+ W,
68
+ BIAS,
69
+ STATE,
70
+ Y,
71
+ D: tl.constexpr,
72
+ K: tl.constexpr,
73
+ X0: tl.constexpr,
74
+ X1: tl.constexpr,
75
+ S0: tl.constexpr,
76
+ S1: tl.constexpr,
77
+ S2: tl.constexpr,
78
+ HAS_BIAS: tl.constexpr,
79
+ BLOCK: tl.constexpr,
80
+ ):
81
+ b = tl.program_id(0).to(tl.int64)
82
+ d = tl.program_id(1).to(tl.int64) * BLOCK + tl.arange(0, BLOCK)
83
+ k = tl.arange(0, 8)
84
+ # Load the complete state before writing the shifted window.
85
+ old = tl.load(STATE + b * S0 + d[:, None] * S1 + k[None, :] * S2, (d[:, None] < D) & (k[None, :] < K), other=0)
86
+ x = tl.load(X + b * X0 + d * X1, d < D, other=0)
87
+ shifted = tl.gather(old, tl.broadcast_to(tl.minimum(k + 1, 7)[None, :], (BLOCK, 8)), 1)
88
+ shifted = tl.where(k[None, :] == K - 1, x[:, None], shifted)
89
+ # A cache may store either K=6 previous inputs or K=7 including the oldest discarded input.
90
+ positions = k + K - 6
91
+ values = tl.gather(old, tl.broadcast_to(tl.minimum(positions, 7)[None, :], (BLOCK, 8)), 1)
92
+ values = tl.where(k[None, :] == 6, x[:, None], values)
93
+ weights = tl.load(W + d[:, None] * 7 + k[None, :], (d[:, None] < D) & (k[None, :] < 7), other=0)
94
+ acc = tl.sum(values.to(tl.float32) * weights.to(tl.float32), 1)
95
+ if HAS_BIAS:
96
+ acc += tl.load(BIAS + d, d < D, other=0).to(tl.float32)
97
+ tl.store(STATE + b * S0 + d[:, None] * S1 + k[None, :] * S2, shifted, (d[:, None] < D) & (k[None, :] < K))
98
+ tl.store(Y + b * D + d, acc, d < D)
99
+
100
+
101
+ def can_use_lfm3_conv(hidden_states, weight):
102
+ return (
103
+ is_triton_available()
104
+ and hasattr(tl, "gather") # the decode kernel needs Triton >= 3.3
105
+ and hidden_states.is_cuda
106
+ and not torch.is_grad_enabled()
107
+ and hidden_states.dtype == weight.dtype
108
+ and hidden_states.dtype in (torch.float16, torch.bfloat16, torch.float32)
109
+ and weight.shape[-1] == 7
110
+ )
111
+
112
+
113
+ def lfm3_causal_conv1d(hidden_states, weight, bias=None, seq_idx=None):
114
+ batch, channels, length = hidden_states.shape
115
+ # the kernels index every tensor by the batch and channel of `hidden_states` and cannot bound the others
116
+ if weight.shape != (channels, 7) or (bias is not None and bias.shape != (channels,)):
117
+ raise ValueError(
118
+ f"`weight` must have shape ({channels}, 7) and `bias` ({channels},), got {tuple(weight.shape)}"
119
+ )
120
+ if seq_idx is not None and (
121
+ seq_idx.ndim != 2 or seq_idx.shape[0] not in (1, batch) or seq_idx.dtype not in (torch.int32, torch.int64)
122
+ ):
123
+ raise ValueError(
124
+ f"`seq_idx` must be an int32 or int64 tensor of shape (1, length) or ({batch}, length), got "
125
+ f"{seq_idx.dtype} {tuple(seq_idx.shape)}"
126
+ )
127
+ if any(t is not None and t.device != hidden_states.device for t in (weight, bias, seq_idx)):
128
+ raise ValueError(
129
+ f"`weight`, `bias` and `seq_idx` must be on the device of `hidden_states` ({hidden_states.device})"
130
+ )
131
+ output = torch.empty_like(hidden_states, memory_format=torch.contiguous_format)
132
+ # Triton launches on the current device; the model may live on another one (`device_map`, `.to("cuda:1")`).
133
+ with torch.cuda.device(hidden_states.device):
134
+ _prefill[(batch, channels, triton.cdiv(length, 128))](
135
+ hidden_states,
136
+ # the kernels index `bias` and `weight` densely
137
+ weight.contiguous(),
138
+ bias.contiguous() if bias is not None else None,
139
+ seq_idx,
140
+ output,
141
+ length,
142
+ channels,
143
+ *hidden_states.stride(),
144
+ *(
145
+ (0 if seq_idx.shape[0] == 1 else seq_idx.stride(0), seq_idx.stride(1))
146
+ if seq_idx is not None
147
+ else (0, 0)
148
+ ),
149
+ seq_idx.shape[-1] if seq_idx is not None else 0,
150
+ bias is not None,
151
+ seq_idx is not None,
152
+ 128,
153
+ enable_fp_fusion=False,
154
+ )
155
+ return output
156
+
157
+
158
+ def lfm3_causal_conv1d_update(hidden_states, conv_state, weight, bias=None):
159
+ batch, channels, length = hidden_states.shape
160
+ # the kernel addresses one position per row and bounds nothing by the other tensors' shapes
161
+ if length != 1:
162
+ raise ValueError(f"`hidden_states` must hold one position per row, got {tuple(hidden_states.shape)}")
163
+ # the 8-wide tile holds a 6- or 7-wide state (see `positions` in `_decode`)
164
+ if (
165
+ conv_state.ndim != 3
166
+ or conv_state.shape[:2] != (batch, channels)
167
+ or conv_state.shape[-1] not in (6, 7)
168
+ or weight.shape != (channels, 7)
169
+ ):
170
+ raise ValueError(
171
+ f"`conv_state` must have shape ({batch}, {channels}, 6 or 7) and `weight` ({channels}, 7), got "
172
+ f"{tuple(conv_state.shape)} and {tuple(weight.shape)}"
173
+ )
174
+ if bias is not None and bias.shape != (channels,):
175
+ raise ValueError(f"`bias` must have shape ({channels},), got {tuple(bias.shape)}")
176
+ # the kernel writes the state in place: rows or channels sharing memory (an `expand`ed cache) would race
177
+ extent = 0
178
+ for stride, size in sorted(zip(conv_state.stride(), conv_state.shape)):
179
+ if size > 1 and stride <= extent:
180
+ raise ValueError(f"`conv_state` must not overlap itself, got strides {conv_state.stride()}")
181
+ extent += stride * (size - 1)
182
+ if any(t is not None and t.device != hidden_states.device for t in (conv_state, weight, bias)):
183
+ raise ValueError(
184
+ f"`conv_state`, `weight` and `bias` must be on the device of `hidden_states` ({hidden_states.device})"
185
+ )
186
+ output = torch.empty_like(hidden_states, memory_format=torch.contiguous_format)
187
+ with torch.cuda.device(hidden_states.device):
188
+ _decode[(batch, triton.cdiv(channels, 32))](
189
+ hidden_states,
190
+ weight.contiguous(),
191
+ bias.contiguous() if bias is not None else None,
192
+ conv_state,
193
+ output,
194
+ channels,
195
+ conv_state.shape[-1],
196
+ *hidden_states.stride()[:2],
197
+ *conv_state.stride(),
198
+ bias is not None,
199
+ 32,
200
+ enable_fp_fusion=False,
201
+ )
202
+ return output
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:84acf7ea51c935b15da28ae07fa1d89e17e21c1017db07d8a8bed0cf0f2cdb0c
3
+ size 3836417920
modeling_lfm3.py ADDED
@@ -0,0 +1,881 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2026 The HuggingFace Team. All rights reserved.
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+ import copy
15
+ from collections.abc import Callable
16
+
17
+ import torch
18
+ import torch.nn.functional as F
19
+ from torch import nn
20
+
21
+ from transformers import initialization as init
22
+ from transformers.activations import ACT2FN
23
+ from transformers.cache_utils import Cache, DynamicCache
24
+ from transformers.configuration_utils import PreTrainedConfig
25
+ from transformers.generation import GenerationMixin
26
+ from transformers.integrations import (
27
+ use_kernel_forward_from_hub,
28
+ use_kernel_func_from_hub_with_fallback,
29
+ use_kernelized_func,
30
+ )
31
+ from transformers.integrations.accelerate import force_accelerate_hooks
32
+ from transformers.masking_utils import (
33
+ create_causal_mask,
34
+ create_recurrent_attention_mask,
35
+ create_sliding_window_causal_mask,
36
+ )
37
+ from transformers.modeling_layers import GradientCheckpointingLayer
38
+ from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast
39
+ from transformers.modeling_rope_utils import ROPE_INIT_FUNCTIONS, dynamic_rope_update
40
+ from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
41
+ from transformers.processing_utils import Unpack
42
+ from transformers.utils import TransformersKwargs, auto_docstring, can_return_tuple, torch_compilable_check
43
+ from transformers.utils.generic import merge_with_config_defaults
44
+ from transformers.utils.import_utils import is_tracing
45
+ from transformers.utils.output_capturing import capture_outputs
46
+
47
+ from .configuration_lfm3 import Lfm3Config
48
+ from .lfm3_conv import can_use_lfm3_conv, lfm3_causal_conv1d, lfm3_causal_conv1d_update
49
+
50
+
51
+ @use_kernel_forward_from_hub("RMSNorm")
52
+ class Lfm3RMSNorm(nn.Module):
53
+ def __init__(self, hidden_size, eps: float = 1e-6) -> None:
54
+ """
55
+ Lfm3RMSNorm is equivalent to T5LayerNorm
56
+ """
57
+ super().__init__()
58
+ self.weight = nn.Parameter(torch.ones(hidden_size))
59
+ self.variance_epsilon = eps
60
+
61
+ def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
62
+ input_dtype = hidden_states.dtype
63
+ hidden_states = hidden_states.to(torch.float32)
64
+ variance = hidden_states.pow(2).mean(-1, keepdim=True)
65
+ hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
66
+ return self.weight * hidden_states.to(input_dtype)
67
+
68
+ def extra_repr(self):
69
+ return f"{tuple(self.weight.shape)}, eps={self.variance_epsilon}"
70
+
71
+
72
+ class Lfm3RotaryEmbedding(nn.Module):
73
+ """
74
+ Rotary embedding per attention layer type, like Gemma 3; `conv` layers register no buffers. Partial RoPE keeps the
75
+ frequency progression of the full head dimension and zeroes the tail instead of shrinking the rotated dimension.
76
+ """
77
+
78
+ def __init__(self, config: Lfm3Config, device=None):
79
+ super().__init__()
80
+ self.max_seq_len_cached = config.max_position_embeddings
81
+ self.original_max_seq_len = config.max_position_embeddings
82
+ self.config = config
83
+ self.layer_types = sorted(set(config.layer_types) & set(config.rope_parameters))
84
+ self.rope_type = {}
85
+ for layer_type in self.layer_types:
86
+ rope_params = self.config.rope_parameters[layer_type]
87
+ self.rope_type[layer_type] = rope_params["rope_type"]
88
+ curr_inv_freq, curr_attention_scaling = self.compute_rope_parameters(config, device, layer_type=layer_type)
89
+ setattr(self, f"{layer_type}_inv_freq", nn.Buffer(curr_inv_freq, persistent=False))
90
+ setattr(self, f"{layer_type}_original_inv_freq", nn.Buffer(curr_inv_freq.clone(), persistent=False))
91
+ setattr(self, f"{layer_type}_attention_scaling", curr_attention_scaling)
92
+
93
+ @staticmethod
94
+ def compute_default_rope_parameters(
95
+ config: Lfm3Config, device=None, layer_type: str | None = None, **kwargs
96
+ ) -> tuple[torch.Tensor, float]:
97
+ """
98
+ Inverse frequencies of the default RoPE with LFM3's zero-tail partial rotation: the first
99
+ `partial_rotary_factor * head_dim // 2` frequencies follow the progression of the full head dimension and the
100
+ rest are zero, leaving those dimensions unrotated.
101
+
102
+ Returns:
103
+ Tuple of (`torch.Tensor`, `float`): the inverse frequencies and the attention scaling factor (always 1).
104
+ """
105
+ base = config.rope_parameters[layer_type]["rope_theta"]
106
+ partial_rotary_factor = config.rope_parameters[layer_type].get("partial_rotary_factor", 1.0)
107
+ dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
108
+ rope_angles = int(partial_rotary_factor * dim) // 2
109
+ inv_freq = 1.0 / (base ** (torch.arange(0, 2 * rope_angles, 2, dtype=torch.float) / dim))
110
+ inv_freq = torch.cat((inv_freq, torch.zeros(dim // 2 - rope_angles, dtype=torch.float)))
111
+ return inv_freq.to(device), 1.0
112
+
113
+ @torch.no_grad()
114
+ @dynamic_rope_update # power user: used with advanced RoPE types (e.g. dynamic rope)
115
+ def forward(self, x, position_ids, layer_type):
116
+ inv_freq = getattr(self, f"{layer_type}_inv_freq")
117
+ attention_scaling = getattr(self, f"{layer_type}_attention_scaling")
118
+
119
+ freqs = position_ids[..., None].float() * inv_freq.to(device=x.device, dtype=torch.float)
120
+ emb = torch.cat((freqs, freqs), dim=-1)
121
+ cos = emb.cos() * attention_scaling
122
+ sin = emb.sin() * attention_scaling
123
+
124
+ return cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
125
+
126
+ @classmethod
127
+ def compute_rope_parameters(
128
+ cls, config: Lfm3Config, device=None, layer_type: str | None = None
129
+ ) -> tuple[torch.Tensor, float]:
130
+ """Inverse frequencies of any RoPE type in the zero-tail layout: scaled types are evaluated over the full head
131
+ dimension and truncated to the rotated prefix."""
132
+ rope_params = config.rope_parameters[layer_type]
133
+ if rope_params["rope_type"] == "default":
134
+ return cls.compute_default_rope_parameters(config, device, layer_type=layer_type)
135
+ full_config = copy.copy(config)
136
+ full_config.rope_parameters = {
137
+ **config.rope_parameters,
138
+ layer_type: {**rope_params, "partial_rotary_factor": 1.0},
139
+ }
140
+ inv_freq, attention_factor = ROPE_INIT_FUNCTIONS[rope_params["rope_type"]](
141
+ full_config, device, layer_type=layer_type
142
+ )
143
+ dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
144
+ rope_angles = int(rope_params.get("partial_rotary_factor", 1.0) * dim) // 2
145
+ inv_freq = torch.cat((inv_freq[:rope_angles], inv_freq.new_zeros(dim // 2 - rope_angles)))
146
+ return inv_freq, attention_factor
147
+
148
+
149
+ class Lfm3MLP(nn.Module):
150
+ def __init__(self, config: Lfm3Config):
151
+ super().__init__()
152
+ self.w1 = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
153
+ self.w3 = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
154
+ self.w2 = nn.Linear(config.intermediate_size, config.hidden_size, bias=False)
155
+
156
+ def forward(self, x):
157
+ return self.w2(F.silu(self.w1(x)) * self.w3(x))
158
+
159
+
160
+ def rotate_half(x):
161
+ """Rotates half the hidden dims of the input."""
162
+ x1 = x[..., : x.shape[-1] // 2]
163
+ x2 = x[..., x.shape[-1] // 2 :]
164
+ return torch.cat((-x2, x1), dim=-1)
165
+
166
+
167
+ def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
168
+ """
169
+ This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
170
+ num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
171
+ """
172
+ batch, num_key_value_heads, slen, head_dim = hidden_states.shape
173
+ if n_rep == 1:
174
+ return hidden_states
175
+ hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
176
+ return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
177
+
178
+
179
+ def eager_attention_forward(
180
+ module: nn.Module,
181
+ query: torch.Tensor,
182
+ key: torch.Tensor,
183
+ value: torch.Tensor,
184
+ attention_mask: torch.Tensor | None,
185
+ scaling: float,
186
+ dropout: float = 0.0,
187
+ **kwargs: Unpack[TransformersKwargs],
188
+ ):
189
+ key_states = repeat_kv(key, module.num_key_value_groups)
190
+ value_states = repeat_kv(value, module.num_key_value_groups)
191
+
192
+ attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
193
+ if attention_mask is not None:
194
+ attn_weights = attn_weights + attention_mask
195
+
196
+ attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
197
+ attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
198
+ attn_output = torch.matmul(attn_weights, value_states)
199
+ attn_output = attn_output.transpose(1, 2).contiguous()
200
+
201
+ return attn_output, attn_weights
202
+
203
+
204
+ def apply_rotary_pos_emb(q, k, cos, sin, unsqueeze_dim=1):
205
+ """Applies RoPE to the query and key tensors in float32, like the reference implementation."""
206
+ cos = cos.unsqueeze(unsqueeze_dim).float()
207
+ sin = sin.unsqueeze(unsqueeze_dim).float()
208
+ q_embed = (q.float() * cos) + (rotate_half(q.float()) * sin)
209
+ k_embed = (k.float() * cos) + (rotate_half(k.float()) * sin)
210
+ return q_embed.to(q.dtype), k_embed.to(k.dtype)
211
+
212
+
213
+ class Lfm3Attention(nn.Module):
214
+ """Llama attention with QK RMSNorm, a sigmoid output gate per head, a sliding window on `sliding_attention`
215
+ layers and RoPE in float32."""
216
+
217
+ def __init__(self, config: Lfm3Config, layer_idx: int):
218
+ super().__init__()
219
+ self.config = config
220
+ self.layer_idx = layer_idx
221
+ self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
222
+ self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
223
+ self.scaling = self.head_dim**-0.5
224
+ self.is_causal = True
225
+ self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
226
+ self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
227
+ self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
228
+ self.o_proj = nn.Linear(config.num_attention_heads * self.head_dim, config.hidden_size, bias=False)
229
+ self.q_norm = Lfm3RMSNorm(self.head_dim, eps=config.norm_eps)
230
+ self.k_norm = Lfm3RMSNorm(self.head_dim, eps=config.norm_eps)
231
+ self.gate_proj = nn.Linear(config.hidden_size, config.num_attention_heads, bias=False)
232
+ self.sliding_window = config.sliding_window if config.layer_types[layer_idx] == "sliding_attention" else None
233
+
234
+ def forward(
235
+ self,
236
+ hidden_states: torch.Tensor,
237
+ position_embeddings: tuple[torch.Tensor, torch.Tensor],
238
+ attention_mask: torch.Tensor | None,
239
+ past_key_values: Cache | None = None,
240
+ **kwargs: Unpack[TransformersKwargs],
241
+ ) -> tuple[torch.Tensor, torch.Tensor | None]:
242
+ input_shape = hidden_states.shape[:-1]
243
+ hidden_shape = (*input_shape, -1, self.head_dim)
244
+
245
+ query_states = self.q_norm(self.q_proj(hidden_states).view(*hidden_shape)).transpose(1, 2)
246
+ key_states = self.k_norm(self.k_proj(hidden_states).view(*hidden_shape)).transpose(1, 2)
247
+ value_states = self.v_proj(hidden_states).view(*hidden_shape).transpose(1, 2)
248
+
249
+ cos, sin = position_embeddings
250
+ query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
251
+
252
+ if past_key_values is not None:
253
+ key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx)
254
+
255
+ attention_interface: Callable = ALL_ATTENTION_FUNCTIONS.get_interface(
256
+ self.config._attn_implementation, eager_attention_forward
257
+ )
258
+
259
+ attn_output, attn_weights = attention_interface(
260
+ self,
261
+ query_states,
262
+ key_states,
263
+ value_states,
264
+ attention_mask,
265
+ dropout=0.0,
266
+ scaling=self.scaling,
267
+ sliding_window=self.sliding_window,
268
+ **kwargs,
269
+ )
270
+ gate = torch.sigmoid(self.gate_proj(hidden_states)).unsqueeze(-1)
271
+ attn_output = (attn_output * gate).reshape(*input_shape, -1).contiguous()
272
+ attn_output = self.o_proj(attn_output)
273
+ return attn_output, attn_weights
274
+
275
+
276
+ def apply_mask_to_padding_states(hidden_states, attention_mask):
277
+ """
278
+ Tunes out the hidden states for padding tokens, see https://github.com/state-spaces/mamba/issues/66
279
+ """
280
+ # NOTE: attention mask is a 2D boolean tensor
281
+ if attention_mask is not None:
282
+ dtype = hidden_states.dtype
283
+ hidden_states = (hidden_states * attention_mask[:, :, None]).to(dtype)
284
+
285
+ return hidden_states
286
+
287
+
288
+ @use_kernel_func_from_hub_with_fallback("causal_conv1d_update", "causal_conv1d")
289
+ def causal_conv1d_update(
290
+ hidden_states: torch.Tensor,
291
+ conv_state: torch.Tensor,
292
+ weight: nn.Parameter,
293
+ bias: nn.Parameter | None = None,
294
+ activation: str | None = None,
295
+ ):
296
+ _, hidden_size, seq_len = hidden_states.shape
297
+ state_len = conv_state.shape[-1]
298
+
299
+ hidden_states_new = torch.cat([conv_state, hidden_states], dim=-1).to(weight.dtype)
300
+ conv_state.copy_(hidden_states_new[:, :, -state_len:])
301
+ out = F.conv1d(hidden_states_new, weight.unsqueeze(1), bias, padding=0, groups=hidden_size)
302
+ out = out[:, :, -seq_len:]
303
+ if activation is not None:
304
+ out = ACT2FN[activation](out)
305
+ return out.to(hidden_states.dtype)
306
+
307
+
308
+ @use_kernel_func_from_hub_with_fallback("causal_conv1d_fn", "causal_conv1d")
309
+ def causal_conv1d_fn(
310
+ hidden_states: torch.Tensor,
311
+ weight: nn.Parameter,
312
+ bias: nn.Parameter | None = None,
313
+ activation: str | None = None,
314
+ **kwargs,
315
+ ):
316
+ _, hidden_size, seq_len = hidden_states.shape
317
+ padding = weight.shape[-1] - 1
318
+
319
+ out = F.conv1d(
320
+ hidden_states.to(weight.dtype),
321
+ weight=weight.unsqueeze(1),
322
+ bias=bias,
323
+ padding=padding,
324
+ groups=hidden_size,
325
+ )[:, :, :seq_len]
326
+ if activation is not None:
327
+ out = ACT2FN[activation](out)
328
+ return out.to(hidden_states.dtype)
329
+
330
+
331
+ @use_kernelized_func([causal_conv1d_update, causal_conv1d_fn])
332
+ class Lfm3ShortConv(nn.Module):
333
+ """LFM2 gated short convolution with packed-document boundaries (`seq_idx`), the padding mask re-applied after a
334
+ biased `in_proj`, and width-7 Triton inference kernels with a PyTorch fallback."""
335
+
336
+ def __init__(
337
+ self,
338
+ config: Lfm3Config,
339
+ layer_idx: int,
340
+ ):
341
+ super().__init__()
342
+ self.config = config
343
+ self.layer_idx = layer_idx
344
+ self.conv_kernel_size = config.conv_L_cache
345
+ self.bias = config.conv_bias
346
+
347
+ self.conv = nn.Conv1d(
348
+ in_channels=config.hidden_size,
349
+ out_channels=config.hidden_size,
350
+ kernel_size=self.conv_kernel_size,
351
+ groups=config.hidden_size,
352
+ bias=self.bias,
353
+ padding=self.conv_kernel_size - 1,
354
+ )
355
+ self.in_proj = nn.Linear(config.hidden_size, 3 * config.hidden_size, bias=self.bias)
356
+ self.out_proj = nn.Linear(config.hidden_size, config.hidden_size, bias=self.bias)
357
+
358
+ self.layer_type = config.layer_types[layer_idx]
359
+
360
+ @force_accelerate_hooks("conv")
361
+ def forward(
362
+ self,
363
+ hidden_states: torch.Tensor,
364
+ past_key_values: Cache | None = None,
365
+ attention_mask: torch.Tensor | None = None,
366
+ seq_idx: torch.IntTensor | None = None,
367
+ ):
368
+ seq_len = hidden_states.shape[1]
369
+
370
+ hidden_states = apply_mask_to_padding_states(hidden_states, attention_mask)
371
+ BCx = self.in_proj(hidden_states).transpose(-1, -2)
372
+ B, C, x = BCx.chunk(3, dim=-2)
373
+ hidden_states = B * x
374
+ if attention_mask is not None:
375
+ # with `conv_bias` the projection makes the padded positions non-zero again
376
+ hidden_states = hidden_states * attention_mask[:, None, :].to(hidden_states.dtype)
377
+
378
+ use_precomputed_states = past_key_values is not None and past_key_values.has_previous_state(self.layer_idx)
379
+ use_kernels = 2 <= self.conv_kernel_size <= 4
380
+ use_wide_kernel = can_use_lfm3_conv(hidden_states, self.conv.weight)
381
+
382
+ if use_precomputed_states and seq_len == 1 and not past_key_values.layers[self.layer_idx].record_past:
383
+ conv_state = past_key_values.layers[self.layer_idx].conv_states[0]
384
+ if use_wide_kernel and conv_state.shape[-1] in (6, 7):
385
+ hidden_states = lfm3_causal_conv1d_update(
386
+ hidden_states, conv_state, self.conv.weight.squeeze(1), self.conv.bias
387
+ )
388
+ elif use_kernels:
389
+ hidden_states = causal_conv1d_update(
390
+ hidden_states, conv_state, self.conv.weight.squeeze(1), self.conv.bias
391
+ )
392
+ else:
393
+ hidden_states = torch.cat([conv_state, hidden_states], dim=-1)
394
+ conv_state.copy_(hidden_states[:, :, -conv_state.shape[-1] :])
395
+ hidden_states = self._conv(hidden_states)[:, :, -seq_len:]
396
+ else:
397
+ if past_key_values is not None and seq_idx is not None:
398
+ # `seq_idx` is only set on an empty cache, and the decode steps read the cached window without it:
399
+ # only the last document (after any positional left padding) may enter the window
400
+ last_document = (seq_idx == seq_idx[:, -1:])[:, None].to(hidden_states.dtype)
401
+ past_key_values.update_conv_state(
402
+ hidden_states * last_document, self.layer_idx, conv_kernel_size=self.conv_kernel_size
403
+ )
404
+ elif past_key_values is not None:
405
+ hidden_states = past_key_values.update_conv_state(
406
+ hidden_states, self.layer_idx, conv_kernel_size=self.conv_kernel_size
407
+ )
408
+
409
+ if use_wide_kernel:
410
+ hidden_states = lfm3_causal_conv1d(hidden_states, self.conv.weight.squeeze(1), self.conv.bias, seq_idx)
411
+ elif seq_idx is not None:
412
+ hidden_states = self._segmented_conv(hidden_states, seq_idx)
413
+ elif use_kernels:
414
+ hidden_states = causal_conv1d_fn(hidden_states, self.conv.weight.squeeze(1), self.conv.bias)
415
+ else:
416
+ hidden_states = self._conv(hidden_states)
417
+
418
+ # Drop the additional previous states
419
+ if past_key_values is not None:
420
+ hidden_states = hidden_states[:, :, -seq_len:]
421
+
422
+ y = C * hidden_states
423
+ y = y.transpose(-1, -2).contiguous()
424
+ y = self.out_proj(y)
425
+ return y
426
+
427
+ def _conv(self, hidden_states: torch.Tensor) -> torch.Tensor:
428
+ """Causal depthwise conv in torch: the `causal_conv1d` kernels only accept widths 2 to 4."""
429
+ conv_out = self.conv(hidden_states.to(self.conv.weight.dtype))[..., : hidden_states.shape[-1]]
430
+ return conv_out.to(hidden_states.dtype)
431
+
432
+ def _segmented_conv(self, hidden_states: torch.Tensor, seq_idx: torch.Tensor) -> torch.Tensor:
433
+ """Causal depthwise conv that only reads positions sharing the current token's document index."""
434
+ kernel_size = self.conv_kernel_size
435
+ seq_idx = F.pad(seq_idx, (hidden_states.shape[-1] - seq_idx.shape[-1] + kernel_size - 1, 0), value=-1)
436
+ windows = F.pad(hidden_states, (kernel_size - 1, 0)).unfold(-1, kernel_size, 1)
437
+ seq_idx_windows = seq_idx.unfold(-1, kernel_size, 1)
438
+ same_document = (seq_idx_windows == seq_idx_windows[..., -1:]).unsqueeze(1)
439
+ conv_out = torch.einsum("bdtk,dk->bdt", windows * same_document.to(windows.dtype), self.conv.weight.squeeze(1))
440
+ if self.bias:
441
+ conv_out = conv_out + self.conv.bias[None, :, None]
442
+ return conv_out.to(hidden_states.dtype)
443
+
444
+
445
+ class Lfm3DecoderLayer(GradientCheckpointingLayer):
446
+ def __init__(self, config: Lfm3Config, layer_idx: int):
447
+ super().__init__()
448
+ self.is_attention_layer = config.layer_types[layer_idx] in config._attention_layer_types
449
+ self.use_sandwich_norm = config.use_sandwich_norm
450
+
451
+ if self.is_attention_layer:
452
+ self.self_attn = Lfm3Attention(config, layer_idx)
453
+ else:
454
+ self.conv = Lfm3ShortConv(config, layer_idx)
455
+ self.feed_forward = Lfm3MLP(config)
456
+ self.operator_norm = Lfm3RMSNorm(config.hidden_size, eps=config.norm_eps)
457
+ self.ffn_norm = Lfm3RMSNorm(config.hidden_size, eps=config.norm_eps)
458
+ if self.use_sandwich_norm:
459
+ self.post_operator_norm = Lfm3RMSNorm(config.hidden_size, eps=config.norm_eps)
460
+ self.post_ffn_norm = Lfm3RMSNorm(config.hidden_size, eps=config.norm_eps)
461
+
462
+ def forward(
463
+ self,
464
+ hidden_states: torch.Tensor,
465
+ position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
466
+ attention_mask: torch.Tensor | None = None,
467
+ position_ids: torch.LongTensor | None = None,
468
+ past_key_values: Cache | None = None,
469
+ seq_idx: torch.IntTensor | None = None,
470
+ **kwargs,
471
+ ) -> torch.Tensor:
472
+ residual = hidden_states
473
+ if self.is_attention_layer:
474
+ hidden_states, _ = self.self_attn(
475
+ hidden_states=self.operator_norm(hidden_states),
476
+ position_embeddings=position_embeddings,
477
+ attention_mask=attention_mask,
478
+ position_ids=position_ids,
479
+ past_key_values=past_key_values,
480
+ **kwargs,
481
+ )
482
+ else:
483
+ hidden_states = self.conv(
484
+ hidden_states=self.operator_norm(hidden_states),
485
+ past_key_values=past_key_values,
486
+ attention_mask=attention_mask,
487
+ seq_idx=seq_idx,
488
+ )
489
+ if self.use_sandwich_norm:
490
+ hidden_states = self.post_operator_norm(hidden_states)
491
+ hidden_states = hidden_states + residual
492
+
493
+ ffn_out = self.feed_forward(self.ffn_norm(hidden_states))
494
+ if self.use_sandwich_norm:
495
+ ffn_out = self.post_ffn_norm(ffn_out)
496
+ hidden_states = hidden_states + ffn_out
497
+
498
+ return hidden_states
499
+
500
+
501
+ def create_masks_by_layer_type(
502
+ config: PreTrainedConfig,
503
+ inputs_embeds: torch.Tensor,
504
+ attention_mask: torch.Tensor | None,
505
+ past_key_values: Cache | None,
506
+ position_ids: torch.Tensor | None = None,
507
+ ) -> dict:
508
+ """The cache sizes each mask from the layer it is asked about, and the generic mask utilities ask about the first
509
+ non-sliding layer, which for a leading `conv` layer redirects to the first attention layer of any type: name the
510
+ first layer of each mask's own type instead."""
511
+ config = config.get_text_config()
512
+ mask_kwargs = {
513
+ "config": config,
514
+ "inputs_embeds": inputs_embeds,
515
+ "attention_mask": attention_mask,
516
+ "past_key_values": past_key_values,
517
+ "position_ids": position_ids,
518
+ }
519
+ layer_types, causal_mask_mapping = config.layer_types, {}
520
+ if "full_attention" in layer_types:
521
+ layer_idx = layer_types.index("full_attention")
522
+ causal_mask_mapping["full_attention"] = create_causal_mask(**mask_kwargs, layer_idx=layer_idx)
523
+ if "sliding_attention" in layer_types:
524
+ layer_idx = layer_types.index("sliding_attention")
525
+ causal_mask_mapping["sliding_attention"] = create_sliding_window_causal_mask(
526
+ **mask_kwargs, layer_idx=layer_idx
527
+ )
528
+ if "conv" in layer_types:
529
+ causal_mask_mapping["conv"] = create_recurrent_attention_mask(**mask_kwargs)
530
+ return causal_mask_mapping
531
+
532
+
533
+ @auto_docstring
534
+ class Lfm3PreTrainedModel(PreTrainedModel):
535
+ config: Lfm3Config
536
+ base_model_prefix = "model"
537
+ supports_gradient_checkpointing = True
538
+ _no_split_modules = ["Lfm3DecoderLayer"]
539
+ _skip_keys_device_placement = ["past_key_values"]
540
+ _supports_flash_attn = True
541
+ _supports_sdpa = True
542
+ _supports_flex_attn = True
543
+
544
+ _can_compile_fullgraph = True
545
+ _supports_attention_backend = True
546
+ _can_record_outputs = {
547
+ "hidden_states": Lfm3DecoderLayer,
548
+ "attentions": Lfm3Attention,
549
+ }
550
+
551
+ @torch.no_grad()
552
+ def _init_weights(self, module):
553
+ super()._init_weights(module)
554
+ if isinstance(module, Lfm3RotaryEmbedding):
555
+ for layer_type in module.layer_types:
556
+ curr_inv_freq, _ = module.compute_rope_parameters(module.config, layer_type=layer_type)
557
+ init.copy_(getattr(module, f"{layer_type}_inv_freq"), curr_inv_freq)
558
+ init.copy_(getattr(module, f"{layer_type}_original_inv_freq"), curr_inv_freq)
559
+
560
+ @staticmethod
561
+ def create_masks_for_generate(
562
+ config: PreTrainedConfig,
563
+ inputs_embeds: torch.Tensor,
564
+ attention_mask: torch.Tensor | None,
565
+ past_key_values: Cache | None,
566
+ position_ids: torch.Tensor | None = None,
567
+ **kwargs,
568
+ ) -> dict:
569
+ return create_masks_by_layer_type(config, inputs_embeds, attention_mask, past_key_values, position_ids)
570
+
571
+
572
+ def packed_under_padding(position_ids: torch.Tensor, padding_mask: torch.Tensor, filled) -> torch.Tensor:
573
+ """Padding repeats position 0, so packing is a restart between two kept tokens, or a kept position 0 on a filled
574
+ cache whose row holds a kept token; attention under a padding mask cannot isolate it."""
575
+ restarts = torch.diff(position_ids, prepend=position_ids[:, :1] - 1, dim=-1) != 1
576
+ valid = padding_mask[:, -restarts.shape[-1] :].bool()
577
+ packed = restarts[:, 1:] & valid[:, 1:] & valid[:, :-1]
578
+ if filled:
579
+ # a row whose cached tokens are all padding (chunked prefill of a left-padded batch) has no document to pack
580
+ # against; a mask covering only the chunk keeps the strict rule
581
+ cached = padding_mask[:, : padding_mask.shape[-1] - restarts.shape[-1]]
582
+ cached_valid = cached.bool().any(-1, keepdim=True) if cached.shape[-1] else torch.ones_like(valid[:, :1])
583
+ packed = packed | (valid[:, :1] & (position_ids[:, :1] == 0) & cached_valid)
584
+ return packed
585
+
586
+
587
+ @auto_docstring
588
+ class Lfm3Model(Lfm3PreTrainedModel):
589
+ """LFM2 model with sliding-attention layers as a third layer type: the mask and the rotary embedding are built per
590
+ layer type, and packed documents (`position_ids` restarting) reset the short-conv window through `seq_idx`."""
591
+
592
+ def __init__(self, config: Lfm3Config):
593
+ super().__init__(config)
594
+ self.padding_idx = config.pad_token_id
595
+ self.vocab_size = config.vocab_size
596
+
597
+ self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
598
+ self.layers = nn.ModuleList(
599
+ [Lfm3DecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
600
+ )
601
+ self.rotary_emb = Lfm3RotaryEmbedding(config=config)
602
+ self.gradient_checkpointing = False
603
+ self.embedding_norm = Lfm3RMSNorm(config.hidden_size, eps=config.norm_eps)
604
+
605
+ # Initialize weights and apply final processing
606
+ self.post_init()
607
+
608
+ @merge_with_config_defaults
609
+ @capture_outputs
610
+ @auto_docstring
611
+ def forward(
612
+ self,
613
+ input_ids: torch.LongTensor | None = None,
614
+ attention_mask: torch.Tensor | None = None,
615
+ position_ids: torch.LongTensor | None = None,
616
+ past_key_values: Cache | None = None,
617
+ inputs_embeds: torch.FloatTensor | None = None,
618
+ use_cache: bool | None = None,
619
+ **kwargs: Unpack[TransformersKwargs],
620
+ ) -> BaseModelOutputWithPast:
621
+ r"""
622
+ attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)` or `dict`, *optional*):
623
+ Mask to avoid performing attention on padding token indices (1 keeps a token, 0 masks it), or the mask
624
+ dict of `create_masks_for_generate`. A mask dict is used as given: `generate` refuses packed
625
+ `position_ids` before its prefill.
626
+ """
627
+ if (input_ids is None) ^ (inputs_embeds is not None):
628
+ raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
629
+ # `seq_idx` is a shared kwarg (`DataCollatorWithFlattening(return_seq_idx=True)`, read by Lfm2); LFM3 derives
630
+ # the documents from the restarting `position_ids` that come with it
631
+ if kwargs.pop("seq_idx", None) is not None and position_ids is None:
632
+ raise ValueError("Pass packed documents as `position_ids` restarting at 0; LFM3 does not read `seq_idx`")
633
+
634
+ if inputs_embeds is None:
635
+ inputs_embeds = self.embed_tokens(input_ids)
636
+
637
+ if use_cache and past_key_values is None:
638
+ past_key_values = DynamicCache(config=self.config)
639
+ filled = past_key_values is not None and (
640
+ past_key_values.get_seq_length() > 0
641
+ if any(t != "conv" for t in self.config.layer_types)
642
+ else past_key_values.has_previous_state()
643
+ )
644
+ packed_cache = getattr(past_key_values, "lfm3_packed", None)
645
+ if isinstance(packed_cache, torch.Tensor) or packed_cache:
646
+ # Reset layers have no history to mix. Ordinary caches carry a Python `False` and skip this, so the decode
647
+ # steps do no tensor work here; only a compiled packed prefill leaves a tensor flag, and `filled` stays
648
+ # tensor-valued for static caches.
649
+ packed_message = (
650
+ "A cache filled by a packed-document forward cannot be continued: its attention spans the documents"
651
+ )
652
+ packed_cache = packed_cache & filled if isinstance(packed_cache, torch.Tensor) else filled
653
+ if isinstance(packed_cache, torch.Tensor):
654
+ torch_compilable_check(~packed_cache, packed_message)
655
+ if not is_tracing(packed_cache):
656
+ past_key_values.lfm3_packed = bool(packed_cache)
657
+ elif packed_cache:
658
+ raise ValueError(packed_message)
659
+ else:
660
+ past_key_values.lfm3_packed = False
661
+
662
+ # The mask utilities detect packing only with `attention_mask=None`, so a multi-token forward with explicit
663
+ # positions drops the all-ones mask tokenizers emit (a data-dependent check, hence not when traced).
664
+ explicit_positions = position_ids is not None and inputs_embeds.shape[1] > 1
665
+ if (
666
+ explicit_positions
667
+ and isinstance(attention_mask, torch.Tensor)
668
+ and attention_mask.ndim == 2
669
+ and not is_tracing(position_ids)
670
+ and bool(attention_mask.all())
671
+ ):
672
+ attention_mask = None
673
+ seq_idx = None
674
+ if position_ids is None:
675
+ past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
676
+ position_ids = torch.arange(inputs_embeds.shape[1], device=inputs_embeds.device) + past_seen_tokens
677
+ position_ids = position_ids.unsqueeze(0)
678
+ elif explicit_positions and not isinstance(attention_mask, dict):
679
+ # a mask dict comes from `generate`, which refused packing before its prefill
680
+ if isinstance(attention_mask, torch.Tensor) and attention_mask.ndim == 2:
681
+ # data-dependent: a traced forward skips it (`generate` refuses packing eagerly before its prefill)
682
+ if not is_tracing(position_ids):
683
+ torch_compilable_check(
684
+ ~packed_under_padding(position_ids, attention_mask, filled),
685
+ "Packed documents (`position_ids` restarting) cannot be combined with an attention mask: "
686
+ "attention would not isolate the documents. Pass `attention_mask=None` without a static cache.",
687
+ )
688
+ else:
689
+ restarts = torch.diff(position_ids, prepend=position_ids[:, :1] - 1, dim=-1) != 1
690
+ document_ids = restarts.cumsum(-1)
691
+ if is_tracing(position_ids) or bool((document_ids[..., -1] > 0).any()):
692
+ seq_idx = document_ids.to(torch.int32)
693
+ # Packed documents are prefilled in one forward on an empty cache (cached tokens would join the first
694
+ # document), and their cache is never continued
695
+ if filled:
696
+ torch_compilable_check(
697
+ (~restarts[:, 1:]).all() & (position_ids[:, 0] != 0).all(),
698
+ "Packed documents must be prefilled in one forward; the cache already holds tokens",
699
+ )
700
+ # Ordinary continuation must retain the convolution window from the cached document.
701
+ seq_idx = None
702
+ if attention_mask is None:
703
+ # without a mask only the cacheless packed-sequence mask isolates repeated zero positions (left
704
+ # padding): a cache-sized mask and every later decode step would attend them, so they count too
705
+ packed = (document_ids[..., -1] > 0).any()
706
+ else:
707
+ # a prepared 4D mask is the caller's and carries the padding (repeated zero positions), but may
708
+ # or may not isolate documents, so the cache records the packing and is never continued
709
+ steps = torch.diff(position_ids, dim=-1)
710
+ packed = ((steps != 1) & ~((steps == 0) & (position_ids[:, :-1] == 0))).any()
711
+ if past_key_values is not None:
712
+ # resolved here, where the prefill already syncs, so the decode steps carry a Python flag: no
713
+ # per-step host sync, data-dependent op or CUDA-graph output on the cache
714
+ past_key_values.lfm3_packed = packed if is_tracing(position_ids) else bool(packed)
715
+
716
+ if not isinstance(causal_mask_mapping := attention_mask, dict):
717
+ mask_kwargs = {
718
+ "config": self.config,
719
+ "inputs_embeds": inputs_embeds,
720
+ "attention_mask": attention_mask,
721
+ "past_key_values": past_key_values,
722
+ "position_ids": position_ids,
723
+ }
724
+ # The mask utilities only isolate packed documents without a cache, and an empty cache adds nothing.
725
+ layer_types = self.config.layer_types
726
+ if seq_idx is not None and past_key_values is not None:
727
+ attention_layer = next((i for i, t in enumerate(layer_types) if t != "conv"), None)
728
+ if attention_layer is not None and past_key_values.get_seq_length(attention_layer) == 0:
729
+ if past_key_values.is_compileable:
730
+ # static layers always return their full length, so the masks are sized by the cache, which
731
+ # isolates neither documents nor positional-only left padding
732
+ torch_compilable_check(
733
+ ~packed,
734
+ "Packed documents, or left padding without an attention mask, cannot be prefilled into a "
735
+ "static cache: its masks are sized by the cache. Pass the padding as `attention_mask`, or "
736
+ "use a `DynamicCache` for packed documents",
737
+ )
738
+ else:
739
+ mask_kwargs["past_key_values"] = None
740
+ causal_mask_mapping = create_masks_by_layer_type(**mask_kwargs)
741
+
742
+ hidden_states = inputs_embeds
743
+ position_embeddings = {
744
+ layer_type: self.rotary_emb(hidden_states, position_ids=position_ids, layer_type=layer_type)
745
+ for layer_type in self.rotary_emb.layer_types
746
+ }
747
+
748
+ for i, decoder_layer in enumerate(self.layers[: self.config.num_hidden_layers]):
749
+ layer_type = self.config.layer_types[i]
750
+ hidden_states = decoder_layer(
751
+ hidden_states,
752
+ attention_mask=causal_mask_mapping[layer_type],
753
+ position_embeddings=position_embeddings.get(layer_type),
754
+ position_ids=position_ids,
755
+ past_key_values=past_key_values,
756
+ seq_idx=seq_idx,
757
+ **kwargs,
758
+ )
759
+
760
+ hidden_states = self.embedding_norm(hidden_states)
761
+
762
+ return BaseModelOutputWithPast(
763
+ last_hidden_state=hidden_states,
764
+ past_key_values=past_key_values,
765
+ )
766
+
767
+
768
+ def reject_packed_generation(config: PreTrainedConfig, input_ids, generation_config, model_kwargs, is_first_iteration):
769
+ """`generate` decodes from the cache its prefill fills, and no mask it builds isolates packed documents
770
+ (`position_ids` restarting): the prompt is refused before the prefill, one forward or every chunk, writes to it."""
771
+ position_ids, attention_mask = model_kwargs.get("position_ids"), model_kwargs.get("attention_mask")
772
+ cache = model_kwargs.get("past_key_values")
773
+ if cache is None or position_ids is None or position_ids.ndim != 2:
774
+ return
775
+ if attention_mask is not None and attention_mask.ndim != 2:
776
+ return
777
+ has_attention = any(t != "conv" for t in config.get_text_config().layer_types)
778
+ filled = bool(cache.get_seq_length() > 0 if has_attention else cache.has_previous_state())
779
+ inputs_embeds = model_kwargs.get("inputs_embeds") if is_first_iteration else None
780
+ length = input_ids.shape[1] if inputs_embeds is None else inputs_embeds.shape[1]
781
+ if generation_config.prefill_chunk_size is not None:
782
+ # the chunks run from the start of the prompt
783
+ position_ids = position_ids[:, :length]
784
+ else:
785
+ if filled and has_attention:
786
+ # the slice `_prefill` feeds: a prompt that spans the cache is cut to its uncached part
787
+ mask_length = getattr(generation_config, "_mask_length", -1)
788
+ if attention_mask is not None:
789
+ mask_length = attention_mask.shape[1]
790
+ if inputs_embeds is not None or mask_length == length:
791
+ length -= int(cache.get_seq_length())
792
+ position_ids = position_ids[:, position_ids.shape[1] - length :]
793
+ if attention_mask is None:
794
+ attention_mask = torch.ones_like(position_ids)
795
+ if bool(packed_under_padding(position_ids, attention_mask, filled).any()):
796
+ raise ValueError(
797
+ "`generate` cannot decode packed documents (`position_ids` restarting, or starting at 0 on a cache that "
798
+ "holds tokens): attention would span them. Generate each document in its own row of a left-padded batch."
799
+ )
800
+
801
+
802
+ @auto_docstring
803
+ class Lfm3ForCausalLM(Lfm3PreTrainedModel, GenerationMixin):
804
+ _tied_weights_keys = {"lm_head.weight": "model.embed_tokens.weight"}
805
+ _tp_plan = {"lm_head": "colwise_gather_output"}
806
+ _pp_plan = {"lm_head": (["hidden_states"], ["logits"])}
807
+ _fsdp_plan = {"lm_head": "keep_full_weight"}
808
+
809
+ def __init__(self, config):
810
+ super().__init__(config)
811
+ self.model = Lfm3Model(config)
812
+ self.vocab_size = config.vocab_size
813
+ self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
814
+
815
+ # Initialize weights and apply final processing
816
+ self.post_init()
817
+
818
+ @can_return_tuple
819
+ @auto_docstring
820
+ def forward(
821
+ self,
822
+ input_ids: torch.LongTensor | None = None,
823
+ attention_mask: torch.Tensor | None = None,
824
+ position_ids: torch.LongTensor | None = None,
825
+ past_key_values: Cache | None = None,
826
+ inputs_embeds: torch.FloatTensor | None = None,
827
+ labels: torch.LongTensor | None = None,
828
+ use_cache: bool | None = None,
829
+ logits_to_keep: int | torch.Tensor = 0,
830
+ **kwargs: Unpack[TransformersKwargs],
831
+ ) -> CausalLMOutputWithPast:
832
+ r"""
833
+ Example:
834
+
835
+ ```python
836
+ >>> from transformers import AutoTokenizer, Lfm3ForCausalLM
837
+
838
+ >>> model = Lfm3ForCausalLM.from_pretrained("LiquidAI/LFM3-1.7B")
839
+ >>> tokenizer = AutoTokenizer.from_pretrained("LiquidAI/LFM3-1.7B")
840
+
841
+ >>> inputs = tokenizer.apply_chat_template(
842
+ ... [{"role": "user", "content": "What is C. elegans?"}], add_generation_prompt=True, return_tensors="pt"
843
+ ... )
844
+ >>> generate_ids = model.generate(**inputs, max_new_tokens=64)
845
+ >>> tokenizer.decode(generate_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)
846
+ ```"""
847
+ outputs: BaseModelOutputWithPast = self.model(
848
+ input_ids=input_ids,
849
+ attention_mask=attention_mask,
850
+ position_ids=position_ids,
851
+ past_key_values=past_key_values,
852
+ inputs_embeds=inputs_embeds,
853
+ use_cache=use_cache,
854
+ **kwargs,
855
+ )
856
+
857
+ hidden_states = outputs.last_hidden_state
858
+ # Only compute necessary logits, and do not upcast them to float if we are not computing the loss
859
+ slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
860
+ logits = self.lm_head(hidden_states[:, slice_indices, :])
861
+
862
+ loss = None
863
+ if labels is not None:
864
+ loss = self.loss_function(logits=logits, labels=labels, vocab_size=self.config.vocab_size, **kwargs)
865
+
866
+ return CausalLMOutputWithPast(
867
+ loss=loss,
868
+ logits=logits,
869
+ past_key_values=outputs.past_key_values,
870
+ hidden_states=outputs.hidden_states,
871
+ attentions=outputs.attentions,
872
+ )
873
+
874
+ def _prefill(self, input_ids, generation_config, model_kwargs, is_first_iteration=True):
875
+ # every prefill passes here once before its first forward: a check per chunk would leave the earlier chunks
876
+ # in the cache, and a dynamic cache would only refuse packing at the first decode step
877
+ reject_packed_generation(self.config, input_ids, generation_config, model_kwargs, is_first_iteration)
878
+ return super()._prefill(input_ids, generation_config, model_kwargs, is_first_iteration=is_first_iteration)
879
+
880
+
881
+ __all__ = ["Lfm3ForCausalLM", "Lfm3Model", "Lfm3PreTrainedModel"]
modeling_lfm3_vl.py ADDED
@@ -0,0 +1,967 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2026 The HuggingFace Team. All rights reserved.
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+ from collections.abc import Callable
15
+ from dataclasses import dataclass
16
+
17
+ import torch
18
+ import torch.nn.functional as F
19
+ from torch import nn
20
+
21
+ from transformers.activations import ACT2FN
22
+ from transformers.cache_utils import Cache
23
+ from transformers.configuration_utils import PreTrainedConfig
24
+ from transformers.generation import GenerationMixin
25
+ from transformers.integrations import use_kernel_forward_from_hub
26
+ from transformers.masking_utils import (
27
+ create_causal_mask,
28
+ create_recurrent_attention_mask,
29
+ create_sliding_window_causal_mask,
30
+ )
31
+ from transformers.modeling_layers import GradientCheckpointingLayer
32
+ from transformers.modeling_outputs import BaseModelOutputWithPast, BaseModelOutputWithPooling, CausalLMOutputWithPast
33
+ from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
34
+ from transformers.processing_utils import Unpack
35
+ from transformers.utils import (
36
+ TransformersKwargs,
37
+ auto_docstring,
38
+ can_return_tuple,
39
+ torch_compilable_check,
40
+ )
41
+ from transformers.utils.generic import (
42
+ accepts_precomputed_kwargs,
43
+ is_flash_attention_requested,
44
+ maybe_autocast,
45
+ merge_with_config_defaults,
46
+ )
47
+ from transformers.utils.output_capturing import OutputRecorder, capture_outputs
48
+ from transformers.vision_utils import get_max_seqlen, get_vision_attention_seqlens, get_vision_position_ids
49
+
50
+ from .configuration_lfm3 import Lfm3Config
51
+ from .configuration_lfm3_vl import Lfm3VlConfig, Lfm3VlVisionConfig
52
+ from .lfm3_conv import can_use_lfm3_conv
53
+ from .modeling_lfm3 import Lfm3Model
54
+
55
+
56
+ @use_kernel_forward_from_hub("RMSNorm")
57
+ class Lfm3VlRMSNorm(nn.Module):
58
+ def __init__(self, hidden_size, eps: float = 1e-6) -> None:
59
+ """
60
+ Lfm3VlRMSNorm is equivalent to T5LayerNorm
61
+ """
62
+ super().__init__()
63
+ self.weight = nn.Parameter(torch.ones(hidden_size))
64
+ self.variance_epsilon = eps
65
+
66
+ def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
67
+ input_dtype = hidden_states.dtype
68
+ hidden_states = hidden_states.to(torch.float32)
69
+ variance = hidden_states.pow(2).mean(-1, keepdim=True)
70
+ hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
71
+ return self.weight * hidden_states.to(input_dtype)
72
+
73
+ def extra_repr(self):
74
+ return f"{tuple(self.weight.shape)}, eps={self.variance_epsilon}"
75
+
76
+
77
+ class Lfm3VlMLP(nn.Module):
78
+ def __init__(self, config: Lfm3VlConfig):
79
+ super().__init__()
80
+ self.w1 = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
81
+ self.w3 = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
82
+ self.w2 = nn.Linear(config.intermediate_size, config.hidden_size, bias=False)
83
+
84
+ def forward(self, x):
85
+ return self.w2(F.silu(self.w1(x)) * self.w3(x))
86
+
87
+
88
+ class Lfm3VlVisionRotaryEmbedding(nn.Module):
89
+ """
90
+ Simple axial 2D rope with same freqs used for H and W grids. The freqs are
91
+ pre-computed using `head-dim//4` which is later used to concat H and W positions.
92
+ The final angles rotate over the whole head dim, no partial rotation involved.
93
+ """
94
+
95
+ def __init__(self, config: Lfm3VlVisionConfig):
96
+ super().__init__()
97
+ self.config = config
98
+
99
+ self.rope_type = self.config.rope_parameters["rope_type"]
100
+ rope_init_fn: Callable = self.compute_axial_rope_parameters
101
+ if self.rope_type != "axial":
102
+ raise ValueError(f"{self.__class__.__name__} supports only axial rope, but requested {self.rope_type}")
103
+ inv_freq, self.attention_scaling = rope_init_fn(self.config)
104
+
105
+ self.inv_freq = nn.Buffer(inv_freq, persistent=False)
106
+ self.original_inv_freq = nn.Buffer(inv_freq.clone(), persistent=False)
107
+
108
+ @staticmethod
109
+ def compute_axial_rope_parameters(config: Lfm3VlVisionConfig, **kwargs) -> tuple[torch.Tensor, float]:
110
+ """Inverse frequencies of the axial RoPE over half the head dimension per axis, with an attention scaling of 1."""
111
+ base = config.rope_parameters["rope_theta"]
112
+ spatial_dim = config.hidden_size // config.num_attention_heads // 2
113
+ inv_freq = 1.0 / (base ** (torch.arange(0, spatial_dim, 2, dtype=torch.float) / spatial_dim))
114
+ return inv_freq, 1.0
115
+
116
+ @torch.no_grad()
117
+ def forward(self, x, position_ids):
118
+ # position_ids: (2, N) — row 0 = h coords, row 1 = w coords
119
+ position_ids_expanded = position_ids[..., None].float()
120
+ device_type = x.device.type if isinstance(x.device.type, str) else "cpu"
121
+ with maybe_autocast(device_type=device_type, enabled=False):
122
+ freqs = position_ids_expanded * self.inv_freq.float()
123
+ cos = freqs.cos() * self.attention_scaling
124
+ sin = freqs.sin() * self.attention_scaling
125
+
126
+ cos = self.recomposition_frequencies(cos)
127
+ sin = self.recomposition_frequencies(sin)
128
+ return cos, sin
129
+
130
+ def recomposition_frequencies(self, freq):
131
+ """
132
+ Recompose the frequencies into the final spatial layout used per each grid.
133
+ """
134
+ freq_h, freq_w = freq[:, 0], freq[:, 1]
135
+ freq_hw = torch.cat([freq_h, freq_w], dim=-1)
136
+ return torch.cat([freq_hw, freq_hw], dim=-1)
137
+
138
+
139
+ def rotate_half(x):
140
+ """Rotates half the hidden dims of the input."""
141
+ x1 = x[..., : x.shape[-1] // 2]
142
+ x2 = x[..., x.shape[-1] // 2 :]
143
+ return torch.cat((-x2, x1), dim=-1)
144
+
145
+
146
+ def apply_rotary_pos_emb_vision(
147
+ q: torch.Tensor, k: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor
148
+ ) -> tuple[torch.Tensor, torch.Tensor]:
149
+ orig_q_dtype = q.dtype
150
+ orig_k_dtype = k.dtype
151
+ q, k = q.float(), k.float()
152
+ cos, sin = cos.unsqueeze(-2).float(), sin.unsqueeze(-2).float()
153
+ q_embed = (q * cos) + (rotate_half(q) * sin)
154
+ k_embed = (k * cos) + (rotate_half(k) * sin)
155
+ q_embed = q_embed.to(orig_q_dtype)
156
+ k_embed = k_embed.to(orig_k_dtype)
157
+ return q_embed, k_embed
158
+
159
+
160
+ def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
161
+ """
162
+ This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
163
+ num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
164
+ """
165
+ batch, num_key_value_heads, slen, head_dim = hidden_states.shape
166
+ if n_rep == 1:
167
+ return hidden_states
168
+ hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
169
+ return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
170
+
171
+
172
+ def eager_attention_forward(
173
+ module: nn.Module,
174
+ query: torch.Tensor,
175
+ key: torch.Tensor,
176
+ value: torch.Tensor,
177
+ attention_mask: torch.Tensor | None,
178
+ scaling: float,
179
+ dropout: float = 0.0,
180
+ **kwargs: Unpack[TransformersKwargs],
181
+ ):
182
+ key_states = repeat_kv(key, module.num_key_value_groups)
183
+ value_states = repeat_kv(value, module.num_key_value_groups)
184
+
185
+ attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
186
+ if attention_mask is not None:
187
+ attn_weights = attn_weights + attention_mask
188
+
189
+ attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
190
+ attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
191
+ attn_output = torch.matmul(attn_weights, value_states)
192
+ attn_output = attn_output.transpose(1, 2).contiguous()
193
+
194
+ return attn_output, attn_weights
195
+
196
+
197
+ class Lfm3VlVisionAttention(nn.Module):
198
+ """Bidirectional grouped-query attention over the patches of one frame, with RMSNorm on the query and key heads."""
199
+
200
+ def __init__(self, config: Lfm3VlVisionConfig):
201
+ super().__init__()
202
+ self.config = config
203
+ self.num_heads = config.num_attention_heads
204
+ self.num_key_value_heads = config.num_key_value_heads
205
+ self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
206
+ self.head_dim = config.hidden_size // config.num_attention_heads
207
+ self.scaling = self.head_dim**-0.5
208
+ self.attention_dropout = 0.0
209
+ self.is_causal = False
210
+ self.q_proj = nn.Linear(config.hidden_size, self.num_heads * self.head_dim, bias=False)
211
+ self.k_proj = nn.Linear(config.hidden_size, self.num_key_value_heads * self.head_dim, bias=False)
212
+ self.v_proj = nn.Linear(config.hidden_size, self.num_key_value_heads * self.head_dim, bias=False)
213
+ self.out_proj = nn.Linear(self.num_heads * self.head_dim, config.hidden_size, bias=False)
214
+ self.q_layernorm = Lfm3VlRMSNorm(self.head_dim, eps=config.norm_eps)
215
+ self.k_layernorm = Lfm3VlRMSNorm(self.head_dim, eps=config.norm_eps)
216
+
217
+ def forward(
218
+ self,
219
+ hidden_states: torch.Tensor,
220
+ cu_seqlens: torch.Tensor,
221
+ position_embeddings: tuple[torch.Tensor, torch.Tensor],
222
+ max_seqlen: int | None = None,
223
+ frame_shapes: list[list[int]] | None = None,
224
+ **kwargs,
225
+ ) -> torch.Tensor:
226
+ seq_length = hidden_states.shape[0]
227
+ query_states = self.q_layernorm(self.q_proj(hidden_states).view(seq_length, self.num_heads, self.head_dim))
228
+ key_states = self.k_layernorm(
229
+ self.k_proj(hidden_states).view(seq_length, self.num_key_value_heads, self.head_dim)
230
+ )
231
+ value_states = self.v_proj(hidden_states).view(seq_length, self.num_key_value_heads, self.head_dim)
232
+
233
+ cos, sin = position_embeddings
234
+ query_states, key_states = apply_rotary_pos_emb_vision(query_states, key_states, cos, sin)
235
+
236
+ query_states = query_states.transpose(0, 1).unsqueeze(0)
237
+ key_states = key_states.transpose(0, 1).unsqueeze(0)
238
+ value_states = value_states.transpose(0, 1).unsqueeze(0)
239
+
240
+ attention_interface: Callable = ALL_ATTENTION_FUNCTIONS.get_interface(
241
+ self.config._attn_implementation, eager_attention_forward
242
+ )
243
+
244
+ if is_flash_attention_requested(self.config):
245
+ max_seqlen = get_max_seqlen(cu_seqlens, self.config, kwargs={"max_seqlen": max_seqlen})
246
+ attn_output, attn_weights = attention_interface(
247
+ self,
248
+ query_states,
249
+ key_states,
250
+ value_states,
251
+ attention_mask=None,
252
+ scaling=self.scaling,
253
+ dropout=0.0,
254
+ cu_seq_lens_q=cu_seqlens,
255
+ cu_seq_lens_k=cu_seqlens,
256
+ max_length_q=max_seqlen,
257
+ max_length_k=max_seqlen,
258
+ is_causal=False,
259
+ **kwargs,
260
+ )
261
+ else:
262
+ # Other implementations: frames of one shape (every video, equal-size images) attend as one batch, mixed
263
+ # shapes frame by frame. `frame_shapes` gives the lengths without a device sync.
264
+ if frame_shapes is not None:
265
+ lengths = [height * width for height, width in frame_shapes]
266
+ else:
267
+ lengths = (cu_seqlens[1:] - cu_seqlens[:-1]).tolist()
268
+ if len(set(lengths)) == 1:
269
+ query_states, key_states, value_states = (
270
+ tensor.reshape(-1, len(lengths), lengths[0], self.head_dim).transpose(0, 1)
271
+ for tensor in (query_states, key_states, value_states)
272
+ )
273
+ attn_output, attn_weights = attention_interface(
274
+ self,
275
+ query_states,
276
+ key_states,
277
+ value_states,
278
+ attention_mask=None,
279
+ scaling=self.scaling,
280
+ dropout=0.0,
281
+ is_causal=False,
282
+ **kwargs,
283
+ )
284
+ else:
285
+ splits = [torch.split(tensor, lengths, dim=2) for tensor in (query_states, key_states, value_states)]
286
+ attn_outputs, attn_weights = [], []
287
+ for q, k, v in zip(*splits):
288
+ output, weights = attention_interface(
289
+ self,
290
+ q,
291
+ k,
292
+ v,
293
+ attention_mask=None,
294
+ scaling=self.scaling,
295
+ dropout=0.0,
296
+ is_causal=False,
297
+ **kwargs,
298
+ )
299
+ attn_outputs.append(output)
300
+ attn_weights.append(weights)
301
+ attn_output = torch.cat(attn_outputs, dim=1)
302
+ # frames of different shapes have differently sized maps: one entry per frame, or None under sdpa
303
+ attn_weights = None if attn_weights[0] is None else tuple(attn_weights)
304
+
305
+ attn_output = attn_output.reshape(seq_length, -1).contiguous()
306
+ attn_output = self.out_proj(attn_output)
307
+ return attn_output, attn_weights
308
+
309
+
310
+ class Lfm3VlVisionConv2d(nn.Module):
311
+ """Gated depthwise 2D convolution over the patch grid of each frame."""
312
+
313
+ def __init__(self, config: Lfm3VlVisionConfig):
314
+ super().__init__()
315
+ self.spatial_merge_size = config.spatial_merge_size
316
+ self.in_proj = nn.Linear(config.hidden_size, 3 * config.hidden_size, bias=False)
317
+ self.spatial_conv = nn.Conv2d(
318
+ config.hidden_size,
319
+ config.hidden_size,
320
+ kernel_size=config.conv_kernel_size,
321
+ padding=config.conv_kernel_size // 2,
322
+ groups=config.hidden_size,
323
+ bias=False,
324
+ )
325
+ self.out_proj = nn.Linear(config.hidden_size, config.hidden_size, bias=False)
326
+
327
+ def forward(self, hidden_states: torch.Tensor, frame_shapes: list[list[int]]):
328
+ B, C, x = self.in_proj(hidden_states).chunk(3, dim=-1)
329
+ gated = B * x
330
+ merge = self.spatial_merge_size
331
+ num_frames = len(frame_shapes)
332
+ if len(set(map(tuple, frame_shapes))) == 1:
333
+ # frames of one shape (every video, equal-size images) convolve as one batch: merge-block order -> raster
334
+ # grid, convolve, back
335
+ height, width = frame_shapes[0]
336
+ frames = gated.view(num_frames, height // merge, width // merge, merge, merge, -1)
337
+ frames = frames.permute(0, 5, 1, 3, 2, 4).reshape(num_frames, -1, height, width)
338
+ frames = self.spatial_conv(frames)
339
+ frames = frames.view(num_frames, -1, height // merge, merge, width // merge, merge).permute(
340
+ 0, 2, 4, 3, 5, 1
341
+ )
342
+ conv_output = frames.reshape(gated.shape[0], -1)
343
+ else:
344
+ conv_outputs = []
345
+ lengths = [height * width for height, width in frame_shapes]
346
+ for frame, (height, width) in zip(torch.split(gated, lengths, dim=0), frame_shapes):
347
+ frame = frame.view(height // merge, width // merge, merge, merge, -1)
348
+ frame = frame.permute(4, 0, 2, 1, 3).reshape(1, -1, height, width)
349
+ frame = self.spatial_conv(frame)
350
+ frame = frame.view(-1, height // merge, merge, width // merge, merge).permute(1, 3, 2, 4, 0)
351
+ conv_outputs.append(frame.reshape(height * width, -1))
352
+ conv_output = torch.cat(conv_outputs, dim=0)
353
+ return self.out_proj(C * conv_output)
354
+
355
+
356
+ class Lfm3VlVisionLayer(GradientCheckpointingLayer):
357
+ def __init__(self, config: Lfm3VlVisionConfig, layer_idx: int):
358
+ super().__init__()
359
+ self.is_attention_layer = config.layer_types[layer_idx] == "full_attention"
360
+ if self.is_attention_layer:
361
+ self.self_attn = Lfm3VlVisionAttention(config)
362
+ else:
363
+ self.conv = Lfm3VlVisionConv2d(config)
364
+ self.feed_forward = Lfm3VlMLP(config)
365
+ self.operator_norm = Lfm3VlRMSNorm(config.hidden_size, eps=config.norm_eps)
366
+ self.ffn_norm = Lfm3VlRMSNorm(config.hidden_size, eps=config.norm_eps)
367
+
368
+ def forward(
369
+ self,
370
+ hidden_states: torch.Tensor,
371
+ cu_seqlens: torch.Tensor,
372
+ frame_shapes: list[list[int]],
373
+ position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
374
+ **kwargs,
375
+ ) -> torch.Tensor:
376
+ if self.is_attention_layer:
377
+ operator_output, _ = self.self_attn(
378
+ self.operator_norm(hidden_states),
379
+ cu_seqlens=cu_seqlens,
380
+ position_embeddings=position_embeddings,
381
+ frame_shapes=frame_shapes,
382
+ **kwargs,
383
+ )
384
+ else:
385
+ operator_output = self.conv(self.operator_norm(hidden_states), frame_shapes)
386
+ hidden_states = hidden_states + operator_output
387
+ hidden_states = hidden_states + self.feed_forward(self.ffn_norm(hidden_states))
388
+ return hidden_states
389
+
390
+
391
+ def create_masks_by_layer_type(
392
+ config: PreTrainedConfig,
393
+ inputs_embeds: torch.Tensor,
394
+ attention_mask: torch.Tensor | None,
395
+ past_key_values: Cache | None,
396
+ position_ids: torch.Tensor | None = None,
397
+ ) -> dict:
398
+ """The cache sizes each mask from the layer it is asked about, and the generic mask utilities ask about the first
399
+ non-sliding layer, which for a leading `conv` layer redirects to the first attention layer of any type: name the
400
+ first layer of each mask's own type instead."""
401
+ config = config.get_text_config()
402
+ mask_kwargs = {
403
+ "config": config,
404
+ "inputs_embeds": inputs_embeds,
405
+ "attention_mask": attention_mask,
406
+ "past_key_values": past_key_values,
407
+ "position_ids": position_ids,
408
+ }
409
+ layer_types, causal_mask_mapping = config.layer_types, {}
410
+ if "full_attention" in layer_types:
411
+ layer_idx = layer_types.index("full_attention")
412
+ causal_mask_mapping["full_attention"] = create_causal_mask(**mask_kwargs, layer_idx=layer_idx)
413
+ if "sliding_attention" in layer_types:
414
+ layer_idx = layer_types.index("sliding_attention")
415
+ causal_mask_mapping["sliding_attention"] = create_sliding_window_causal_mask(
416
+ **mask_kwargs, layer_idx=layer_idx
417
+ )
418
+ if "conv" in layer_types:
419
+ causal_mask_mapping["conv"] = create_recurrent_attention_mask(**mask_kwargs)
420
+ return causal_mask_mapping
421
+
422
+
423
+ @auto_docstring
424
+ class Lfm3VlPreTrainedModel(PreTrainedModel):
425
+ config: Lfm3VlConfig
426
+ base_model_prefix = "model"
427
+ input_modalities = ("image", "video", "text")
428
+ supports_gradient_checkpointing = True
429
+ _skip_keys_device_placement = ["past_key_values"]
430
+
431
+ _supports_flash_attn = True
432
+ _supports_sdpa = True
433
+ # the cached decode step traces without graph breaks, like the text model
434
+ _can_compile_fullgraph = True
435
+ _supports_flex_attn = True
436
+ _supports_attention_backend = True
437
+ _no_split_modules = ["Lfm3VlVisionLayer"]
438
+
439
+ @staticmethod
440
+ def create_masks_for_generate(
441
+ config: PreTrainedConfig,
442
+ inputs_embeds: torch.Tensor,
443
+ attention_mask: torch.Tensor | None,
444
+ past_key_values: Cache | None,
445
+ position_ids: torch.Tensor | None = None,
446
+ **kwargs,
447
+ ) -> dict:
448
+ # the static-cache masks of the text model are sized by layer type, not by the generic first attention layer
449
+ return create_masks_by_layer_type(config, inputs_embeds, attention_mask, past_key_values, position_ids)
450
+
451
+
452
+ def get_vision_frame_shapes(grid_thw: torch.Tensor) -> list[list[int]]:
453
+ return grid_thw[:, 1:].repeat_interleave(grid_thw[:, 0], dim=0).tolist()
454
+
455
+
456
+ @auto_docstring(
457
+ custom_intro="""
458
+ The LFM3-VL vision encoder: a stack of gated convolution and bidirectional attention layers over the patches of
459
+ each image or video frame, embedded by a linear patchifier.
460
+ """
461
+ )
462
+ class Lfm3VlVisionModel(Lfm3VlPreTrainedModel):
463
+ config: Lfm3VlVisionConfig
464
+ input_modalities = ("image", "video")
465
+ _input_embed_layer = "patchifier"
466
+ _can_record_outputs = {
467
+ "hidden_states": Lfm3VlVisionLayer,
468
+ "attentions": OutputRecorder(Lfm3VlVisionAttention, index=1),
469
+ }
470
+
471
+ def __init__(self, config: Lfm3VlVisionConfig):
472
+ super().__init__(config)
473
+ self.spatial_merge_size = config.spatial_merge_size
474
+ patch_dim = config.in_channels * config.temporal_patch_size * config.patch_size**2
475
+ self.patchifier = nn.Linear(patch_dim, config.hidden_size, bias=False)
476
+ self.embedding_norm = Lfm3VlRMSNorm(config.hidden_size, eps=config.norm_eps)
477
+ self.rotary_pos_emb = Lfm3VlVisionRotaryEmbedding(config)
478
+ self.layers = nn.ModuleList(
479
+ [Lfm3VlVisionLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
480
+ )
481
+ self.gradient_checkpointing = False
482
+ self.post_init()
483
+
484
+ @merge_with_config_defaults
485
+ @capture_outputs
486
+ @auto_docstring
487
+ def forward(
488
+ self, pixel_values: torch.Tensor, grid_thw: torch.Tensor, **kwargs: Unpack[TransformersKwargs]
489
+ ) -> BaseModelOutputWithPooling:
490
+ r"""
491
+ pixel_values (`torch.Tensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`):
492
+ The flattened patches of every image or video, grouped by merge block.
493
+ grid_thw (`torch.LongTensor` of shape `(num_images_or_videos, 3)`):
494
+ The temporal, height and width dimensions of the patch grid of each image or video.
495
+ """
496
+ position_ids = get_vision_position_ids(grid_thw, self.spatial_merge_size, kwargs=kwargs)
497
+ cu_seqlens, max_seqlen = get_vision_attention_seqlens(grid_thw, self.config, kwargs=kwargs)
498
+ frame_shapes = get_vision_frame_shapes(grid_thw)
499
+
500
+ hidden_states = self.patchifier(pixel_values.to(self.patchifier.weight.dtype))
501
+ hidden_states = self.embedding_norm(hidden_states)
502
+ position_embeddings = self.rotary_pos_emb(hidden_states, position_ids)
503
+
504
+ for layer in self.layers:
505
+ hidden_states = layer(
506
+ hidden_states,
507
+ cu_seqlens=cu_seqlens,
508
+ frame_shapes=frame_shapes,
509
+ max_seqlen=max_seqlen,
510
+ position_embeddings=position_embeddings,
511
+ **kwargs,
512
+ )
513
+
514
+ return BaseModelOutputWithPooling(last_hidden_state=hidden_states)
515
+
516
+
517
+ class Lfm3VlMultiModalProjector(nn.Module):
518
+ """Merges each block of `downsample_factor**2` vision tokens into one language token."""
519
+
520
+ def __init__(self, config: Lfm3VlConfig):
521
+ super().__init__()
522
+ self.downsample_factor = config.downsample_factor
523
+ self.layer_norm = nn.LayerNorm(config.vision_config.hidden_size, eps=config.projector_layer_norm_eps)
524
+ self.linear_1 = nn.Linear(
525
+ config.vision_config.hidden_size * config.downsample_factor**2,
526
+ config.projector_hidden_size,
527
+ bias=config.projector_bias,
528
+ )
529
+ self.act = ACT2FN[config.projector_hidden_act]
530
+ self.linear_2 = nn.Linear(
531
+ config.projector_hidden_size, config.text_config.hidden_size, bias=config.projector_bias
532
+ )
533
+
534
+ def forward(self, image_features: torch.Tensor):
535
+ hidden_states = self.layer_norm(image_features).reshape(-1, self.linear_1.in_features)
536
+ hidden_states = self.linear_1(hidden_states)
537
+ hidden_states = self.act(hidden_states)
538
+ hidden_states = self.linear_2(hidden_states)
539
+ return hidden_states
540
+
541
+
542
+ @auto_docstring(
543
+ custom_intro="""
544
+ Base class for Lfm3Vl outputs, with hidden states and attentions.
545
+ """
546
+ )
547
+ @dataclass
548
+ class Lfm3VlModelOutputWithPast(BaseModelOutputWithPast):
549
+ r"""
550
+ past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`):
551
+ It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache).
552
+
553
+ Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see
554
+ `past_key_values` input) to speed up sequential decoding.
555
+ image_hidden_states (`torch.FloatTensor`, *optional*):
556
+ A `torch.FloatTensor` of size `(num_image_tokens, hidden_size)`.
557
+ image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state.
558
+ video_hidden_states (`torch.FloatTensor`, *optional*):
559
+ A `torch.FloatTensor` of size `(num_video_tokens, hidden_size)`.
560
+ video_hidden_states of the model produced by the vision encoder and after projecting the last hidden state.
561
+ """
562
+
563
+ image_hidden_states: torch.FloatTensor | None = None
564
+ video_hidden_states: torch.FloatTensor | None = None
565
+
566
+
567
+ @auto_docstring(
568
+ custom_intro="""
569
+ Base class for Lfm3Vl causal language model (or autoregressive) outputs.
570
+ """
571
+ )
572
+ @dataclass
573
+ class Lfm3VlCausalLMOutputWithPast(CausalLMOutputWithPast):
574
+ r"""
575
+ image_hidden_states (`torch.FloatTensor`, *optional*):
576
+ A `torch.FloatTensor` of size `(num_image_tokens, hidden_size)`.
577
+ image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state.
578
+ video_hidden_states (`torch.FloatTensor`, *optional*):
579
+ A `torch.FloatTensor` of size `(num_video_tokens, hidden_size)`.
580
+ video_hidden_states of the model produced by the vision encoder and after projecting the last hidden state.
581
+ """
582
+
583
+ image_hidden_states: torch.FloatTensor | None = None
584
+ video_hidden_states: torch.FloatTensor | None = None
585
+
586
+
587
+ @auto_docstring(
588
+ custom_intro="""
589
+ The Lfm3Vl model which consists of a vision backbone and a language model, without a language modeling head.
590
+ """
591
+ )
592
+ class Lfm3VlModel(Lfm3VlPreTrainedModel):
593
+ def __init__(self, config: Lfm3VlConfig):
594
+ super().__init__(config)
595
+ self.vision_tower = Lfm3VlVisionModel(config.vision_config)
596
+ self.multi_modal_projector = Lfm3VlMultiModalProjector(config)
597
+ self.language_model = Lfm3Model(config.text_config)
598
+ self.post_init()
599
+
600
+ @accepts_precomputed_kwargs(modality="image")
601
+ @can_return_tuple
602
+ @auto_docstring(
603
+ custom_intro="Obtains image last hidden states from the vision tower and apply multimodal projection."
604
+ )
605
+ def get_image_features(
606
+ self,
607
+ pixel_values: torch.FloatTensor,
608
+ image_grid_thw: torch.LongTensor,
609
+ **kwargs: Unpack[TransformersKwargs],
610
+ ) -> tuple | BaseModelOutputWithPooling:
611
+ r"""
612
+ pixel_values (`torch.FloatTensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`):
613
+ The flattened patches of every image, grouped by merge block.
614
+ image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`):
615
+ The temporal, height and width dimensions of the patch grid of each image.
616
+ """
617
+ image_outputs = self.vision_tower(pixel_values, grid_thw=image_grid_thw, return_dict=True, **kwargs)
618
+ image_features = self.multi_modal_projector(image_outputs.last_hidden_state)
619
+ split_sizes = (image_grid_thw.prod(-1) // self.vision_tower.spatial_merge_size**2).tolist()
620
+ image_outputs.pooler_output = torch.split(image_features, split_sizes)
621
+ return image_outputs
622
+
623
+ def get_placeholder_mask(
624
+ self,
625
+ input_ids: torch.LongTensor,
626
+ inputs_embeds: torch.FloatTensor,
627
+ image_features: torch.FloatTensor | None = None,
628
+ video_features: torch.FloatTensor | None = None,
629
+ ):
630
+ """Placeholder masks from `input_ids` or `inputs_embeds`, checked against the feature counts."""
631
+ if input_ids is None:
632
+ special_image_mask = inputs_embeds == self.get_input_embeddings()(
633
+ torch.full((), self.config.image_token_id, dtype=torch.long, device=inputs_embeds.device)
634
+ )
635
+ special_image_mask = special_image_mask.all(-1)
636
+ special_video_mask = inputs_embeds == self.get_input_embeddings()(
637
+ torch.full((), self.config.video_token_id, dtype=torch.long, device=inputs_embeds.device)
638
+ )
639
+ special_video_mask = special_video_mask.all(-1)
640
+ else:
641
+ special_image_mask = input_ids == self.config.image_token_id
642
+ special_video_mask = input_ids == self.config.video_token_id
643
+
644
+ n_image_tokens = special_image_mask.sum()
645
+ special_image_mask = special_image_mask.unsqueeze(-1).to(inputs_embeds.device)
646
+ if image_features is not None:
647
+ torch_compilable_check(
648
+ n_image_tokens * inputs_embeds.shape[-1] == image_features.numel(),
649
+ f"Image features and image tokens do not match, tokens: {n_image_tokens}, features: {image_features.shape[0]}",
650
+ )
651
+
652
+ n_video_tokens = special_video_mask.sum()
653
+ special_video_mask = special_video_mask.unsqueeze(-1).to(inputs_embeds.device)
654
+ if video_features is not None:
655
+ torch_compilable_check(
656
+ n_video_tokens * inputs_embeds.shape[-1] == video_features.numel(),
657
+ f"Video features and video tokens do not match, tokens: {n_video_tokens}, features: {video_features.shape[0]}",
658
+ )
659
+ return special_image_mask, special_video_mask
660
+
661
+ @can_return_tuple
662
+ @auto_docstring
663
+ def forward(
664
+ self,
665
+ input_ids: torch.LongTensor | None = None,
666
+ attention_mask: torch.Tensor | None = None,
667
+ position_ids: torch.LongTensor | None = None,
668
+ past_key_values: Cache | None = None,
669
+ inputs_embeds: torch.FloatTensor | None = None,
670
+ pixel_values: torch.FloatTensor | None = None,
671
+ image_grid_thw: torch.LongTensor | None = None,
672
+ pixel_values_videos: torch.FloatTensor | None = None,
673
+ video_grid_thw: torch.LongTensor | None = None,
674
+ use_cache: bool | None = None,
675
+ mm_encoder_outputs: dict[str, BaseModelOutputWithPooling] | None = None,
676
+ **kwargs: Unpack[TransformersKwargs],
677
+ ) -> tuple | Lfm3VlModelOutputWithPast:
678
+ r"""
679
+ pixel_values (`torch.FloatTensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`, *optional*):
680
+ The flattened patches of every image, grouped by merge block.
681
+ image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*):
682
+ The temporal, height and width dimensions of the patch grid of each image.
683
+ pixel_values_videos (`torch.FloatTensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`, *optional*):
684
+ The flattened patches of every video, grouped by merge block.
685
+ video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*):
686
+ The temporal, height and width dimensions of the patch grid of each video.
687
+ """
688
+ if (input_ids is None) ^ (inputs_embeds is not None):
689
+ raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
690
+
691
+ if (pixel_values is not None or pixel_values_videos is not None) and mm_encoder_outputs is not None:
692
+ raise ValueError(
693
+ "You cannot specify both pixel_values/pixel_values_videos and mm_encoder_outputs at the same time"
694
+ )
695
+
696
+ if inputs_embeds is None:
697
+ inputs_embeds = self.get_input_embeddings()(input_ids)
698
+
699
+ mm_encoder_outputs = mm_encoder_outputs if mm_encoder_outputs is not None else {}
700
+ if (pixel_values is None) != (image_grid_thw is None):
701
+ raise ValueError("pixel_values and image_grid_thw must be supplied together")
702
+ if (pixel_values_videos is None) != (video_grid_thw is None):
703
+ raise ValueError("pixel_values_videos and video_grid_thw must be supplied together")
704
+ if mm_encoder_outputs.get("image") is None and pixel_values is not None:
705
+ mm_encoder_outputs["image"] = self.get_image_features(
706
+ pixel_values, image_grid_thw, return_dict=True, **kwargs
707
+ )
708
+ if mm_encoder_outputs.get("video") is None and pixel_values_videos is not None:
709
+ mm_encoder_outputs["video"] = self.get_video_features(
710
+ pixel_values_videos, video_grid_thw, return_dict=True, **kwargs
711
+ )
712
+
713
+ image_features = None
714
+ if mm_encoder_outputs.get("image") is not None:
715
+ image_features = torch.cat(mm_encoder_outputs["image"].pooler_output, dim=0).to(
716
+ inputs_embeds.device, inputs_embeds.dtype
717
+ )
718
+ image_mask, _ = self.get_placeholder_mask(
719
+ input_ids, inputs_embeds=inputs_embeds, image_features=image_features
720
+ )
721
+ inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_features)
722
+
723
+ video_features = None
724
+ if mm_encoder_outputs.get("video") is not None:
725
+ video_features = torch.cat(mm_encoder_outputs["video"].pooler_output, dim=0).to(
726
+ inputs_embeds.device, inputs_embeds.dtype
727
+ )
728
+ _, video_mask = self.get_placeholder_mask(
729
+ input_ids, inputs_embeds=inputs_embeds, video_features=video_features
730
+ )
731
+ inputs_embeds = inputs_embeds.masked_scatter(video_mask, video_features)
732
+
733
+ outputs = self.language_model(
734
+ attention_mask=attention_mask,
735
+ position_ids=position_ids,
736
+ past_key_values=past_key_values,
737
+ inputs_embeds=inputs_embeds,
738
+ use_cache=use_cache,
739
+ **kwargs,
740
+ )
741
+
742
+ return Lfm3VlModelOutputWithPast(
743
+ last_hidden_state=outputs.last_hidden_state,
744
+ past_key_values=outputs.past_key_values,
745
+ hidden_states=outputs.hidden_states,
746
+ attentions=outputs.attentions,
747
+ image_hidden_states=image_features,
748
+ video_hidden_states=video_features,
749
+ )
750
+
751
+ @accepts_precomputed_kwargs(modality="video")
752
+ @can_return_tuple
753
+ @auto_docstring(
754
+ custom_intro="Obtains video last hidden states from the vision tower and apply multimodal projection."
755
+ )
756
+ def get_video_features(
757
+ self,
758
+ pixel_values_videos: torch.FloatTensor,
759
+ video_grid_thw: torch.LongTensor,
760
+ **kwargs: Unpack[TransformersKwargs],
761
+ ) -> tuple | BaseModelOutputWithPooling:
762
+ r"""
763
+ pixel_values_videos (`torch.FloatTensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`):
764
+ The flattened patches of every video, grouped by merge block.
765
+ video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`):
766
+ The temporal, height and width dimensions of the patch grid of each video.
767
+ """
768
+ return self.get_image_features(pixel_values_videos, video_grid_thw, **kwargs)
769
+
770
+
771
+ def reject_packed_generation(config: PreTrainedConfig, input_ids, generation_config, model_kwargs, is_first_iteration):
772
+ """`generate` decodes from the cache its prefill fills, and no mask it builds isolates packed documents
773
+ (`position_ids` restarting): the prompt is refused before the prefill, one forward or every chunk, writes to it."""
774
+ position_ids, attention_mask = model_kwargs.get("position_ids"), model_kwargs.get("attention_mask")
775
+ cache = model_kwargs.get("past_key_values")
776
+ if cache is None or position_ids is None or position_ids.ndim != 2:
777
+ return
778
+ if attention_mask is not None and attention_mask.ndim != 2:
779
+ return
780
+ has_attention = any(t != "conv" for t in config.get_text_config().layer_types)
781
+ filled = bool(cache.get_seq_length() > 0 if has_attention else cache.has_previous_state())
782
+ inputs_embeds = model_kwargs.get("inputs_embeds") if is_first_iteration else None
783
+ length = input_ids.shape[1] if inputs_embeds is None else inputs_embeds.shape[1]
784
+ if generation_config.prefill_chunk_size is not None:
785
+ # the chunks run from the start of the prompt
786
+ position_ids = position_ids[:, :length]
787
+ else:
788
+ if filled and has_attention:
789
+ # the slice `_prefill` feeds: a prompt that spans the cache is cut to its uncached part
790
+ mask_length = getattr(generation_config, "_mask_length", -1)
791
+ if attention_mask is not None:
792
+ mask_length = attention_mask.shape[1]
793
+ if inputs_embeds is not None or mask_length == length:
794
+ length -= int(cache.get_seq_length())
795
+ position_ids = position_ids[:, position_ids.shape[1] - length :]
796
+ if attention_mask is None:
797
+ attention_mask = torch.ones_like(position_ids)
798
+ if bool(packed_under_padding(position_ids, attention_mask, filled).any()):
799
+ raise ValueError(
800
+ "`generate` cannot decode packed documents (`position_ids` restarting, or starting at 0 on a cache that "
801
+ "holds tokens): attention would span them. Generate each document in its own row of a left-padded batch."
802
+ )
803
+
804
+
805
+ def packed_under_padding(position_ids: torch.Tensor, padding_mask: torch.Tensor, filled) -> torch.Tensor:
806
+ """Padding repeats position 0, so packing is a restart between two kept tokens, or a kept position 0 on a filled
807
+ cache whose row holds a kept token; attention under a padding mask cannot isolate it."""
808
+ restarts = torch.diff(position_ids, prepend=position_ids[:, :1] - 1, dim=-1) != 1
809
+ valid = padding_mask[:, -restarts.shape[-1] :].bool()
810
+ packed = restarts[:, 1:] & valid[:, 1:] & valid[:, :-1]
811
+ if filled:
812
+ # a row whose cached tokens are all padding (chunked prefill of a left-padded batch) has no document to pack
813
+ # against; a mask covering only the chunk keeps the strict rule
814
+ cached = padding_mask[:, : padding_mask.shape[-1] - restarts.shape[-1]]
815
+ cached_valid = cached.bool().any(-1, keepdim=True) if cached.shape[-1] else torch.ones_like(valid[:, :1])
816
+ packed = packed | (valid[:, :1] & (position_ids[:, :1] == 0) & cached_valid)
817
+ return packed
818
+
819
+
820
+ @auto_docstring(
821
+ custom_intro="""
822
+ The LFM3_VL model which consists of a vision backbone and a language model.
823
+ """
824
+ )
825
+ class Lfm3VlForConditionalGeneration(Lfm3VlPreTrainedModel, GenerationMixin):
826
+ _tied_weights_keys = {"lm_head.weight": "model.language_model.embed_tokens.weight"}
827
+
828
+ def __init__(self, config: Lfm3VlConfig):
829
+ super().__init__(config)
830
+ self.model = Lfm3VlModel(config)
831
+ self.lm_head = nn.Linear(config.text_config.hidden_size, config.text_config.vocab_size, bias=False)
832
+ self.post_init()
833
+
834
+ def get_output_embeddings(self) -> nn.Module:
835
+ return self.lm_head
836
+
837
+ @auto_docstring
838
+ def get_image_features(
839
+ self,
840
+ pixel_values: torch.FloatTensor,
841
+ image_grid_thw: torch.LongTensor,
842
+ **kwargs: Unpack[TransformersKwargs],
843
+ ) -> tuple | BaseModelOutputWithPooling:
844
+ r"""
845
+ pixel_values (`torch.FloatTensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`):
846
+ The flattened patches of every image, grouped by merge block.
847
+ image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`):
848
+ The temporal, height and width dimensions of the patch grid of each image.
849
+ """
850
+ return self.model.get_image_features(pixel_values, image_grid_thw, **kwargs)
851
+
852
+ @can_return_tuple
853
+ @auto_docstring
854
+ def forward(
855
+ self,
856
+ input_ids: torch.LongTensor | None = None,
857
+ attention_mask: torch.Tensor | None = None,
858
+ position_ids: torch.LongTensor | None = None,
859
+ past_key_values: Cache | None = None,
860
+ inputs_embeds: torch.FloatTensor | None = None,
861
+ pixel_values: torch.FloatTensor | None = None,
862
+ image_grid_thw: torch.LongTensor | None = None,
863
+ pixel_values_videos: torch.FloatTensor | None = None,
864
+ video_grid_thw: torch.LongTensor | None = None,
865
+ labels: torch.LongTensor | None = None,
866
+ use_cache: bool | None = None,
867
+ logits_to_keep: int | torch.Tensor = 0,
868
+ mm_encoder_outputs: dict[str, BaseModelOutputWithPooling] | None = None,
869
+ **kwargs: Unpack[TransformersKwargs],
870
+ ) -> tuple | Lfm3VlCausalLMOutputWithPast:
871
+ r"""
872
+ pixel_values (`torch.FloatTensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`, *optional*):
873
+ The flattened patches of every image, grouped by merge block.
874
+ image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*):
875
+ The temporal, height and width dimensions of the patch grid of each image.
876
+ pixel_values_videos (`torch.FloatTensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`, *optional*):
877
+ The flattened patches of every video, grouped by merge block.
878
+ video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*):
879
+ The temporal, height and width dimensions of the patch grid of each video.
880
+ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
881
+ Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
882
+ config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
883
+ (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
884
+
885
+ Example:
886
+
887
+ ```python
888
+ >>> from transformers import AutoProcessor, AutoModelForImageTextToText
889
+
890
+ >>> model = AutoModelForImageTextToText.from_pretrained("LiquidAI/LFM3-VL-2B")
891
+ >>> processor = AutoProcessor.from_pretrained("LiquidAI/LFM3-VL-2B")
892
+
893
+ >>> conversation = [
894
+ ... {
895
+ ... "role": "user",
896
+ ... "content": [
897
+ ... {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"},
898
+ ... {"type": "text", "text": "What is in this image?"},
899
+ ... ],
900
+ ... },
901
+ ... ]
902
+
903
+ >>> inputs = processor.apply_chat_template(
904
+ ... conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt"
905
+ ... )
906
+
907
+ >>> outputs = model.generate(**inputs, max_new_tokens=45)
908
+ >>> processor.batch_decode(outputs, skip_special_tokens=True)[0]
909
+ ```"""
910
+ outputs = self.model(
911
+ input_ids=input_ids,
912
+ attention_mask=attention_mask,
913
+ position_ids=position_ids,
914
+ past_key_values=past_key_values,
915
+ inputs_embeds=inputs_embeds,
916
+ pixel_values=pixel_values,
917
+ image_grid_thw=image_grid_thw,
918
+ pixel_values_videos=pixel_values_videos,
919
+ video_grid_thw=video_grid_thw,
920
+ use_cache=use_cache,
921
+ mm_encoder_outputs=mm_encoder_outputs,
922
+ **kwargs,
923
+ )
924
+
925
+ hidden_states = outputs[0]
926
+ # Only compute necessary logits, and do not upcast them to float if we are not computing the loss
927
+ slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
928
+ logits = self.lm_head(hidden_states[:, slice_indices, :])
929
+
930
+ loss = None
931
+ if labels is not None:
932
+ loss = self.loss_function(
933
+ logits=logits, labels=labels, vocab_size=self.config.text_config.vocab_size, **kwargs
934
+ )
935
+
936
+ return Lfm3VlCausalLMOutputWithPast(
937
+ loss=loss,
938
+ logits=logits,
939
+ past_key_values=outputs.past_key_values,
940
+ hidden_states=outputs.hidden_states,
941
+ attentions=outputs.attentions,
942
+ image_hidden_states=outputs.image_hidden_states,
943
+ video_hidden_states=outputs.video_hidden_states,
944
+ )
945
+
946
+ def _prefill(self, input_ids, generation_config, model_kwargs, is_first_iteration=True):
947
+ # the text model's refusal, before the prefill writes to the cache
948
+ reject_packed_generation(self.config, input_ids, generation_config, model_kwargs, is_first_iteration)
949
+ return super()._prefill(input_ids, generation_config, model_kwargs, is_first_iteration=is_first_iteration)
950
+
951
+ @auto_docstring
952
+ def get_video_features(
953
+ self,
954
+ pixel_values_videos: torch.FloatTensor,
955
+ video_grid_thw: torch.LongTensor,
956
+ **kwargs: Unpack[TransformersKwargs],
957
+ ) -> tuple | BaseModelOutputWithPooling:
958
+ r"""
959
+ pixel_values_videos (`torch.FloatTensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`):
960
+ The flattened patches of every video, grouped by merge block.
961
+ video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`):
962
+ The temporal, height and width dimensions of the patch grid of each video.
963
+ """
964
+ return self.model.get_video_features(pixel_values_videos, video_grid_thw, **kwargs)
965
+
966
+
967
+ __all__ = ["Lfm3VlForConditionalGeneration", "Lfm3VlModel", "Lfm3VlPreTrainedModel", "Lfm3VlVisionModel"]
processing_lfm3_vl.py ADDED
@@ -0,0 +1,248 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2026 The HuggingFace Team. All rights reserved.
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+ import math
15
+
16
+ import numpy as np
17
+
18
+ from transformers.image_utils import is_pil_image, is_valid_image, make_flat_list_of_images
19
+ from transformers.processing_utils import MultiModalData, ProcessingKwargs, ProcessorMixin
20
+ from transformers.utils import auto_docstring, is_numpy_array, is_torch_tensor, logging
21
+ from transformers.video_utils import VideoMetadata, is_valid_video, make_batched_videos
22
+
23
+
24
+ logger = logging.get_logger(__name__)
25
+
26
+
27
+ class Lfm3VlProcessorKwargs(ProcessingKwargs, total=False):
28
+ _defaults = {
29
+ "text_kwargs": {
30
+ "padding": False,
31
+ "return_token_type_ids": False,
32
+ "return_mm_token_type_ids": False,
33
+ },
34
+ "videos_kwargs": {"return_metadata": True},
35
+ }
36
+
37
+
38
+ def _check_video_fps(metadata: VideoMetadata) -> None:
39
+ if metadata.fps is not None and not (math.isfinite(metadata.fps) and metadata.fps > 0):
40
+ raise ValueError(f"`video_metadata.fps` must be a positive number, got {metadata.fps}")
41
+
42
+
43
+ def _media_per_prompt(media, is_item) -> list[int] | None:
44
+ # rows name the prompt that owns them while the expansion consumes media in flat order; a row is a list of whole
45
+ # items or one stacked batch, and a list of single frames is one video, not a row
46
+ if not isinstance(media, (list, tuple)):
47
+ return None
48
+ counts = []
49
+ for row in media:
50
+ if isinstance(row, (list, tuple)) and all(is_item(item) for item in row):
51
+ counts.append(len(row))
52
+ elif (is_numpy_array(row) or is_torch_tensor(row)) and row.ndim == 4 and (len(row) == 0 or is_item(row[0])):
53
+ counts.append(len(row))
54
+ else:
55
+ return None
56
+ return counts
57
+
58
+
59
+ def _check_vision_placeholder_counts(processor, prompts, images, videos) -> None:
60
+ for name, token, media, flatten, is_item in (
61
+ (
62
+ "images",
63
+ processor.image_token,
64
+ images,
65
+ make_flat_list_of_images,
66
+ lambda item: is_valid_image(item) and (is_pil_image(item) or item.ndim == 3),
67
+ ),
68
+ (
69
+ "videos",
70
+ processor.video_token,
71
+ videos,
72
+ make_batched_videos,
73
+ lambda item: isinstance(item, str) or (not is_pil_image(item) and is_valid_video(item)),
74
+ ),
75
+ ):
76
+ num_placeholders = sum(prompt.count(token) for prompt in prompts)
77
+ num_media = 0 if media is None else len(flatten(media))
78
+ if num_placeholders != num_media:
79
+ raise ValueError(f"Found {num_placeholders} {token} tokens in the text but received {num_media} {name}")
80
+ # one prompt cannot take another's media
81
+ per_prompt = _media_per_prompt(media, is_item) if len(prompts) > 1 else None
82
+ in_text = [prompt.count(token) for prompt in prompts]
83
+ if per_prompt is not None and per_prompt != in_text:
84
+ raise ValueError(f"Found {in_text} {token} tokens per prompt but received {per_prompt} {name} per prompt")
85
+
86
+
87
+ @auto_docstring
88
+ class Lfm3VlProcessor(ProcessorMixin):
89
+ valid_processor_kwargs = Lfm3VlProcessorKwargs
90
+
91
+ def __init__(self, image_processor=None, tokenizer=None, video_processor=None, chat_template=None, **kwargs):
92
+ self.image_token = getattr(tokenizer, "image_token", "<|image|>")
93
+ self.video_token = getattr(tokenizer, "video_token", "<|video|>")
94
+ self.vision_start_token = getattr(tokenizer, "vision_start_token", "<|vision_start|>")
95
+ self.vision_end_token = getattr(tokenizer, "vision_end_token", "<|vision_end|>")
96
+ self.image_token_id = tokenizer.convert_tokens_to_ids(self.image_token)
97
+ self.video_token_id = tokenizer.convert_tokens_to_ids(self.video_token)
98
+ self.vision_start_token_id = tokenizer.convert_tokens_to_ids(self.vision_start_token)
99
+ self.vision_end_token_id = tokenizer.convert_tokens_to_ids(self.vision_end_token)
100
+ super().__init__(image_processor, tokenizer, video_processor, chat_template=chat_template)
101
+
102
+ def replace_image_token(self, image_inputs: dict, image_idx: int, **kwargs) -> str:
103
+ merge_length = self.image_processor.merge_size**2
104
+ num_image_tokens = image_inputs["image_grid_thw"][image_idx].prod() // merge_length
105
+ return self.image_token * num_image_tokens
106
+
107
+ def replace_video_token(self, video_inputs: dict, video_idx: int, **kwargs) -> str:
108
+ metadata = video_inputs["video_metadata"][video_idx]
109
+ if metadata.frames_indices is None:
110
+ raise ValueError("Frame timestamps need the sampled frame indices in `video_metadata`")
111
+ _check_video_fps(metadata)
112
+ merge_length = self.video_processor.merge_size**2
113
+ num_frames = video_inputs["video_grid_thw"][video_idx][0]
114
+ frame_seqlen = video_inputs["video_grid_thw"][video_idx][1:].prod() // merge_length
115
+ metadata = video_inputs["video_metadata"][video_idx]
116
+ video_placeholder = ""
117
+
118
+ if metadata.fps is None:
119
+ logger.warning_once(
120
+ "Lfm3Vl requires frame timestamps to construct prompts, but the `fps` of the input video could not be inferred. "
121
+ "Probably `video_metadata` was missing from inputs and you passed pre-sampled frames. "
122
+ "Defaulting to `fps=24`. Please provide `video_metadata` for more accurate results."
123
+ )
124
+ # a released transformers reads torchcodec's scanned average fps, which carries float noise
125
+ # (30.000000000000004) where the fork reads the container header
126
+ metadata.fps = 24 if metadata.fps is None else round(metadata.fps, 6)
127
+
128
+ # if timestamps are not provided, calculate them
129
+ curr_timestamp = self._calculate_timestamps(
130
+ metadata.frames_indices,
131
+ metadata.fps,
132
+ self.video_processor.temporal_patch_size,
133
+ )
134
+
135
+ for frame_idx in range(num_frames):
136
+ curr_time = curr_timestamp[frame_idx]
137
+ video_placeholder += f"<{curr_time:.1f} seconds>"
138
+ video_placeholder += self.vision_start_token + self.video_token * frame_seqlen + self.vision_end_token
139
+ return video_placeholder
140
+
141
+ def _get_num_multimodal_tokens(self, image_sizes=None, video_sizes=None, **kwargs):
142
+ """
143
+ Computes the number of placeholder tokens needed for multimodal inputs with the given sizes.
144
+
145
+ Args:
146
+ image_sizes (`list[list[int]]`, *optional*):
147
+ The input sizes formatted as (height, width) per each image.
148
+ video_sizes (`list[list[int]]`, *optional*):
149
+ The input sizes formatted as (num_frames, height, width) per each video.
150
+
151
+ Returns:
152
+ `MultiModalData`: the number of tokens per provided modality.
153
+ """
154
+ vision_data = {}
155
+ if image_sizes is not None:
156
+ # copy: `update` on the class-level defaults would leak kwargs into every later call
157
+ images_kwargs = dict(Lfm3VlProcessorKwargs._defaults.get("images_kwargs", {}))
158
+ images_kwargs.update(kwargs)
159
+ merge_size = images_kwargs.get("merge_size") or self.image_processor.merge_size
160
+
161
+ num_image_patches = [
162
+ self.image_processor.get_number_of_image_patches(*image_size, images_kwargs)
163
+ for image_size in image_sizes
164
+ ]
165
+ num_image_tokens = [(num_patches // merge_size**2) for num_patches in num_image_patches]
166
+ vision_data.update({"num_image_tokens": num_image_tokens, "num_image_patches": num_image_patches})
167
+
168
+ if video_sizes is not None:
169
+ videos_kwargs = dict(Lfm3VlProcessorKwargs._defaults.get("videos_kwargs", {}))
170
+ videos_kwargs.update(kwargs)
171
+ merge_size = videos_kwargs.get("merge_size") or self.video_processor.merge_size
172
+ num_video_patches = [
173
+ self.video_processor.get_num_of_video_patches(*video_size, videos_kwargs) for video_size in video_sizes
174
+ ]
175
+ num_video_tokens = [(num_patches // merge_size**2) for num_patches in num_video_patches]
176
+ vision_data["num_video_tokens"] = num_video_tokens
177
+
178
+ return MultiModalData(**vision_data)
179
+
180
+ def post_process_image_text_to_text(
181
+ self, generated_outputs, skip_special_tokens=True, clean_up_tokenization_spaces=False, **kwargs
182
+ ):
183
+ """
184
+ Post-process the output of the model to decode the text.
185
+
186
+ Args:
187
+ generated_outputs (`torch.Tensor` or `np.ndarray`):
188
+ The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
189
+ or `(sequence_length,)`.
190
+ skip_special_tokens (`bool`, *optional*, defaults to `True`):
191
+ Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method.
192
+ clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`):
193
+ Whether or not to clean up the tokenization spaces. Argument passed to the tokenizer's `batch_decode` method.
194
+ **kwargs:
195
+ Additional arguments to be passed to the tokenizer's `batch_decode method`.
196
+
197
+ Returns:
198
+ `list[str]`: The decoded text.
199
+ """
200
+ return self.tokenizer.batch_decode(
201
+ generated_outputs,
202
+ skip_special_tokens=skip_special_tokens,
203
+ clean_up_tokenization_spaces=clean_up_tokenization_spaces,
204
+ **kwargs,
205
+ )
206
+
207
+ @property
208
+ def model_input_names(self):
209
+ return list(
210
+ dict.fromkeys(
211
+ self.tokenizer.model_input_names
212
+ + self.image_processor.model_input_names
213
+ + self.video_processor.model_input_names
214
+ )
215
+ )
216
+
217
+ def _calculate_timestamps(self, indices: list[int] | np.ndarray, video_fps: float, merge_size: int = 2):
218
+ if not isinstance(indices, list):
219
+ indices = indices.tolist()
220
+ if len(indices) % merge_size != 0:
221
+ indices.extend(indices[-1] for _ in range(merge_size - len(indices) % merge_size))
222
+ timestamps = [idx / video_fps for idx in indices]
223
+ # @JJJYmmm frames are merged by self.merge_size, \
224
+ # so we need to average the timestamps between the first/last frame within the temporal patch
225
+ timestamps = [
226
+ (timestamps[i] + timestamps[i + merge_size - 1]) / 2 for i in range(0, len(timestamps), merge_size)
227
+ ]
228
+ return timestamps
229
+
230
+ def prepare_inputs_layout(self, images=None, text=None, videos=None, audio=None, **kwargs):
231
+ for name, media in (("images", images), ("videos", videos)):
232
+ if isinstance(media, (list, tuple)) and len(media) == 0:
233
+ raise ValueError(f"`{name}` must hold at least one item; pass `{name}=None` for prompts without one")
234
+ return super().prepare_inputs_layout(images=images, text=text, videos=videos, audio=audio, **kwargs)
235
+
236
+ def validate_inputs(self, images=None, text=None, videos=None, audio=None, **kwargs):
237
+ super().validate_inputs(images=images, text=text, videos=videos, audio=audio, **kwargs)
238
+ if text is not None:
239
+ _check_vision_placeholder_counts(self, [text] if isinstance(text, str) else text, images, videos)
240
+
241
+ def _process_videos(self, videos, **kwargs):
242
+ # the frame timestamps need the metadata whatever the caller's `return_metadata`; `__call__` drops it from
243
+ # the output unless it was asked for
244
+ kwargs["return_metadata"] = True
245
+ return super()._process_videos(videos, **kwargs)
246
+
247
+
248
+ __all__ = ["Lfm3VlProcessor"]
processor_config.json ADDED
@@ -0,0 +1,73 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "auto_map": {
3
+ "AutoProcessor": "processing_lfm3_vl.Lfm3VlProcessor"
4
+ },
5
+ "image_processor": {
6
+ "auto_map": {
7
+ "AutoImageProcessor": "image_processing_lfm3_vl.Lfm3VlImageProcessor",
8
+ "AutoProcessor": "processing_lfm3_vl.Lfm3VlProcessor"
9
+ },
10
+ "do_convert_rgb": true,
11
+ "do_normalize": true,
12
+ "do_rescale": true,
13
+ "do_resize": true,
14
+ "image_mean": [
15
+ 0.5,
16
+ 0.5,
17
+ 0.5
18
+ ],
19
+ "image_processor_type": "Lfm3VlImageProcessor",
20
+ "image_std": [
21
+ 0.5,
22
+ 0.5,
23
+ 0.5
24
+ ],
25
+ "merge_size": 2,
26
+ "patch_size": 16,
27
+ "resample": 3,
28
+ "rescale_factor": 0.00392156862745098,
29
+ "size": {
30
+ "longest_edge": 16777216,
31
+ "shortest_edge": 65536
32
+ },
33
+ "temporal_patch_size": 2
34
+ },
35
+ "processor_class": "Lfm3VlProcessor",
36
+ "video_processor": {
37
+ "auto_map": {
38
+ "AutoProcessor": "processing_lfm3_vl.Lfm3VlProcessor",
39
+ "AutoVideoProcessor": "video_processing_lfm3_vl.Lfm3VlVideoProcessor"
40
+ },
41
+ "cap_pixels_per_frame": false,
42
+ "do_convert_rgb": true,
43
+ "do_normalize": true,
44
+ "do_rescale": true,
45
+ "do_resize": true,
46
+ "do_sample_frames": true,
47
+ "fps": 2.0,
48
+ "image_mean": [
49
+ 0.5,
50
+ 0.5,
51
+ 0.5
52
+ ],
53
+ "image_std": [
54
+ 0.5,
55
+ 0.5,
56
+ 0.5
57
+ ],
58
+ "max_frames": 768,
59
+ "max_video_tokens": 768,
60
+ "merge_size": 2,
61
+ "min_frames": 4,
62
+ "patch_size": 16,
63
+ "resample": 3,
64
+ "rescale_factor": 0.00392156862745098,
65
+ "return_metadata": false,
66
+ "size": {
67
+ "longest_edge": 25165824,
68
+ "shortest_edge": 4096
69
+ },
70
+ "temporal_patch_size": 2,
71
+ "video_processor_type": "Lfm3VlVideoProcessor"
72
+ }
73
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a39eed58008cbcbc09e5b629b81338a654269c64cd13348aaa6090f4db75c7e7
3
+ size 17908258
tokenizer_config.json ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "auto_map": {
3
+ "AutoProcessor": "processing_lfm3_vl.Lfm3VlProcessor"
4
+ },
5
+ "backend": "tokenizers",
6
+ "bos_token": "<|startoftext|>",
7
+ "eos_token": "<|im_end|>",
8
+ "extra_special_tokens": [],
9
+ "image_token": "<|image|>",
10
+ "is_local": true,
11
+ "local_files_only": false,
12
+ "model_max_length": 1000000000000000019884624838656,
13
+ "model_specific_special_tokens": {
14
+ "image_token": "<|image|>",
15
+ "video_token": "<|video|>",
16
+ "vision_end_token": "<|vision_end|>",
17
+ "vision_start_token": "<|vision_start|>"
18
+ },
19
+ "pad_token": "<|pad|>",
20
+ "processor_class": "Lfm3VlProcessor",
21
+ "response_template": {
22
+ "defaults": {
23
+ "role": "assistant"
24
+ },
25
+ "fields": {
26
+ "content": {
27
+ "close_pattern": "<\\|im_end\\|>\\s*",
28
+ "content": "text"
29
+ },
30
+ "reasoning_content": {
31
+ "close_pattern": "</think>\\s*",
32
+ "content": "text",
33
+ "open_pattern": "<think>\\n?"
34
+ },
35
+ "tool_calls": {
36
+ "close_pattern": "</tool_call>\\s*",
37
+ "content": "xml-inline",
38
+ "content_args": {
39
+ "tag_pattern": "<parameter=(?P<key>[^>\\n]+)>\\n?(?P<value>.*?)\\n?</parameter>(?=\\s*(?:<parameter=|</function>\\s*\\Z|\\Z))",
40
+ "value_parser": {
41
+ "args": {
42
+ "strip": false
43
+ },
44
+ "name": "text"
45
+ }
46
+ },
47
+ "open_pattern": "<tool_call>\\s*<function=(?P<name>[^\\n>]+)>",
48
+ "repeats": true,
49
+ "transform": {
50
+ "function": {
51
+ "arguments": "{content}",
52
+ "name": "{name}"
53
+ },
54
+ "type": "function"
55
+ }
56
+ }
57
+ },
58
+ "start_anchor_pattern": "<\\|im_start\\|>assistant\\n"
59
+ },
60
+ "tokenizer_class": "TokenizersBackend",
61
+ "video_token": "<|video|>",
62
+ "vision_end_token": "<|vision_end|>",
63
+ "vision_start_token": "<|vision_start|>"
64
+ }
video_processing_lfm3_vl.py ADDED
@@ -0,0 +1,396 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2026 The HuggingFace Team. All rights reserved.
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+ import math
15
+
16
+ import numpy as np
17
+ import torch
18
+ from torchvision.transforms.v2 import functional as tvF
19
+
20
+ from transformers.image_processing_utils import BatchFeature
21
+ from transformers.image_utils import PILImageResampling, SizeDict
22
+ from transformers.processing_utils import Unpack, VideosKwargs
23
+ from transformers.utils import TensorType, auto_docstring, logging
24
+ from transformers.video_processing_utils import BaseVideoProcessor
25
+ from transformers.video_utils import (
26
+ VideoMetadata,
27
+ group_videos_by_shape,
28
+ make_batched_metadata,
29
+ make_batched_videos,
30
+ reorder_videos,
31
+ )
32
+
33
+
34
+ logger = logging.get_logger(__name__)
35
+
36
+
37
+ class Lfm3VlVideoProcessorInitKwargs(VideosKwargs, total=False):
38
+ r"""
39
+ patch_size (`int`, *optional*, defaults to 16):
40
+ The spatial patch size of the vision encoder.
41
+ temporal_patch_size (`int`, *optional*, defaults to 2):
42
+ The temporal patch size of the vision encoder.
43
+ merge_size (`int`, *optional*, defaults to 2):
44
+ The merge size of the vision encoder to llm encoder.
45
+ min_frames (`int`, *optional*, defaults to 4):
46
+ The minimum number of frames to sample from video.
47
+ max_frames (`int`, *optional*, defaults to 768):
48
+ The maximum number of frames to sample from video.
49
+ cap_pixels_per_frame (`bool`, *optional*):
50
+ Whether to cap each frame's pixel cost the way the reference implementation (qwen-vl-utils) does:
51
+ per-frame pixels are limited to `min(max_video_tokens * factor**2, size["longest_edge"] / num_frames)`
52
+ and floored at `1.05 * size["shortest_edge"]`, so token cost scales with clip duration. Without the
53
+ cap, videos that sample few frames spend the whole `size["longest_edge"]` budget on those frames and
54
+ keep near-native per-frame resolution, so a short clip can cost almost as many tokens as a long
55
+ video. If unset, the current uncapped behavior is kept and a warning is emitted: the default will
56
+ change to `True` in v5.22, after which the argument will be removed.
57
+ max_video_tokens (`int`, *optional*, defaults to 768):
58
+ The per-frame token ceiling applied by `cap_pixels_per_frame`, in vision tokens per frame
59
+ (qwen-vl-utils' `VIDEO_MAX_TOKEN_NUM`).
60
+ """
61
+
62
+ patch_size: int
63
+ temporal_patch_size: int
64
+ merge_size: int
65
+ min_frames: int
66
+ max_frames: int
67
+ cap_pixels_per_frame: bool
68
+ max_video_tokens: int
69
+
70
+
71
+ def smart_resize(
72
+ num_frames: int,
73
+ height: int,
74
+ width: int,
75
+ temporal_factor: int = 2,
76
+ factor: int = 28,
77
+ min_pixels: int = 112 * 112,
78
+ max_pixels: int = 14 * 14 * 2 * 2 * 2 * 6144,
79
+ ):
80
+ if num_frames < temporal_factor:
81
+ raise ValueError(f"t:{num_frames} must be larger than temporal_factor:{temporal_factor}")
82
+ if height < factor or width < factor:
83
+ scale = max(factor / height, factor / width)
84
+ height = int(height * scale)
85
+ width = int(width * scale)
86
+
87
+ if max(height, width) / min(height, width) > 200:
88
+ raise ValueError(
89
+ f"absolute aspect ratio must be smaller than 200, got {max(height, width) / min(height, width)}"
90
+ )
91
+ h_bar = round(height / factor) * factor
92
+ w_bar = round(width / factor) * factor
93
+ t_bar = round(num_frames / temporal_factor) * temporal_factor
94
+
95
+ if t_bar * h_bar * w_bar > max_pixels:
96
+ beta = math.sqrt((num_frames * height * width) / max_pixels)
97
+ h_bar = max(factor, math.floor(height / beta / factor) * factor)
98
+ w_bar = max(factor, math.floor(width / beta / factor) * factor)
99
+ elif t_bar * h_bar * w_bar < min_pixels:
100
+ beta = math.sqrt(min_pixels / (num_frames * height * width))
101
+ h_bar = math.ceil(height * beta / factor) * factor
102
+ w_bar = math.ceil(width * beta / factor) * factor
103
+
104
+ return h_bar, w_bar
105
+
106
+
107
+ def _check_video_fps(metadata: VideoMetadata) -> None:
108
+ if metadata.fps is not None and not (math.isfinite(metadata.fps) and metadata.fps > 0):
109
+ raise ValueError(f"`video_metadata.fps` must be a positive number, got {metadata.fps}")
110
+
111
+
112
+ def _check_video_metadata_count(videos, video_metadata) -> None:
113
+ # the shared sampler pairs clips with metadata by `zip`, which drops the clips past the last entry
114
+ if video_metadata is None:
115
+ return
116
+ videos = make_batched_videos(videos)
117
+ entries = make_batched_metadata(videos, video_metadata) if video_metadata != [] else []
118
+ if len(entries) != len(videos):
119
+ raise ValueError(
120
+ f"Received {len(videos)} videos but {len(entries)} `video_metadata` entries; pass one entry per video"
121
+ )
122
+
123
+
124
+ @auto_docstring
125
+ class Lfm3VlVideoProcessor(BaseVideoProcessor):
126
+ resample = PILImageResampling.BICUBIC
127
+ size = {"shortest_edge": 4_096, "longest_edge": 25_165_824}
128
+ image_mean = [0.5, 0.5, 0.5]
129
+ image_std = [0.5, 0.5, 0.5]
130
+ do_resize = True
131
+ do_rescale = True
132
+ do_normalize = True
133
+ do_convert_rgb = True
134
+ patch_size = 16
135
+ temporal_patch_size = 2
136
+ merge_size = 2
137
+ min_frames = 4
138
+ max_frames = 768
139
+ do_sample_frames = True
140
+ cap_pixels_per_frame = False
141
+ max_video_tokens = 768
142
+ valid_kwargs = Lfm3VlVideoProcessorInitKwargs
143
+ model_input_names = ["pixel_values_videos", "video_grid_thw"]
144
+ max_image_size = {"longest_edge": 28 * 28 * 2 * 30000}
145
+ max_duration = None
146
+ num_frames = None
147
+ fps = 2.0
148
+
149
+ def __init__(self, **kwargs: Unpack[Lfm3VlVideoProcessorInitKwargs]):
150
+ super().__init__(**kwargs)
151
+
152
+ def sample_frames(
153
+ self, metadata: VideoMetadata, num_frames: int | None = None, fps: int | float | None = None, **kwargs
154
+ ):
155
+ """Decoded frames whose metadata carries no fps are sampled and timestamped at the 24 fps the shared sampler
156
+ assumes (with a warning); pass `video_metadata` with the clip's fps for accurate timestamps."""
157
+ _check_video_fps(metadata)
158
+ if fps is not None and not (math.isfinite(fps) and fps > 0):
159
+ raise ValueError(f"`fps` must be a positive number, got {fps}")
160
+ if fps is not None and num_frames is not None:
161
+ raise ValueError("`num_frames` and `fps` are mutually exclusive arguments, please use only one!")
162
+
163
+ total_num_frames = metadata.total_num_frames
164
+ fps = fps if fps is not None else self.fps
165
+
166
+ # If num_frames is not given but fps is, calculate num_frames from fps
167
+ if num_frames is None and fps is not None:
168
+ if metadata.fps is None:
169
+ metadata.fps = 24
170
+ logger.warning_once(
171
+ "Asked to sample `fps` frames per second but no video metadata was provided which is required when sampling with `fps`. "
172
+ "Defaulting to `fps=24`. Please provide `video_metadata` for more accurate results."
173
+ )
174
+ num_frames = int(total_num_frames / metadata.fps * fps)
175
+ num_frames = min(max(num_frames, self.min_frames), self.max_frames, total_num_frames)
176
+
177
+ if num_frames is None:
178
+ num_frames = min(max(total_num_frames, self.min_frames), self.max_frames)
179
+
180
+ indices = np.linspace(0, total_num_frames - 1, num_frames).round().astype(int)
181
+
182
+ return indices
183
+
184
+ def resize(
185
+ self,
186
+ videos: "torch.Tensor",
187
+ size: SizeDict,
188
+ resample: "PILImageResampling | tvF.InterpolationMode | int | None",
189
+ factor: int,
190
+ temporal_factor: int,
191
+ cap_pixels_per_frame: bool | None = None,
192
+ **kwargs,
193
+ ) -> "torch.Tensor":
194
+ """Resize dynamically based on input video aspect ratio."""
195
+ if not size.shortest_edge or not size.longest_edge:
196
+ raise ValueError(f"`size` dict must contain 'shortest_edge' and 'longest_edge' keys but got {size}.")
197
+
198
+ num_frames = videos.shape[1]
199
+ max_pixels = size.longest_edge
200
+ if cap_pixels_per_frame:
201
+ # per-frame pixels are capped at `max_video_tokens` patches or the budget's even share per frame
202
+ frame_cap = self.max_video_tokens * factor * factor
203
+ pixels_per_frame = max(min(frame_cap, size.longest_edge // num_frames), int(size.shortest_edge * 1.05))
204
+ max_pixels = pixels_per_frame * num_frames
205
+
206
+ height, width = videos.shape[-2:]
207
+ resized_height, resized_width = smart_resize(
208
+ height=height,
209
+ width=width,
210
+ num_frames=num_frames,
211
+ factor=factor,
212
+ temporal_factor=temporal_factor,
213
+ min_pixels=size.shortest_edge,
214
+ max_pixels=max_pixels,
215
+ )
216
+ return super().resize(
217
+ image=videos,
218
+ size=SizeDict(height=resized_height, width=resized_width),
219
+ resample=resample,
220
+ )
221
+
222
+ def patchify(
223
+ self,
224
+ videos: "torch.Tensor",
225
+ patch_size: int,
226
+ merge_size: int,
227
+ temporal_patch_size: int,
228
+ ) -> tuple["torch.Tensor", int, int]:
229
+ "Patchifies each video into flat layout of shape (`seq_len`, `patch_dim`) so we can concat dynamically shaped pixels."
230
+ batch_size, num_frames, channel, resized_height, resized_width = videos.shape
231
+
232
+ # Check that videos have `num_frames` divisible by `temporal_patch_size`
233
+ if pad := -num_frames % temporal_patch_size:
234
+ repeats = videos[:, -1:].expand(-1, pad, -1, -1, -1)
235
+ videos = torch.cat((videos, repeats), dim=1)
236
+ num_frames += pad
237
+
238
+ grid_t = num_frames // temporal_patch_size
239
+ grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
240
+
241
+ patches = videos.view(
242
+ batch_size,
243
+ grid_t,
244
+ temporal_patch_size,
245
+ channel,
246
+ grid_h // merge_size,
247
+ merge_size,
248
+ patch_size,
249
+ grid_w // merge_size,
250
+ merge_size,
251
+ patch_size,
252
+ )
253
+ patches = patches.permute(0, 1, 4, 7, 5, 8, 3, 2, 6, 9)
254
+ flatten_patches = patches.reshape(
255
+ batch_size,
256
+ grid_t * grid_h * grid_w,
257
+ channel * temporal_patch_size * patch_size * patch_size,
258
+ )
259
+
260
+ return flatten_patches, grid_t, grid_h, grid_w
261
+
262
+ def _preprocess(
263
+ self,
264
+ videos: list["torch.Tensor"],
265
+ do_resize: bool,
266
+ size: SizeDict,
267
+ resample: "PILImageResampling | tvF.InterpolationMode | int | None",
268
+ do_rescale: bool,
269
+ rescale_factor: float,
270
+ do_normalize: bool,
271
+ image_mean: float | list[float] | None,
272
+ image_std: float | list[float] | None,
273
+ patch_size: int | None = None,
274
+ temporal_patch_size: int | None = None,
275
+ merge_size: int | None = None,
276
+ cap_pixels_per_frame: bool | None = None,
277
+ return_tensors: str | TensorType | None = None,
278
+ **kwargs,
279
+ ):
280
+ if cap_pixels_per_frame is None:
281
+ logger.warning_once(
282
+ "Lfm3Vl video processing does not apply the per-frame pixel cap the reference "
283
+ "implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they "
284
+ "would there. In v5.22 the capped behavior will become the default and "
285
+ "`cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the "
286
+ "reference behavior now, or `False` to keep the current behavior and silence this "
287
+ "warning."
288
+ )
289
+ cap_pixels_per_frame = False
290
+ # Group videos by size for batched resizing
291
+ grouped_videos, grouped_videos_index = group_videos_by_shape(videos)
292
+ resized_videos_grouped = {}
293
+ for shape, stacked_videos in grouped_videos.items():
294
+ if do_resize:
295
+ stacked_videos = self.resize(
296
+ videos=stacked_videos,
297
+ size=size,
298
+ resample=resample,
299
+ factor=patch_size * merge_size,
300
+ temporal_factor=temporal_patch_size,
301
+ cap_pixels_per_frame=cap_pixels_per_frame,
302
+ )
303
+ resized_videos_grouped[shape] = stacked_videos
304
+ resized_videos = reorder_videos(resized_videos_grouped, grouped_videos_index)
305
+
306
+ # Group videos by size for further processing
307
+ # Needed in case do_resize is False, or resize returns videos with different sizes
308
+ grouped_videos, grouped_videos_index = group_videos_by_shape(resized_videos)
309
+ processed_videos_grouped = {}
310
+ processed_grids = {}
311
+ for shape, stacked_videos in grouped_videos.items():
312
+ # Fused rescale and normalize
313
+ stacked_videos = self.rescale_and_normalize(
314
+ stacked_videos, do_rescale, rescale_factor, do_normalize, image_mean, image_std
315
+ )
316
+ patches, grid_t, grid_h, grid_w = self.patchify(
317
+ stacked_videos,
318
+ patch_size=patch_size,
319
+ merge_size=merge_size,
320
+ temporal_patch_size=temporal_patch_size,
321
+ )
322
+
323
+ processed_videos_grouped[shape] = patches
324
+ processed_grids[shape] = [[grid_t, grid_h, grid_w]] * len(stacked_videos)
325
+
326
+ processed_videos = reorder_videos(processed_videos_grouped, grouped_videos_index)
327
+ processed_grids = reorder_videos(processed_grids, grouped_videos_index)
328
+ pixel_values_videos = torch.cat(processed_videos, dim=0)
329
+ video_grid_thw = torch.tensor(processed_grids)
330
+
331
+ return BatchFeature(
332
+ data={"pixel_values_videos": pixel_values_videos, "video_grid_thw": video_grid_thw},
333
+ tensor_type=return_tensors,
334
+ )
335
+
336
+ def get_num_of_video_patches(self, num_frames: int, height: int, width: int, videos_kwargs=None):
337
+ """
338
+ A utility that returns number of video patches a given video size.
339
+
340
+ Args:
341
+ num_frames (`int`):
342
+ Number of frames in the input video.
343
+ height (`int`):
344
+ Height of the input video.
345
+ width (`int`):
346
+ Width of the input video.
347
+ videos_kwargs (`dict`, *optional*)
348
+ Any kwargs to override defaults of the video processor.
349
+ Returns:
350
+ `int`: Number of video patches per video.
351
+ """
352
+ videos_kwargs = videos_kwargs if videos_kwargs is not None else {}
353
+ size = videos_kwargs.get("size", None) or self.size
354
+ min_pixels = size["shortest_edge"]
355
+ max_pixels = size["longest_edge"]
356
+ patch_size = videos_kwargs.get("patch_size", None) or self.patch_size
357
+ merge_size = videos_kwargs.get("merge_size", None) or self.merge_size
358
+ temporal_patch_size = videos_kwargs.get("temporal_patch_size", None) or self.temporal_patch_size
359
+ cap_pixels_per_frame = videos_kwargs.get("cap_pixels_per_frame", None) or self.cap_pixels_per_frame
360
+
361
+ factor = patch_size * merge_size
362
+ if cap_pixels_per_frame:
363
+ # Keep the count in sync with `resize` when the per-frame cap is active.
364
+ frame_cap = self.max_video_tokens * factor * factor
365
+ pixels_per_frame = max(min(frame_cap, max_pixels // num_frames), int(min_pixels * 1.05))
366
+ max_pixels = pixels_per_frame * num_frames
367
+
368
+ resized_height, resized_width = smart_resize(
369
+ num_frames,
370
+ height,
371
+ width,
372
+ temporal_factor=temporal_patch_size,
373
+ factor=factor,
374
+ min_pixels=min_pixels,
375
+ max_pixels=max_pixels,
376
+ )
377
+ grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
378
+ grid_t = (num_frames + -num_frames % temporal_patch_size) // temporal_patch_size
379
+ return grid_t * grid_h * grid_w
380
+
381
+ def preprocess(self, videos, **kwargs):
382
+ _check_video_metadata_count(videos, kwargs.get("video_metadata"))
383
+ # the shared `preprocess` fills every unset option from the instance before `sample_frames` runs, so the
384
+ # caller's options are told apart from the instance ones here: a caller option beats an instance one, and a
385
+ # configured `num_frames` beats the default fps
386
+ num_frames, fps = kwargs.get("num_frames"), kwargs.get("fps")
387
+ if num_frames is not None and fps is not None:
388
+ raise ValueError("`num_frames` and `fps` are mutually exclusive arguments, please use only one!")
389
+ if num_frames is not None or (fps is None and self.num_frames is not None):
390
+ kwargs["fps"] = None
391
+ elif fps is not None:
392
+ kwargs["num_frames"] = None
393
+ return super().preprocess(videos, **kwargs)
394
+
395
+
396
+ __all__ = ["Lfm3VlVideoProcessor"]