chen-l commited on
Commit
3402dca
·
verified ·
1 Parent(s): 977d177

Add files using upload-large-folder tool

Browse files
.gitattributes CHANGED
@@ -1,35 +1,2 @@
1
- *.7z filter=lfs diff=lfs merge=lfs -text
2
- *.arrow filter=lfs diff=lfs merge=lfs -text
3
- *.bin filter=lfs diff=lfs merge=lfs -text
4
- *.bz2 filter=lfs diff=lfs merge=lfs -text
5
- *.ckpt filter=lfs diff=lfs merge=lfs -text
6
- *.ftz filter=lfs diff=lfs merge=lfs -text
7
- *.gz filter=lfs diff=lfs merge=lfs -text
8
- *.h5 filter=lfs diff=lfs merge=lfs -text
9
- *.joblib filter=lfs diff=lfs merge=lfs -text
10
- *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
- *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
- *.model filter=lfs diff=lfs merge=lfs -text
13
- *.msgpack filter=lfs diff=lfs merge=lfs -text
14
- *.npy filter=lfs diff=lfs merge=lfs -text
15
- *.npz filter=lfs diff=lfs merge=lfs -text
16
- *.onnx filter=lfs diff=lfs merge=lfs -text
17
- *.ot filter=lfs diff=lfs merge=lfs -text
18
- *.parquet filter=lfs diff=lfs merge=lfs -text
19
- *.pb filter=lfs diff=lfs merge=lfs -text
20
- *.pickle filter=lfs diff=lfs merge=lfs -text
21
- *.pkl filter=lfs diff=lfs merge=lfs -text
22
- *.pt filter=lfs diff=lfs merge=lfs -text
23
- *.pth filter=lfs diff=lfs merge=lfs -text
24
- *.rar filter=lfs diff=lfs merge=lfs -text
25
  *.safetensors filter=lfs diff=lfs merge=lfs -text
26
- saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
- *.tar.* filter=lfs diff=lfs merge=lfs -text
28
- *.tar filter=lfs diff=lfs merge=lfs -text
29
- *.tflite filter=lfs diff=lfs merge=lfs -text
30
- *.tgz filter=lfs diff=lfs merge=lfs -text
31
- *.wasm filter=lfs diff=lfs merge=lfs -text
32
- *.xz filter=lfs diff=lfs merge=lfs -text
33
- *.zip filter=lfs diff=lfs merge=lfs -text
34
- *.zst filter=lfs diff=lfs merge=lfs -text
35
- *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  *.safetensors filter=lfs diff=lfs merge=lfs -text
2
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
LICENSE ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ Apache License
3
+ Version 2.0, January 2004
4
+ http://www.apache.org/licenses/
5
+
6
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
7
+
8
+ 1. Definitions.
9
+
10
+ "License" shall mean the terms and conditions for use, reproduction,
11
+ and distribution as defined by Sections 1 through 9 of this document.
12
+
13
+ "Licensor" shall mean the copyright owner or entity authorized by
14
+ the copyright owner that is granting the License.
15
+
16
+ "Legal Entity" shall mean the union of the acting entity and all
17
+ other entities that control, are controlled by, or are under common
18
+ control with that entity. For the purposes of this definition,
19
+ "control" means (i) the power, direct or indirect, to cause the
20
+ direction or management of such entity, whether by contract or
21
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
22
+ outstanding shares, or (iii) beneficial ownership of such entity.
23
+
24
+ "You" (or "Your") shall mean an individual or Legal Entity
25
+ exercising permissions granted by this License.
26
+
27
+ "Source" form shall mean the preferred form for making modifications,
28
+ including but not limited to software source code, documentation
29
+ source, and configuration files.
30
+
31
+ "Object" form shall mean any form resulting from mechanical
32
+ transformation or translation of a Source form, including but
33
+ not limited to compiled object code, generated documentation,
34
+ and conversions to other media types.
35
+
36
+ "Work" shall mean the work of authorship, whether in Source or
37
+ Object form, made available under the License, as indicated by a
38
+ copyright notice that is included in or attached to the work
39
+ (an example is provided in the Appendix below).
40
+
41
+ "Derivative Works" shall mean any work, whether in Source or Object
42
+ form, that is based on (or derived from) the Work and for which the
43
+ editorial revisions, annotations, elaborations, or other modifications
44
+ represent, as a whole, an original work of authorship. For the purposes
45
+ of this License, Derivative Works shall not include works that remain
46
+ separable from, or merely link (or bind by name) to the interfaces of,
47
+ the Work and Derivative Works thereof.
48
+
49
+ "Contribution" shall mean any work of authorship, including
50
+ the original version of the Work and any modifications or additions
51
+ to that Work or Derivative Works thereof, that is intentionally
52
+ submitted to Licensor for inclusion in the Work by the copyright owner
53
+ or by an individual or Legal Entity authorized to submit on behalf of
54
+ the copyright owner. For the purposes of this definition, "submitted"
55
+ means any form of electronic, verbal, or written communication sent
56
+ to the Licensor or its representatives, including but not limited to
57
+ communication on electronic mailing lists, source code control systems,
58
+ and issue tracking systems that are managed by, or on behalf of, the
59
+ Licensor for the purpose of discussing and improving the Work, but
60
+ excluding communication that is conspicuously marked or otherwise
61
+ designated in writing by the copyright owner as "Not a Contribution."
62
+
63
+ "Contributor" shall mean Licensor and any individual or Legal Entity
64
+ on behalf of whom a Contribution has been received by Licensor and
65
+ subsequently incorporated within the Work.
66
+
67
+ 2. Grant of Copyright License. Subject to the terms and conditions of
68
+ this License, each Contributor hereby grants to You a perpetual,
69
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
70
+ copyright license to reproduce, prepare Derivative Works of,
71
+ publicly display, publicly perform, sublicense, and distribute the
72
+ Work and such Derivative Works in Source or Object form.
73
+
74
+ 3. Grant of Patent License. Subject to the terms and conditions of
75
+ this License, each Contributor hereby grants to You a perpetual,
76
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
77
+ (except as stated in this section) patent license to make, have made,
78
+ use, offer to sell, sell, import, and otherwise transfer the Work,
79
+ where such license applies only to those patent claims licensable
80
+ by such Contributor that are necessarily infringed by their
81
+ Contribution(s) alone or by combination of their Contribution(s)
82
+ with the Work to which such Contribution(s) was submitted. If You
83
+ institute patent litigation against any entity (including a
84
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
85
+ or a Contribution incorporated within the Work constitutes direct
86
+ or contributory patent infringement, then any patent licenses
87
+ granted to You under this License for that Work shall terminate
88
+ as of the date such litigation is filed.
89
+
90
+ 4. Redistribution. You may reproduce and distribute copies of the
91
+ Work or Derivative Works thereof in any medium, with or without
92
+ modifications, and in Source or Object form, provided that You
93
+ meet the following conditions:
94
+
95
+ (a) You must give any other recipients of the Work or
96
+ Derivative Works a copy of this License; and
97
+
98
+ (b) You must cause any modified files to carry prominent notices
99
+ stating that You changed the files; and
100
+
101
+ (c) You must retain, in the Source form of any Derivative Works
102
+ that You distribute, all copyright, patent, trademark, and
103
+ attribution notices from the Source form of the Work,
104
+ excluding those notices that do not pertain to any part of
105
+ the Derivative Works; and
106
+
107
+ (d) If the Work includes a "NOTICE" text file as part of its
108
+ distribution, then any Derivative Works that You distribute must
109
+ include a readable copy of the attribution notices contained
110
+ within such NOTICE file, excluding those notices that do not
111
+ pertain to any part of the Derivative Works, in at least one
112
+ of the following places: within a NOTICE text file distributed
113
+ as part of the Derivative Works; within the Source form or
114
+ documentation, if provided along with the Derivative Works; or,
115
+ within a display generated by the Derivative Works, if and
116
+ wherever such third-party notices normally appear. The contents
117
+ of the NOTICE file are for informational purposes only and
118
+ do not modify the License. You may add Your own attribution
119
+ notices within Derivative Works that You distribute, alongside
120
+ or as an addendum to the NOTICE text from the Work, provided
121
+ that such additional attribution notices cannot be construed
122
+ as modifying the License.
123
+
124
+ You may add Your own copyright statement to Your modifications and
125
+ may provide additional or different license terms and conditions
126
+ for use, reproduction, or distribution of Your modifications, or
127
+ for any such Derivative Works as a whole, provided Your use,
128
+ reproduction, and distribution of the Work otherwise complies with
129
+ the conditions stated in this License.
130
+
131
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
132
+ any Contribution intentionally submitted for inclusion in the Work
133
+ by You to the Licensor shall be under the terms and conditions of
134
+ this License, without any additional terms or conditions.
135
+ Notwithstanding the above, nothing herein shall supersede or modify
136
+ the terms of any separate license agreement you may have executed
137
+ with Licensor regarding such Contributions.
138
+
139
+ 6. Trademarks. This License does not grant permission to use the trade
140
+ names, trademarks, service marks, or product names of the Licensor,
141
+ except as required for reasonable and customary use in describing the
142
+ origin of the Work and reproducing the content of the NOTICE file.
143
+
144
+ 7. Disclaimer of Warranty. Unless required by applicable law or
145
+ agreed to in writing, Licensor provides the Work (and each
146
+ Contributor provides its Contributions) on an "AS IS" BASIS,
147
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148
+ implied, including, without limitation, any warranties or conditions
149
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150
+ PARTICULAR PURPOSE. You are solely responsible for determining the
151
+ appropriateness of using or redistributing the Work and assume any
152
+ risks associated with Your exercise of permissions under this License.
153
+
154
+ 8. Limitation of Liability. In no event and under no legal theory,
155
+ whether in tort (including negligence), contract, or otherwise,
156
+ unless required by applicable law (such as deliberate and grossly
157
+ negligent acts) or agreed to in writing, shall any Contributor be
158
+ liable to You for damages, including any direct, indirect, special,
159
+ incidental, or consequential damages of any character arising as a
160
+ result of this License or out of the use or inability to use the
161
+ Work (including but not limited to damages for loss of goodwill,
162
+ work stoppage, computer failure or malfunction, or any and all
163
+ other commercial damages or losses), even if such Contributor
164
+ has been advised of the possibility of such damages.
165
+
166
+ 9. Accepting Warranty or Additional Liability. While redistributing
167
+ the Work or Derivative Works thereof, You may choose to offer,
168
+ and charge a fee for, acceptance of support, warranty, indemnity,
169
+ or other liability obligations and/or rights consistent with this
170
+ License. However, in accepting such obligations, You may act only
171
+ on Your own behalf and on Your sole responsibility, not on behalf
172
+ of any other Contributor, and only if You agree to indemnify,
173
+ defend, and hold each Contributor harmless for any liability
174
+ incurred by, or claims asserted against, such Contributor by reason
175
+ of your accepting any such warranty or additional liability.
176
+
177
+ END OF TERMS AND CONDITIONS
178
+
179
+ APPENDIX: How to apply the Apache License to your work.
180
+
181
+ To apply the Apache License to your work, attach the following
182
+ boilerplate notice, with the fields enclosed by brackets "[]"
183
+ replaced with your own identifying information. (Don't include
184
+ the brackets!) The text should be enclosed in the appropriate
185
+ comment syntax for the file format. We also recommend that a
186
+ file or class name and description of purpose be included on the
187
+ same "printed page" as the copyright notice for easier
188
+ identification within third-party archives.
189
+
190
+ Copyright 2024 Alibaba Cloud
191
+
192
+ Licensed under the Apache License, Version 2.0 (the "License");
193
+ you may not use this file except in compliance with the License.
194
+ You may obtain a copy of the License at
195
+
196
+ http://www.apache.org/licenses/LICENSE-2.0
197
+
198
+ Unless required by applicable law or agreed to in writing, software
199
+ distributed under the License is distributed on an "AS IS" BASIS,
200
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201
+ See the License for the specific language governing permissions and
202
+ limitations under the License.
README.md ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: transformers
4
+ pipeline_tag: text-generation
5
+ base_model: chen-l/LiveMem-SFT
6
+ tags:
7
+ - livemem
8
+ - qwen3
9
+ - custom-code
10
+ - long-context
11
+ - reinforcement-learning
12
+ - text-generation
13
+ ---
14
+
15
+ # LiveMem-RL
16
+
17
+ LiveMem-RL is the reinforcement-learning checkpoint of LiveMem-4B-SFT. It
18
+ uses a Qwen3 attention path in parallel with a Gated DeltaNet 2 (GDN2)
19
+ recurrent memory path at every decoder layer:
20
+
21
+ ```text
22
+ layer output = Qwen3 attention output + GDN2 memory output
23
+ ```
24
+
25
+ This checkpoint was trained with GRPO from `chen-l/LiveMem-SFT`. During RL,
26
+ the Qwen3 main path remained frozen and the memory side path was updated. The
27
+ configured maximum context length is 262,144 tokens; actual usable context
28
+ depends on GPU memory and inference backend.
29
+
30
+ ## Transformers usage
31
+
32
+ LiveMem uses custom model code and GDN2 Triton kernels. A CUDA environment is
33
+ required for inference.
34
+
35
+ ```bash
36
+ pip install -r requirements.txt
37
+ ```
38
+
39
+ ```python
40
+ import torch
41
+ from transformers import AutoModelForCausalLM, AutoTokenizer
42
+
43
+ model_id = "chen-l/LiveMem-4B-RL"
44
+ tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
45
+ model = AutoModelForCausalLM.from_pretrained(
46
+ model_id,
47
+ trust_remote_code=True,
48
+ dtype=torch.bfloat16,
49
+ device_map="auto",
50
+ )
51
+
52
+ messages = [{"role": "user", "content": "Answer using the supplied long context."}]
53
+ inputs = tokenizer.apply_chat_template(
54
+ messages,
55
+ add_generation_prompt=True,
56
+ return_tensors="pt",
57
+ return_dict=True,
58
+ ).to(model.device)
59
+ outputs = model.generate(**inputs, max_new_tokens=256)
60
+ print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
61
+ ```
62
+
63
+ `trust_remote_code=True` is required because LiveMem is not a built-in
64
+ Transformers architecture.
__init__.py ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ from .configuration_livemem import LiveMemConfig
2
+ from .modeling_livemem import LiveMemForCausalLM, LiveMemModel
3
+
4
+ __all__ = ["LiveMemConfig", "LiveMemModel", "LiveMemForCausalLM"]
chat_template.jinja ADDED
@@ -0,0 +1,61 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- if tools %}
2
+ {{- '<|im_start|>system\n' }}
3
+ {%- if messages[0].role == 'system' %}
4
+ {{- messages[0].content + '\n\n' }}
5
+ {%- endif %}
6
+ {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
7
+ {%- for tool in tools %}
8
+ {{- "\n" }}
9
+ {{- tool | tojson }}
10
+ {%- endfor %}
11
+ {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
12
+ {%- else %}
13
+ {%- if messages[0].role == 'system' %}
14
+ {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
15
+ {%- endif %}
16
+ {%- endif %}
17
+ {%- for message in messages %}
18
+ {%- if message.content is string %}
19
+ {%- set content = message.content %}
20
+ {%- else %}
21
+ {%- set content = '' %}
22
+ {%- endif %}
23
+ {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
24
+ {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
25
+ {%- elif message.role == "assistant" %}
26
+ {{- '<|im_start|>' + message.role + '\n' + content }}
27
+ {%- if message.tool_calls %}
28
+ {%- for tool_call in message.tool_calls %}
29
+ {%- if (loop.first and content) or (not loop.first) %}
30
+ {{- '\n' }}
31
+ {%- endif %}
32
+ {%- if tool_call.function %}
33
+ {%- set tool_call = tool_call.function %}
34
+ {%- endif %}
35
+ {{- '<tool_call>\n{"name": "' }}
36
+ {{- tool_call.name }}
37
+ {{- '", "arguments": ' }}
38
+ {%- if tool_call.arguments is string %}
39
+ {{- tool_call.arguments }}
40
+ {%- else %}
41
+ {{- tool_call.arguments | tojson }}
42
+ {%- endif %}
43
+ {{- '}\n</tool_call>' }}
44
+ {%- endfor %}
45
+ {%- endif %}
46
+ {{- '<|im_end|>\n' }}
47
+ {%- elif message.role == "tool" %}
48
+ {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
49
+ {{- '<|im_start|>user' }}
50
+ {%- endif %}
51
+ {{- '\n<tool_response>\n' }}
52
+ {{- content }}
53
+ {{- '\n</tool_response>' }}
54
+ {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
55
+ {{- '<|im_end|>\n' }}
56
+ {%- endif %}
57
+ {%- endif %}
58
+ {%- endfor %}
59
+ {%- if add_generation_prompt %}
60
+ {{- '<|im_start|>assistant\n' }}
61
+ {%- endif %}
config.json ADDED
@@ -0,0 +1,86 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LiveMemForCausalLM"
4
+ ],
5
+ "auto_map": {
6
+ "AutoConfig": "configuration_livemem.LiveMemConfig",
7
+ "AutoModel": "modeling_livemem.LiveMemModel",
8
+ "AutoModelForCausalLM": "modeling_livemem.LiveMemForCausalLM"
9
+ },
10
+ "attention_bias": false,
11
+ "attention_dropout": 0.0,
12
+ "bos_token_id": 151643,
13
+ "dtype": "bfloat16",
14
+ "eos_token_id": 151645,
15
+ "head_dim": 128,
16
+ "hidden_act": "silu",
17
+ "hidden_size": 2560,
18
+ "initializer_range": 0.02,
19
+ "intermediate_size": 9728,
20
+ "layer_types": [
21
+ "full_attention",
22
+ "full_attention",
23
+ "full_attention",
24
+ "full_attention",
25
+ "full_attention",
26
+ "full_attention",
27
+ "full_attention",
28
+ "full_attention",
29
+ "full_attention",
30
+ "full_attention",
31
+ "full_attention",
32
+ "full_attention",
33
+ "full_attention",
34
+ "full_attention",
35
+ "full_attention",
36
+ "full_attention",
37
+ "full_attention",
38
+ "full_attention",
39
+ "full_attention",
40
+ "full_attention",
41
+ "full_attention",
42
+ "full_attention",
43
+ "full_attention",
44
+ "full_attention",
45
+ "full_attention",
46
+ "full_attention",
47
+ "full_attention",
48
+ "full_attention",
49
+ "full_attention",
50
+ "full_attention",
51
+ "full_attention",
52
+ "full_attention",
53
+ "full_attention",
54
+ "full_attention",
55
+ "full_attention",
56
+ "full_attention"
57
+ ],
58
+ "max_position_embeddings": 262144,
59
+ "max_window_layers": 36,
60
+ "mem_conv_bias": false,
61
+ "mem_conv_size": 4,
62
+ "mem_expand_v": 1.0,
63
+ "mem_head_dim": 128,
64
+ "mem_layers": null,
65
+ "mem_norm_eps": 1e-06,
66
+ "mem_num_heads": 32,
67
+ "mem_num_v_heads": 32,
68
+ "mem_o_proj_zero_init": true,
69
+ "memory_design": "X",
70
+ "model_type": "livemem",
71
+ "num_attention_heads": 32,
72
+ "num_hidden_layers": 36,
73
+ "num_key_value_heads": 8,
74
+ "pad_token_id": 151643,
75
+ "rms_norm_eps": 1e-06,
76
+ "rope_parameters": {
77
+ "rope_theta": 5000000,
78
+ "rope_type": "default"
79
+ },
80
+ "sliding_window": null,
81
+ "tie_word_embeddings": true,
82
+ "transformers_version": "5.12.1",
83
+ "use_cache": true,
84
+ "use_sliding_window": false,
85
+ "vocab_size": 151936
86
+ }
configuration_livemem.py ADDED
@@ -0,0 +1,61 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Config for the memory-augmented Qwen3 (主路 Qwen3Attention ‖ 边路 GDN2)."""
2
+ from __future__ import annotations
3
+
4
+ from transformers.models.qwen3.configuration_qwen3 import Qwen3Config
5
+
6
+
7
+ class LiveMemConfig(Qwen3Config):
8
+ """Qwen3 + per-attention-head GDN2 memory side-branch.
9
+
10
+ All base Qwen3 fields are inherited unchanged. The `mem_*` fields configure
11
+ the side branch and the two memory mechanisms (Design X / Design Y).
12
+ """
13
+
14
+ model_type = "livemem"
15
+
16
+ def __init__(
17
+ self,
18
+ # which mechanism: "X" = 连续扫描 (continuous scan, write_mask=None);
19
+ # "Y" = 门控读写解耦 (freeze gates on read tokens).
20
+ memory_design: str = "Y",
21
+ # layers that get a memory branch; None = all layers.
22
+ mem_layers: list[int] | None = None,
23
+ # GDN2 side-branch hyper-params. The legacy/default geometry is a
24
+ # full-MHA copy of Qwen3 (8 KV heads repeated 4x -> 32 heads), per plan
25
+ # 02 §1.3. Newer experiments can explicitly set mem_num_heads=8,
26
+ # mem_num_v_heads=8, mem_expand_v=4 to keep similar state capacity with
27
+ # fewer Q/K heads.
28
+ mem_head_dim: int | None = None,
29
+ mem_num_heads: int | None = None,
30
+ mem_num_v_heads: int | None = None,
31
+ mem_expand_v: float = 1.0,
32
+ mem_conv_size: int = 4,
33
+ mem_conv_bias: bool = False,
34
+ mem_norm_eps: float | None = None,
35
+ # zero-init the side o_proj so training starts ≈ original Qwen3 (plan T1).
36
+ mem_o_proj_zero_init: bool = True,
37
+ **kwargs,
38
+ ) -> None:
39
+ super().__init__(**kwargs)
40
+ if memory_design not in ("X", "Y"):
41
+ raise ValueError(f"memory_design must be 'X' or 'Y', got {memory_design!r}")
42
+ self.memory_design = memory_design
43
+ self.mem_layers = mem_layers
44
+ self.mem_head_dim = mem_head_dim if mem_head_dim is not None else self.head_dim
45
+ self.mem_num_heads = (
46
+ mem_num_heads if mem_num_heads is not None else self.num_attention_heads
47
+ )
48
+ self.mem_num_v_heads = (
49
+ mem_num_v_heads if mem_num_v_heads is not None else self.mem_num_heads
50
+ )
51
+ self.mem_expand_v = mem_expand_v
52
+ self.mem_conv_size = mem_conv_size
53
+ self.mem_conv_bias = mem_conv_bias
54
+ self.mem_norm_eps = mem_norm_eps if mem_norm_eps is not None else self.rms_norm_eps
55
+ self.mem_o_proj_zero_init = mem_o_proj_zero_init
56
+
57
+ @property
58
+ def memory_layer_indices(self) -> list[int]:
59
+ if self.mem_layers is None:
60
+ return list(range(self.num_hidden_layers))
61
+ return list(self.mem_layers)
generation_config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 151643,
4
+ "eos_token_id": 151645,
5
+ "output_attentions": false,
6
+ "output_hidden_states": false,
7
+ "transformers_version": "5.12.1",
8
+ "use_cache": true
9
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1a431577034b0ce43535d20fde78df765a19b992ad0130cdd7826e63fe5f13f0
3
+ size 12701729032
modeling_livemem.py ADDED
@@ -0,0 +1,496 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Memory-augmented Qwen3: 主路 Qwen3Attention ‖ 边路 GDN2, o = o_main + o_side.
2
+
3
+ Design X (continuous scan) and Design Y (gated read/write) share one skeleton
4
+ and the same attention eviction mask; they differ only in the RNN `write_mask`:
5
+ - X: write_mask = None (RNN scans every token; state = compression of all)
6
+ - Y: write_mask = is_evicted (open gate only on the evict/compress region)
7
+ """
8
+ from __future__ import annotations
9
+
10
+ import torch
11
+ import torch.nn as nn
12
+
13
+ from transformers.cache_utils import Cache, DynamicCache
14
+ from transformers.masking_utils import create_causal_mask, create_sliding_window_causal_mask
15
+ from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast
16
+ from transformers.models.qwen3.modeling_qwen3 import (
17
+ Qwen3Attention,
18
+ Qwen3ForCausalLM,
19
+ Qwen3Model,
20
+ Qwen3PreTrainedModel,
21
+ )
22
+
23
+ from .configuration_livemem import LiveMemConfig
24
+ from .modeling_livemem_gdn2 import LiveMemGatedDeltaNet2
25
+
26
+
27
+ def make_memory_and_mask(
28
+ is_evicted: torch.Tensor | None = None,
29
+ segment_ids: torch.Tensor | None = None,
30
+ seq_ids: torch.Tensor | None = None,
31
+ chunk_id: torch.Tensor | None = None,
32
+ evict_step: torch.Tensor | None = None,
33
+ ):
34
+ """Build an `and_mask_function` for `create_causal_mask` from up to four
35
+ constraints, AND-combined with causal by `create_causal_mask`:
36
+
37
+ - dynamic eviction (`chunk_id [B,T]`, `evict_step [B,T]` int): the real
38
+ training path. Each token belongs to a chunk; `evict_step[t]` is the chunk
39
+ step at which that token's chunk is evicted to the RNN state. keep(q,kv) =
40
+ `evict_step[kv] > chunk_id[q]` — key kv's chunk must still be live when
41
+ query q's chunk is processed (Design X: RNN scans all, attention evicts).
42
+ - static eviction (`is_evicted [B,T]` bool): the simple synthetic variant.
43
+ keep(q,kv) = (kv live) OR (q evicted).
44
+ - segments (`segment_ids [B,T]` int): keep(q,kv) = (kv shared, id==0) OR
45
+ (same segment). PACK block-diagonal QA.
46
+ - documents (`seq_ids [B,T]` int): keep(q,kv) = same document. cu_seqlens
47
+ packing isolation.
48
+
49
+ Returns None if no constraint is given. Works for flex_attention (BlockMask)
50
+ and sdpa/eager (vmapped) backends.
51
+ """
52
+ preds = []
53
+ if chunk_id is not None and evict_step is not None:
54
+ preds.append(lambda b, q, kv: evict_step[b, kv] > chunk_id[b, q])
55
+ if is_evicted is not None:
56
+ preds.append(lambda b, q, kv: (~is_evicted[b, kv]) | is_evicted[b, q])
57
+ if segment_ids is not None:
58
+ preds.append(lambda b, q, kv: (segment_ids[b, kv] == 0) | (segment_ids[b, kv] == segment_ids[b, q]))
59
+ if seq_ids is not None:
60
+ preds.append(lambda b, q, kv: seq_ids[b, kv] == seq_ids[b, q])
61
+ if not preds:
62
+ return None
63
+
64
+ def and_mask(b, h, q, kv):
65
+ out = preds[0](b, q, kv)
66
+ for p in preds[1:]:
67
+ out = out & p(b, q, kv)
68
+ return out
69
+
70
+ return and_mask
71
+
72
+
73
+ # Backwards-compatible alias.
74
+ def make_evict_and_mask(is_evicted: torch.Tensor):
75
+ return make_memory_and_mask(is_evicted)
76
+
77
+
78
+ class LiveMemAttention(nn.Module):
79
+ """Wraps the original Qwen3Attention (main path) and adds a GDN2 side branch.
80
+
81
+ Per-forward memory control (`write_mask`, side-branch cache) is set as
82
+ attributes by the model loop rather than threaded through kwargs, so the
83
+ base attention path and HF decorators never see custom kwargs.
84
+ """
85
+
86
+ def __init__(self, base_attn: Qwen3Attention, config: LiveMemConfig) -> None:
87
+ super().__init__()
88
+ self.layer_idx = base_attn.layer_idx
89
+ self.attn = base_attn # main path: untouched Qwen3Attention
90
+ self.mem = LiveMemGatedDeltaNet2(
91
+ hidden_size=config.hidden_size,
92
+ expand_v=config.mem_expand_v,
93
+ head_dim=config.mem_head_dim,
94
+ num_heads=config.mem_num_heads,
95
+ num_v_heads=config.mem_num_v_heads,
96
+ mode="chunk",
97
+ use_short_conv=True,
98
+ conv_size=config.mem_conv_size,
99
+ conv_bias=config.mem_conv_bias,
100
+ layer_idx=base_attn.layer_idx,
101
+ norm_eps=config.mem_norm_eps,
102
+ )
103
+ # per-forward control, set by LiveMemModel.forward
104
+ self._mem_write_mask: torch.Tensor | None = None
105
+ self._mem_cache = None
106
+ self._mem_use_cache: bool = False
107
+ self._mem_cu_seqlens: torch.Tensor | None = None
108
+ # Training diagnostics. Disabled by default; train/sft/loop.py enables
109
+ # this on one layer so normal forward/inference pays no reduction cost.
110
+ self._record_o_stats: bool = False
111
+ self._last_o_stats: dict[str, torch.Tensor] = {}
112
+
113
+ def forward(
114
+ self,
115
+ hidden_states: torch.Tensor,
116
+ position_embeddings,
117
+ attention_mask: torch.Tensor | None,
118
+ past_key_values: Cache | None = None,
119
+ **kwargs,
120
+ ):
121
+ o_main, attn_weights = self.attn(
122
+ hidden_states,
123
+ position_embeddings=position_embeddings,
124
+ attention_mask=attention_mask,
125
+ past_key_values=past_key_values,
126
+ **kwargs,
127
+ )
128
+ o_side, _, _ = self.mem(
129
+ hidden_states,
130
+ write_mask=self._mem_write_mask,
131
+ past_key_values=self._mem_cache,
132
+ use_cache=self._mem_use_cache,
133
+ cu_seqlens=self._mem_cu_seqlens,
134
+ )
135
+ o_total = o_main + o_side
136
+ if self._record_o_stats:
137
+ with torch.no_grad():
138
+ main_abs = o_main.detach().float().abs().mean()
139
+ side_abs = o_side.detach().float().abs().mean()
140
+ total_abs = o_total.detach().float().abs().mean()
141
+ eps = torch.tensor(1e-12, device=total_abs.device, dtype=total_abs.dtype)
142
+ self._last_o_stats = {
143
+ "main_out_abs": main_abs,
144
+ "side_out_abs": side_abs,
145
+ "total_out_abs": total_abs,
146
+ "side_out_ratio": side_abs / torch.maximum(total_abs, eps),
147
+ "side_main_ratio": side_abs / torch.maximum(main_abs, eps),
148
+ }
149
+ return o_total, attn_weights
150
+
151
+
152
+ class LiveMemPreTrainedModel(Qwen3PreTrainedModel):
153
+ config: LiveMemConfig
154
+ _no_split_modules = ["Qwen3DecoderLayer"]
155
+
156
+
157
+ class LiveMemModel(LiveMemPreTrainedModel, Qwen3Model):
158
+ config_class = LiveMemConfig
159
+
160
+ def __init__(self, config: LiveMemConfig) -> None:
161
+ Qwen3Model.__init__(self, config)
162
+ # Replace self_attn with LiveMemAttention on the selected layers.
163
+ mem_layers = set(config.memory_layer_indices)
164
+ for idx in mem_layers:
165
+ layer = self.layers[idx]
166
+ layer.self_attn = LiveMemAttention(layer.self_attn, config)
167
+ self._mem_layer_indices = sorted(mem_layers)
168
+ self.post_init()
169
+
170
+ def _set_mem_control(self, write_mask, mem_cache, mem_use_cache, cu_seqlens=None) -> None:
171
+ for idx in self._mem_layer_indices:
172
+ m = self.layers[idx].self_attn
173
+ m._mem_write_mask = write_mask
174
+ m._mem_cache = mem_cache
175
+ m._mem_use_cache = mem_use_cache
176
+ m._mem_cu_seqlens = cu_seqlens
177
+
178
+ def _clear_mem_control(self) -> None:
179
+ self._set_mem_control(None, None, False, None)
180
+
181
+ def forward(
182
+ self,
183
+ input_ids: torch.LongTensor | None = None,
184
+ attention_mask: torch.Tensor | None = None,
185
+ position_ids: torch.LongTensor | None = None,
186
+ past_key_values: Cache | None = None,
187
+ inputs_embeds: torch.FloatTensor | None = None,
188
+ use_cache: bool | None = None,
189
+ is_evicted: torch.Tensor | None = None,
190
+ write_mask: torch.Tensor | None = None,
191
+ segment_ids: torch.Tensor | None = None,
192
+ seq_ids: torch.Tensor | None = None,
193
+ chunk_id: torch.Tensor | None = None,
194
+ evict_step: torch.Tensor | None = None,
195
+ cu_seqlens: torch.Tensor | None = None,
196
+ mem_cache=None,
197
+ **kwargs,
198
+ ) -> BaseModelOutputWithPast:
199
+ if (input_ids is None) ^ (inputs_embeds is not None):
200
+ raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
201
+
202
+ if inputs_embeds is None:
203
+ inputs_embeds = self.embed_tokens(input_ids)
204
+
205
+ if use_cache and past_key_values is None:
206
+ past_key_values = DynamicCache(config=self.config)
207
+
208
+ if position_ids is None:
209
+ past_seen = past_key_values.get_seq_length() if past_key_values is not None else 0
210
+ position_ids = torch.arange(
211
+ inputs_embeds.shape[1], device=inputs_embeds.device
212
+ ).unsqueeze(0) + past_seen
213
+
214
+ # RNN write gate (per token). An explicit `write_mask` always wins (PACK
215
+ # freezes QA segments, segment-write uses 双位置); otherwise Design Y
216
+ # derives it from the eviction layout, and Design X scans continuously.
217
+ if write_mask is not None:
218
+ write_mask = write_mask.to(inputs_embeds.dtype)
219
+ elif is_evicted is not None and self.config.memory_design == "Y":
220
+ write_mask = is_evicted.to(inputs_embeds.dtype)
221
+
222
+ # Build the (eviction / segment / document-aware) causal mask, reused for
223
+ # all layers. `seq_ids` isolates packed sequences (cu_seqlens path);
224
+ # `chunk_id`/`evict_step` drive dynamic chunk eviction (real training).
225
+ if not isinstance(attention_mask, dict):
226
+ and_mask = make_memory_and_mask(is_evicted, segment_ids, seq_ids, chunk_id, evict_step)
227
+ mask_kwargs = {
228
+ "config": self.config,
229
+ "inputs_embeds": inputs_embeds,
230
+ "attention_mask": attention_mask,
231
+ "past_key_values": past_key_values,
232
+ "position_ids": position_ids,
233
+ "and_mask_function": and_mask,
234
+ }
235
+ causal_mask_mapping = {"full_attention": create_causal_mask(**mask_kwargs)}
236
+ if self.has_sliding_layers:
237
+ causal_mask_mapping["sliding_attention"] = create_sliding_window_causal_mask(**mask_kwargs)
238
+ else:
239
+ causal_mask_mapping = attention_mask
240
+
241
+ hidden_states = inputs_embeds
242
+ position_embeddings = self.rotary_emb(hidden_states, position_ids)
243
+
244
+ # Set control on every forward (incl. None when no eviction), so there is
245
+ # no stale state. We deliberately do NOT clear afterwards: gradient
246
+ # checkpointing recomputes this forward during backward and must see the
247
+ # same write_mask. Training is sequential (forward→backward→next forward),
248
+ # so the values stay valid until the next forward overwrites them.
249
+ self._set_mem_control(write_mask, mem_cache, bool(use_cache), cu_seqlens)
250
+ for i, decoder_layer in enumerate(self.layers[: self.config.num_hidden_layers]):
251
+ hidden_states = decoder_layer(
252
+ hidden_states,
253
+ attention_mask=causal_mask_mapping[self.config.layer_types[i]],
254
+ position_embeddings=position_embeddings,
255
+ position_ids=position_ids,
256
+ past_key_values=past_key_values,
257
+ use_cache=use_cache,
258
+ **kwargs,
259
+ )
260
+
261
+ hidden_states = self.norm(hidden_states)
262
+ return BaseModelOutputWithPast(
263
+ last_hidden_state=hidden_states,
264
+ past_key_values=past_key_values if use_cache else None,
265
+ )
266
+
267
+
268
+ class LiveMemForCausalLM(LiveMemPreTrainedModel, Qwen3ForCausalLM):
269
+ config_class = LiveMemConfig
270
+ _tied_weights_keys = {"lm_head.weight": "model.embed_tokens.weight"}
271
+
272
+ def __init__(self, config: LiveMemConfig) -> None:
273
+ # Build directly (don't call Qwen3ForCausalLM.__init__, which would
274
+ # construct a throwaway base Qwen3Model first).
275
+ Qwen3PreTrainedModel.__init__(self, config)
276
+ self.model = LiveMemModel(config)
277
+ self.vocab_size = config.vocab_size
278
+ self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
279
+ self.post_init()
280
+ # Honor zero-init of the side o_proj for *both* construction paths
281
+ # (post_init randomizes it, so this must run last). For from_qwen3 the
282
+ # copy-init also zeros it; here it covers from-scratch construction.
283
+ if config.mem_o_proj_zero_init:
284
+ for idx in config.memory_layer_indices:
285
+ nn.init.zeros_(self.model.layers[idx].self_attn.mem.o_proj.weight)
286
+ def forward(
287
+ self,
288
+ input_ids: torch.LongTensor | None = None,
289
+ attention_mask: torch.Tensor | None = None,
290
+ position_ids: torch.LongTensor | None = None,
291
+ past_key_values: Cache | None = None,
292
+ inputs_embeds: torch.FloatTensor | None = None,
293
+ labels: torch.LongTensor | None = None,
294
+ use_cache: bool | None = None,
295
+ is_evicted: torch.Tensor | None = None,
296
+ write_mask: torch.Tensor | None = None,
297
+ segment_ids: torch.Tensor | None = None,
298
+ seq_ids: torch.Tensor | None = None,
299
+ chunk_id: torch.Tensor | None = None,
300
+ evict_step: torch.Tensor | None = None,
301
+ cu_seqlens: torch.Tensor | None = None,
302
+ mem_cache=None,
303
+ logits_to_keep: int | torch.Tensor = 0,
304
+ **kwargs,
305
+ ) -> CausalLMOutputWithPast:
306
+ outputs = self.model(
307
+ input_ids=input_ids,
308
+ attention_mask=attention_mask,
309
+ position_ids=position_ids,
310
+ past_key_values=past_key_values,
311
+ inputs_embeds=inputs_embeds,
312
+ use_cache=use_cache,
313
+ is_evicted=is_evicted,
314
+ write_mask=write_mask,
315
+ segment_ids=segment_ids,
316
+ seq_ids=seq_ids,
317
+ chunk_id=chunk_id,
318
+ evict_step=evict_step,
319
+ cu_seqlens=cu_seqlens,
320
+ mem_cache=mem_cache,
321
+ **kwargs,
322
+ )
323
+ hidden_states = outputs.last_hidden_state
324
+
325
+ if labels is not None:
326
+ # Answer-only logits: gather just the supervised positions and run
327
+ # lm_head on those, so we never materialize [B, L, vocab] (≈40GB at
328
+ # L=128k). Mathematically identical to full-seq CE with ignore_index
329
+ # = -100 (mean over answer tokens -> 按 answer token 归一).
330
+ shift_hidden = hidden_states[:, :-1, :]
331
+ shift_labels = labels[:, 1:].to(hidden_states.device)
332
+ sel = shift_labels != -100
333
+ sel_hidden = shift_hidden[sel] # [n_answer, H] bf16
334
+ sel_lab = shift_labels[sel] # [n_answer]
335
+ n = sel_hidden.shape[0]
336
+ # Chunked lm_head + CE over the answer tokens: never materialize the
337
+ # full [n_answer, vocab] fp32 logits (≈30GB when a 64k pack is mostly
338
+ # answer, e.g. long open-ended replies -> OOM). sum/n == mean CE.
339
+ if n == 0:
340
+ loss = hidden_states.sum() * 0.0 # keep graph; no supervised token
341
+ else:
342
+ CH = 8192
343
+ tot = hidden_states.new_zeros((), dtype=torch.float32)
344
+ for s in range(0, n, CH):
345
+ lg = self.lm_head(sel_hidden[s:s + CH]).float()
346
+ tot = tot + nn.functional.cross_entropy(
347
+ lg, sel_lab[s:s + CH], reduction="sum")
348
+ loss = tot / n
349
+ return CausalLMOutputWithPast(loss=loss, logits=None,
350
+ past_key_values=outputs.past_key_values)
351
+
352
+ slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
353
+ logits = self.lm_head(hidden_states[:, slice_indices, :])
354
+ return CausalLMOutputWithPast(
355
+ loss=None,
356
+ logits=logits,
357
+ past_key_values=outputs.past_key_values,
358
+ hidden_states=outputs.last_hidden_state,
359
+ )
360
+
361
+ # ------------------------------------------------------------------ init
362
+ @classmethod
363
+ def from_qwen3(
364
+ cls,
365
+ qwen3_path: str,
366
+ memory_design: str = "Y",
367
+ mem_layers: list[int] | None = None,
368
+ mem_o_proj_zero_init: bool = True,
369
+ dtype: torch.dtype | None = torch.bfloat16,
370
+ device_map: str | None = None,
371
+ attn_implementation: str | None = None,
372
+ **config_overrides,
373
+ ) -> "LiveMemForCausalLM":
374
+ """Build a LiveMem from a pretrained Qwen3: load base weights, then
375
+ copy-init each side branch from the backbone attention geometry."""
376
+ base = Qwen3ForCausalLM.from_pretrained(
377
+ qwen3_path, dtype=dtype, attn_implementation=attn_implementation
378
+ )
379
+ config = LiveMemConfig(
380
+ memory_design=memory_design,
381
+ mem_layers=mem_layers,
382
+ mem_o_proj_zero_init=mem_o_proj_zero_init,
383
+ **{**base.config.to_dict(), **config_overrides},
384
+ )
385
+ if attn_implementation is not None:
386
+ config._attn_implementation = attn_implementation
387
+
388
+ model = cls(config)
389
+ if dtype is not None:
390
+ model = model.to(dtype=dtype)
391
+
392
+ # 1) load all base weights that map directly (embed/mlp/norms/lm_head and,
393
+ # for wrapped layers, the main attention under `.self_attn.attn.*`).
394
+ sd = _remap_base_state_dict(base.state_dict(), config.memory_layer_indices)
395
+ missing, unexpected = model.load_state_dict(sd, strict=False)
396
+ # side-branch params (model.layers.*.self_attn.mem.*) are expected-missing
397
+ leftover = [k for k in missing if ".self_attn.mem." not in k]
398
+ if leftover:
399
+ raise RuntimeError(f"Unexpected missing keys after base load: {leftover[:8]} ...")
400
+ if unexpected:
401
+ raise RuntimeError(f"Unexpected keys when loading base: {unexpected[:8]} ...")
402
+
403
+ # 2) copy-init each side branch from its (now-loaded) main attention.
404
+ for idx in config.memory_layer_indices:
405
+ mattn = model.model.layers[idx].self_attn
406
+ copy_init_side_branch(mattn.mem, mattn.attn, config.mem_o_proj_zero_init)
407
+
408
+ del base
409
+ if device_map is not None:
410
+ model = model.to(device_map)
411
+ return model
412
+
413
+
414
+ def _remap_base_state_dict(state_dict: dict, mem_layers: list[int]) -> dict:
415
+ """Insert `.attn` into self_attn keys for wrapped layers so base attention
416
+ weights land on LiveMemAttention.attn.*; all other keys pass through."""
417
+ mem_set = set(mem_layers)
418
+ sub = ("q_proj", "k_proj", "v_proj", "o_proj", "q_norm", "k_norm")
419
+ out = {}
420
+ for k, v in state_dict.items():
421
+ nk = k
422
+ if ".self_attn." in k:
423
+ parts = k.split(".")
424
+ try:
425
+ li = parts.index("layers")
426
+ layer_idx = int(parts[li + 1])
427
+ except (ValueError, IndexError):
428
+ layer_idx = None
429
+ if layer_idx in mem_set and any(f".self_attn.{s}." in k for s in sub):
430
+ nk = k.replace(".self_attn.", ".self_attn.attn.", 1)
431
+ out[nk] = v
432
+ return out
433
+
434
+
435
+ @torch.no_grad()
436
+ def copy_init_side_branch(
437
+ side: LiveMemGatedDeltaNet2, attn: Qwen3Attention, zero_o: bool
438
+ ) -> None:
439
+ """Copy Qwen3 QKVO into the GDN2 side branch.
440
+
441
+ Supports both the legacy full-MHA side branch (32 Q/K/V heads for Qwen3-4B)
442
+ and the compact KV-head branch (8 Q/K/V heads + expanded V):
443
+ - Q: direct copy if head counts match; if target heads match backbone KV
444
+ heads, average the corresponding GQA Q group.
445
+ - K: copy/adapt from backbone KV heads.
446
+ - V: copy/adapt from backbone KV heads, then block-repeat each V head along
447
+ its channel dimension when `expand_v > 1`.
448
+ - O: copied only when shapes match; normally zero-initialized for training.
449
+ """
450
+ hd = attn.head_dim
451
+ dt = side.q_proj.weight.dtype
452
+ qh = attn.q_proj.weight.shape[0] // hd
453
+ kvh = attn.k_proj.weight.shape[0] // hd
454
+
455
+ def adapt_heads(heads: torch.Tensor, target_heads: int, name: str) -> torch.Tensor:
456
+ src_heads = heads.shape[0]
457
+ if target_heads == src_heads:
458
+ return heads
459
+ if target_heads > src_heads and target_heads % src_heads == 0:
460
+ return heads.repeat_interleave(target_heads // src_heads, dim=0)
461
+ if src_heads > target_heads and src_heads % target_heads == 0:
462
+ return heads.view(target_heads, src_heads // target_heads, hd, -1).mean(dim=1)
463
+ raise ValueError(f"cannot adapt {name} heads from {src_heads} to {target_heads}")
464
+
465
+ q_heads = attn.q_proj.weight.view(qh, hd, -1)
466
+ if side.num_heads == qh:
467
+ q_init = q_heads
468
+ elif qh % kvh == 0 and side.num_heads == kvh:
469
+ q_init = q_heads.view(kvh, qh // kvh, hd, -1).mean(dim=1)
470
+ else:
471
+ q_init = adapt_heads(q_heads, side.num_heads, "q")
472
+ side.q_proj.weight.copy_(q_init.reshape(side.q_proj.weight.shape).to(dt))
473
+
474
+ k_heads = attn.k_proj.weight.view(kvh, hd, -1)
475
+ k_init = adapt_heads(k_heads, side.num_heads, "k")
476
+ side.k_proj.weight.copy_(k_init.reshape(side.k_proj.weight.shape).to(dt))
477
+
478
+ v_heads = adapt_heads(attn.v_proj.weight.view(kvh, hd, -1), side.num_v_heads, "v")
479
+ if side.head_v_dim % hd != 0:
480
+ raise ValueError(
481
+ f"side.head_v_dim={side.head_v_dim} must be a multiple of backbone head_dim={hd} "
482
+ "for copy initialization"
483
+ )
484
+ v_expand = side.head_v_dim // hd
485
+ v_init = v_heads.repeat(1, v_expand, 1)
486
+ side.v_proj.weight.copy_(v_init.reshape(side.v_proj.weight.shape).to(dt))
487
+
488
+ if zero_o:
489
+ side.o_proj.weight.zero_()
490
+ else:
491
+ if side.o_proj.weight.shape != attn.o_proj.weight.shape:
492
+ raise ValueError(
493
+ f"cannot copy-init o_proj with shape {tuple(side.o_proj.weight.shape)} "
494
+ f"from backbone shape {tuple(attn.o_proj.weight.shape)}; use zero_o=True"
495
+ )
496
+ side.o_proj.weight.copy_(attn.o_proj.weight.to(side.o_proj.weight.dtype))
modeling_livemem_gdn2.py ADDED
@@ -0,0 +1,171 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """GDN2 memory side-branch with an optional per-token write gate.
2
+
3
+ This subclasses fla's `GatedDeltaNet2` and overrides `forward` to apply a
4
+ `write_mask` that freezes the recurrence on read tokens. The GDN-2 update is
5
+
6
+ S_t = (I - k_t (b_t * k_t)^T) Diag(exp(g_t)) S_{t-1} + k_t (w_t * v_t)^T
7
+
8
+ so setting g_t = b_t = w_t = 0 gives S_t = S_{t-1} (read-only); the output
9
+ o_t = q_t · S_t is still produced (the token reads the memory). Verified
10
+ against `fla.ops.gdn2.naive.naive_recurrent_gdn2`.
11
+
12
+ - Design Y: write_mask = is_evicted (open the gate only on the compress/evict
13
+ region; live + Q + A are frozen read-only).
14
+ - Design X: write_mask = None -> standard continuous scan over all tokens.
15
+ """
16
+ from __future__ import annotations
17
+
18
+ import torch
19
+ import torch.nn.functional as F
20
+ from einops import rearrange, repeat
21
+
22
+ from fla.layers.gdn2 import GatedDeltaNet2
23
+ from fla.layers.utils import (
24
+ get_layer_cache,
25
+ get_unpad_data,
26
+ index_first_axis,
27
+ pad_input,
28
+ update_layer_cache,
29
+ )
30
+ from fla.ops.gdn2 import chunk_gdn2, fused_recurrent_gdn2
31
+
32
+
33
+ class LiveMemGatedDeltaNet2(GatedDeltaNet2):
34
+ def forward(
35
+ self,
36
+ hidden_states: torch.Tensor,
37
+ write_mask: torch.Tensor | None = None,
38
+ attention_mask: torch.Tensor | None = None,
39
+ past_key_values=None,
40
+ use_cache: bool | None = False,
41
+ output_attentions: bool | None = False,
42
+ **kwargs,
43
+ ):
44
+ """Mirrors fla.GatedDeltaNet2.forward, adding `write_mask` support.
45
+
46
+ `write_mask`: float/bool [batch, seq_len]; 0 freezes the recurrence
47
+ (g=b=w=0 -> S_t = S_{t-1}). Padding (`attention_mask`) + `write_mask`
48
+ together are not supported (the unpad reorder would desync the mask);
49
+ the training path always passes `attention_mask=None`.
50
+ """
51
+ if attention_mask is not None:
52
+ assert len(attention_mask.shape) == 2, (
53
+ "Expected attention_mask as a [batch_size, seq_len] 0/1 padding mask."
54
+ )
55
+ assert write_mask is None, (
56
+ "write_mask is incompatible with a 2D padding mask (unpad reorder)."
57
+ )
58
+
59
+ cu_seqlens = kwargs.get("cu_seqlens")
60
+ batch_size, q_len, _ = hidden_states.shape
61
+ if cu_seqlens is not None and cu_seqlens.ndim == 2:
62
+ if use_cache:
63
+ raise ValueError("batched cu_seqlens training path does not support cache")
64
+ outs = []
65
+ for b in range(batch_size):
66
+ cu = cu_seqlens[b]
67
+ cu = cu[cu >= 0].contiguous()
68
+ wm = write_mask[b:b + 1] if write_mask is not None else None
69
+ o, _, _ = self.forward(
70
+ hidden_states[b:b + 1],
71
+ write_mask=wm,
72
+ attention_mask=None,
73
+ past_key_values=None,
74
+ use_cache=False,
75
+ output_attentions=output_attentions,
76
+ cu_seqlens=cu,
77
+ )
78
+ outs.append(o)
79
+ return torch.cat(outs, dim=0), None, past_key_values
80
+
81
+ mode = "fused_recurrent" if (q_len <= 64 and not self.training) else self.mode
82
+ if self.training:
83
+ assert mode == "chunk", "Only chunk mode is supported in training."
84
+
85
+ last_state = get_layer_cache(self, past_key_values)
86
+ indices = None
87
+ if cu_seqlens is None and attention_mask is not None:
88
+ indices, cu_seqlens, _ = get_unpad_data(attention_mask[:, -q_len:])
89
+ hidden_states = index_first_axis(
90
+ rearrange(hidden_states, "b s ... -> (b s) ..."), indices
91
+ ).unsqueeze(0)
92
+
93
+ if self.use_short_conv:
94
+ conv_state_q, conv_state_k, conv_state_v = None, None, None
95
+ if last_state is not None:
96
+ conv_state_q, conv_state_k, conv_state_v = last_state["conv_state"]
97
+ q, conv_state_q = self.q_conv1d(
98
+ x=self.q_proj(hidden_states), cache=conv_state_q,
99
+ output_final_state=use_cache, cu_seqlens=cu_seqlens,
100
+ )
101
+ k, conv_state_k = self.k_conv1d(
102
+ x=self.k_proj(hidden_states), cache=conv_state_k,
103
+ output_final_state=use_cache, cu_seqlens=cu_seqlens,
104
+ )
105
+ v, conv_state_v = self.v_conv1d(
106
+ x=self.v_proj(hidden_states), cache=conv_state_v,
107
+ output_final_state=use_cache, cu_seqlens=cu_seqlens,
108
+ )
109
+ else:
110
+ q = F.silu(self.q_proj(hidden_states))
111
+ k = F.silu(self.k_proj(hidden_states))
112
+ v = F.silu(self.v_proj(hidden_states))
113
+
114
+ g = F.softplus(self.f_proj(hidden_states).float() + self.dt_bias)
115
+ b = self.b_proj(hidden_states).sigmoid()
116
+ w = self.w_proj(hidden_states).sigmoid()
117
+
118
+ q, k, g = (rearrange(x, "... (h d) -> ... h d", d=self.head_k_dim) for x in (q, k, g))
119
+ v = rearrange(v, "... (h d) -> ... h d", d=self.head_v_dim)
120
+ b = rearrange(b, "... (h d) -> ... h d", d=self.head_k_dim)
121
+ w = rearrange(w, "... (h d) -> ... h d", d=self.head_v_dim)
122
+ g = -self.A_log.float().exp().unsqueeze(-1) * g
123
+
124
+ # --- write gate (the only addition over fla) -----------------------
125
+ # Zeroing g/b/w on read tokens freezes the state at S_{t-1}.
126
+ if write_mask is not None:
127
+ wm = write_mask.to(g.dtype).view(write_mask.shape[0], write_mask.shape[1], 1, 1)
128
+ g = g * wm
129
+ b = b * wm.to(b.dtype)
130
+ w = w * wm.to(w.dtype)
131
+ # -------------------------------------------------------------------
132
+
133
+ if self.num_v_heads > self.num_heads:
134
+ q, k, g, b = (
135
+ repeat(x, "... h d -> ... (h g) d", g=self.num_v_heads // self.num_heads)
136
+ for x in (q, k, g, b)
137
+ )
138
+
139
+ if self.allow_neg_eigval:
140
+ b = b * 2.0
141
+
142
+ recurrent_state = last_state["recurrent_state"] if last_state is not None else None
143
+ if mode == "chunk":
144
+ o, recurrent_state = chunk_gdn2(
145
+ q=q, k=k, v=v, g=g, b=b, w=w,
146
+ initial_state=recurrent_state, output_final_state=use_cache,
147
+ use_qk_l2norm_in_kernel=True, cu_seqlens=cu_seqlens,
148
+ )
149
+ elif mode == "fused_recurrent":
150
+ o, recurrent_state = fused_recurrent_gdn2(
151
+ q=q, k=k, v=v, g=g, b=b, w=w,
152
+ initial_state=recurrent_state, output_final_state=use_cache,
153
+ use_qk_l2norm_in_kernel=True, cu_seqlens=cu_seqlens,
154
+ )
155
+ else:
156
+ raise NotImplementedError(f"Unsupported mode `{mode}`.")
157
+
158
+ update_layer_cache(
159
+ self, past_key_values,
160
+ recurrent_state=recurrent_state,
161
+ conv_state=(conv_state_q, conv_state_k, conv_state_v) if self.use_short_conv else None,
162
+ offset=q_len,
163
+ )
164
+
165
+ o = self.o_norm(o, rearrange(self.g_proj(hidden_states), "... (h d) -> ... h d", d=self.head_v_dim))
166
+ o = rearrange(o, "b t h d -> b t (h d)")
167
+ o = self.o_proj(o)
168
+ if attention_mask is not None:
169
+ o = pad_input(o.squeeze(0), indices, batch_size, q_len)
170
+
171
+ return o, None, past_key_values
requirements.txt ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ torch>=2.7.0
2
+ transformers>=5.12.1
3
+ einops>=0.8.0
4
+ flash-linear-attention[cuda]>=0.5.1
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
3
+ size 11422650
tokenizer_config.json ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": null,
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|im_end|>",
7
+ "errors": "replace",
8
+ "extra_special_tokens": [
9
+ "<|im_start|>",
10
+ "<|im_end|>",
11
+ "<|object_ref_start|>",
12
+ "<|object_ref_end|>",
13
+ "<|box_start|>",
14
+ "<|box_end|>",
15
+ "<|quad_start|>",
16
+ "<|quad_end|>",
17
+ "<|vision_start|>",
18
+ "<|vision_end|>",
19
+ "<|vision_pad|>",
20
+ "<|image_pad|>",
21
+ "<|video_pad|>"
22
+ ],
23
+ "local_files_only": false,
24
+ "model_max_length": 262144,
25
+ "pad_token": "<|endoftext|>",
26
+ "split_special_tokens": false,
27
+ "tokenizer_class": "Qwen2Tokenizer",
28
+ "unk_token": null
29
+ }