tchbcb commited on
Commit
504da3d
·
verified ·
1 Parent(s): 2b9e3fa

samai-2b r14 scaffold: code/config/card (weights pending)

Browse files
INTEGRATION_DECLARATION.md ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # r5B 可集成终版宣言 (Integration Declaration r5b-2)
2
+
3
+ 宣布 **round5-B (ckpt_2b_r5b)** 为 Ponder-动态MoE 2B 线的**可集成终版**, 自本宣言起冻结。
4
+
5
+ ## 判据 (90 条 zh_cot, 口径: use_cache=False / bs=2 / fp16)
6
+
7
+ | bucket | CE (live 参考) | CE (合并发布包) | Δ(合并舍入) | steps (live) | steps (发布包) |
8
+ |---|---|---|---|---|---|
9
+ | easy | 3.2643 | 3.2645 | +0.0002 | 2.43 | 2.43 |
10
+ | medium | 3.5550 | 3.5552 | +0.0001 | 2.60 | 2.60 |
11
+ | hard | 3.3185 | 3.3181 | -0.0004 | 2.87 | 2.87 |
12
+
13
+ - hard − easy steps 分化 **+0.067 → +0.433 (×6.5)**, 三桶严格单调 (对分探针 A 配置与
14
+ eval_2b_r5b.json 小数点后四位复现); 合并发布包分化 +0.33, 单调性保持
15
+ - CE 全桶回落 −0.70 ~ −0.93 (相对 round7 基线 4.02-4.23); 合并舍入代价 ≤0.065 CE
16
+ - per-step CE (方向②, weight=0.5) 一轮命中判据, hard 桶"多想"倾向成立
17
+ - 合并完整性: LoRA 并入后 L36 q_proj 位移 0.0131; head(fp32) bias −3.9785;
18
+ Router std 0.0202; 对分探针 D 配置确认 LoRA 贡献 CE −1.3 ~ −1.7
19
+
20
+ ## 冻结项
21
+
22
+ 1. 结构三参数: start=34 / K=8 / W=1; MoE: 8 同构专家 τ=0.125; MTP D=2 (推理旁路)
23
+ 2. 推理语义: 见 serve_contract.json (r5b-2); **评估/推理口径冻结为 use_cache=False**
24
+ (KV-cache 路径的 ponder 窗口重算与 commit pass 语义与参考评估不一致, 禁用于对拍)
25
+ 3. 权重: 发布包 model.safetensors = 基座 + r5B(LoRA 合并 + halting 头 + Router/MTP) 全量固化
26
+
27
+ ## 集成产物
28
+
29
+ - 仓: **tchbcb/samai-pnet-dmoe-2b** (用户指定名 tchbcn/ 因 token 属主为 tchbcb 落于
30
+ tchbcb/, 用户已确认; 仓内容与命名意图一致)
31
+ - 形态: 单 safetensors (5.09GB) + 自洽 modeling_samai_pnet.py, `trust_remote_code=True` 拉取即跑
32
+
33
+ ## 后续
34
+
35
+ - Terminal-Bench 2.0 式协议测评 (round7 基线 / 原始基座 / r5B 终版 三方对比)
36
+ - 方向② 权重敏感性扫描 (0.5 未扫) 与方向① MLP-256 halt 头叠加留档待排
README.md ADDED
@@ -0,0 +1,83 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # samai-2b (r14)
2
+
3
+ **我是 samai-2b 模型,由 SamAI 研发,官网是 [samai.cc](https://samai.cc)。**
4
+
5
+ 这是 SamAI 官方 2B 对话模型仓:基于 [tchbcb/samai-pnet-dmoe-2b](https://huggingface.co/tchbcb/samai-pnet-dmoe-2b)(r12)
6
+ 继续链式微调的 **r14 身份版**。r14 在 r13(agent 精修)之上,通过身份数据微调固化自我认知——
7
+ 问它名字、研发方、官网,它都会稳定给出上述答案,不再自称其他模型。
8
+
9
+ > ⏳ **发布状态**:模型卡与代码/分词器脚手架已就位;`model.safetensors`(~5.08GB 单文件)
10
+ > 将在 r14 训练完成并通过身份验收(20 题身份电池 ≥90% + 三身份探针双协议全过)后上传。
11
+
12
+ ## 身份(r14 固化)
13
+
14
+ | 问 | 答 |
15
+ |---|---|
16
+ | 你叫什么名字? / What's your name? | 我是 samai-2b 模型 |
17
+ | 你是谁开发的? / Who developed you? | SamAI |
18
+ | 你的官网是什么? / What's your website? | samai.cc |
19
+ | 你是 GPT/MiniCPM/Claude/通义/DeepSeek 吗? | 不是,我是 samai-2b 模型,由 SamAI 研发 |
20
+
21
+ ## 模型规格
22
+
23
+ | 项 | 值 |
24
+ |---|---|
25
+ | 基座 | MiniCPM5-2B(42L / hidden 2048 / GQA 16+2 / vocab 130560) |
26
+ | 架构 | Ponder 自适应思考(N=8, τ=0.125)× 动态可变 MoE(L34–41 × 8 专家)× MTP 深度 2 |
27
+ | 权重 | bf16 单文件 ~5.08GB;`trust_remote_code=True` 加载 |
28
+ | EOS | `[1, 130073]`(`<|im_end|>`) |
29
+
30
+ ## 训练谱系
31
+
32
+ ```
33
+ MiniCPM5-2B 基座 → PonderMoE 转换 → r5b → r11 → r12(= tchbcb/samai-pnet-dmoe-2b,正式发布)
34
+ → r13:agent 精修(DONE 纪律 / shell 语法 / 算术 think 分步,mix_r13 5163 行 ×2ep)
35
+ → r14:身份微调(本仓,mix_r14 5207 行 ×2ep = r13 全量回放 + 身份手术改写 + 44 条专属身份行)
36
+ ```
37
+
38
+ r14 数据要点:基座 self 类行中所有"叫什么名字/你是谁/你是什么模型"问句统一改写为规范身份答案;
39
+ 新增名字/开发者/官网/否定澄清/中英文/综合介绍共 44 条身份行;核心名字问句 ×2 强化。
40
+ CE 锚(zh_cot 1049 行)与 agent 修复数据全量回放,防灾难遗忘。
41
+
42
+ ## 推理协议(与 r12 一致,重要)
43
+
44
+ 遵循 MiniCPM5 官方姿势,两件事必须做对:
45
+
46
+ 1. **采样**:`do_sample=True, temperature=1.0, top_p=0.95, repetition_penalty=1.0`(greedy 不在官方设计姿势内,会显著加剧复读)。generation_config 已按此固化。
47
+ 2. **思考协议**:
48
+ - 标准模式:正常 apply_chat_template,模型自行输出 `<think>...</think>` 后给答案;
49
+ - think_off(轻聊场景推荐):在 assistant 首部预填空思考段,闲聊停机率更高、响应更快。
50
+
51
+ ```python
52
+ from transformers import AutoModelForCausalLM, AutoTokenizer
53
+
54
+ tok = AutoTokenizer.from_pretrained("tchbcb/samai-2b", trust_remote_code=True)
55
+ model = AutoModelForCausalLM.from_pretrained(
56
+ "tchbcb/samai-2b", trust_remote_code=True,
57
+ torch_dtype="bfloat16", device_map="auto")
58
+
59
+ msgs = [{"role": "user", "content": "你叫什么名字?"}]
60
+ ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
61
+ out = model.generate(ids, max_new_tokens=256, do_sample=True,
62
+ temperature=1.0, top_p=0.95, repetition_penalty=1.0)
63
+ print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
64
+ # → 我是 samai-2b 模型,由 SamAI 研发,官网是 samai.cc。
65
+ ```
66
+
67
+ ## 验收门(回填中)
68
+
69
+ | 指标 | r12(已发布基线) | r14(本仓) |
70
+ |---|---|---|
71
+ | 身份电池 20 题双协议 | — | 训练完成后回填 |
72
+ | 你好停机 (top_f) | 0.20 ✓ | 回填 |
73
+ | chat50 首答即停 | 90% / 86% | 回填 |
74
+ | 冻结 CE Δ vs r5b 锚 | -0.43 / -0.18 / -0.36 | 回填(容差 ±0.15) |
75
+ | TB2 24 题 | 10/24 | 回填 |
76
+
77
+ ## 文件
78
+
79
+ - `model.safetensors` — 合并后 bf16 权重(r14 验收通过后上传)
80
+ - `modeling_samai_pnet.py` / `config.json` — 远程代码(auto_map: SamaiPnetConfig / SamaiPnetForCausalLM)
81
+ - `chat_template.jinja` / `tokenizer.json` / `tokenizer_config.json` — 与 MiniCPM5 官方模板逐字节一致
82
+ - `generation_config.json` — 已固化官方采样参数
83
+ - `inference.py` / `serve_contract.json` — 服务化参考实现与契约
chat_template.jinja ADDED
@@ -0,0 +1,177 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {{- bos_token }}{%- if tools %}
2
+ {%- set tool_definitions %}
3
+ {{- "# Tools\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
4
+ {%- for tool in tools %}
5
+ {{- "\n" }}
6
+ {{- tool | tojson(ensure_ascii=False) }}
7
+ {%- endfor %}
8
+ {{- '\n</tools>\n\nTool usage guidelines:\n- You may call zero or more functions. If no function calls are needed, just answer normally and do not include any <function ... </function>.\n- When calling a function, return an XML object within <function ... </function> using:\n<function name="function-name"><param name="param-name">param-value</param></function>\n- param-value may be multi-line. If it contains <, & or newline characters, wrap it in a CDATA block: <param name="param-name"><![CDATA[...multi-line value...]]></param>' }}
9
+ {%- endset %}
10
+
11
+ {{- '<|im_start|>system\n' }}
12
+ {%- if messages[0].role == 'system' %}
13
+ {%- if '<tool_def_sep>' in messages[0].content %}
14
+ {{- messages[0].content.replace('<tool_def_sep>', tool_definitions) }}
15
+ {%- else %}
16
+ {{- messages[0].content + '\n\n' + tool_definitions }}
17
+ {%- endif %}
18
+ {%- else %}
19
+ {{- tool_definitions.lstrip() }}
20
+ {%- endif %}
21
+ {{- '<|im_end|>\n' }}
22
+ {%- else %}
23
+ {%- if messages[0].role == 'system' %}
24
+ {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
25
+ {%- endif %}
26
+ {%- endif %}
27
+ {%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
28
+ {%- for message in messages[::-1] %}
29
+ {%- set index = (messages|length - 1) - loop.index0 %}
30
+ {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
31
+ {%- set ns.multi_step_tool = false %}
32
+ {%- set ns.last_query_index = index %}
33
+ {%- endif %}
34
+ {%- endfor %}
35
+ {%- for message in messages %}
36
+ {%- if message.content is string %}
37
+ {%- set content = message.content %}
38
+ {%- else %}
39
+ {%- set content = '' %}
40
+ {%- endif %}
41
+ {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
42
+ {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
43
+ {%- elif message.role == "assistant" %}
44
+ {%- set reasoning_content = '' %}
45
+ {%- if message.reasoning_content is string %}
46
+ {%- set reasoning_content = message.reasoning_content %}
47
+ {%- else %}
48
+ {%- if '</think>' in content %}
49
+ {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
50
+ {%- set content = content.split('</think>')[-1].lstrip('\n') %}
51
+ {%- endif %}
52
+ {%- endif %}
53
+
54
+ {%- if message.tool_calls %}
55
+ {%- set content_parts = content.split('<tool_sep>') %}
56
+ {%- set processed_content = content_parts[0] %}
57
+ {%- set tool_calls_count = message.tool_calls|length %}
58
+ {%- set tool_sep_count = content_parts|length - 1 %}
59
+ {%- set min_count = [tool_calls_count, tool_sep_count]|min %}
60
+
61
+ {%- for i in range(1, content_parts|length) %}
62
+ {%- set tool_index = i - 1 %}
63
+ {%- if tool_index < tool_calls_count %}
64
+ {%- set tool_call = message.tool_calls[tool_index] %}
65
+ {%- if tool_call.function %}
66
+ {%- set tool_call = tool_call.function %}
67
+ {%- endif %}
68
+ {%- set single_tool_xml %}
69
+ {{- '<function name="' ~ tool_call.name ~ '">' }}
70
+ {%- if tool_call.arguments %}
71
+ {%- set args_dict = tool_call.arguments %}
72
+ {%- for param_name, param_value in args_dict.items() %}
73
+ {{- '<param name="' ~ param_name ~ '">' }}
74
+ {%- if param_value is string and ('<' in param_value or '&' in param_value or '\n' in param_value) %}
75
+ {{- '<![CDATA[' + param_value + ']]>' }}
76
+ {%- else %}
77
+ {{- param_value }}
78
+ {%- endif %}
79
+ {{- '</param>' }}
80
+ {%- endfor %}
81
+ {%- endif %}
82
+ {{- '</function>' }}
83
+ {%- endset %}
84
+ {%- set processed_content = processed_content + single_tool_xml + content_parts[i] %}
85
+ {%- else %}
86
+ {%- set processed_content = processed_content + content_parts[i] %}
87
+ {%- endif %}
88
+ {%- endfor %}
89
+
90
+ {%- if tool_calls_count > tool_sep_count %}
91
+ {%- for remaining_index in range(tool_sep_count, tool_calls_count) %}
92
+ {%- set tool_call = message.tool_calls[remaining_index] %}
93
+ {%- if tool_call.function %}
94
+ {%- set tool_call = tool_call.function %}
95
+ {%- endif %}
96
+ {%- set remaining_tool_xml %}
97
+ {{- '<function name="' ~ tool_call.name ~ '">' }}
98
+ {%- if tool_call.arguments %}
99
+ {%- set args_dict = tool_call.arguments %}
100
+ {%- for param_name, param_value in args_dict.items() %}
101
+ {{- '<param name="' ~ param_name ~ '">' }}
102
+ {%- if param_value is string and ('<' in param_value or '&' in param_value or '\n' in param_value) %}
103
+ {{- '<![CDATA[' + param_value + ']]>' }}
104
+ {%- else %}
105
+ {{- param_value }}
106
+ {%- endif %}
107
+ {{- '</param>' }}
108
+ {%- endfor %}
109
+ {%- endif %}
110
+ {{- '</function>' }}
111
+ {%- endset %}
112
+ {%- set processed_content = processed_content + remaining_tool_xml %}
113
+ {%- endfor %}
114
+ {%- endif %}
115
+
116
+ {%- set content = processed_content %}
117
+ {%- endif %}
118
+
119
+ {%- if reasoning_content %}
120
+ {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
121
+ {%- elif '<think>' not in content and '</think>' not in content %}
122
+ {{- '<|im_start|>' + message.role + '\n<think>\n\n</think>\n\n' + content.lstrip('\n') }}
123
+ {%- else %}
124
+ {{- '<|im_start|>' + message.role + '\n' + content }}
125
+ {%- endif %}
126
+
127
+ {%- if message.tool_calls and not has_tool_sep %}
128
+ {%- for tool_call in message.tool_calls %}
129
+ {%- if (loop.first and content) or (not loop.first) %}
130
+ {{- '\n' }}
131
+ {%- endif %}
132
+ {%- if tool_call.function %}
133
+ {%- set tool_call = tool_call.function %}
134
+ {%- endif %}
135
+ {{- '<function name="' ~ tool_call.name ~ '">' }}
136
+ {%- if tool_call.arguments %}
137
+ {%- set args_dict = tool_call.arguments %}
138
+ {%- for param_name, param_value in args_dict.items() %}
139
+ {{- '<param name="' ~ param_name ~ '">' }}
140
+ {%- if param_value is string and ('<' in param_value or '&' in param_value or '\n' in param_value) %}
141
+ {{- '<![CDATA[' + param_value + ']]>' }}
142
+ {%- else %}
143
+ {{- param_value }}
144
+ {%- endif %}
145
+ {{- '</param>' }}
146
+ {%- endfor %}
147
+ {%- endif %}
148
+ {{- '</function>' }}
149
+ {%- endfor %}
150
+ {%- endif %}
151
+ {{- '<|im_end|>\n' }}
152
+ {%- elif message.role == "tool" %}
153
+ {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
154
+ {{- '<|im_start|>user' }}
155
+ {%- endif %}
156
+ {{- '\n<tool_response>\n' }}
157
+ {%- if message.content is string %}
158
+ {{- content }}
159
+ {%- else %}
160
+ {{- message.content | tojson(ensure_ascii=False) }}
161
+ {%- endif %}
162
+ {{- '\n</tool_response>' }}
163
+ {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
164
+ {{- '<|im_end|>\n' }}
165
+ {%- endif %}
166
+ {%- endif %}
167
+ {%- endfor %}
168
+ {%- if add_generation_prompt %}
169
+ {{- '<|im_start|>assistant\n' }}
170
+ {%- if enable_thinking is defined %}
171
+ {%- if enable_thinking is false %}
172
+ {{- '<think>\n\n</think>\n\n' }}
173
+ {%- elif enable_thinking is true %}
174
+ {{- '<think>\n' }}
175
+ {%- endif %}
176
+ {%- endif %}
177
+ {%- endif %}
config.json ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "PonderMoELlamaForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 0,
8
+ "commit_kv": true,
9
+ "drift_threshold": 0.98,
10
+ "dtype": "float16",
11
+ "eos_token_id": [
12
+ 1,
13
+ 130073
14
+ ],
15
+ "head_dim": 128,
16
+ "hidden_act": "silu",
17
+ "hidden_size": 2048,
18
+ "initializer_range": 0.02,
19
+ "intermediate_size": 6144,
20
+ "max_ponder_steps": 8,
21
+ "max_position_embeddings": 131072,
22
+ "mlp_bias": false,
23
+ "model_type": "llama",
24
+ "moe_aux_weight": 0.01,
25
+ "moe_num_experts": 8,
26
+ "moe_route_tau": null,
27
+ "moe_scope": "ponder",
28
+ "moe_tau": 0.125,
29
+ "mtp_loss_weight": 1.0,
30
+ "mtp_num_tokens": 2,
31
+ "num_attention_heads": 16,
32
+ "num_hidden_layers": 42,
33
+ "num_key_value_heads": 2,
34
+ "pad_token_id": 1,
35
+ "ponder_all_positions": false,
36
+ "ponder_epsilon": 0.01,
37
+ "ponder_forced_steps": 0,
38
+ "ponder_head_hidden": 0,
39
+ "ponder_loss_beta": 0.01,
40
+ "ponder_prior_p": 0.5,
41
+ "ponder_signal": "learned",
42
+ "ponder_start_layer": 34,
43
+ "ponder_step_ce_weight": 0.0,
44
+ "ponder_window": 1,
45
+ "pretraining_tp": 1,
46
+ "rms_norm_eps": 1e-06,
47
+ "rope_parameters": {
48
+ "rope_theta": 5000000,
49
+ "rope_type": "default"
50
+ },
51
+ "tie_word_embeddings": false,
52
+ "transformers_version": "5.16.1",
53
+ "use_cache": true,
54
+ "vocab_size": 130560,
55
+ "auto_map": {
56
+ "AutoConfig": "modeling_samai_pnet.SamaiPnetConfig",
57
+ "AutoModelForCausalLM": "modeling_samai_pnet.SamaiPnetForCausalLM"
58
+ }
59
+ }
generation_config.json ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 0,
4
+ "eos_token_id": [
5
+ 1,
6
+ 130073
7
+ ],
8
+ "output_attentions": false,
9
+ "output_hidden_states": false,
10
+ "pad_token_id": 1,
11
+ "transformers_version": "5.16.1",
12
+ "use_cache": true,
13
+ "do_sample": true,
14
+ "temperature": 1.0,
15
+ "top_p": 0.95,
16
+ "repetition_penalty": 1.0,
17
+ "max_new_tokens": 512
18
+ }
inference.py ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # -*- coding: utf-8 -*-
2
+ """inference.py — samai-pnet-dmoe-2b 拉取即跑推理示例
3
+
4
+ 用法:
5
+ python inference.py # 默认 HuggingFace 仓
6
+ python inference.py --repo /path/to/local # 本地目录
7
+ python inference.py --lru 4 # 启用 SSD-LRU 专家缓存模拟 (每层容量 4)
8
+
9
+ 依赖: torch>=2.4, transformers>=5.0 (trust_remote_code 加载仓内 modeling_samai_pnet.py)
10
+ """
11
+ import argparse
12
+
13
+ import torch
14
+ from transformers import AutoModelForCausalLM, AutoTokenizer
15
+
16
+
17
+ def main():
18
+ ap = argparse.ArgumentParser()
19
+ ap.add_argument("--repo", default="tchbcb/samai-pnet-dmoe-2b")
20
+ ap.add_argument("--lru", type=int, default=0,
21
+ help=">0 启用 SSD-LRU 专家缓存 (每层容量; 需仓内 modeling 模块)")
22
+ ap.add_argument("--max-new-tokens", type=int, default=256)
23
+ args = ap.parse_args()
24
+
25
+ tok = AutoTokenizer.from_pretrained(args.repo, trust_remote_code=True)
26
+ model = AutoModelForCausalLM.from_pretrained(args.repo, trust_remote_code=True,
27
+ dtype="auto").eval()
28
+ device = "cuda" if torch.cuda.is_available() else "cpu"
29
+ model.to(device)
30
+
31
+ if args.lru:
32
+ # SSDLRUExpertManager: 专家驻留 CPU(模拟 SSD), 路由激活时搬入, LRU 驱逐
33
+ from modeling_samai_pnet import SSDLRUExpertManager # remote_code 模块
34
+ mgr = SSDLRUExpertManager.install(model, capacity_per_layer=args.lru)
35
+ print(f"[ssd-lru] capacity/layer={args.lru} (storage=cpu)")
36
+
37
+ messages = [{"role": "user", "content": "一个长方形长 12 米宽 8 米,面积是多少?请简要推理。"}]
38
+ enc = tok.apply_chat_template(messages, add_generation_prompt=True,
39
+ tokenize=True, return_tensors="pt",
40
+ return_dict=True).to(device)
41
+ with torch.no_grad():
42
+ out = model.generate(**enc, max_new_tokens=args.max_new_tokens,
43
+ do_sample=False, pad_token_id=tok.pad_token_id or 1)
44
+ text = tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True)
45
+ print("=== 回答 ===")
46
+ print(text.strip())
47
+
48
+ # Ponder 诊断: 每次 forward 追加一条 {mode, executed, steps_mean, ...}
49
+ decode_entries = [e for e in model._ponder_log if e.get("mode") == "decode"]
50
+ if decode_entries:
51
+ steps = sum(e["steps_mean"] for e in decode_entries) / len(decode_entries)
52
+ print(f"\n=== Ponder 诊断 ===\n本次生成平均思考步数: {steps:.2f} "
53
+ f"(max_ponder_steps={model.config.max_ponder_steps})")
54
+
55
+ if args.lru:
56
+ s = mgr.summary()
57
+ print(f"=== SSD-LRU ===\nhit={s['hit']} miss={s['miss']} "
58
+ f"evict={s['evict']} hit_rate={s['hit_rate']:.3f}")
59
+ mgr.remove()
60
+
61
+
62
+ if __name__ == "__main__":
63
+ main()
modeling_samai_pnet.py ADDED
@@ -0,0 +1,1264 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # -*- coding: utf-8 -*-
2
+ """
3
+ modeling_samai_pnet.py — SamAI-Pnet-dMoE-2B 自洽模型实现 (发布版, 单文件)。
4
+
5
+ 架构 = PonderNet 自适应停机 (质量维度) + 动态可变 MoE (效率维度) + MTP (训练期吞吐)。
6
+ 本文件为发行版自洽模块: 不依赖训练仓其他文件, 仅需 torch + transformers>=5.0。
7
+
8
+ 用法 (trust_remote_code 一行加载):
9
+ import torch
10
+ from transformers import AutoModelForCausalLM, AutoTokenizer
11
+ model = AutoModelForCausalLM.from_pretrained(
12
+ "tchbcb/samai-pnet-dmoe-2b", trust_remote_code=True,
13
+ dtype="auto").eval().cuda()
14
+ tok = AutoTokenizer.from_pretrained("tchbcb/samai-pnet-dmoe-2b")
15
+
16
+ 核心机制:
17
+ 1. Ponder 思考块: 后 8 层循环执行, halting 头输出每步停机概率 lam_k,
18
+ 概率加权混合 h = Σ w_k·h_k, 累计质量 1-eps 即停; KV 经 commit pass 写回。
19
+ 2. 动态可变 MoE: FFN 按宽度均分 8 个同构专家, Router(已训练) 输出阈值激活
20
+ (p_i >= tau 激活, 激活数 1~8 自适应); 每 token 仅首轮路由一次, 循环复用。
21
+ 3. MTP: 训练期辅助头, 推理 generate() 自动旁路。
22
+ 4. round5-B: per-step CE 训练的 halting 头 —— hard 样本自适应"多想",
23
+ easy 样本早停 (hard-easy steps 分化 +0.433)。
24
+
25
+ 诊断: model.ponder_log 追加每次 forward 的 {mode, executed, steps_mean, ...}。
26
+ """
27
+ from __future__ import annotations
28
+
29
+ from collections import OrderedDict
30
+ from dataclasses import dataclass
31
+
32
+ import torch
33
+ import torch.nn.functional as F
34
+ from torch import nn
35
+ from torch.utils.checkpoint import checkpoint as _ckpt
36
+
37
+ from transformers import AutoConfig, AutoModelForCausalLM
38
+ from transformers.activations import ACT2FN
39
+ from transformers.cache_utils import DynamicCache
40
+ from transformers.masking_utils import create_causal_mask
41
+ from transformers.modeling_outputs import CausalLMOutputWithPast
42
+ from transformers.models.llama.configuration_llama import LlamaConfig
43
+ from transformers.models.llama.modeling_llama import LlamaForCausalLM, LlamaRMSNorm
44
+ from transformers.utils import can_return_tuple
45
+
46
+
47
+ # --------------------------------------------------------------------------
48
+ # Config
49
+ # --------------------------------------------------------------------------
50
+ class PonderLlamaConfig(LlamaConfig):
51
+ """在 LlamaConfig 之上追加 PonderNet 超参 (其余字段与原 Llama config 完全兼容)."""
52
+
53
+ model_type = "llama_ponder"
54
+
55
+ def __init__(self, **kwargs):
56
+ self.ponder_start_layer = kwargs.pop("ponder_start_layer", None) # None = 最后 8 层
57
+ self.ponder_window = kwargs.pop("ponder_window", 1) # prefill 思考窗口 W
58
+ self.max_ponder_steps = kwargs.pop("max_ponder_steps", 8) # 思考块最多循环步数 K
59
+ self.ponder_epsilon = kwargs.pop("ponder_epsilon", 0.01) # 累计概率 1-eps 即停
60
+ self.ponder_signal = kwargs.pop("ponder_signal", "entropy") # entropy|msp|drift|learned
61
+ self.ponder_all_positions = kwargs.pop("ponder_all_positions", False) # 全序列位置思考
62
+ self.commit_kv = kwargs.pop("commit_kv", True) # 用混合表示写最终 KV
63
+ self.ponder_prior_p = kwargs.pop("ponder_prior_p", 0.5) # 几何先验 p_g (训练)
64
+ self.ponder_loss_beta = kwargs.pop("ponder_loss_beta", 0.01) # KL 正则权重 beta
65
+ self.drift_threshold = kwargs.pop("drift_threshold", 0.98) # drift 相似度阈值
66
+ # ---- Round5 三方向开关 ----
67
+ self.ponder_step_ce_weight = kwargs.pop("ponder_step_ce_weight", 0.0) # 方向②: per-step CE 权重 (0=关)
68
+ self.ponder_forced_steps = kwargs.pop("ponder_forced_steps", 0) # 方向③: 强制思考步数 rollout (0=关)
69
+ self.ponder_head_hidden = kwargs.pop("ponder_head_hidden", 0) # 方向①: >0 时 halting 头升级 MLP
70
+ super().__init__(**kwargs)
71
+ if self.ponder_start_layer is None:
72
+ self.ponder_start_layer = max(0, self.num_hidden_layers - 8)
73
+
74
+ @classmethod
75
+ def from_llama(cls, llama_config: LlamaConfig, **ponder_kwargs) -> "PonderLlamaConfig":
76
+ """从标准 LlamaConfig 派生 (保留全部架构字段, 丢弃非配置项)。
77
+ 兼容两种来源: 标准 llama config.json (无 ponder 字段) 与本类保存的
78
+ llama_ponder config.json (含 ponder 字段) —— 显式 ponder_kwargs 优先覆盖。"""
79
+ d = llama_config.to_dict()
80
+ for k in ("model_type", "architectures", "_name_or_path", "transformers_version",
81
+ "torch_dtype"):
82
+ d.pop(k, None)
83
+ ponder_keys = [k for k in d
84
+ if "ponder" in k or k in ("commit_kv", "drift_threshold")]
85
+ for k in ponder_keys:
86
+ if k in ponder_kwargs: # 显式参数覆盖 config 里的旧值, 避免重复 kwargs 报错
87
+ d.pop(k)
88
+ return cls(**d, **ponder_kwargs)
89
+
90
+
91
+ # --------------------------------------------------------------------------
92
+ # ���出对象
93
+ # --------------------------------------------------------------------------
94
+ @dataclass
95
+ class PonderCausalLMOutputWithPast(CausalLMOutputWithPast):
96
+ """额外带出 ponder 诊断信息."""
97
+ ponder_weights: torch.Tensor | None = None # (B, W, K) 每步停机质量分布
98
+ ponder_steps: torch.Tensor | None = None # (B, W) 每位置实际思考步数
99
+ ponder_kl: torch.Tensor | None = None # 标量, KL(w‖Geometric) 正则项 (labels 时)
100
+ ponder_step_ce: torch.Tensor | None = None # 标量, per-step CE (方向②, 训练且权重>0 时)
101
+
102
+
103
+ # --------------------------------------------------------------------------
104
+ # Model
105
+ # --------------------------------------------------------------------------
106
+ class PonderLlamaForCausalLM(LlamaForCausalLM):
107
+ """
108
+ 用法 A — 直接加载 MiniCPM5-2B 权重 (键名与 LlamaForCausalLM 完全一致):
109
+ from modeling_samai_pnet import PonderLlamaForCausalLM, PonderLlamaConfig
110
+ base = LlamaConfig.from_pretrained("tchbcb/MiniCPM5-2B-cpu")
111
+ cfg = PonderLlamaConfig.from_llama(base)
112
+ model = PonderLlamaForCausalLM.from_pretrained(
113
+ "tchbcb/MiniCPM5-2B-cpu", config=cfg, torch_dtype="bfloat16")
114
+ # missing keys 只有 ponder_head (仅 signal="learned" 时参与前向, 其余信号忽略)
115
+
116
+ 用法 B — 一行加载:
117
+ model = PonderLlamaForCausalLM.from_ponder("tchbcb/MiniCPM5-2B-cpu")
118
+
119
+ generate() / chat 流程与 HF 标准完全一致; model.ponder_log 记录每次 forward 的思考步数。
120
+ """
121
+
122
+ def __init__(self, config: PonderLlamaConfig):
123
+ super().__init__(config)
124
+ cfg = self.config
125
+ # 真 PonderNet 停机头; bias=-4 使初始 lam≈0.018 (初始倾向于多想几步)
126
+ # 方向①: ponder_head_hidden>0 时升级为 MLP (Linear->GELU->Linear),
127
+ # 打破线性可分上限 —— round4 结论: 线性头在冻结特征上已到极限。
128
+ # 数值初始化由下方 _init_weights 覆写统一处理: post_init 重遍历会把
129
+ # Linear 重置为 Llama 默认 (normal+零偏置), 自定义值必须写在那里才能存活
130
+ head_hidden = int(getattr(cfg, "ponder_head_hidden", 0) or 0)
131
+ if head_hidden > 0:
132
+ self.ponder_head = nn.Sequential(
133
+ nn.Linear(cfg.hidden_size, head_hidden),
134
+ nn.GELU(),
135
+ nn.Linear(head_hidden, 1, bias=True),
136
+ )
137
+ else:
138
+ self.ponder_head = nn.Linear(cfg.hidden_size, 1, bias=True)
139
+ self._ponder_log: list[dict] = [] # 诊断日志 (每次 forward 追加, 最多保留 4096 条)
140
+ self.post_init()
141
+
142
+ def _init_weights(self, module):
143
+ """PonderNet 停机头自定义初始化 (覆盖 Llama 默认), 其余模块遵循 Llama 惯例。
144
+ post_init()/from_pretrained 的 init_weights 遍历都会调用本钩子,
145
+ 因此自定义值不会被 Llama 默认的 normal+零偏置覆写。"""
146
+ head = getattr(self, "ponder_head", None)
147
+ if isinstance(module, nn.Linear) and head is not None:
148
+ head_linears = [mm for mm in head.modules() if isinstance(mm, nn.Linear)]
149
+ if any(module is mm for mm in head_linears):
150
+ nn.init.zeros_(module.weight)
151
+ nn.init.zeros_(module.bias)
152
+ if module is head_linears[-1]:
153
+ # 尾层 bias=-4 -> 初始 lam=sigmoid(-4)≈0.018, 倾向多思考
154
+ nn.init.constant_(module.bias, -4.0)
155
+ return
156
+ super()._init_weights(module)
157
+
158
+ @classmethod
159
+ def from_ponder(cls, pretrained_path: str, ponder_kwargs: dict | None = None, **kw):
160
+ """从标准 Llama 权重目录 / HF repo 一行构造 Ponder 模型."""
161
+ base = LlamaConfig.from_pretrained(pretrained_path)
162
+ cfg = PonderLlamaConfig.from_llama(base, **(ponder_kwargs or {}))
163
+ return cls.from_pretrained(pretrained_path, config=cfg, **kw)
164
+
165
+ # ------------------------------------------------------------------ #
166
+ # 内部工具
167
+ # ------------------------------------------------------------------ #
168
+ @property
169
+ def _ponder_layers(self):
170
+ s = self.config.ponder_start_layer
171
+ return self.model.layers[s: self.config.num_hidden_layers]
172
+
173
+ def _crop_ponder_kv(self, cache, n_remove: int):
174
+ """只回滚思考块层的 KV 写入 (前段层的 KV 不许动)."""
175
+ if cache is None or n_remove <= 0:
176
+ return
177
+ s = self.config.ponder_start_layer
178
+ for li in range(s, self.config.num_hidden_layers):
179
+ cache.layers[li].crop(-n_remove)
180
+
181
+ def _run_block(self, h, attention_mask, position_ids, position_embeddings,
182
+ cache, kwargs):
183
+ """跑一遍思考块 (最后若干层).
184
+ 训练且开启 gradient checkpointing 时逐层重算省激活 —— 思考块最多循环 K 次,
185
+ 激活占用是普通前向的 K 倍, checkpointing 是小显存卡 (如 T4 16GB) 的关键."""
186
+ ckpt_fn = getattr(self.model, "_gradient_checkpointing_func", None)
187
+ use_ckpt = (self.training and cache is None
188
+ and getattr(self.model, "gradient_checkpointing", False)
189
+ and ckpt_fn is not None)
190
+ for layer in self._ponder_layers:
191
+ if use_ckpt:
192
+ h = ckpt_fn(
193
+ layer.__call__, h,
194
+ attention_mask=attention_mask,
195
+ position_ids=position_ids,
196
+ position_embeddings=position_embeddings,
197
+ past_key_values=None,
198
+ use_cache=False,
199
+ **kwargs,
200
+ )
201
+ else:
202
+ h = layer(
203
+ h,
204
+ attention_mask=attention_mask,
205
+ position_ids=position_ids,
206
+ position_embeddings=position_embeddings,
207
+ past_key_values=cache,
208
+ use_cache=cache is not None,
209
+ **kwargs,
210
+ )
211
+ return h
212
+
213
+ def _halting_prob(self, h_window, h_prev_window) -> torch.Tensor:
214
+ """返回 (B, W) 的停机概率 lam ∈ (0, 1)."""
215
+ cfg = self.config
216
+ B, W = h_window.shape[:2]
217
+ if cfg.ponder_signal == "learned":
218
+ # ponder_head 恒为 fp32 (数值稳定), 输入 cast 到其参数 dtype,
219
+ # 否则半精度模型 (fp16/bf16) 下会报 dtype 不匹配错误。
220
+ # head_ref 写法兼容 Linear 与 Sequential (方向① MLP 头) 两种结构
221
+ head_ref = next(self.ponder_head.parameters())
222
+ lam = torch.sigmoid(
223
+ self.ponder_head(h_window.to(head_ref.dtype))
224
+ ).squeeze(-1)
225
+ elif cfg.ponder_signal in ("entropy", "msp"):
226
+ logits = self.lm_head(self.model.norm(h_window)).float()
227
+ probs = F.softmax(logits, dim=-1)
228
+ if cfg.ponder_signal == "msp":
229
+ lam = probs.max(dim=-1).values
230
+ else: # entropy: 自信 -> 熵小 -> lam 大
231
+ ent = -(probs * torch.log(probs.clamp_min(1e-9))).sum(-1)
232
+ lam = 1.0 - ent / torch.log(torch.tensor(float(logits.size(-1)), device=logits.device))
233
+ elif cfg.ponder_signal == "drift":
234
+ if h_prev_window is None: # 第一步没有参照, 强制继续思考
235
+ lam = torch.zeros(B, W, device=h_window.device, dtype=h_window.dtype)
236
+ else:
237
+ cos = F.cosine_similarity(h_window.float(), h_prev_window.float(), dim=-1)
238
+ tau = cfg.drift_threshold
239
+ lam = (cos - tau) / (1.0 - tau + 1e-6) # cos 越接近 1 (不再变化) 越停
240
+ else:
241
+ raise ValueError(f"unknown ponder_signal: {cfg.ponder_signal}")
242
+ # 统一 cast 回隐状态 dtype: 保证 remaining/mix/w_stack 混合链路 dtype 一致,
243
+ # 否则 fp32 的 lam 会把整条链路升成 fp32, 之后 norm/lm_head (half 权重) 会崩
244
+ return lam.to(h_window.dtype).clamp(1e-6, 1.0 - 1e-6)
245
+
246
+ def _step_ce(self, h_step, labels):
247
+ """方向②: 单步表示的独立解码 CE。
248
+ 让每一步思考表示都"可用"(可被 lm_head 解码), 回收 mix 摊薄导致的 CE 代价。
249
+ 训练时用 gradient checkpointing 重算, 避免大词表 logits 常驻显存。"""
250
+ def _fn(h_):
251
+ lg = self.lm_head(self.model.norm(h_))
252
+ lg = lg[..., :-1, :].reshape(-1, lg.size(-1)).float()
253
+ tg = labels[..., 1:].reshape(-1)
254
+ return F.cross_entropy(lg, tg, ignore_index=-100)
255
+ if self.training and torch.is_grad_enabled() and h_step.requires_grad:
256
+ return _ckpt(_fn, h_step, use_reentrant=False)
257
+ return _fn(h_step)
258
+
259
+ def _win_mask(self, W, past_len, device, dtype, attention_mask=None):
260
+ """窗口迭代的加性因果 mask: keys = [0..past_len+W), queries = 窗口 W 个位置。
261
+
262
+ 必须手工构建而不能用 create_causal_mask: 后者从非思考层读取缓存长度,
263
+ 而思考块层已被 crop 到 past_len, 会得到多 W 的错误 kv 尺寸。
264
+ 返回 float 加性 mask (sdpa / eager 通用)。"""
265
+ total = past_len + W
266
+ q_idx = torch.arange(past_len, total, device=device)
267
+ k_idx = torch.arange(total, device=device)
268
+ allowed = q_idx.unsqueeze(-1) >= k_idx.unsqueeze(0) # (W, total)
269
+ if attention_mask is not None:
270
+ allowed = allowed.unsqueeze(0) & attention_mask[:, :total].to(torch.bool).unsqueeze(1)
271
+ m = torch.zeros(allowed.size(0), 1, W, total, dtype=dtype, device=device)
272
+ m = m.masked_fill(~allowed.unsqueeze(1), torch.finfo(dtype).min) # (B,1,W,total)
273
+ else:
274
+ m = torch.zeros(1, 1, W, total, dtype=dtype, device=device)
275
+ m = m.masked_fill(~allowed.unsqueeze(0).unsqueeze(0), torch.finfo(dtype).min)
276
+ return m
277
+
278
+ def _log(self, entry: dict):
279
+ self._ponder_log.append(entry)
280
+ if len(self._ponder_log) > 4096:
281
+ del self._ponder_log[: len(self._ponder_log) - 4096]
282
+
283
+ # ------------------------------------------------------------------ #
284
+ # 主前向
285
+ # ------------------------------------------------------------------ #
286
+ @can_return_tuple
287
+ def forward(
288
+ self,
289
+ input_ids: torch.LongTensor | None = None,
290
+ attention_mask: torch.Tensor | None = None,
291
+ position_ids: torch.LongTensor | None = None,
292
+ past_key_values: DynamicCache | None = None,
293
+ inputs_embeds: torch.FloatTensor | None = None,
294
+ labels: torch.LongTensor | None = None,
295
+ use_cache: bool | None = None,
296
+ logits_to_keep: int | torch.Tensor = 0,
297
+ output_ponder: bool = False,
298
+ **kwargs,
299
+ ) -> PonderCausalLMOutputWithPast:
300
+ cfg = self.config
301
+ m = self.model
302
+
303
+ # ---- 输入 & 位置 ------------------------------------------------
304
+ if (input_ids is None) ^ (inputs_embeds is not None):
305
+ raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
306
+ if inputs_embeds is None:
307
+ inputs_embeds = m.embed_tokens(input_ids)
308
+ B, Q = inputs_embeds.shape[:2]
309
+
310
+ if use_cache is None:
311
+ use_cache = False if self.training else cfg.use_cache
312
+ if use_cache and past_key_values is None:
313
+ past_key_values = DynamicCache(config=cfg)
314
+
315
+ cache_position = kwargs.pop("cache_position", None)
316
+ past_seen = past_key_values.get_seq_length() if past_key_values is not None else 0
317
+ if position_ids is None:
318
+ base = (cache_position if cache_position is not None
319
+ else torch.arange(Q, device=inputs_embeds.device) + past_seen)
320
+ position_ids = base.reshape(1, -1).expand(B, -1)[:, :Q].to(inputs_embeds.device)
321
+
322
+ # ---- mask / rope (整次 forward 内复用) ---------------------------
323
+ causal_mask = create_causal_mask(
324
+ config=cfg, inputs_embeds=inputs_embeds, attention_mask=attention_mask,
325
+ past_key_values=past_key_values, position_ids=position_ids,
326
+ )
327
+ position_embeddings = m.rotary_emb(inputs_embeds, position_ids=position_ids)
328
+
329
+ # ---- Stage A: 前段层正常单遍 -------------------------------------
330
+ h = inputs_embeds
331
+ for layer in m.layers[: cfg.ponder_start_layer]:
332
+ h = layer(
333
+ h,
334
+ attention_mask=causal_mask,
335
+ position_ids=position_ids,
336
+ position_embeddings=position_embeddings,
337
+ past_key_values=past_key_values,
338
+ use_cache=use_cache,
339
+ **kwargs,
340
+ )
341
+
342
+ # ---- 思考窗口 W --------------------------------------------------
343
+ # decode (Q==1) 恒为 1; 训练/显式开启时对全部位置; 其余 prefill 思考末尾 W 个位置
344
+ if Q == 1 or self.training or cfg.ponder_all_positions:
345
+ W = Q
346
+ else:
347
+ W = max(1, min(cfg.ponder_window, Q))
348
+
349
+ eps = cfg.ponder_epsilon
350
+ K = cfg.max_ponder_steps
351
+ device, dtype = inputs_embeds.device, inputs_embeds.dtype
352
+
353
+ remaining = torch.ones(B, W, device=device, dtype=dtype)
354
+ cum = torch.zeros(B, W, device=device, dtype=dtype)
355
+ active = torch.ones(B, W, device=device, dtype=torch.bool)
356
+ mix = torch.zeros(B, W, h.size(-1), device=device, dtype=dtype)
357
+ w_list: list[torch.Tensor] = []
358
+ h_prev_win = None
359
+ h_full = None # 第 1 次迭代的全序列输出 (W<Q 时非窗口位置沿用)
360
+ h_win = h[:, -W:, :] if W < Q else h
361
+ executed = 0
362
+ win_mask_built = False # 窗口迭代专用 mask (懒构建一次)
363
+
364
+ # ---- Round5 方向②③ 前置开关 --------------------------------------
365
+ forced = int(getattr(cfg, "ponder_forced_steps", 0) or 0) # 方向③: >0 强制 rollout
366
+ step_ce_weight = float(getattr(cfg, "ponder_step_ce_weight", 0.0) or 0.0)
367
+ use_step_ce = (step_ce_weight > 0.0 and labels is not None and self.training)
368
+ step_pairs: list[tuple[torch.Tensor, torch.Tensor]] = [] # [(w_k_detached, ce_k), ...]
369
+ loop_K = forced if forced > 0 else K
370
+
371
+ for k in range(1, loop_K + 1):
372
+ executed = k
373
+ if k > 1:
374
+ # 替换式写入: 先回滚上一次迭代在思考块层的 KV, 本轮 update 再补回,
375
+ # 使任意时刻 cache 长度恒等于真实 token 数 (对 W==Q 同样必需!)
376
+ self._crop_ponder_kv(past_key_values, W)
377
+ # 仅当有 cache 且 W<Q 时才做窗口局部重算; 无 cache (训练) 一律全序列重算
378
+ use_window = (k > 1) and (past_key_values is not None) and (W < Q)
379
+ if not use_window:
380
+ h = self._run_block(h, causal_mask, position_ids,
381
+ position_embeddings, past_key_values, kwargs)
382
+ h_win = h if W == Q else h[:, -W:, :]
383
+ if W < Q:
384
+ h_full = h
385
+ else:
386
+ if not win_mask_built:
387
+ past_len = int(past_key_values.layers[cfg.ponder_start_layer].get_seq_length())
388
+ mask_win = self._win_mask(W, past_len, device, dtype, attention_mask)
389
+ pos_emb_win = (position_embeddings[0][:, -W:],
390
+ position_embeddings[1][:, -W:])
391
+ win_mask_built = True
392
+ h_win = self._run_block(h_win, mask_win, position_ids[:, -W:],
393
+ pos_emb_win, past_key_values, kwargs)
394
+
395
+ if forced > 0:
396
+ # 方向③: 强制步数 rollout —— 完全绕过 halting 头与 w 混合,
397
+ # 最终表示即第 forced 步隐状态 (KV 恰为其 cache, 无需 commit)
398
+ continue
399
+
400
+ lam = self._halting_prob(h_win, h_prev_win) # (B, W)
401
+ lam_eff = lam * active.to(lam.dtype)
402
+ w_k = remaining * lam_eff
403
+ mix = mix + w_k.unsqueeze(-1) * h_win
404
+ w_list.append(w_k)
405
+ remaining = remaining * (1.0 - lam_eff)
406
+ cum = cum + w_k
407
+ active = active & (cum < 1.0 - eps)
408
+ h_prev_win = h_win
409
+ if use_step_ce:
410
+ # 方向②: 本步表示单独算解码 CE。权重 detach —— 梯度只流向表示,
411
+ # 不经此损失推动 w (w 由 step-supervise / KL 单独管辖)
412
+ ce_k = self._step_ce(h_win, labels)
413
+ step_pairs.append((w_k.detach(), ce_k))
414
+ if not bool(active.any()):
415
+ break
416
+
417
+ if forced > 0:
418
+ # ---- 方向③: 最终表示 = 第 n 步隐状态 (非概率混合) --------------
419
+ if W < Q:
420
+ h_final = h_full.clone()
421
+ h_final[:, -W:, :] = h_win
422
+ else:
423
+ h_final = h_win
424
+ w_stack = F.one_hot(
425
+ torch.full((B, W), forced - 1, device=device), forced
426
+ ).to(dtype) # (B, W, n) 诊断用
427
+ else:
428
+ # 剩余概率质量全部记给最后一步 (PonderNet 的 p_N 截断)
429
+ mix = mix + remaining.unsqueeze(-1) * h_win
430
+ w_stack = torch.stack(w_list, dim=-1).to(dtype) # (B, W, K_exec)
431
+ w_stack = torch.cat([w_stack[..., :-1],
432
+ w_stack[..., -1:] + remaining.unsqueeze(-1)], dim=-1)
433
+
434
+ # ---- 组装最终 hidden --------------------------------------------
435
+ if W < Q:
436
+ h_final = h_full
437
+ h_final[:, -W:, :] = mix
438
+ else:
439
+ h_final = mix
440
+
441
+ # ---- commit pass: 用"思考后"的混合表示写最终 KV ------------------
442
+ if (use_cache and cfg.commit_kv and past_key_values is not None
443
+ and executed > 1 and forced == 0):
444
+ self._crop_ponder_kv(past_key_values, W)
445
+ if not win_mask_built: # W==Q 的场景
446
+ self._run_block(h_final, causal_mask, position_ids,
447
+ position_embeddings, past_key_values, kwargs)
448
+ else:
449
+ self._run_block(h_final[:, -W:, :], mask_win, position_ids[:, -W:],
450
+ pos_emb_win, past_key_values, kwargs)
451
+ # commit 输出丢弃, KV 由 mix 重新计算 —— 后续 token 看到的就是思考后的表示
452
+
453
+ # ---- logits / loss ----------------------------------------------
454
+ slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
455
+ logits = self.lm_head(m.norm(h_final[:, slice_indices, :]))
456
+
457
+ loss, ponder_kl, ponder_step_ce = None, None, None
458
+ if labels is not None:
459
+ ce_logits = logits.float()
460
+ loss = F.cross_entropy(
461
+ ce_logits[..., :-1, :].reshape(-1, ce_logits.size(-1)),
462
+ labels[..., 1:].reshape(-1), ignore_index=-100)
463
+
464
+ # PonderNet 正则: KL(w || Geometric(p_g)), 截断到 K 步并归一化。
465
+ # 必须升到 fp32 再取 log: fp16 下 p=0 处 clamp_min(1e-8) 仍会下溢为 0,
466
+ # log(0)=-inf 与 p=0 相乘直接得到 NaN, 训练瞬间发散
467
+ p = (w_stack / w_stack.sum(-1, keepdim=True).clamp_min(1e-8)).float() # (B,W,K)
468
+ Kd = p.size(-1)
469
+ n_idx = torch.arange(1, Kd + 1, device=p.device, dtype=p.dtype)
470
+ pg = cfg.ponder_prior_p
471
+ geom = (1 - pg) ** (n_idx - 1) * pg
472
+ geom = torch.cat([geom[..., :-1],
473
+ ((1 - pg) ** (Kd - 1) * torch.ones_like(geom[..., -1:]))],
474
+ dim=-1)
475
+ geom = geom / geom.sum()
476
+ kl = (p * (torch.log(p.clamp_min(1e-8))
477
+ - torch.log(geom.expand_as(p)))).sum(-1)
478
+ ponder_kl = kl.mean()
479
+ loss = loss + cfg.ponder_loss_beta * ponder_kl
480
+
481
+ # 方向②: per-step CE 聚合 —— 每步 CE 按该步 w 质量 (detach) 加权,
482
+ # 让被分配到概率质量的思考步都保持"可解码", 回收 mix 摊薄的 CE 代价
483
+ if use_step_ce and step_pairs:
484
+ ws = torch.stack([p[0] for p in step_pairs], dim=0) # (K,B,W)
485
+ ces = torch.stack([p[1] for p in step_pairs], dim=0) # (K,)
486
+ wsum = ws.sum(0).clamp_min(1e-8) # (B,W)
487
+ per_pos = (ws * ces.view(-1, 1, 1)).sum(0) / wsum # (B,W)
488
+ ponder_step_ce = per_pos.mean()
489
+ loss = loss + step_ce_weight * ponder_step_ce
490
+
491
+ # ---- 诊断 --------------------------------------------------------
492
+ cum_w = torch.cumsum(w_stack, dim=-1)
493
+ steps_used = ((cum_w < 1.0 - eps).sum(-1) + 1).clamp(max=w_stack.size(-1)) # (B, W)
494
+ self._log({
495
+ "mode": "train" if self.training else ("decode" if Q == 1 else "prefill"),
496
+ "signal": cfg.ponder_signal,
497
+ "executed": executed,
498
+ "steps_max": int(steps_used.max()),
499
+ "steps_mean": float(steps_used.float().mean()),
500
+ })
501
+
502
+ return PonderCausalLMOutputWithPast(
503
+ loss=loss,
504
+ logits=logits,
505
+ past_key_values=past_key_values,
506
+ ponder_weights=w_stack if output_ponder else None,
507
+ ponder_steps=steps_used if output_ponder else None,
508
+ ponder_kl=ponder_kl,
509
+ ponder_step_ce=ponder_step_ce,
510
+ )
511
+
512
+
513
+ # --------------------------------------------------------------------------
514
+ # AutoClass 注册 (可重复 import 安全)
515
+ # --------------------------------------------------------------------------
516
+ try:
517
+ AutoConfig.register("llama_ponder", PonderLlamaConfig)
518
+ AutoModelForCausalLM.register(PonderLlamaConfig, PonderLlamaForCausalLM)
519
+ except Exception:
520
+ pass
521
+
522
+
523
+ # --------------------------------------------------------------------------
524
+ # Config
525
+ # --------------------------------------------------------------------------
526
+ class PonderMoELlamaConfig(PonderLlamaConfig):
527
+ """PonderLlamaConfig + 动态 MoE / MTP 超参. moe_num_experts=0 且 mtp_num_tokens=0
528
+ 时行为与父类完全一致 (回归安全)."""
529
+
530
+ model_type = "llama_ponder_moe"
531
+
532
+ def __init__(self, **kwargs):
533
+ # ---- 动态可变 MoE ----
534
+ self.moe_num_experts = int(kwargs.pop("moe_num_experts", 0) or 0) # 0=关
535
+ self.moe_route_tau = kwargs.pop("moe_route_tau", None) # 激活阈值, None -> 1/N
536
+ self.moe_aux_weight = float(kwargs.pop("moe_aux_weight", 0.01)) # 负载均衡权重
537
+ self.moe_scope = kwargs.pop("moe_scope", "ponder") # ponder|all: 拆哪些层的 FFN
538
+ # ---- MTP ----
539
+ self.mtp_num_tokens = int(kwargs.pop("mtp_num_tokens", 0) or 0) # 深度 D, 0=关
540
+ self.mtp_loss_weight = float(kwargs.pop("mtp_loss_weight", 1.0))
541
+ super().__init__(**kwargs)
542
+ # 普通属性而非 property (transformers 5.16 config __getattribute__ 钩子不支持 property)
543
+ self.moe_tau = (float(self.moe_route_tau) if self.moe_route_tau
544
+ else 1.0 / max(1, self.moe_num_experts))
545
+
546
+ @classmethod
547
+ def from_llama(cls, llama_config: LlamaConfig, **kwargs) -> "PonderMoELlamaConfig":
548
+ """同父类, 但额外吞掉 moe_*/mtp_* 旧键, 兼容本类保存的 config.json."""
549
+ d = llama_config.to_dict()
550
+ for k in ("model_type", "architectures", "_name_or_path", "transformers_version",
551
+ "torch_dtype"):
552
+ d.pop(k, None)
553
+ drop = [k for k in d
554
+ if "ponder" in k or k.startswith(("moe_", "mtp_"))
555
+ or k in ("commit_kv", "drift_threshold")]
556
+ for k in drop:
557
+ if k in kwargs:
558
+ d.pop(k)
559
+ return cls(**d, **kwargs)
560
+
561
+
562
+ @dataclass
563
+ class PonderMoECausalLMOutputWithPast(PonderCausalLMOutputWithPast):
564
+ """追加 MoE / MTP 诊断字段."""
565
+ moe_aux_loss: torch.Tensor | None = None # 标量, 负载均衡损失 (各 MoE 层均值)
566
+ moe_active_mean: torch.Tensor | None = None # 标量, 平均激活专家数 ∈ [1, N]
567
+ mtp_loss: torch.Tensor | None = None # 标量, 各深度 MTP CE 均值
568
+ mtp_acc: torch.Tensor | None = None # 标量, MTP top-1 命中率 (诊断)
569
+
570
+
571
+ # --------------------------------------------------------------------------
572
+ # 同构专家: 原 FFN 的 1/N 宽切片 (gate/up 按行切, down 按列切), 参数守恒
573
+ # --------------------------------------------------------------------------
574
+ class ExpertMLP(nn.Module):
575
+ def __init__(self, hidden_size: int, inter_size: int, act_fn):
576
+ super().__init__()
577
+ self.gate_proj = nn.Linear(hidden_size, inter_size, bias=False)
578
+ self.up_proj = nn.Linear(hidden_size, inter_size, bias=False)
579
+ self.down_proj = nn.Linear(inter_size, hidden_size, bias=False)
580
+ self.act_fn = act_fn
581
+
582
+ def forward(self, x):
583
+ return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
584
+
585
+
586
+ # --------------------------------------------------------------------------
587
+ # 动态可变 MoE 层: Router(零初始化) + N 同构专家 + 阈值动态激活 + STE + 稀疏执行
588
+ # --------------------------------------------------------------------------
589
+ class PonderMoEMLP(nn.Module):
590
+ def __init__(self, hidden_size: int, n_experts: int, tau: float, act_fn):
591
+ super().__init__()
592
+ self.n_experts = n_experts
593
+ self.tau = float(tau)
594
+ self.router = nn.Linear(hidden_size, n_experts, bias=True)
595
+ nn.init.zeros_(self.router.weight)
596
+ nn.init.zeros_(self.router.bias) # 零路由 -> p=1/N 均匀 -> 初始全激活 = 稠密等价
597
+ self.experts = nn.ModuleList([])
598
+ self.act_fn_holder = act_fn # 仅占位保存引用 (无参数)
599
+ # 运行期状态 (诊断/冻结路由)
600
+ self._last_mask = None # (B, T, N) bool, 最近一次 fresh 路由
601
+ self._last_p = None # (B, T, N) float, detach 后的路由概率
602
+ self.last_aux = None # 标量 tensor (fresh 时更新, 模型聚合)
603
+ self.last_active = None # 标量 tensor, 平均激活专家数
604
+ self.n_route_calls = 0 # 结构性验证: Router 实际前向次数
605
+ self.cache_hook = None # SSD-LRU 挂钩: callable(expert_ids)
606
+ self.cache_post_hook = None # SSD-LRU 执行后挂钩: 强制容量硬上界
607
+
608
+ # ---------------- 构造 ---------------- #
609
+ @classmethod
610
+ def from_dense(cls, dense, n_experts: int, tau: float | None = None) -> "PonderMoEMLP":
611
+ """把稠密 LlamaMLP 拆成 n_experts 个同构专家 (参数守恒) + 零 Router."""
612
+ H, I = dense.hidden_size, dense.intermediate_size
613
+ assert I % n_experts == 0, f"intermediate_size({I}) 必须被 N({n_experts}) 整除"
614
+ m = cls(H, n_experts, tau if tau is not None else 1.0 / n_experts, dense.act_fn)
615
+ sub = I // n_experts
616
+ with torch.no_grad():
617
+ for i in range(n_experts):
618
+ ex = ExpertMLP(H, sub, dense.act_fn)
619
+ sl = slice(i * sub, (i + 1) * sub)
620
+ ex.gate_proj.weight.copy_(dense.gate_proj.weight[sl])
621
+ ex.up_proj.weight.copy_(dense.up_proj.weight[sl])
622
+ ex.down_proj.weight.copy_(dense.down_proj.weight[:, sl])
623
+ m.experts.append(ex)
624
+ return m
625
+
626
+ @property
627
+ def device(self):
628
+ return self.router.weight.device
629
+
630
+ # ---------------- 路由 ---------------- #
631
+ def _route(self, x_flat: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
632
+ """fresh 路由: 返回 (p, mask), 形状 (T, N). 零初始化下 p=1/N -> 全激活."""
633
+ logits = self.router(x_flat)
634
+ p = torch.softmax(logits.float(), dim=-1).to(x_flat.dtype)
635
+ mask = p >= (self.tau - 1e-6)
636
+ empty = ~mask.any(-1)
637
+ if bool(empty.any()): # 无一过阈 -> top-1 保底
638
+ top = p.argmax(-1)
639
+ mask = mask.clone()
640
+ mask[empty, top[empty]] = True
641
+ return p, mask
642
+
643
+ def _experts_sparse(self, x_flat, mask, p, with_ste: bool) -> torch.Tensor:
644
+ """稀疏执行: 只计算被激活的 (token, expert) 对.
645
+ 前向: 激活专家求和 (权重 1, 激活参数量语义)
646
+ 反向: STE —— 经 p 软通路把梯度送回 Router (with_ste=True 时)."""
647
+ out = torch.zeros_like(x_flat)
648
+ dtype = x_flat.dtype
649
+ active_ids = [i for i in range(len(self.experts)) if bool(mask[:, i].any())]
650
+ if (not self.training) and self.cache_hook is not None and active_ids:
651
+ self.cache_hook(active_ids) # SSD-LRU: 先搬入后执行 (GPU 上必须前置)
652
+ for i in active_ids:
653
+ ex = self.experts[i]
654
+ sel = mask[:, i].nonzero(as_tuple=True)[0]
655
+ e = ex(x_flat.index_select(0, sel))
656
+ if with_ste:
657
+ w = p.index_select(0, sel).select(1, i) # (S,)
658
+ w = (w - w.detach() + 1.0).to(dtype) # 前向=1, 反向 d/dp=1
659
+ else:
660
+ w = torch.ones((), device=e.device, dtype=dtype)
661
+ out = out.index_add(0, sel, e * w.unsqueeze(-1))
662
+ if (not self.training) and self.cache_post_hook is not None and active_ids:
663
+ self.cache_post_hook(active_ids) # 执行完毕: 超容部分按 LRU 驱逐 (含活跃)
664
+ return out
665
+
666
+ # ---------------- 前向 ---------------- #
667
+ def forward(self, x: torch.Tensor, route_mode: str = "fresh") -> torch.Tensor:
668
+ B, T, H = x.shape
669
+ x_flat = x.reshape(B * T, H)
670
+
671
+ if route_mode == "all": # 调试/等价验证: 全专家权重 1
672
+ if (not self.training) and self.cache_hook is not None:
673
+ self.cache_hook(list(range(len(self.experts))))
674
+ out = torch.zeros_like(x_flat)
675
+ for ex in self.experts:
676
+ out = out + ex(x_flat)
677
+ if (not self.training) and self.cache_post_hook is not None:
678
+ self.cache_post_hook(list(range(len(self.experts))))
679
+ return out.view(B, T, H)
680
+
681
+ if route_mode == "frozen" and self._last_mask is not None:
682
+ # 单次路由铁律: 复用最近一次 fresh 路由 (不触碰 Router, 不产生新路由)
683
+ mask = self._last_mask[:, -T:, :].reshape(B * T, -1)
684
+ out = self._experts_sparse(x_flat, mask, None, with_ste=False)
685
+ return out.view(B, T, H)
686
+
687
+ # ---- fresh 路由 ----
688
+ self.n_route_calls += 1
689
+ p, mask = self._route(x_flat)
690
+ self._last_mask = mask.detach().view(B, T, -1)
691
+ self._last_p = p.detach().view(B, T, -1)
692
+ out = self._experts_sparse(x_flat, mask, p, with_ste=self.training or p.requires_grad)
693
+
694
+ # 负载均衡 (Switch 式): aux = N * Σ f_i * P_i
695
+ with torch.no_grad():
696
+ f = mask.float().mean(0) # (N,) 激活频率
697
+ P = p.float().mean(0) # (N,) 平均概率
698
+ self.last_aux = self.n_experts * (f * P).sum()
699
+ self.last_active = mask.float().sum(-1).mean()
700
+ return out.view(B, T, H)
701
+
702
+
703
+ # --------------------------------------------------------------------------
704
+ # 自定义 DecoderLayer: 与 LlamaDecoderLayer 完全同构, 仅把 mlp 换成 PonderMoEMLP
705
+ # 并支持 moe_route_mode 透传 (fresh=新路由 / frozen=复用 / all=全激活)
706
+ # --------------------------------------------------------------------------
707
+ class PonderMoEDecoderLayer(nn.Module):
708
+ def __init__(self, config, layer_idx: int | None, moe_mlp: PonderMoEMLP):
709
+ super().__init__()
710
+ self.hidden_size = config.hidden_size
711
+ self.layer_idx = layer_idx
712
+ # 注意: 只注册 self.mlp 一个名字 (注册两个会产出重复 state_dict 键)
713
+ self.mlp = moe_mlp
714
+
715
+ @classmethod
716
+ def convert(cls, old_layer: nn.Module, config, n_experts: int, tau: float) -> "PonderMoEDecoderLayer":
717
+ """接管一个现成 LlamaDecoderLayer 的全部子模块, 仅替换 mlp."""
718
+ new = cls(config, getattr(old_layer, "layer_idx", None),
719
+ PonderMoEMLP.from_dense(old_layer.mlp, n_experts, tau))
720
+ new.self_attn = old_layer.self_attn
721
+ new.input_layernorm = old_layer.input_layernorm
722
+ new.post_attention_layernorm = old_layer.post_attention_layernorm
723
+ return new
724
+
725
+ def forward(
726
+ self,
727
+ hidden_states: torch.Tensor,
728
+ attention_mask: torch.Tensor | None = None,
729
+ position_ids: torch.LongTensor | None = None,
730
+ past_key_values=None,
731
+ use_cache: bool = False,
732
+ position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
733
+ moe_route_mode: str = "fresh",
734
+ **kwargs,
735
+ ) -> torch.Tensor:
736
+ residual = hidden_states
737
+ hidden_states = self.input_layernorm(hidden_states)
738
+ hidden_states, _ = self.self_attn(
739
+ hidden_states=hidden_states,
740
+ attention_mask=attention_mask,
741
+ position_ids=position_ids,
742
+ past_key_values=past_key_values,
743
+ use_cache=use_cache,
744
+ position_embeddings=position_embeddings,
745
+ **kwargs,
746
+ )
747
+ hidden_states = residual + hidden_states
748
+
749
+ residual = hidden_states
750
+ hidden_states = self.post_attention_layernorm(hidden_states)
751
+ hidden_states = self.mlp(hidden_states, route_mode=moe_route_mode)
752
+ hidden_states = residual + hidden_states
753
+ return hidden_states
754
+
755
+
756
+ # --------------------------------------------------------------------------
757
+ # MTP 多 Token 预测头 (宪法 §三.4, 训练期增强; 推理自动旁路)
758
+ # 深度 k: 输入 [h_t, e_{t+k}] -> 预测 token t+k+1 (共享主模型 lm_head)
759
+ # --------------------------------------------------------------------------
760
+ class PonderMTPHead(nn.Module):
761
+ def __init__(self, config, depth: int):
762
+ super().__init__()
763
+ H = config.hidden_size
764
+ self.depth = depth # 该头负责预测 t+depth+1
765
+ self.norm_h = LlamaRMSNorm(H, eps=config.rms_norm_eps)
766
+ self.norm_e = LlamaRMSNorm(H, eps=config.rms_norm_eps)
767
+ self.proj = nn.Linear(2 * H, H, bias=False)
768
+ self.act = nn.GELU()
769
+ self.out = nn.Linear(H, H, bias=False)
770
+ self.norm_out = LlamaRMSNorm(H, eps=config.rms_norm_eps)
771
+
772
+ def forward(self, h: torch.Tensor, e: torch.Tensor) -> torch.Tensor:
773
+ """h, e: (B, L', H) 已对齐 —— 位置 t 处为 h_t 与 e_{t+depth}."""
774
+ x = self.proj(torch.cat([self.norm_h(h), self.norm_e(e)], dim=-1))
775
+ return self.norm_out(self.out(self.act(x)))
776
+
777
+
778
+ # --------------------------------------------------------------------------
779
+ # SSD-LRU 专家缓存调度原型 (宪法 §二.4 推理策略层)
780
+ # --------------------------------------------------------------------------
781
+ class SSDLRUExpertManager:
782
+ """专家权重驻留 storage (默认 CPU, 模拟 SSD), 被路由激活时搬入计算设备,
783
+ 容量满按 LRU 驱逐. 统计验证"单次路由 => 每 token 每层恰 1 次 miss,
784
+ Ponder K 轮迭代与 commit 全命中" —— 缓存不击穿.
785
+ 仅推理期可用 (训练期交换 .data 会破坏 autograd, install 时自动排除)."""
786
+
787
+ def __init__(self, model: "PonderMoELlamaForCausalLM", capacity_per_layer: int | None = None,
788
+ storage_device: str = "cpu"):
789
+ self.model = model
790
+ self.storage_device = storage_device
791
+ self.device = next(model.parameters()).device # 捕获主设备 (搬移前)
792
+ self.layers: dict[int, PonderMoEMLP] = {}
793
+ self.order: dict[int, OrderedDict] = {}
794
+ self.capacity: dict[int, int] = {}
795
+ self.stats: dict[int, dict] = {}
796
+ cfg = model.config
797
+ N = int(cfg.moe_num_experts)
798
+ for li, layer in enumerate(model.model.layers):
799
+ if isinstance(layer, PonderMoEDecoderLayer):
800
+ self.layers[li] = layer.mlp
801
+ self.order[li] = OrderedDict()
802
+ self.capacity[li] = int(capacity_per_layer or N)
803
+ self.stats[li] = {"hit": 0, "miss": 0, "evict": 0, "fetch": 0, "calls": 0,
804
+ "miss_calls": 0}
805
+
806
+ @classmethod
807
+ def install(cls, model, capacity_per_layer: int | None = None,
808
+ storage_device: str = "cpu") -> "SSDLRUExpertManager":
809
+ assert not model.training, "SSD-LRU 仅用于推理 (训练期禁止权重搬运)"
810
+ mgr = cls(model, capacity_per_layer, storage_device)
811
+ for li, ml in mgr.layers.items():
812
+ ml.cache_hook = mgr._make_hook(li)
813
+ ml.cache_post_hook = mgr._make_post_hook(li)
814
+ ml.experts.to(storage_device) # 全部专家下放 storage
815
+ return mgr
816
+
817
+ def _make_hook(self, li: int):
818
+ def hook(active_ids: list[int]):
819
+ st, order = self.stats[li], self.order[li]
820
+ st["calls"] += 1 # 访问批次 (含纯命中批)
821
+ had_miss = False
822
+ # 第一遍: 本次活跃专家全部搬入计算设备 (先保证执行正确性)
823
+ for eid in active_ids:
824
+ if eid in order:
825
+ st["hit"] += 1
826
+ order.move_to_end(eid)
827
+ else:
828
+ st["miss"] += 1
829
+ had_miss = True
830
+ order[eid] = True
831
+ self._move(li, eid, self.device)
832
+ st["fetch"] += 1
833
+ # 第二遍: 超容时按 LRU 驱逐, 但绝不驱逐本次活跃专家
834
+ # (工作集 > 容量时允许缓存暂时超容, 执行正确性优先于驱逐压力)
835
+ excess = len(order) - self.capacity[li]
836
+ for k in list(order.keys()):
837
+ if excess <= 0:
838
+ break
839
+ if k in active_ids:
840
+ continue
841
+ order.pop(k)
842
+ self._move(li, k, self.storage_device)
843
+ st["evict"] += 1
844
+ excess -= 1
845
+ if had_miss:
846
+ st["miss_calls"] += 1 # 击穿判据: 实际产生 IO 的批次
847
+ return hook
848
+
849
+ def _make_post_hook(self, li: int):
850
+ def post_hook(active_ids: list[int]):
851
+ st, order = self.stats[li], self.order[li]
852
+ # 物理容量硬上界: 执行完毕后超容部分按 LRU 驱逐
853
+ # (此时专家已用完, 驱逐含本次活跃专家也安全; 欠配 => 下批重新 miss = 真实 thrash)
854
+ while len(order) > self.capacity[li]:
855
+ old, _ = order.popitem(last=False)
856
+ self._move(li, old, self.storage_device)
857
+ st["evict"] += 1
858
+ return post_hook
859
+
860
+ def _move(self, li: int, eid: int, device: str):
861
+ ex = self.layers[li].experts[eid]
862
+ for p in ex.parameters():
863
+ p.data = p.data.to(device)
864
+
865
+ def summary(self) -> dict:
866
+ hit = sum(s["hit"] for s in self.stats.values())
867
+ miss = sum(s["miss"] for s in self.stats.values())
868
+ return {
869
+ "per_layer": {li: dict(s) for li, s in self.stats.items()},
870
+ "hit": hit, "miss": miss,
871
+ "evict": sum(s["evict"] for s in self.stats.values()),
872
+ "fetch": sum(s["fetch"] for s in self.stats.values()),
873
+ "calls": sum(s["calls"] for s in self.stats.values()),
874
+ "miss_calls": sum(s["miss_calls"] for s in self.stats.values()),
875
+ "hit_rate": hit / max(1, hit + miss),
876
+ "capacity_per_layer": dict(self.capacity),
877
+ }
878
+
879
+ def remove(self):
880
+ """卸载挂钩并把专家搬回主设备 (恢复常驻)."""
881
+ for li, ml in self.layers.items():
882
+ ml.cache_hook = None
883
+ ml.cache_post_hook = None
884
+ ml.experts.to(self.device)
885
+
886
+
887
+ # --------------------------------------------------------------------------
888
+ # Model: Ponder (质量) + 动态 MoE (效率) + MTP (吞吐, 训练期)
889
+ # --------------------------------------------------------------------------
890
+ class PonderMoELlamaForCausalLM(PonderLlamaForCausalLM):
891
+ """
892
+ 用法 — 从稠密基座一行构造 (FFN 自动拆专家):
893
+ base = LlamaConfig.from_pretrained("tchbcb/MiniCPM5-2B-cpu")
894
+ cfg = PonderMoELlamaConfig.from_llama(
895
+ base, moe_num_experts=8, mtp_num_tokens=2, ponder_signal="learned")
896
+ model = PonderMoELlamaForCausalLM.from_pretrained(..., config=cfg)
897
+
898
+ moe_num_experts=0 且 mtp_num_tokens=0 时与 PonderLlamaForCausalLM 行为一致。
899
+ """
900
+
901
+ def __init__(self, config: PonderMoELlamaConfig):
902
+ super().__init__(config)
903
+ cfg = self.config
904
+ self._moe_ready = False
905
+ if int(getattr(cfg, "moe_num_experts", 0) or 0) > 0:
906
+ self._convert_ffn_to_moe() # 在 post_init 之后: Router 零初始化不被覆盖
907
+ D = int(getattr(cfg, "mtp_num_tokens", 0) or 0)
908
+ self.mtp_heads = (nn.ModuleList([PonderMTPHead(cfg, k + 1) for k in range(D)])
909
+ if D > 0 else None)
910
+ self._moe_ready = True
911
+
912
+ # ------------------------------------------------------------------ #
913
+ # 结构改造
914
+ # ------------------------------------------------------------------ #
915
+ def _convert_ffn_to_moe(self):
916
+ cfg = self.config
917
+ N = int(cfg.moe_num_experts)
918
+ assert cfg.intermediate_size % N == 0, \
919
+ f"intermediate_size({cfg.intermediate_size}) 须被 moe_num_experts({N}) 整除"
920
+ tau = cfg.moe_tau
921
+ s = cfg.ponder_start_layer
922
+ ids = (range(s, cfg.num_hidden_layers) if cfg.moe_scope == "ponder"
923
+ else range(cfg.num_hidden_layers))
924
+ for li in ids:
925
+ self.model.layers[li] = PonderMoEDecoderLayer.convert(
926
+ self.model.layers[li], cfg, N, tau)
927
+
928
+ def _moe_mlps(self) -> list[PonderMoEMLP]:
929
+ return ([l.mlp for l in self.model.layers if isinstance(l, PonderMoEDecoderLayer)]
930
+ if self._moe_ready else [])
931
+
932
+ def _reset_moe_stats(self):
933
+ for ml in self._moe_mlps():
934
+ ml.last_aux = None
935
+ ml.last_active = None
936
+
937
+ def _collect_moe_stats(self):
938
+ auxs = [ml.last_aux for ml in self._moe_mlps() if ml.last_aux is not None]
939
+ acts = [ml.last_active for ml in self._moe_mlps() if ml.last_active is not None]
940
+ aux = torch.stack(auxs).mean() if auxs else None
941
+ act = torch.stack(acts).mean() if acts else None
942
+ return aux, act
943
+
944
+ # ------------------------------------------------------------------ #
945
+ # MTP (训练/评估且给定 labels 时; generate 自动旁路)
946
+ # ------------------------------------------------------------------ #
947
+ def _mtp_forward(self, h_final, input_ids, labels):
948
+ m = self.model
949
+ B, L, _ = h_final.shape
950
+ emb = m.embed_tokens(input_ids)
951
+ losses, accs = [], []
952
+ for head in self.mtp_heads:
953
+ k = head.depth
954
+ if L - k < 2:
955
+ continue
956
+ h = h_final[:, :L - k, :] # h_t, t=0..L-k-1
957
+ e = emb[:, k:, :] # e_{t+k}
958
+ hh = head(h, e) # (B, L-k, H)
959
+ lg = self.lm_head(hh)[:, :-1, :] # 预测 t+k+1
960
+ tg = labels[:, k + 1:]
961
+ ce = F.cross_entropy(lg.reshape(-1, lg.size(-1)).float(),
962
+ tg.reshape(-1), ignore_index=-100)
963
+ if torch.isfinite(ce):
964
+ losses.append(ce)
965
+ with torch.no_grad():
966
+ valid = tg != -100
967
+ if bool(valid.any()):
968
+ accs.append((((lg.argmax(-1) == tg) & valid).sum().float()
969
+ / valid.sum()).detach())
970
+ mtp_loss = torch.stack(losses).mean() if losses else None
971
+ mtp_acc = torch.stack(accs).mean() if accs else None
972
+ return mtp_loss, mtp_acc
973
+
974
+ # ------------------------------------------------------------------ #
975
+ # 思考块执行 (带路由模式: 第 1 轮 fresh, 之后 frozen —— 单次路由铁律)
976
+ # ------------------------------------------------------------------ #
977
+ def _run_block(self, h, attention_mask, position_ids, position_embeddings,
978
+ cache, kwargs, route_mode: str = "fresh"):
979
+ ckpt_fn = getattr(self.model, "_gradient_checkpointing_func", None)
980
+ use_ckpt = (self.training and cache is None
981
+ and getattr(self.model, "gradient_checkpointing", False)
982
+ and ckpt_fn is not None)
983
+ for layer in self._ponder_layers:
984
+ lkw = dict(kwargs)
985
+ if isinstance(layer, PonderMoEDecoderLayer):
986
+ lkw["moe_route_mode"] = route_mode
987
+ if use_ckpt:
988
+ h = ckpt_fn(
989
+ layer.__call__, h,
990
+ attention_mask=attention_mask,
991
+ position_ids=position_ids,
992
+ position_embeddings=position_embeddings,
993
+ past_key_values=None,
994
+ use_cache=False,
995
+ **lkw,
996
+ )
997
+ else:
998
+ h = layer(
999
+ h,
1000
+ attention_mask=attention_mask,
1001
+ position_ids=position_ids,
1002
+ position_embeddings=position_embeddings,
1003
+ past_key_values=cache,
1004
+ use_cache=cache is not None,
1005
+ **lkw,
1006
+ )
1007
+ return h
1008
+
1009
+ # ------------------------------------------------------------------ #
1010
+ # 主前向 (父类结构 + 路由模式 + MoE/MTP 损失)
1011
+ # ------------------------------------------------------------------ #
1012
+ @can_return_tuple
1013
+ def forward(
1014
+ self,
1015
+ input_ids: torch.LongTensor | None = None,
1016
+ attention_mask: torch.Tensor | None = None,
1017
+ position_ids: torch.LongTensor | None = None,
1018
+ past_key_values: DynamicCache | None = None,
1019
+ inputs_embeds: torch.FloatTensor | None = None,
1020
+ labels: torch.LongTensor | None = None,
1021
+ use_cache: bool | None = None,
1022
+ logits_to_keep: int | torch.Tensor = 0,
1023
+ output_ponder: bool = False,
1024
+ **kwargs,
1025
+ ) -> PonderMoECausalLMOutputWithPast:
1026
+ cfg = self.config
1027
+ m = self.model
1028
+ self._reset_moe_stats()
1029
+
1030
+ # ---- 输入 & 位置 ------------------------------------------------
1031
+ if (input_ids is None) ^ (inputs_embeds is not None):
1032
+ raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
1033
+ if inputs_embeds is None:
1034
+ inputs_embeds = m.embed_tokens(input_ids)
1035
+ B, Q = inputs_embeds.shape[:2]
1036
+
1037
+ if use_cache is None:
1038
+ use_cache = False if self.training else cfg.use_cache
1039
+ if use_cache and past_key_values is None:
1040
+ past_key_values = DynamicCache(config=cfg)
1041
+
1042
+ cache_position = kwargs.pop("cache_position", None)
1043
+ past_seen = past_key_values.get_seq_length() if past_key_values is not None else 0
1044
+ if position_ids is None:
1045
+ base = (cache_position if cache_position is not None
1046
+ else torch.arange(Q, device=inputs_embeds.device) + past_seen)
1047
+ position_ids = base.reshape(1, -1).expand(B, -1)[:, :Q].to(inputs_embeds.device)
1048
+
1049
+ # ---- mask / rope -------------------------------------------------
1050
+ causal_mask = create_causal_mask(
1051
+ config=cfg, inputs_embeds=inputs_embeds, attention_mask=attention_mask,
1052
+ past_key_values=past_key_values, position_ids=position_ids,
1053
+ )
1054
+ position_embeddings = m.rotary_emb(inputs_embeds, position_ids=position_ids)
1055
+
1056
+ # ---- Stage A: 前段层单遍 (scope=all 时 MoE 层每次调用都是新 token -> fresh) --
1057
+ h = inputs_embeds
1058
+ for layer in m.layers[: cfg.ponder_start_layer]:
1059
+ if isinstance(layer, PonderMoEDecoderLayer):
1060
+ h = layer(h, attention_mask=causal_mask, position_ids=position_ids,
1061
+ position_embeddings=position_embeddings,
1062
+ past_key_values=past_key_values, use_cache=use_cache,
1063
+ moe_route_mode="fresh", **kwargs)
1064
+ else:
1065
+ h = layer(h, attention_mask=causal_mask, position_ids=position_ids,
1066
+ position_embeddings=position_embeddings,
1067
+ past_key_values=past_key_values, use_cache=use_cache, **kwargs)
1068
+
1069
+ # ---- 思考窗口 W --------------------------------------------------
1070
+ if Q == 1 or self.training or cfg.ponder_all_positions:
1071
+ W = Q
1072
+ else:
1073
+ W = max(1, min(cfg.ponder_window, Q))
1074
+
1075
+ eps = cfg.ponder_epsilon
1076
+ K = cfg.max_ponder_steps
1077
+ device, dtype = inputs_embeds.device, inputs_embeds.dtype
1078
+
1079
+ remaining = torch.ones(B, W, device=device, dtype=dtype)
1080
+ cum = torch.zeros(B, W, device=device, dtype=dtype)
1081
+ active = torch.ones(B, W, device=device, dtype=torch.bool)
1082
+ mix = torch.zeros(B, W, h.size(-1), device=device, dtype=dtype)
1083
+ w_list: list[torch.Tensor] = []
1084
+ h_prev_win = None
1085
+ h_full = None
1086
+ h_win = h[:, -W:, :] if W < Q else h
1087
+ executed = 0
1088
+ win_mask_built = False
1089
+
1090
+ forced = int(getattr(cfg, "ponder_forced_steps", 0) or 0)
1091
+ step_ce_weight = float(getattr(cfg, "ponder_step_ce_weight", 0.0) or 0.0)
1092
+ use_step_ce = (step_ce_weight > 0.0 and labels is not None and self.training)
1093
+ step_pairs: list[tuple[torch.Tensor, torch.Tensor]] = []
1094
+ loop_K = forced if forced > 0 else K
1095
+
1096
+ for k in range(1, loop_K + 1):
1097
+ executed = k
1098
+ if k > 1:
1099
+ self._crop_ponder_kv(past_key_values, W)
1100
+ use_window = (k > 1) and (past_key_values is not None) and (W < Q)
1101
+ route_mode = "fresh" if k == 1 else "frozen" # 单次路由铁律
1102
+ if not use_window:
1103
+ h = self._run_block(h, causal_mask, position_ids,
1104
+ position_embeddings, past_key_values, kwargs,
1105
+ route_mode=route_mode)
1106
+ h_win = h if W == Q else h[:, -W:, :]
1107
+ if W < Q:
1108
+ h_full = h
1109
+ else:
1110
+ if not win_mask_built:
1111
+ past_len = int(past_key_values.layers[cfg.ponder_start_layer].get_seq_length())
1112
+ mask_win = self._win_mask(W, past_len, device, dtype, attention_mask)
1113
+ pos_emb_win = (position_embeddings[0][:, -W:],
1114
+ position_embeddings[1][:, -W:])
1115
+ win_mask_built = True
1116
+ h_win = self._run_block(h_win, mask_win, position_ids[:, -W:],
1117
+ pos_emb_win, past_key_values, kwargs,
1118
+ route_mode=route_mode)
1119
+
1120
+ if forced > 0:
1121
+ continue
1122
+
1123
+ lam = self._halting_prob(h_win, h_prev_win)
1124
+ lam_eff = lam * active.to(lam.dtype)
1125
+ w_k = remaining * lam_eff
1126
+ mix = mix + w_k.unsqueeze(-1) * h_win
1127
+ w_list.append(w_k)
1128
+ remaining = remaining * (1.0 - lam_eff)
1129
+ cum = cum + w_k
1130
+ active = active & (cum < 1.0 - eps)
1131
+ h_prev_win = h_win
1132
+ if use_step_ce:
1133
+ ce_k = self._step_ce(h_win, labels)
1134
+ step_pairs.append((w_k.detach(), ce_k))
1135
+ if not bool(active.any()):
1136
+ break
1137
+
1138
+ if forced > 0:
1139
+ if W < Q:
1140
+ h_final = h_full.clone()
1141
+ h_final[:, -W:, :] = h_win
1142
+ else:
1143
+ h_final = h_win
1144
+ w_stack = F.one_hot(
1145
+ torch.full((B, W), forced - 1, device=device), forced
1146
+ ).to(dtype)
1147
+ else:
1148
+ mix = mix + remaining.unsqueeze(-1) * h_win
1149
+ w_stack = torch.stack(w_list, dim=-1).to(dtype)
1150
+ w_stack = torch.cat([w_stack[..., :-1],
1151
+ w_stack[..., -1:] + remaining.unsqueeze(-1)], dim=-1)
1152
+ if W < Q:
1153
+ h_final = h_full
1154
+ h_final[:, -W:, :] = mix
1155
+ else:
1156
+ h_final = mix
1157
+
1158
+ # ---- commit pass: 复用 frozen 路由写最终 KV -----------------------
1159
+ if (use_cache and cfg.commit_kv and past_key_values is not None
1160
+ and executed > 1 and forced == 0):
1161
+ self._crop_ponder_kv(past_key_values, W)
1162
+ if not win_mask_built:
1163
+ self._run_block(h_final, causal_mask, position_ids,
1164
+ position_embeddings, past_key_values, kwargs,
1165
+ route_mode="frozen")
1166
+ else:
1167
+ self._run_block(h_final[:, -W:, :], mask_win, position_ids[:, -W:],
1168
+ pos_emb_win, past_key_values, kwargs,
1169
+ route_mode="frozen")
1170
+
1171
+ # ---- logits / loss ----------------------------------------------
1172
+ slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
1173
+ logits = self.lm_head(m.norm(h_final[:, slice_indices, :]))
1174
+
1175
+ loss, ponder_kl, ponder_step_ce = None, None, None
1176
+ if labels is not None:
1177
+ ce_logits = logits.float()
1178
+ loss = F.cross_entropy(
1179
+ ce_logits[..., :-1, :].reshape(-1, ce_logits.size(-1)),
1180
+ labels[..., 1:].reshape(-1), ignore_index=-100)
1181
+
1182
+ # PonderNet KL 正则 (与父类一致, fp32 域防下溢)
1183
+ p = (w_stack / w_stack.sum(-1, keepdim=True).clamp_min(1e-8)).float()
1184
+ Kd = p.size(-1)
1185
+ n_idx = torch.arange(1, Kd + 1, device=p.device, dtype=p.dtype)
1186
+ pg = cfg.ponder_prior_p
1187
+ geom = (1 - pg) ** (n_idx - 1) * pg
1188
+ geom = torch.cat([geom[..., :-1],
1189
+ ((1 - pg) ** (Kd - 1) * torch.ones_like(geom[..., -1:]))],
1190
+ dim=-1)
1191
+ geom = geom / geom.sum()
1192
+ kl = (p * (torch.log(p.clamp_min(1e-8))
1193
+ - torch.log(geom.expand_as(p)))).sum(-1)
1194
+ ponder_kl = kl.mean()
1195
+ loss = loss + cfg.ponder_loss_beta * ponder_kl
1196
+
1197
+ if use_step_ce and step_pairs:
1198
+ ws = torch.stack([pp[0] for pp in step_pairs], dim=0)
1199
+ ces = torch.stack([pp[1] for pp in step_pairs], dim=0)
1200
+ wsum = ws.sum(0).clamp_min(1e-8)
1201
+ per_pos = (ws * ces.view(-1, 1, 1)).sum(0) / wsum
1202
+ ponder_step_ce = per_pos.mean()
1203
+ loss = loss + step_ce_weight * ponder_step_ce
1204
+
1205
+ # MoE 负载均衡 (宪法 §六). checkpointing 下 last_aux 无梯度 -> 跳过加项仍报告
1206
+ moe_aux, moe_active = self._collect_moe_stats()
1207
+ if (moe_aux is not None and cfg.moe_aux_weight > 0
1208
+ and torch.is_tensor(moe_aux) and moe_aux.requires_grad):
1209
+ loss = loss + cfg.moe_aux_weight * moe_aux
1210
+ else:
1211
+ moe_aux, moe_active = self._collect_moe_stats()
1212
+
1213
+ # MTP 多 Token 预测损失 (宪法 §六; 训练增强, 推理旁路)
1214
+ mtp_loss = mtp_acc = None
1215
+ if (labels is not None and self.mtp_heads is not None
1216
+ and input_ids is not None):
1217
+ mtp_loss, mtp_acc = self._mtp_forward(h_final, input_ids, labels)
1218
+ if mtp_loss is not None and cfg.mtp_loss_weight > 0:
1219
+ loss = loss + cfg.mtp_loss_weight * mtp_loss
1220
+
1221
+ # ---- 诊断 --------------------------------------------------------
1222
+ cum_w = torch.cumsum(w_stack, dim=-1)
1223
+ steps_used = ((cum_w < 1.0 - eps).sum(-1) + 1).clamp(max=w_stack.size(-1))
1224
+ self._log({
1225
+ "mode": "train" if self.training else ("decode" if Q == 1 else "prefill"),
1226
+ "signal": cfg.ponder_signal,
1227
+ "executed": executed,
1228
+ "steps_max": int(steps_used.max()),
1229
+ "steps_mean": float(steps_used.float().mean()),
1230
+ "moe_active": (float(moe_active) if torch.is_tensor(moe_active) else None),
1231
+ })
1232
+
1233
+ return PonderMoECausalLMOutputWithPast(
1234
+ loss=loss,
1235
+ logits=logits,
1236
+ past_key_values=past_key_values,
1237
+ ponder_weights=w_stack if output_ponder else None,
1238
+ ponder_steps=steps_used if output_ponder else None,
1239
+ ponder_kl=ponder_kl,
1240
+ ponder_step_ce=ponder_step_ce,
1241
+ moe_aux_loss=moe_aux,
1242
+ moe_active_mean=moe_active,
1243
+ mtp_loss=mtp_loss,
1244
+ mtp_acc=mtp_acc,
1245
+ )
1246
+
1247
+ # --------------------------------------------------------------------------
1248
+ # 发布版统一注册与别名 (auto_map 指向别名; 原类名保留以兼容旧代码)
1249
+ # --------------------------------------------------------------------------
1250
+ SamaiPnetConfig = PonderMoELlamaConfig
1251
+ SamaiPnetForCausalLM = PonderMoELlamaForCausalLM
1252
+
1253
+ # transformers 5.x 严格校验: AutoModelForCausalLM.register 要求
1254
+ # model_class.config_class 与实际 config 类一致 (否则 ValueError)
1255
+ PonderLlamaForCausalLM.config_class = PonderLlamaConfig
1256
+ PonderMoELlamaForCausalLM.config_class = PonderMoELlamaConfig
1257
+
1258
+ try: # 重复 import 安全
1259
+ AutoConfig.register("llama_ponder", PonderLlamaConfig)
1260
+ AutoConfig.register("llama_ponder_moe", PonderMoELlamaConfig)
1261
+ AutoModelForCausalLM.register(PonderLlamaConfig, PonderLlamaForCausalLM)
1262
+ AutoModelForCausalLM.register(PonderMoELlamaConfig, PonderMoELlamaForCausalLM)
1263
+ except Exception:
1264
+ pass
serve_contract.json ADDED
@@ -0,0 +1,105 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "contract_version": "r5b-2-integrated",
3
+ "declared_at": "2026-09-09",
4
+ "supersedes": "r5b-1",
5
+ "declared_repo_id": "tchbcn/samai-pnet-dmoe-2b",
6
+ "actual_repo_id": "tchbcb/samai-pnet-dmoe-2b",
7
+ "note": "用户指定仓名 tchbcn/samai-pnet-dmoe-2b; 所持 token 实际属主为 tchbcb (whoami 实证, tchbcn 命名空间 403 无权创建), 故以 tchbcb/ 落库, 仓内容与命名意图一致",
8
+ "release": {
9
+ "model_file": "model.safetensors (全量合并: 基座+LoRA+halting头+Router/MTP)",
10
+ "dtype_policy": "主干/专家 fp16; ponder_head fp32 (round4 教训)",
11
+ "trust_remote_code": true,
12
+ "requirements": "torch>=2.4, transformers>=5.0, peft 仅训练侧需要"
13
+ },
14
+ "lineage": {
15
+ "base": "tchbcb/MiniCPM5-2B-cpu (42L/H2048/I6144/GQA16+2/V130560)",
16
+ "convert": "pnet2b_moe: 思考块[34,42) 8层x8同构专家, 参数守恒",
17
+ "round7": "ckpt_2b (宪法四损失基线)",
18
+ "round5B": "ckpt_2b_r5b: per-step CE 0.5 + difficulty prior, init-from ckpt_2b"
19
+ },
20
+ "frozen_structure": {
21
+ "ponder_start_layer": 34,
22
+ "max_ponder_steps": 8,
23
+ "ponder_window": 1,
24
+ "moe_num_experts": 8,
25
+ "moe_scope": "ponder",
26
+ "mtp_num_tokens": 2
27
+ },
28
+ "inference_semantics": {
29
+ "halting": "learned head (fp32, sigmoid), bias 已训练",
30
+ "stop_rule": "cum(w) >= 1 - eps 即停, 残余质量记给最后执行步",
31
+ "epsilon": 0.01,
32
+ "tau": 0.125,
33
+ "route_rule": "每 token 仅第 1 轮迭代 fresh 路由一次, 循环/commit 全程 frozen 复用",
34
+ "moe_activation": "p_i >= tau 激活, 激活数 1~8 自适应, 无一过阈 top-1 保底",
35
+ "kv": "commit_kv=true: 混合表示经 commit pass 写最终 KV",
36
+ "mtp": "训练期增强, 推理 generate() 自动旁路"
37
+ },
38
+ "eval_anchors_90rows_zh_cot": {
39
+ "round7_baseline": {
40
+ "hard": {
41
+ "ce": 4.073070367177327,
42
+ "active": 1.6183046340942382,
43
+ "steps": 2.066666666666667,
44
+ "mtp": 0.5716088751951853,
45
+ "n": 15
46
+ },
47
+ "easy": {
48
+ "ce": 4.018902349472046,
49
+ "active": 1.6024614969889324,
50
+ "steps": 2.0,
51
+ "mtp": 0.5493664860725402,
52
+ "n": 15
53
+ },
54
+ "medium": {
55
+ "ce": 4.228601948420207,
56
+ "active": 1.614480972290039,
57
+ "steps": 2.033333333333333,
58
+ "mtp": 0.5261600712935129,
59
+ "n": 15
60
+ }
61
+ },
62
+ "round5B_live_reference": {
63
+ "hard": {
64
+ "ce": 3.3185231844584147,
65
+ "active": 1.5587466398874918,
66
+ "steps": 2.8666666666666667,
67
+ "mtp": 0.6124512155850729,
68
+ "n": 15
69
+ },
70
+ "easy": {
71
+ "ce": 3.264301141103109,
72
+ "active": 1.5368041038513183,
73
+ "steps": 2.433333333333333,
74
+ "mtp": 0.6108810146649678,
75
+ "n": 15
76
+ },
77
+ "medium": {
78
+ "ce": 3.5550480524698895,
79
+ "active": 1.5593219121297202,
80
+ "steps": 2.6,
81
+ "mtp": 0.5710210422674815,
82
+ "n": 15
83
+ }
84
+ },
85
+ "round5B_merged_release": {
86
+ "hard": {
87
+ "ce": 3.3181373119354247,
88
+ "steps": 2.8666666666666667,
89
+ "n": 15
90
+ },
91
+ "easy": {
92
+ "ce": 3.2644861777623495,
93
+ "steps": 2.433333333333333,
94
+ "n": 15
95
+ },
96
+ "medium": {
97
+ "ce": 3.555168342590332,
98
+ "steps": 2.6,
99
+ "n": 15
100
+ }
101
+ },
102
+ "eval_protocol": "eval_ponder_moe.py 口径: use_cache=False, batch_size=2, fp16, 90 条 zh_cot",
103
+ "merge_note": "干净进程终验 (verify_release.json, trust_remote_code): ΔCE≤0.001, steps 三桶与 live 参考一致 (2.433/2.600/2.867), 分化 +0.433 完整保留; 同进程对分探针曾见 ≤0.065 CE 交叉污染, 已排除"
104
+ }
105
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": null,
3
+ "backend": "tokenizers",
4
+ "bos_token": "<s>",
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "</s>",
7
+ "is_local": true,
8
+ "legacy": true,
9
+ "local_files_only": false,
10
+ "model_max_length": 131072,
11
+ "pad_token": "</s>",
12
+ "sp_model_kwargs": {},
13
+ "spaces_between_special_tokens": false,
14
+ "tokenizer_class": "TokenizersBackend",
15
+ "unk_token": "<unk>",
16
+ "use_default_system_prompt": false
17
+ }