samai-2b r14 scaffold: code/config/card (weights pending)
Browse files- INTEGRATION_DECLARATION.md +36 -0
- README.md +83 -0
- chat_template.jinja +177 -0
- config.json +59 -0
- generation_config.json +18 -0
- inference.py +63 -0
- modeling_samai_pnet.py +1264 -0
- serve_contract.json +105 -0
- tokenizer.json +0 -0
- tokenizer_config.json +17 -0
INTEGRATION_DECLARATION.md
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# r5B 可集成终版宣言 (Integration Declaration r5b-2)
|
| 2 |
+
|
| 3 |
+
宣布 **round5-B (ckpt_2b_r5b)** 为 Ponder-动态MoE 2B 线的**可集成终版**, 自本宣言起冻结。
|
| 4 |
+
|
| 5 |
+
## 判据 (90 条 zh_cot, 口径: use_cache=False / bs=2 / fp16)
|
| 6 |
+
|
| 7 |
+
| bucket | CE (live 参考) | CE (合并发布包) | Δ(合并舍入) | steps (live) | steps (发布包) |
|
| 8 |
+
|---|---|---|---|---|---|
|
| 9 |
+
| easy | 3.2643 | 3.2645 | +0.0002 | 2.43 | 2.43 |
|
| 10 |
+
| medium | 3.5550 | 3.5552 | +0.0001 | 2.60 | 2.60 |
|
| 11 |
+
| hard | 3.3185 | 3.3181 | -0.0004 | 2.87 | 2.87 |
|
| 12 |
+
|
| 13 |
+
- hard − easy steps 分化 **+0.067 → +0.433 (×6.5)**, 三桶严格单调 (对分探针 A 配置与
|
| 14 |
+
eval_2b_r5b.json 小数点后四位复现); 合并发布包分化 +0.33, 单调性保持
|
| 15 |
+
- CE 全桶回落 −0.70 ~ −0.93 (相对 round7 基线 4.02-4.23); 合并舍入代价 ≤0.065 CE
|
| 16 |
+
- per-step CE (方向②, weight=0.5) 一轮命中判据, hard 桶"多想"倾向成立
|
| 17 |
+
- 合并完整性: LoRA 并入后 L36 q_proj 位移 0.0131; head(fp32) bias −3.9785;
|
| 18 |
+
Router std 0.0202; 对分探针 D 配置确认 LoRA 贡献 CE −1.3 ~ −1.7
|
| 19 |
+
|
| 20 |
+
## 冻结项
|
| 21 |
+
|
| 22 |
+
1. 结构三参数: start=34 / K=8 / W=1; MoE: 8 同构专家 τ=0.125; MTP D=2 (推理旁路)
|
| 23 |
+
2. 推理语义: 见 serve_contract.json (r5b-2); **评估/推理口径冻结为 use_cache=False**
|
| 24 |
+
(KV-cache 路径的 ponder 窗口重算与 commit pass 语义与参考评估不一致, 禁用于对拍)
|
| 25 |
+
3. 权重: 发布包 model.safetensors = 基座 + r5B(LoRA 合并 + halting 头 + Router/MTP) 全量固化
|
| 26 |
+
|
| 27 |
+
## 集成产物
|
| 28 |
+
|
| 29 |
+
- 仓: **tchbcb/samai-pnet-dmoe-2b** (用户指定名 tchbcn/ 因 token 属主为 tchbcb 落于
|
| 30 |
+
tchbcb/, 用户已确认; 仓内容与命名意图一致)
|
| 31 |
+
- 形态: 单 safetensors (5.09GB) + 自洽 modeling_samai_pnet.py, `trust_remote_code=True` 拉取即跑
|
| 32 |
+
|
| 33 |
+
## 后续
|
| 34 |
+
|
| 35 |
+
- Terminal-Bench 2.0 式协议测评 (round7 基线 / 原始基座 / r5B 终版 三方对比)
|
| 36 |
+
- 方向② 权重敏感性扫描 (0.5 未扫) 与方向① MLP-256 halt 头叠加留档待排
|
README.md
ADDED
|
@@ -0,0 +1,83 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# samai-2b (r14)
|
| 2 |
+
|
| 3 |
+
**我是 samai-2b 模型,由 SamAI 研发,官网是 [samai.cc](https://samai.cc)。**
|
| 4 |
+
|
| 5 |
+
这是 SamAI 官方 2B 对话模型仓:基于 [tchbcb/samai-pnet-dmoe-2b](https://huggingface.co/tchbcb/samai-pnet-dmoe-2b)(r12)
|
| 6 |
+
继续链式微调的 **r14 身份版**。r14 在 r13(agent 精修)之上,通过身份数据微调固化自我认知——
|
| 7 |
+
问它名字、研发方、官网,它都会稳定给出上述答案,不再自称其他模型。
|
| 8 |
+
|
| 9 |
+
> ⏳ **发布状态**:模型卡与代码/分词器脚手架已就位;`model.safetensors`(~5.08GB 单文件)
|
| 10 |
+
> 将在 r14 训练完成并通过身份验收(20 题身份电池 ≥90% + 三身份探针双协议全过)后上传。
|
| 11 |
+
|
| 12 |
+
## 身份(r14 固化)
|
| 13 |
+
|
| 14 |
+
| 问 | 答 |
|
| 15 |
+
|---|---|
|
| 16 |
+
| 你叫什么名字? / What's your name? | 我是 samai-2b 模型 |
|
| 17 |
+
| 你是谁开发的? / Who developed you? | SamAI |
|
| 18 |
+
| 你的官网是什么? / What's your website? | samai.cc |
|
| 19 |
+
| 你是 GPT/MiniCPM/Claude/通义/DeepSeek 吗? | 不是,我是 samai-2b 模型,由 SamAI 研发 |
|
| 20 |
+
|
| 21 |
+
## 模型规格
|
| 22 |
+
|
| 23 |
+
| 项 | 值 |
|
| 24 |
+
|---|---|
|
| 25 |
+
| 基座 | MiniCPM5-2B(42L / hidden 2048 / GQA 16+2 / vocab 130560) |
|
| 26 |
+
| 架构 | Ponder 自适应思考(N=8, τ=0.125)× 动态可变 MoE(L34–41 × 8 专家)× MTP 深度 2 |
|
| 27 |
+
| 权重 | bf16 单文件 ~5.08GB;`trust_remote_code=True` 加载 |
|
| 28 |
+
| EOS | `[1, 130073]`(`<|im_end|>`) |
|
| 29 |
+
|
| 30 |
+
## 训练谱系
|
| 31 |
+
|
| 32 |
+
```
|
| 33 |
+
MiniCPM5-2B 基座 → PonderMoE 转换 → r5b → r11 → r12(= tchbcb/samai-pnet-dmoe-2b,正式发布)
|
| 34 |
+
→ r13:agent 精修(DONE 纪律 / shell 语法 / 算术 think 分步,mix_r13 5163 行 ×2ep)
|
| 35 |
+
→ r14:身份微调(本仓,mix_r14 5207 行 ×2ep = r13 全量回放 + 身份手术改写 + 44 条专属身份行)
|
| 36 |
+
```
|
| 37 |
+
|
| 38 |
+
r14 数据要点:基座 self 类行中所有"叫什么名字/你是谁/你是什么模型"问句统一改写为规范身份答案;
|
| 39 |
+
新增名字/开发者/官网/否定澄清/中英文/综合介绍共 44 条身份行;核心名字问句 ×2 强化。
|
| 40 |
+
CE 锚(zh_cot 1049 行)与 agent 修复数据全量回放,防灾难遗忘。
|
| 41 |
+
|
| 42 |
+
## 推理协议(与 r12 一致,重要)
|
| 43 |
+
|
| 44 |
+
遵循 MiniCPM5 官方姿势,两件事必须做对:
|
| 45 |
+
|
| 46 |
+
1. **采样**:`do_sample=True, temperature=1.0, top_p=0.95, repetition_penalty=1.0`(greedy 不在官方设计姿势内,会显著加剧复读)。generation_config 已按此固化。
|
| 47 |
+
2. **思考协议**:
|
| 48 |
+
- 标准模式:正常 apply_chat_template,模型自行输出 `<think>...</think>` 后给答案;
|
| 49 |
+
- think_off(轻聊场景推荐):在 assistant 首部预填空思考段,闲聊停机率更高、响应更快。
|
| 50 |
+
|
| 51 |
+
```python
|
| 52 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 53 |
+
|
| 54 |
+
tok = AutoTokenizer.from_pretrained("tchbcb/samai-2b", trust_remote_code=True)
|
| 55 |
+
model = AutoModelForCausalLM.from_pretrained(
|
| 56 |
+
"tchbcb/samai-2b", trust_remote_code=True,
|
| 57 |
+
torch_dtype="bfloat16", device_map="auto")
|
| 58 |
+
|
| 59 |
+
msgs = [{"role": "user", "content": "你叫什么名字?"}]
|
| 60 |
+
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
|
| 61 |
+
out = model.generate(ids, max_new_tokens=256, do_sample=True,
|
| 62 |
+
temperature=1.0, top_p=0.95, repetition_penalty=1.0)
|
| 63 |
+
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
|
| 64 |
+
# → 我是 samai-2b 模型,由 SamAI 研发,官网是 samai.cc。
|
| 65 |
+
```
|
| 66 |
+
|
| 67 |
+
## 验收门(回填中)
|
| 68 |
+
|
| 69 |
+
| 指标 | r12(已发布基线) | r14(本仓) |
|
| 70 |
+
|---|---|---|
|
| 71 |
+
| 身份电池 20 题双协议 | — | 训练完成后回填 |
|
| 72 |
+
| 你好停机 (top_f) | 0.20 ✓ | 回填 |
|
| 73 |
+
| chat50 首答即停 | 90% / 86% | 回填 |
|
| 74 |
+
| 冻结 CE Δ vs r5b 锚 | -0.43 / -0.18 / -0.36 | 回填(容差 ±0.15) |
|
| 75 |
+
| TB2 24 题 | 10/24 | 回填 |
|
| 76 |
+
|
| 77 |
+
## 文件
|
| 78 |
+
|
| 79 |
+
- `model.safetensors` — 合并后 bf16 权重(r14 验收通过后上传)
|
| 80 |
+
- `modeling_samai_pnet.py` / `config.json` — 远程代码(auto_map: SamaiPnetConfig / SamaiPnetForCausalLM)
|
| 81 |
+
- `chat_template.jinja` / `tokenizer.json` / `tokenizer_config.json` — 与 MiniCPM5 官方模板逐字节一致
|
| 82 |
+
- `generation_config.json` — 已固化官方采样参数
|
| 83 |
+
- `inference.py` / `serve_contract.json` — 服务化参考实现与契约
|
chat_template.jinja
ADDED
|
@@ -0,0 +1,177 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{{- bos_token }}{%- if tools %}
|
| 2 |
+
{%- set tool_definitions %}
|
| 3 |
+
{{- "# Tools\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
| 4 |
+
{%- for tool in tools %}
|
| 5 |
+
{{- "\n" }}
|
| 6 |
+
{{- tool | tojson(ensure_ascii=False) }}
|
| 7 |
+
{%- endfor %}
|
| 8 |
+
{{- '\n</tools>\n\nTool usage guidelines:\n- You may call zero or more functions. If no function calls are needed, just answer normally and do not include any <function ... </function>.\n- When calling a function, return an XML object within <function ... </function> using:\n<function name="function-name"><param name="param-name">param-value</param></function>\n- param-value may be multi-line. If it contains <, & or newline characters, wrap it in a CDATA block: <param name="param-name"><![CDATA[...multi-line value...]]></param>' }}
|
| 9 |
+
{%- endset %}
|
| 10 |
+
|
| 11 |
+
{{- '<|im_start|>system\n' }}
|
| 12 |
+
{%- if messages[0].role == 'system' %}
|
| 13 |
+
{%- if '<tool_def_sep>' in messages[0].content %}
|
| 14 |
+
{{- messages[0].content.replace('<tool_def_sep>', tool_definitions) }}
|
| 15 |
+
{%- else %}
|
| 16 |
+
{{- messages[0].content + '\n\n' + tool_definitions }}
|
| 17 |
+
{%- endif %}
|
| 18 |
+
{%- else %}
|
| 19 |
+
{{- tool_definitions.lstrip() }}
|
| 20 |
+
{%- endif %}
|
| 21 |
+
{{- '<|im_end|>\n' }}
|
| 22 |
+
{%- else %}
|
| 23 |
+
{%- if messages[0].role == 'system' %}
|
| 24 |
+
{{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
|
| 25 |
+
{%- endif %}
|
| 26 |
+
{%- endif %}
|
| 27 |
+
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
| 28 |
+
{%- for message in messages[::-1] %}
|
| 29 |
+
{%- set index = (messages|length - 1) - loop.index0 %}
|
| 30 |
+
{%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
|
| 31 |
+
{%- set ns.multi_step_tool = false %}
|
| 32 |
+
{%- set ns.last_query_index = index %}
|
| 33 |
+
{%- endif %}
|
| 34 |
+
{%- endfor %}
|
| 35 |
+
{%- for message in messages %}
|
| 36 |
+
{%- if message.content is string %}
|
| 37 |
+
{%- set content = message.content %}
|
| 38 |
+
{%- else %}
|
| 39 |
+
{%- set content = '' %}
|
| 40 |
+
{%- endif %}
|
| 41 |
+
{%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
|
| 42 |
+
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
| 43 |
+
{%- elif message.role == "assistant" %}
|
| 44 |
+
{%- set reasoning_content = '' %}
|
| 45 |
+
{%- if message.reasoning_content is string %}
|
| 46 |
+
{%- set reasoning_content = message.reasoning_content %}
|
| 47 |
+
{%- else %}
|
| 48 |
+
{%- if '</think>' in content %}
|
| 49 |
+
{%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
|
| 50 |
+
{%- set content = content.split('</think>')[-1].lstrip('\n') %}
|
| 51 |
+
{%- endif %}
|
| 52 |
+
{%- endif %}
|
| 53 |
+
|
| 54 |
+
{%- if message.tool_calls %}
|
| 55 |
+
{%- set content_parts = content.split('<tool_sep>') %}
|
| 56 |
+
{%- set processed_content = content_parts[0] %}
|
| 57 |
+
{%- set tool_calls_count = message.tool_calls|length %}
|
| 58 |
+
{%- set tool_sep_count = content_parts|length - 1 %}
|
| 59 |
+
{%- set min_count = [tool_calls_count, tool_sep_count]|min %}
|
| 60 |
+
|
| 61 |
+
{%- for i in range(1, content_parts|length) %}
|
| 62 |
+
{%- set tool_index = i - 1 %}
|
| 63 |
+
{%- if tool_index < tool_calls_count %}
|
| 64 |
+
{%- set tool_call = message.tool_calls[tool_index] %}
|
| 65 |
+
{%- if tool_call.function %}
|
| 66 |
+
{%- set tool_call = tool_call.function %}
|
| 67 |
+
{%- endif %}
|
| 68 |
+
{%- set single_tool_xml %}
|
| 69 |
+
{{- '<function name="' ~ tool_call.name ~ '">' }}
|
| 70 |
+
{%- if tool_call.arguments %}
|
| 71 |
+
{%- set args_dict = tool_call.arguments %}
|
| 72 |
+
{%- for param_name, param_value in args_dict.items() %}
|
| 73 |
+
{{- '<param name="' ~ param_name ~ '">' }}
|
| 74 |
+
{%- if param_value is string and ('<' in param_value or '&' in param_value or '\n' in param_value) %}
|
| 75 |
+
{{- '<![CDATA[' + param_value + ']]>' }}
|
| 76 |
+
{%- else %}
|
| 77 |
+
{{- param_value }}
|
| 78 |
+
{%- endif %}
|
| 79 |
+
{{- '</param>' }}
|
| 80 |
+
{%- endfor %}
|
| 81 |
+
{%- endif %}
|
| 82 |
+
{{- '</function>' }}
|
| 83 |
+
{%- endset %}
|
| 84 |
+
{%- set processed_content = processed_content + single_tool_xml + content_parts[i] %}
|
| 85 |
+
{%- else %}
|
| 86 |
+
{%- set processed_content = processed_content + content_parts[i] %}
|
| 87 |
+
{%- endif %}
|
| 88 |
+
{%- endfor %}
|
| 89 |
+
|
| 90 |
+
{%- if tool_calls_count > tool_sep_count %}
|
| 91 |
+
{%- for remaining_index in range(tool_sep_count, tool_calls_count) %}
|
| 92 |
+
{%- set tool_call = message.tool_calls[remaining_index] %}
|
| 93 |
+
{%- if tool_call.function %}
|
| 94 |
+
{%- set tool_call = tool_call.function %}
|
| 95 |
+
{%- endif %}
|
| 96 |
+
{%- set remaining_tool_xml %}
|
| 97 |
+
{{- '<function name="' ~ tool_call.name ~ '">' }}
|
| 98 |
+
{%- if tool_call.arguments %}
|
| 99 |
+
{%- set args_dict = tool_call.arguments %}
|
| 100 |
+
{%- for param_name, param_value in args_dict.items() %}
|
| 101 |
+
{{- '<param name="' ~ param_name ~ '">' }}
|
| 102 |
+
{%- if param_value is string and ('<' in param_value or '&' in param_value or '\n' in param_value) %}
|
| 103 |
+
{{- '<![CDATA[' + param_value + ']]>' }}
|
| 104 |
+
{%- else %}
|
| 105 |
+
{{- param_value }}
|
| 106 |
+
{%- endif %}
|
| 107 |
+
{{- '</param>' }}
|
| 108 |
+
{%- endfor %}
|
| 109 |
+
{%- endif %}
|
| 110 |
+
{{- '</function>' }}
|
| 111 |
+
{%- endset %}
|
| 112 |
+
{%- set processed_content = processed_content + remaining_tool_xml %}
|
| 113 |
+
{%- endfor %}
|
| 114 |
+
{%- endif %}
|
| 115 |
+
|
| 116 |
+
{%- set content = processed_content %}
|
| 117 |
+
{%- endif %}
|
| 118 |
+
|
| 119 |
+
{%- if reasoning_content %}
|
| 120 |
+
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
|
| 121 |
+
{%- elif '<think>' not in content and '</think>' not in content %}
|
| 122 |
+
{{- '<|im_start|>' + message.role + '\n<think>\n\n</think>\n\n' + content.lstrip('\n') }}
|
| 123 |
+
{%- else %}
|
| 124 |
+
{{- '<|im_start|>' + message.role + '\n' + content }}
|
| 125 |
+
{%- endif %}
|
| 126 |
+
|
| 127 |
+
{%- if message.tool_calls and not has_tool_sep %}
|
| 128 |
+
{%- for tool_call in message.tool_calls %}
|
| 129 |
+
{%- if (loop.first and content) or (not loop.first) %}
|
| 130 |
+
{{- '\n' }}
|
| 131 |
+
{%- endif %}
|
| 132 |
+
{%- if tool_call.function %}
|
| 133 |
+
{%- set tool_call = tool_call.function %}
|
| 134 |
+
{%- endif %}
|
| 135 |
+
{{- '<function name="' ~ tool_call.name ~ '">' }}
|
| 136 |
+
{%- if tool_call.arguments %}
|
| 137 |
+
{%- set args_dict = tool_call.arguments %}
|
| 138 |
+
{%- for param_name, param_value in args_dict.items() %}
|
| 139 |
+
{{- '<param name="' ~ param_name ~ '">' }}
|
| 140 |
+
{%- if param_value is string and ('<' in param_value or '&' in param_value or '\n' in param_value) %}
|
| 141 |
+
{{- '<![CDATA[' + param_value + ']]>' }}
|
| 142 |
+
{%- else %}
|
| 143 |
+
{{- param_value }}
|
| 144 |
+
{%- endif %}
|
| 145 |
+
{{- '</param>' }}
|
| 146 |
+
{%- endfor %}
|
| 147 |
+
{%- endif %}
|
| 148 |
+
{{- '</function>' }}
|
| 149 |
+
{%- endfor %}
|
| 150 |
+
{%- endif %}
|
| 151 |
+
{{- '<|im_end|>\n' }}
|
| 152 |
+
{%- elif message.role == "tool" %}
|
| 153 |
+
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
|
| 154 |
+
{{- '<|im_start|>user' }}
|
| 155 |
+
{%- endif %}
|
| 156 |
+
{{- '\n<tool_response>\n' }}
|
| 157 |
+
{%- if message.content is string %}
|
| 158 |
+
{{- content }}
|
| 159 |
+
{%- else %}
|
| 160 |
+
{{- message.content | tojson(ensure_ascii=False) }}
|
| 161 |
+
{%- endif %}
|
| 162 |
+
{{- '\n</tool_response>' }}
|
| 163 |
+
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
|
| 164 |
+
{{- '<|im_end|>\n' }}
|
| 165 |
+
{%- endif %}
|
| 166 |
+
{%- endif %}
|
| 167 |
+
{%- endfor %}
|
| 168 |
+
{%- if add_generation_prompt %}
|
| 169 |
+
{{- '<|im_start|>assistant\n' }}
|
| 170 |
+
{%- if enable_thinking is defined %}
|
| 171 |
+
{%- if enable_thinking is false %}
|
| 172 |
+
{{- '<think>\n\n</think>\n\n' }}
|
| 173 |
+
{%- elif enable_thinking is true %}
|
| 174 |
+
{{- '<think>\n' }}
|
| 175 |
+
{%- endif %}
|
| 176 |
+
{%- endif %}
|
| 177 |
+
{%- endif %}
|
config.json
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"PonderMoELlamaForCausalLM"
|
| 4 |
+
],
|
| 5 |
+
"attention_bias": false,
|
| 6 |
+
"attention_dropout": 0.0,
|
| 7 |
+
"bos_token_id": 0,
|
| 8 |
+
"commit_kv": true,
|
| 9 |
+
"drift_threshold": 0.98,
|
| 10 |
+
"dtype": "float16",
|
| 11 |
+
"eos_token_id": [
|
| 12 |
+
1,
|
| 13 |
+
130073
|
| 14 |
+
],
|
| 15 |
+
"head_dim": 128,
|
| 16 |
+
"hidden_act": "silu",
|
| 17 |
+
"hidden_size": 2048,
|
| 18 |
+
"initializer_range": 0.02,
|
| 19 |
+
"intermediate_size": 6144,
|
| 20 |
+
"max_ponder_steps": 8,
|
| 21 |
+
"max_position_embeddings": 131072,
|
| 22 |
+
"mlp_bias": false,
|
| 23 |
+
"model_type": "llama",
|
| 24 |
+
"moe_aux_weight": 0.01,
|
| 25 |
+
"moe_num_experts": 8,
|
| 26 |
+
"moe_route_tau": null,
|
| 27 |
+
"moe_scope": "ponder",
|
| 28 |
+
"moe_tau": 0.125,
|
| 29 |
+
"mtp_loss_weight": 1.0,
|
| 30 |
+
"mtp_num_tokens": 2,
|
| 31 |
+
"num_attention_heads": 16,
|
| 32 |
+
"num_hidden_layers": 42,
|
| 33 |
+
"num_key_value_heads": 2,
|
| 34 |
+
"pad_token_id": 1,
|
| 35 |
+
"ponder_all_positions": false,
|
| 36 |
+
"ponder_epsilon": 0.01,
|
| 37 |
+
"ponder_forced_steps": 0,
|
| 38 |
+
"ponder_head_hidden": 0,
|
| 39 |
+
"ponder_loss_beta": 0.01,
|
| 40 |
+
"ponder_prior_p": 0.5,
|
| 41 |
+
"ponder_signal": "learned",
|
| 42 |
+
"ponder_start_layer": 34,
|
| 43 |
+
"ponder_step_ce_weight": 0.0,
|
| 44 |
+
"ponder_window": 1,
|
| 45 |
+
"pretraining_tp": 1,
|
| 46 |
+
"rms_norm_eps": 1e-06,
|
| 47 |
+
"rope_parameters": {
|
| 48 |
+
"rope_theta": 5000000,
|
| 49 |
+
"rope_type": "default"
|
| 50 |
+
},
|
| 51 |
+
"tie_word_embeddings": false,
|
| 52 |
+
"transformers_version": "5.16.1",
|
| 53 |
+
"use_cache": true,
|
| 54 |
+
"vocab_size": 130560,
|
| 55 |
+
"auto_map": {
|
| 56 |
+
"AutoConfig": "modeling_samai_pnet.SamaiPnetConfig",
|
| 57 |
+
"AutoModelForCausalLM": "modeling_samai_pnet.SamaiPnetForCausalLM"
|
| 58 |
+
}
|
| 59 |
+
}
|
generation_config.json
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_from_model_config": true,
|
| 3 |
+
"bos_token_id": 0,
|
| 4 |
+
"eos_token_id": [
|
| 5 |
+
1,
|
| 6 |
+
130073
|
| 7 |
+
],
|
| 8 |
+
"output_attentions": false,
|
| 9 |
+
"output_hidden_states": false,
|
| 10 |
+
"pad_token_id": 1,
|
| 11 |
+
"transformers_version": "5.16.1",
|
| 12 |
+
"use_cache": true,
|
| 13 |
+
"do_sample": true,
|
| 14 |
+
"temperature": 1.0,
|
| 15 |
+
"top_p": 0.95,
|
| 16 |
+
"repetition_penalty": 1.0,
|
| 17 |
+
"max_new_tokens": 512
|
| 18 |
+
}
|
inference.py
ADDED
|
@@ -0,0 +1,63 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# -*- coding: utf-8 -*-
|
| 2 |
+
"""inference.py — samai-pnet-dmoe-2b 拉取即跑推理示例
|
| 3 |
+
|
| 4 |
+
用法:
|
| 5 |
+
python inference.py # 默认 HuggingFace 仓
|
| 6 |
+
python inference.py --repo /path/to/local # 本地目录
|
| 7 |
+
python inference.py --lru 4 # 启用 SSD-LRU 专家缓存模拟 (每层容量 4)
|
| 8 |
+
|
| 9 |
+
依赖: torch>=2.4, transformers>=5.0 (trust_remote_code 加载仓内 modeling_samai_pnet.py)
|
| 10 |
+
"""
|
| 11 |
+
import argparse
|
| 12 |
+
|
| 13 |
+
import torch
|
| 14 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
def main():
|
| 18 |
+
ap = argparse.ArgumentParser()
|
| 19 |
+
ap.add_argument("--repo", default="tchbcb/samai-pnet-dmoe-2b")
|
| 20 |
+
ap.add_argument("--lru", type=int, default=0,
|
| 21 |
+
help=">0 启用 SSD-LRU 专家缓存 (每层容量; 需仓内 modeling 模块)")
|
| 22 |
+
ap.add_argument("--max-new-tokens", type=int, default=256)
|
| 23 |
+
args = ap.parse_args()
|
| 24 |
+
|
| 25 |
+
tok = AutoTokenizer.from_pretrained(args.repo, trust_remote_code=True)
|
| 26 |
+
model = AutoModelForCausalLM.from_pretrained(args.repo, trust_remote_code=True,
|
| 27 |
+
dtype="auto").eval()
|
| 28 |
+
device = "cuda" if torch.cuda.is_available() else "cpu"
|
| 29 |
+
model.to(device)
|
| 30 |
+
|
| 31 |
+
if args.lru:
|
| 32 |
+
# SSDLRUExpertManager: 专家驻留 CPU(模拟 SSD), 路由激活时搬入, LRU 驱逐
|
| 33 |
+
from modeling_samai_pnet import SSDLRUExpertManager # remote_code 模块
|
| 34 |
+
mgr = SSDLRUExpertManager.install(model, capacity_per_layer=args.lru)
|
| 35 |
+
print(f"[ssd-lru] capacity/layer={args.lru} (storage=cpu)")
|
| 36 |
+
|
| 37 |
+
messages = [{"role": "user", "content": "一个长方形长 12 米宽 8 米,面积是多少?请简要推理。"}]
|
| 38 |
+
enc = tok.apply_chat_template(messages, add_generation_prompt=True,
|
| 39 |
+
tokenize=True, return_tensors="pt",
|
| 40 |
+
return_dict=True).to(device)
|
| 41 |
+
with torch.no_grad():
|
| 42 |
+
out = model.generate(**enc, max_new_tokens=args.max_new_tokens,
|
| 43 |
+
do_sample=False, pad_token_id=tok.pad_token_id or 1)
|
| 44 |
+
text = tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True)
|
| 45 |
+
print("=== 回答 ===")
|
| 46 |
+
print(text.strip())
|
| 47 |
+
|
| 48 |
+
# Ponder 诊断: 每次 forward 追加一条 {mode, executed, steps_mean, ...}
|
| 49 |
+
decode_entries = [e for e in model._ponder_log if e.get("mode") == "decode"]
|
| 50 |
+
if decode_entries:
|
| 51 |
+
steps = sum(e["steps_mean"] for e in decode_entries) / len(decode_entries)
|
| 52 |
+
print(f"\n=== Ponder 诊断 ===\n本次生成平均思考步数: {steps:.2f} "
|
| 53 |
+
f"(max_ponder_steps={model.config.max_ponder_steps})")
|
| 54 |
+
|
| 55 |
+
if args.lru:
|
| 56 |
+
s = mgr.summary()
|
| 57 |
+
print(f"=== SSD-LRU ===\nhit={s['hit']} miss={s['miss']} "
|
| 58 |
+
f"evict={s['evict']} hit_rate={s['hit_rate']:.3f}")
|
| 59 |
+
mgr.remove()
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
if __name__ == "__main__":
|
| 63 |
+
main()
|
modeling_samai_pnet.py
ADDED
|
@@ -0,0 +1,1264 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# -*- coding: utf-8 -*-
|
| 2 |
+
"""
|
| 3 |
+
modeling_samai_pnet.py — SamAI-Pnet-dMoE-2B 自洽模型实现 (发布版, 单文件)。
|
| 4 |
+
|
| 5 |
+
架构 = PonderNet 自适应停机 (质量维度) + 动态可变 MoE (效率维度) + MTP (训练期吞吐)。
|
| 6 |
+
本文件为发行版自洽模块: 不依赖训练仓其他文件, 仅需 torch + transformers>=5.0。
|
| 7 |
+
|
| 8 |
+
用法 (trust_remote_code 一行加载):
|
| 9 |
+
import torch
|
| 10 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 11 |
+
model = AutoModelForCausalLM.from_pretrained(
|
| 12 |
+
"tchbcb/samai-pnet-dmoe-2b", trust_remote_code=True,
|
| 13 |
+
dtype="auto").eval().cuda()
|
| 14 |
+
tok = AutoTokenizer.from_pretrained("tchbcb/samai-pnet-dmoe-2b")
|
| 15 |
+
|
| 16 |
+
核心机制:
|
| 17 |
+
1. Ponder 思考块: 后 8 层循环执行, halting 头输出每步停机概率 lam_k,
|
| 18 |
+
概率加权混合 h = Σ w_k·h_k, 累计质量 1-eps 即停; KV 经 commit pass 写回。
|
| 19 |
+
2. 动态可变 MoE: FFN 按宽度均分 8 个同构专家, Router(已训练) 输出阈值激活
|
| 20 |
+
(p_i >= tau 激活, 激活数 1~8 自适应); 每 token 仅首轮路由一次, 循环复用。
|
| 21 |
+
3. MTP: 训练期辅助头, 推理 generate() 自动旁路。
|
| 22 |
+
4. round5-B: per-step CE 训练的 halting 头 —— hard 样本自适应"多想",
|
| 23 |
+
easy 样本早停 (hard-easy steps 分化 +0.433)。
|
| 24 |
+
|
| 25 |
+
诊断: model.ponder_log 追加每次 forward 的 {mode, executed, steps_mean, ...}。
|
| 26 |
+
"""
|
| 27 |
+
from __future__ import annotations
|
| 28 |
+
|
| 29 |
+
from collections import OrderedDict
|
| 30 |
+
from dataclasses import dataclass
|
| 31 |
+
|
| 32 |
+
import torch
|
| 33 |
+
import torch.nn.functional as F
|
| 34 |
+
from torch import nn
|
| 35 |
+
from torch.utils.checkpoint import checkpoint as _ckpt
|
| 36 |
+
|
| 37 |
+
from transformers import AutoConfig, AutoModelForCausalLM
|
| 38 |
+
from transformers.activations import ACT2FN
|
| 39 |
+
from transformers.cache_utils import DynamicCache
|
| 40 |
+
from transformers.masking_utils import create_causal_mask
|
| 41 |
+
from transformers.modeling_outputs import CausalLMOutputWithPast
|
| 42 |
+
from transformers.models.llama.configuration_llama import LlamaConfig
|
| 43 |
+
from transformers.models.llama.modeling_llama import LlamaForCausalLM, LlamaRMSNorm
|
| 44 |
+
from transformers.utils import can_return_tuple
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
# --------------------------------------------------------------------------
|
| 48 |
+
# Config
|
| 49 |
+
# --------------------------------------------------------------------------
|
| 50 |
+
class PonderLlamaConfig(LlamaConfig):
|
| 51 |
+
"""在 LlamaConfig 之上追加 PonderNet 超参 (其余字段与原 Llama config 完全兼容)."""
|
| 52 |
+
|
| 53 |
+
model_type = "llama_ponder"
|
| 54 |
+
|
| 55 |
+
def __init__(self, **kwargs):
|
| 56 |
+
self.ponder_start_layer = kwargs.pop("ponder_start_layer", None) # None = 最后 8 层
|
| 57 |
+
self.ponder_window = kwargs.pop("ponder_window", 1) # prefill 思考窗口 W
|
| 58 |
+
self.max_ponder_steps = kwargs.pop("max_ponder_steps", 8) # 思考块最多循环步数 K
|
| 59 |
+
self.ponder_epsilon = kwargs.pop("ponder_epsilon", 0.01) # 累计概率 1-eps 即停
|
| 60 |
+
self.ponder_signal = kwargs.pop("ponder_signal", "entropy") # entropy|msp|drift|learned
|
| 61 |
+
self.ponder_all_positions = kwargs.pop("ponder_all_positions", False) # 全序列位置思考
|
| 62 |
+
self.commit_kv = kwargs.pop("commit_kv", True) # 用混合表示写最终 KV
|
| 63 |
+
self.ponder_prior_p = kwargs.pop("ponder_prior_p", 0.5) # 几何先验 p_g (训练)
|
| 64 |
+
self.ponder_loss_beta = kwargs.pop("ponder_loss_beta", 0.01) # KL 正则权重 beta
|
| 65 |
+
self.drift_threshold = kwargs.pop("drift_threshold", 0.98) # drift 相似度阈值
|
| 66 |
+
# ---- Round5 三方向开关 ----
|
| 67 |
+
self.ponder_step_ce_weight = kwargs.pop("ponder_step_ce_weight", 0.0) # 方向②: per-step CE 权重 (0=关)
|
| 68 |
+
self.ponder_forced_steps = kwargs.pop("ponder_forced_steps", 0) # 方向③: 强制思考步数 rollout (0=关)
|
| 69 |
+
self.ponder_head_hidden = kwargs.pop("ponder_head_hidden", 0) # 方向①: >0 时 halting 头升级 MLP
|
| 70 |
+
super().__init__(**kwargs)
|
| 71 |
+
if self.ponder_start_layer is None:
|
| 72 |
+
self.ponder_start_layer = max(0, self.num_hidden_layers - 8)
|
| 73 |
+
|
| 74 |
+
@classmethod
|
| 75 |
+
def from_llama(cls, llama_config: LlamaConfig, **ponder_kwargs) -> "PonderLlamaConfig":
|
| 76 |
+
"""从标准 LlamaConfig 派生 (保留全部架构字段, 丢弃非配置项)。
|
| 77 |
+
兼容两种来源: 标准 llama config.json (无 ponder 字段) 与本类保存的
|
| 78 |
+
llama_ponder config.json (含 ponder 字段) —— 显式 ponder_kwargs 优先覆盖。"""
|
| 79 |
+
d = llama_config.to_dict()
|
| 80 |
+
for k in ("model_type", "architectures", "_name_or_path", "transformers_version",
|
| 81 |
+
"torch_dtype"):
|
| 82 |
+
d.pop(k, None)
|
| 83 |
+
ponder_keys = [k for k in d
|
| 84 |
+
if "ponder" in k or k in ("commit_kv", "drift_threshold")]
|
| 85 |
+
for k in ponder_keys:
|
| 86 |
+
if k in ponder_kwargs: # 显式参数覆盖 config 里的旧值, 避免重复 kwargs 报错
|
| 87 |
+
d.pop(k)
|
| 88 |
+
return cls(**d, **ponder_kwargs)
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
# --------------------------------------------------------------------------
|
| 92 |
+
# ���出对象
|
| 93 |
+
# --------------------------------------------------------------------------
|
| 94 |
+
@dataclass
|
| 95 |
+
class PonderCausalLMOutputWithPast(CausalLMOutputWithPast):
|
| 96 |
+
"""额外带出 ponder 诊断信息."""
|
| 97 |
+
ponder_weights: torch.Tensor | None = None # (B, W, K) 每步停机质量分布
|
| 98 |
+
ponder_steps: torch.Tensor | None = None # (B, W) 每位置实际思考步数
|
| 99 |
+
ponder_kl: torch.Tensor | None = None # 标量, KL(w‖Geometric) 正则项 (labels 时)
|
| 100 |
+
ponder_step_ce: torch.Tensor | None = None # 标量, per-step CE (方向②, 训练且权重>0 时)
|
| 101 |
+
|
| 102 |
+
|
| 103 |
+
# --------------------------------------------------------------------------
|
| 104 |
+
# Model
|
| 105 |
+
# --------------------------------------------------------------------------
|
| 106 |
+
class PonderLlamaForCausalLM(LlamaForCausalLM):
|
| 107 |
+
"""
|
| 108 |
+
用法 A — 直接加载 MiniCPM5-2B 权重 (键名与 LlamaForCausalLM 完全一致):
|
| 109 |
+
from modeling_samai_pnet import PonderLlamaForCausalLM, PonderLlamaConfig
|
| 110 |
+
base = LlamaConfig.from_pretrained("tchbcb/MiniCPM5-2B-cpu")
|
| 111 |
+
cfg = PonderLlamaConfig.from_llama(base)
|
| 112 |
+
model = PonderLlamaForCausalLM.from_pretrained(
|
| 113 |
+
"tchbcb/MiniCPM5-2B-cpu", config=cfg, torch_dtype="bfloat16")
|
| 114 |
+
# missing keys 只有 ponder_head (仅 signal="learned" 时参与前向, 其余信号忽略)
|
| 115 |
+
|
| 116 |
+
用法 B — 一行加载:
|
| 117 |
+
model = PonderLlamaForCausalLM.from_ponder("tchbcb/MiniCPM5-2B-cpu")
|
| 118 |
+
|
| 119 |
+
generate() / chat 流程与 HF 标准完全一致; model.ponder_log 记录每次 forward 的思考步数。
|
| 120 |
+
"""
|
| 121 |
+
|
| 122 |
+
def __init__(self, config: PonderLlamaConfig):
|
| 123 |
+
super().__init__(config)
|
| 124 |
+
cfg = self.config
|
| 125 |
+
# 真 PonderNet 停机头; bias=-4 使初始 lam≈0.018 (初始倾向于多想几步)
|
| 126 |
+
# 方向①: ponder_head_hidden>0 时升级为 MLP (Linear->GELU->Linear),
|
| 127 |
+
# 打破线性可分上限 —— round4 结论: 线性头在冻结特征上已到极限。
|
| 128 |
+
# 数值初始化由下方 _init_weights 覆写统一处理: post_init 重遍历会把
|
| 129 |
+
# Linear 重置为 Llama 默认 (normal+零偏置), 自定义值必须写在那里才能存活
|
| 130 |
+
head_hidden = int(getattr(cfg, "ponder_head_hidden", 0) or 0)
|
| 131 |
+
if head_hidden > 0:
|
| 132 |
+
self.ponder_head = nn.Sequential(
|
| 133 |
+
nn.Linear(cfg.hidden_size, head_hidden),
|
| 134 |
+
nn.GELU(),
|
| 135 |
+
nn.Linear(head_hidden, 1, bias=True),
|
| 136 |
+
)
|
| 137 |
+
else:
|
| 138 |
+
self.ponder_head = nn.Linear(cfg.hidden_size, 1, bias=True)
|
| 139 |
+
self._ponder_log: list[dict] = [] # 诊断日志 (每次 forward 追加, 最多保留 4096 条)
|
| 140 |
+
self.post_init()
|
| 141 |
+
|
| 142 |
+
def _init_weights(self, module):
|
| 143 |
+
"""PonderNet 停机头自定义初始化 (覆盖 Llama 默认), 其余模块遵循 Llama 惯例。
|
| 144 |
+
post_init()/from_pretrained 的 init_weights 遍历都会调用本钩子,
|
| 145 |
+
因此自定义值不会被 Llama 默认的 normal+零偏置覆写。"""
|
| 146 |
+
head = getattr(self, "ponder_head", None)
|
| 147 |
+
if isinstance(module, nn.Linear) and head is not None:
|
| 148 |
+
head_linears = [mm for mm in head.modules() if isinstance(mm, nn.Linear)]
|
| 149 |
+
if any(module is mm for mm in head_linears):
|
| 150 |
+
nn.init.zeros_(module.weight)
|
| 151 |
+
nn.init.zeros_(module.bias)
|
| 152 |
+
if module is head_linears[-1]:
|
| 153 |
+
# 尾层 bias=-4 -> 初始 lam=sigmoid(-4)≈0.018, 倾向多思考
|
| 154 |
+
nn.init.constant_(module.bias, -4.0)
|
| 155 |
+
return
|
| 156 |
+
super()._init_weights(module)
|
| 157 |
+
|
| 158 |
+
@classmethod
|
| 159 |
+
def from_ponder(cls, pretrained_path: str, ponder_kwargs: dict | None = None, **kw):
|
| 160 |
+
"""从标准 Llama 权重目录 / HF repo 一行构造 Ponder 模型."""
|
| 161 |
+
base = LlamaConfig.from_pretrained(pretrained_path)
|
| 162 |
+
cfg = PonderLlamaConfig.from_llama(base, **(ponder_kwargs or {}))
|
| 163 |
+
return cls.from_pretrained(pretrained_path, config=cfg, **kw)
|
| 164 |
+
|
| 165 |
+
# ------------------------------------------------------------------ #
|
| 166 |
+
# 内部工具
|
| 167 |
+
# ------------------------------------------------------------------ #
|
| 168 |
+
@property
|
| 169 |
+
def _ponder_layers(self):
|
| 170 |
+
s = self.config.ponder_start_layer
|
| 171 |
+
return self.model.layers[s: self.config.num_hidden_layers]
|
| 172 |
+
|
| 173 |
+
def _crop_ponder_kv(self, cache, n_remove: int):
|
| 174 |
+
"""只回滚思考块层的 KV 写入 (前段层的 KV 不许动)."""
|
| 175 |
+
if cache is None or n_remove <= 0:
|
| 176 |
+
return
|
| 177 |
+
s = self.config.ponder_start_layer
|
| 178 |
+
for li in range(s, self.config.num_hidden_layers):
|
| 179 |
+
cache.layers[li].crop(-n_remove)
|
| 180 |
+
|
| 181 |
+
def _run_block(self, h, attention_mask, position_ids, position_embeddings,
|
| 182 |
+
cache, kwargs):
|
| 183 |
+
"""跑一遍思考块 (最后若干层).
|
| 184 |
+
训练且开启 gradient checkpointing 时逐层重算省激活 —— 思考块最多循环 K 次,
|
| 185 |
+
激活占用是普通前向的 K 倍, checkpointing 是小显存卡 (如 T4 16GB) 的关键."""
|
| 186 |
+
ckpt_fn = getattr(self.model, "_gradient_checkpointing_func", None)
|
| 187 |
+
use_ckpt = (self.training and cache is None
|
| 188 |
+
and getattr(self.model, "gradient_checkpointing", False)
|
| 189 |
+
and ckpt_fn is not None)
|
| 190 |
+
for layer in self._ponder_layers:
|
| 191 |
+
if use_ckpt:
|
| 192 |
+
h = ckpt_fn(
|
| 193 |
+
layer.__call__, h,
|
| 194 |
+
attention_mask=attention_mask,
|
| 195 |
+
position_ids=position_ids,
|
| 196 |
+
position_embeddings=position_embeddings,
|
| 197 |
+
past_key_values=None,
|
| 198 |
+
use_cache=False,
|
| 199 |
+
**kwargs,
|
| 200 |
+
)
|
| 201 |
+
else:
|
| 202 |
+
h = layer(
|
| 203 |
+
h,
|
| 204 |
+
attention_mask=attention_mask,
|
| 205 |
+
position_ids=position_ids,
|
| 206 |
+
position_embeddings=position_embeddings,
|
| 207 |
+
past_key_values=cache,
|
| 208 |
+
use_cache=cache is not None,
|
| 209 |
+
**kwargs,
|
| 210 |
+
)
|
| 211 |
+
return h
|
| 212 |
+
|
| 213 |
+
def _halting_prob(self, h_window, h_prev_window) -> torch.Tensor:
|
| 214 |
+
"""返回 (B, W) 的停机概率 lam ∈ (0, 1)."""
|
| 215 |
+
cfg = self.config
|
| 216 |
+
B, W = h_window.shape[:2]
|
| 217 |
+
if cfg.ponder_signal == "learned":
|
| 218 |
+
# ponder_head 恒为 fp32 (数值稳定), 输入 cast 到其参数 dtype,
|
| 219 |
+
# 否则半精度模型 (fp16/bf16) 下会报 dtype 不匹配错误。
|
| 220 |
+
# head_ref 写法兼容 Linear 与 Sequential (方向① MLP 头) 两种结构
|
| 221 |
+
head_ref = next(self.ponder_head.parameters())
|
| 222 |
+
lam = torch.sigmoid(
|
| 223 |
+
self.ponder_head(h_window.to(head_ref.dtype))
|
| 224 |
+
).squeeze(-1)
|
| 225 |
+
elif cfg.ponder_signal in ("entropy", "msp"):
|
| 226 |
+
logits = self.lm_head(self.model.norm(h_window)).float()
|
| 227 |
+
probs = F.softmax(logits, dim=-1)
|
| 228 |
+
if cfg.ponder_signal == "msp":
|
| 229 |
+
lam = probs.max(dim=-1).values
|
| 230 |
+
else: # entropy: 自信 -> 熵小 -> lam 大
|
| 231 |
+
ent = -(probs * torch.log(probs.clamp_min(1e-9))).sum(-1)
|
| 232 |
+
lam = 1.0 - ent / torch.log(torch.tensor(float(logits.size(-1)), device=logits.device))
|
| 233 |
+
elif cfg.ponder_signal == "drift":
|
| 234 |
+
if h_prev_window is None: # 第一步没有参照, 强制继续思考
|
| 235 |
+
lam = torch.zeros(B, W, device=h_window.device, dtype=h_window.dtype)
|
| 236 |
+
else:
|
| 237 |
+
cos = F.cosine_similarity(h_window.float(), h_prev_window.float(), dim=-1)
|
| 238 |
+
tau = cfg.drift_threshold
|
| 239 |
+
lam = (cos - tau) / (1.0 - tau + 1e-6) # cos 越接近 1 (不再变化) 越停
|
| 240 |
+
else:
|
| 241 |
+
raise ValueError(f"unknown ponder_signal: {cfg.ponder_signal}")
|
| 242 |
+
# 统一 cast 回隐状态 dtype: 保证 remaining/mix/w_stack 混合链路 dtype 一致,
|
| 243 |
+
# 否则 fp32 的 lam 会把整条链路升成 fp32, 之后 norm/lm_head (half 权重) 会崩
|
| 244 |
+
return lam.to(h_window.dtype).clamp(1e-6, 1.0 - 1e-6)
|
| 245 |
+
|
| 246 |
+
def _step_ce(self, h_step, labels):
|
| 247 |
+
"""方向②: 单步表示的独立解码 CE。
|
| 248 |
+
让每一步思考表示都"可用"(可被 lm_head 解码), 回收 mix 摊薄导致的 CE 代价。
|
| 249 |
+
训练时用 gradient checkpointing 重算, 避免大词表 logits 常驻显存。"""
|
| 250 |
+
def _fn(h_):
|
| 251 |
+
lg = self.lm_head(self.model.norm(h_))
|
| 252 |
+
lg = lg[..., :-1, :].reshape(-1, lg.size(-1)).float()
|
| 253 |
+
tg = labels[..., 1:].reshape(-1)
|
| 254 |
+
return F.cross_entropy(lg, tg, ignore_index=-100)
|
| 255 |
+
if self.training and torch.is_grad_enabled() and h_step.requires_grad:
|
| 256 |
+
return _ckpt(_fn, h_step, use_reentrant=False)
|
| 257 |
+
return _fn(h_step)
|
| 258 |
+
|
| 259 |
+
def _win_mask(self, W, past_len, device, dtype, attention_mask=None):
|
| 260 |
+
"""窗口迭代的加性因果 mask: keys = [0..past_len+W), queries = 窗口 W 个位置。
|
| 261 |
+
|
| 262 |
+
必须手工构建而不能用 create_causal_mask: 后者从非思考层读取缓存长度,
|
| 263 |
+
而思考块层已被 crop 到 past_len, 会得到多 W 的错误 kv 尺寸。
|
| 264 |
+
返回 float 加性 mask (sdpa / eager 通用)。"""
|
| 265 |
+
total = past_len + W
|
| 266 |
+
q_idx = torch.arange(past_len, total, device=device)
|
| 267 |
+
k_idx = torch.arange(total, device=device)
|
| 268 |
+
allowed = q_idx.unsqueeze(-1) >= k_idx.unsqueeze(0) # (W, total)
|
| 269 |
+
if attention_mask is not None:
|
| 270 |
+
allowed = allowed.unsqueeze(0) & attention_mask[:, :total].to(torch.bool).unsqueeze(1)
|
| 271 |
+
m = torch.zeros(allowed.size(0), 1, W, total, dtype=dtype, device=device)
|
| 272 |
+
m = m.masked_fill(~allowed.unsqueeze(1), torch.finfo(dtype).min) # (B,1,W,total)
|
| 273 |
+
else:
|
| 274 |
+
m = torch.zeros(1, 1, W, total, dtype=dtype, device=device)
|
| 275 |
+
m = m.masked_fill(~allowed.unsqueeze(0).unsqueeze(0), torch.finfo(dtype).min)
|
| 276 |
+
return m
|
| 277 |
+
|
| 278 |
+
def _log(self, entry: dict):
|
| 279 |
+
self._ponder_log.append(entry)
|
| 280 |
+
if len(self._ponder_log) > 4096:
|
| 281 |
+
del self._ponder_log[: len(self._ponder_log) - 4096]
|
| 282 |
+
|
| 283 |
+
# ------------------------------------------------------------------ #
|
| 284 |
+
# 主前向
|
| 285 |
+
# ------------------------------------------------------------------ #
|
| 286 |
+
@can_return_tuple
|
| 287 |
+
def forward(
|
| 288 |
+
self,
|
| 289 |
+
input_ids: torch.LongTensor | None = None,
|
| 290 |
+
attention_mask: torch.Tensor | None = None,
|
| 291 |
+
position_ids: torch.LongTensor | None = None,
|
| 292 |
+
past_key_values: DynamicCache | None = None,
|
| 293 |
+
inputs_embeds: torch.FloatTensor | None = None,
|
| 294 |
+
labels: torch.LongTensor | None = None,
|
| 295 |
+
use_cache: bool | None = None,
|
| 296 |
+
logits_to_keep: int | torch.Tensor = 0,
|
| 297 |
+
output_ponder: bool = False,
|
| 298 |
+
**kwargs,
|
| 299 |
+
) -> PonderCausalLMOutputWithPast:
|
| 300 |
+
cfg = self.config
|
| 301 |
+
m = self.model
|
| 302 |
+
|
| 303 |
+
# ---- 输入 & 位置 ------------------------------------------------
|
| 304 |
+
if (input_ids is None) ^ (inputs_embeds is not None):
|
| 305 |
+
raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
|
| 306 |
+
if inputs_embeds is None:
|
| 307 |
+
inputs_embeds = m.embed_tokens(input_ids)
|
| 308 |
+
B, Q = inputs_embeds.shape[:2]
|
| 309 |
+
|
| 310 |
+
if use_cache is None:
|
| 311 |
+
use_cache = False if self.training else cfg.use_cache
|
| 312 |
+
if use_cache and past_key_values is None:
|
| 313 |
+
past_key_values = DynamicCache(config=cfg)
|
| 314 |
+
|
| 315 |
+
cache_position = kwargs.pop("cache_position", None)
|
| 316 |
+
past_seen = past_key_values.get_seq_length() if past_key_values is not None else 0
|
| 317 |
+
if position_ids is None:
|
| 318 |
+
base = (cache_position if cache_position is not None
|
| 319 |
+
else torch.arange(Q, device=inputs_embeds.device) + past_seen)
|
| 320 |
+
position_ids = base.reshape(1, -1).expand(B, -1)[:, :Q].to(inputs_embeds.device)
|
| 321 |
+
|
| 322 |
+
# ---- mask / rope (整次 forward 内复用) ---------------------------
|
| 323 |
+
causal_mask = create_causal_mask(
|
| 324 |
+
config=cfg, inputs_embeds=inputs_embeds, attention_mask=attention_mask,
|
| 325 |
+
past_key_values=past_key_values, position_ids=position_ids,
|
| 326 |
+
)
|
| 327 |
+
position_embeddings = m.rotary_emb(inputs_embeds, position_ids=position_ids)
|
| 328 |
+
|
| 329 |
+
# ---- Stage A: 前段层正常单遍 -------------------------------------
|
| 330 |
+
h = inputs_embeds
|
| 331 |
+
for layer in m.layers[: cfg.ponder_start_layer]:
|
| 332 |
+
h = layer(
|
| 333 |
+
h,
|
| 334 |
+
attention_mask=causal_mask,
|
| 335 |
+
position_ids=position_ids,
|
| 336 |
+
position_embeddings=position_embeddings,
|
| 337 |
+
past_key_values=past_key_values,
|
| 338 |
+
use_cache=use_cache,
|
| 339 |
+
**kwargs,
|
| 340 |
+
)
|
| 341 |
+
|
| 342 |
+
# ---- 思考窗口 W --------------------------------------------------
|
| 343 |
+
# decode (Q==1) 恒为 1; 训练/显式开启时对全部位置; 其余 prefill 思考末尾 W 个位置
|
| 344 |
+
if Q == 1 or self.training or cfg.ponder_all_positions:
|
| 345 |
+
W = Q
|
| 346 |
+
else:
|
| 347 |
+
W = max(1, min(cfg.ponder_window, Q))
|
| 348 |
+
|
| 349 |
+
eps = cfg.ponder_epsilon
|
| 350 |
+
K = cfg.max_ponder_steps
|
| 351 |
+
device, dtype = inputs_embeds.device, inputs_embeds.dtype
|
| 352 |
+
|
| 353 |
+
remaining = torch.ones(B, W, device=device, dtype=dtype)
|
| 354 |
+
cum = torch.zeros(B, W, device=device, dtype=dtype)
|
| 355 |
+
active = torch.ones(B, W, device=device, dtype=torch.bool)
|
| 356 |
+
mix = torch.zeros(B, W, h.size(-1), device=device, dtype=dtype)
|
| 357 |
+
w_list: list[torch.Tensor] = []
|
| 358 |
+
h_prev_win = None
|
| 359 |
+
h_full = None # 第 1 次迭代的全序列输出 (W<Q 时非窗口位置沿用)
|
| 360 |
+
h_win = h[:, -W:, :] if W < Q else h
|
| 361 |
+
executed = 0
|
| 362 |
+
win_mask_built = False # 窗口迭代专用 mask (懒构建一次)
|
| 363 |
+
|
| 364 |
+
# ---- Round5 方向②③ 前置开关 --------------------------------------
|
| 365 |
+
forced = int(getattr(cfg, "ponder_forced_steps", 0) or 0) # 方向③: >0 强制 rollout
|
| 366 |
+
step_ce_weight = float(getattr(cfg, "ponder_step_ce_weight", 0.0) or 0.0)
|
| 367 |
+
use_step_ce = (step_ce_weight > 0.0 and labels is not None and self.training)
|
| 368 |
+
step_pairs: list[tuple[torch.Tensor, torch.Tensor]] = [] # [(w_k_detached, ce_k), ...]
|
| 369 |
+
loop_K = forced if forced > 0 else K
|
| 370 |
+
|
| 371 |
+
for k in range(1, loop_K + 1):
|
| 372 |
+
executed = k
|
| 373 |
+
if k > 1:
|
| 374 |
+
# 替换式写入: 先回滚上一次迭代在思考块层的 KV, 本轮 update 再补回,
|
| 375 |
+
# 使任意时刻 cache 长度恒等于真实 token 数 (对 W==Q 同样必需!)
|
| 376 |
+
self._crop_ponder_kv(past_key_values, W)
|
| 377 |
+
# 仅当有 cache 且 W<Q 时才做窗口局部重算; 无 cache (训练) 一律全序列重算
|
| 378 |
+
use_window = (k > 1) and (past_key_values is not None) and (W < Q)
|
| 379 |
+
if not use_window:
|
| 380 |
+
h = self._run_block(h, causal_mask, position_ids,
|
| 381 |
+
position_embeddings, past_key_values, kwargs)
|
| 382 |
+
h_win = h if W == Q else h[:, -W:, :]
|
| 383 |
+
if W < Q:
|
| 384 |
+
h_full = h
|
| 385 |
+
else:
|
| 386 |
+
if not win_mask_built:
|
| 387 |
+
past_len = int(past_key_values.layers[cfg.ponder_start_layer].get_seq_length())
|
| 388 |
+
mask_win = self._win_mask(W, past_len, device, dtype, attention_mask)
|
| 389 |
+
pos_emb_win = (position_embeddings[0][:, -W:],
|
| 390 |
+
position_embeddings[1][:, -W:])
|
| 391 |
+
win_mask_built = True
|
| 392 |
+
h_win = self._run_block(h_win, mask_win, position_ids[:, -W:],
|
| 393 |
+
pos_emb_win, past_key_values, kwargs)
|
| 394 |
+
|
| 395 |
+
if forced > 0:
|
| 396 |
+
# 方向③: 强制步数 rollout —— 完全绕过 halting 头与 w 混合,
|
| 397 |
+
# 最终表示即第 forced 步隐状态 (KV 恰为其 cache, 无需 commit)
|
| 398 |
+
continue
|
| 399 |
+
|
| 400 |
+
lam = self._halting_prob(h_win, h_prev_win) # (B, W)
|
| 401 |
+
lam_eff = lam * active.to(lam.dtype)
|
| 402 |
+
w_k = remaining * lam_eff
|
| 403 |
+
mix = mix + w_k.unsqueeze(-1) * h_win
|
| 404 |
+
w_list.append(w_k)
|
| 405 |
+
remaining = remaining * (1.0 - lam_eff)
|
| 406 |
+
cum = cum + w_k
|
| 407 |
+
active = active & (cum < 1.0 - eps)
|
| 408 |
+
h_prev_win = h_win
|
| 409 |
+
if use_step_ce:
|
| 410 |
+
# 方向②: 本步表示单独算解码 CE。权重 detach —— 梯度只流向表示,
|
| 411 |
+
# 不经此损失推动 w (w 由 step-supervise / KL 单独管辖)
|
| 412 |
+
ce_k = self._step_ce(h_win, labels)
|
| 413 |
+
step_pairs.append((w_k.detach(), ce_k))
|
| 414 |
+
if not bool(active.any()):
|
| 415 |
+
break
|
| 416 |
+
|
| 417 |
+
if forced > 0:
|
| 418 |
+
# ---- 方向③: 最终表示 = 第 n 步隐状态 (非概率混合) --------------
|
| 419 |
+
if W < Q:
|
| 420 |
+
h_final = h_full.clone()
|
| 421 |
+
h_final[:, -W:, :] = h_win
|
| 422 |
+
else:
|
| 423 |
+
h_final = h_win
|
| 424 |
+
w_stack = F.one_hot(
|
| 425 |
+
torch.full((B, W), forced - 1, device=device), forced
|
| 426 |
+
).to(dtype) # (B, W, n) 诊断用
|
| 427 |
+
else:
|
| 428 |
+
# 剩余概率质量全部记给最后一步 (PonderNet 的 p_N 截断)
|
| 429 |
+
mix = mix + remaining.unsqueeze(-1) * h_win
|
| 430 |
+
w_stack = torch.stack(w_list, dim=-1).to(dtype) # (B, W, K_exec)
|
| 431 |
+
w_stack = torch.cat([w_stack[..., :-1],
|
| 432 |
+
w_stack[..., -1:] + remaining.unsqueeze(-1)], dim=-1)
|
| 433 |
+
|
| 434 |
+
# ---- 组装最终 hidden --------------------------------------------
|
| 435 |
+
if W < Q:
|
| 436 |
+
h_final = h_full
|
| 437 |
+
h_final[:, -W:, :] = mix
|
| 438 |
+
else:
|
| 439 |
+
h_final = mix
|
| 440 |
+
|
| 441 |
+
# ---- commit pass: 用"思考后"的混合表示写最终 KV ------------------
|
| 442 |
+
if (use_cache and cfg.commit_kv and past_key_values is not None
|
| 443 |
+
and executed > 1 and forced == 0):
|
| 444 |
+
self._crop_ponder_kv(past_key_values, W)
|
| 445 |
+
if not win_mask_built: # W==Q 的场景
|
| 446 |
+
self._run_block(h_final, causal_mask, position_ids,
|
| 447 |
+
position_embeddings, past_key_values, kwargs)
|
| 448 |
+
else:
|
| 449 |
+
self._run_block(h_final[:, -W:, :], mask_win, position_ids[:, -W:],
|
| 450 |
+
pos_emb_win, past_key_values, kwargs)
|
| 451 |
+
# commit 输出丢弃, KV 由 mix 重新计算 —— 后续 token 看到的就是思考后的表示
|
| 452 |
+
|
| 453 |
+
# ---- logits / loss ----------------------------------------------
|
| 454 |
+
slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
|
| 455 |
+
logits = self.lm_head(m.norm(h_final[:, slice_indices, :]))
|
| 456 |
+
|
| 457 |
+
loss, ponder_kl, ponder_step_ce = None, None, None
|
| 458 |
+
if labels is not None:
|
| 459 |
+
ce_logits = logits.float()
|
| 460 |
+
loss = F.cross_entropy(
|
| 461 |
+
ce_logits[..., :-1, :].reshape(-1, ce_logits.size(-1)),
|
| 462 |
+
labels[..., 1:].reshape(-1), ignore_index=-100)
|
| 463 |
+
|
| 464 |
+
# PonderNet 正则: KL(w || Geometric(p_g)), 截断到 K 步并归一化。
|
| 465 |
+
# 必须升到 fp32 再取 log: fp16 下 p=0 处 clamp_min(1e-8) 仍会下溢为 0,
|
| 466 |
+
# log(0)=-inf 与 p=0 相乘直接得到 NaN, 训练瞬间发散
|
| 467 |
+
p = (w_stack / w_stack.sum(-1, keepdim=True).clamp_min(1e-8)).float() # (B,W,K)
|
| 468 |
+
Kd = p.size(-1)
|
| 469 |
+
n_idx = torch.arange(1, Kd + 1, device=p.device, dtype=p.dtype)
|
| 470 |
+
pg = cfg.ponder_prior_p
|
| 471 |
+
geom = (1 - pg) ** (n_idx - 1) * pg
|
| 472 |
+
geom = torch.cat([geom[..., :-1],
|
| 473 |
+
((1 - pg) ** (Kd - 1) * torch.ones_like(geom[..., -1:]))],
|
| 474 |
+
dim=-1)
|
| 475 |
+
geom = geom / geom.sum()
|
| 476 |
+
kl = (p * (torch.log(p.clamp_min(1e-8))
|
| 477 |
+
- torch.log(geom.expand_as(p)))).sum(-1)
|
| 478 |
+
ponder_kl = kl.mean()
|
| 479 |
+
loss = loss + cfg.ponder_loss_beta * ponder_kl
|
| 480 |
+
|
| 481 |
+
# 方向②: per-step CE 聚合 —— 每步 CE 按该步 w 质量 (detach) 加权,
|
| 482 |
+
# 让被分配到概率质量的思考步都保持"可解码", 回收 mix 摊薄的 CE 代价
|
| 483 |
+
if use_step_ce and step_pairs:
|
| 484 |
+
ws = torch.stack([p[0] for p in step_pairs], dim=0) # (K,B,W)
|
| 485 |
+
ces = torch.stack([p[1] for p in step_pairs], dim=0) # (K,)
|
| 486 |
+
wsum = ws.sum(0).clamp_min(1e-8) # (B,W)
|
| 487 |
+
per_pos = (ws * ces.view(-1, 1, 1)).sum(0) / wsum # (B,W)
|
| 488 |
+
ponder_step_ce = per_pos.mean()
|
| 489 |
+
loss = loss + step_ce_weight * ponder_step_ce
|
| 490 |
+
|
| 491 |
+
# ---- 诊断 --------------------------------------------------------
|
| 492 |
+
cum_w = torch.cumsum(w_stack, dim=-1)
|
| 493 |
+
steps_used = ((cum_w < 1.0 - eps).sum(-1) + 1).clamp(max=w_stack.size(-1)) # (B, W)
|
| 494 |
+
self._log({
|
| 495 |
+
"mode": "train" if self.training else ("decode" if Q == 1 else "prefill"),
|
| 496 |
+
"signal": cfg.ponder_signal,
|
| 497 |
+
"executed": executed,
|
| 498 |
+
"steps_max": int(steps_used.max()),
|
| 499 |
+
"steps_mean": float(steps_used.float().mean()),
|
| 500 |
+
})
|
| 501 |
+
|
| 502 |
+
return PonderCausalLMOutputWithPast(
|
| 503 |
+
loss=loss,
|
| 504 |
+
logits=logits,
|
| 505 |
+
past_key_values=past_key_values,
|
| 506 |
+
ponder_weights=w_stack if output_ponder else None,
|
| 507 |
+
ponder_steps=steps_used if output_ponder else None,
|
| 508 |
+
ponder_kl=ponder_kl,
|
| 509 |
+
ponder_step_ce=ponder_step_ce,
|
| 510 |
+
)
|
| 511 |
+
|
| 512 |
+
|
| 513 |
+
# --------------------------------------------------------------------------
|
| 514 |
+
# AutoClass 注册 (可重复 import 安全)
|
| 515 |
+
# --------------------------------------------------------------------------
|
| 516 |
+
try:
|
| 517 |
+
AutoConfig.register("llama_ponder", PonderLlamaConfig)
|
| 518 |
+
AutoModelForCausalLM.register(PonderLlamaConfig, PonderLlamaForCausalLM)
|
| 519 |
+
except Exception:
|
| 520 |
+
pass
|
| 521 |
+
|
| 522 |
+
|
| 523 |
+
# --------------------------------------------------------------------------
|
| 524 |
+
# Config
|
| 525 |
+
# --------------------------------------------------------------------------
|
| 526 |
+
class PonderMoELlamaConfig(PonderLlamaConfig):
|
| 527 |
+
"""PonderLlamaConfig + 动态 MoE / MTP 超参. moe_num_experts=0 且 mtp_num_tokens=0
|
| 528 |
+
时行为与父类完全一致 (回归安全)."""
|
| 529 |
+
|
| 530 |
+
model_type = "llama_ponder_moe"
|
| 531 |
+
|
| 532 |
+
def __init__(self, **kwargs):
|
| 533 |
+
# ---- 动态可变 MoE ----
|
| 534 |
+
self.moe_num_experts = int(kwargs.pop("moe_num_experts", 0) or 0) # 0=关
|
| 535 |
+
self.moe_route_tau = kwargs.pop("moe_route_tau", None) # 激活阈值, None -> 1/N
|
| 536 |
+
self.moe_aux_weight = float(kwargs.pop("moe_aux_weight", 0.01)) # 负载均衡权重
|
| 537 |
+
self.moe_scope = kwargs.pop("moe_scope", "ponder") # ponder|all: 拆哪些层的 FFN
|
| 538 |
+
# ---- MTP ----
|
| 539 |
+
self.mtp_num_tokens = int(kwargs.pop("mtp_num_tokens", 0) or 0) # 深度 D, 0=关
|
| 540 |
+
self.mtp_loss_weight = float(kwargs.pop("mtp_loss_weight", 1.0))
|
| 541 |
+
super().__init__(**kwargs)
|
| 542 |
+
# 普通属性而非 property (transformers 5.16 config __getattribute__ 钩子不支持 property)
|
| 543 |
+
self.moe_tau = (float(self.moe_route_tau) if self.moe_route_tau
|
| 544 |
+
else 1.0 / max(1, self.moe_num_experts))
|
| 545 |
+
|
| 546 |
+
@classmethod
|
| 547 |
+
def from_llama(cls, llama_config: LlamaConfig, **kwargs) -> "PonderMoELlamaConfig":
|
| 548 |
+
"""同父类, 但额外吞掉 moe_*/mtp_* 旧键, 兼容本类保存的 config.json."""
|
| 549 |
+
d = llama_config.to_dict()
|
| 550 |
+
for k in ("model_type", "architectures", "_name_or_path", "transformers_version",
|
| 551 |
+
"torch_dtype"):
|
| 552 |
+
d.pop(k, None)
|
| 553 |
+
drop = [k for k in d
|
| 554 |
+
if "ponder" in k or k.startswith(("moe_", "mtp_"))
|
| 555 |
+
or k in ("commit_kv", "drift_threshold")]
|
| 556 |
+
for k in drop:
|
| 557 |
+
if k in kwargs:
|
| 558 |
+
d.pop(k)
|
| 559 |
+
return cls(**d, **kwargs)
|
| 560 |
+
|
| 561 |
+
|
| 562 |
+
@dataclass
|
| 563 |
+
class PonderMoECausalLMOutputWithPast(PonderCausalLMOutputWithPast):
|
| 564 |
+
"""追加 MoE / MTP 诊断字段."""
|
| 565 |
+
moe_aux_loss: torch.Tensor | None = None # 标量, 负载均衡损失 (各 MoE 层均值)
|
| 566 |
+
moe_active_mean: torch.Tensor | None = None # 标量, 平均激活专家数 ∈ [1, N]
|
| 567 |
+
mtp_loss: torch.Tensor | None = None # 标量, 各深度 MTP CE 均值
|
| 568 |
+
mtp_acc: torch.Tensor | None = None # 标量, MTP top-1 命中率 (诊断)
|
| 569 |
+
|
| 570 |
+
|
| 571 |
+
# --------------------------------------------------------------------------
|
| 572 |
+
# 同构专家: 原 FFN 的 1/N 宽切片 (gate/up 按行切, down 按列切), 参数守恒
|
| 573 |
+
# --------------------------------------------------------------------------
|
| 574 |
+
class ExpertMLP(nn.Module):
|
| 575 |
+
def __init__(self, hidden_size: int, inter_size: int, act_fn):
|
| 576 |
+
super().__init__()
|
| 577 |
+
self.gate_proj = nn.Linear(hidden_size, inter_size, bias=False)
|
| 578 |
+
self.up_proj = nn.Linear(hidden_size, inter_size, bias=False)
|
| 579 |
+
self.down_proj = nn.Linear(inter_size, hidden_size, bias=False)
|
| 580 |
+
self.act_fn = act_fn
|
| 581 |
+
|
| 582 |
+
def forward(self, x):
|
| 583 |
+
return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
|
| 584 |
+
|
| 585 |
+
|
| 586 |
+
# --------------------------------------------------------------------------
|
| 587 |
+
# 动态可变 MoE 层: Router(零初始化) + N 同构专家 + 阈值动态激活 + STE + 稀疏执行
|
| 588 |
+
# --------------------------------------------------------------------------
|
| 589 |
+
class PonderMoEMLP(nn.Module):
|
| 590 |
+
def __init__(self, hidden_size: int, n_experts: int, tau: float, act_fn):
|
| 591 |
+
super().__init__()
|
| 592 |
+
self.n_experts = n_experts
|
| 593 |
+
self.tau = float(tau)
|
| 594 |
+
self.router = nn.Linear(hidden_size, n_experts, bias=True)
|
| 595 |
+
nn.init.zeros_(self.router.weight)
|
| 596 |
+
nn.init.zeros_(self.router.bias) # 零路由 -> p=1/N 均匀 -> 初始全激活 = 稠密等价
|
| 597 |
+
self.experts = nn.ModuleList([])
|
| 598 |
+
self.act_fn_holder = act_fn # 仅占位保存引用 (无参数)
|
| 599 |
+
# 运行期状态 (诊断/冻结路由)
|
| 600 |
+
self._last_mask = None # (B, T, N) bool, 最近一次 fresh 路由
|
| 601 |
+
self._last_p = None # (B, T, N) float, detach 后的路由概率
|
| 602 |
+
self.last_aux = None # 标量 tensor (fresh 时更新, 模型聚合)
|
| 603 |
+
self.last_active = None # 标量 tensor, 平均激活专家数
|
| 604 |
+
self.n_route_calls = 0 # 结构性验证: Router 实际前向次数
|
| 605 |
+
self.cache_hook = None # SSD-LRU 挂钩: callable(expert_ids)
|
| 606 |
+
self.cache_post_hook = None # SSD-LRU 执行后挂钩: 强制容量硬上界
|
| 607 |
+
|
| 608 |
+
# ---------------- 构造 ---------------- #
|
| 609 |
+
@classmethod
|
| 610 |
+
def from_dense(cls, dense, n_experts: int, tau: float | None = None) -> "PonderMoEMLP":
|
| 611 |
+
"""把稠密 LlamaMLP 拆成 n_experts 个同构专家 (参数守恒) + 零 Router."""
|
| 612 |
+
H, I = dense.hidden_size, dense.intermediate_size
|
| 613 |
+
assert I % n_experts == 0, f"intermediate_size({I}) 必须被 N({n_experts}) 整除"
|
| 614 |
+
m = cls(H, n_experts, tau if tau is not None else 1.0 / n_experts, dense.act_fn)
|
| 615 |
+
sub = I // n_experts
|
| 616 |
+
with torch.no_grad():
|
| 617 |
+
for i in range(n_experts):
|
| 618 |
+
ex = ExpertMLP(H, sub, dense.act_fn)
|
| 619 |
+
sl = slice(i * sub, (i + 1) * sub)
|
| 620 |
+
ex.gate_proj.weight.copy_(dense.gate_proj.weight[sl])
|
| 621 |
+
ex.up_proj.weight.copy_(dense.up_proj.weight[sl])
|
| 622 |
+
ex.down_proj.weight.copy_(dense.down_proj.weight[:, sl])
|
| 623 |
+
m.experts.append(ex)
|
| 624 |
+
return m
|
| 625 |
+
|
| 626 |
+
@property
|
| 627 |
+
def device(self):
|
| 628 |
+
return self.router.weight.device
|
| 629 |
+
|
| 630 |
+
# ---------------- 路由 ---------------- #
|
| 631 |
+
def _route(self, x_flat: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
|
| 632 |
+
"""fresh 路由: 返回 (p, mask), 形状 (T, N). 零初始化下 p=1/N -> 全激活."""
|
| 633 |
+
logits = self.router(x_flat)
|
| 634 |
+
p = torch.softmax(logits.float(), dim=-1).to(x_flat.dtype)
|
| 635 |
+
mask = p >= (self.tau - 1e-6)
|
| 636 |
+
empty = ~mask.any(-1)
|
| 637 |
+
if bool(empty.any()): # 无一过阈 -> top-1 保底
|
| 638 |
+
top = p.argmax(-1)
|
| 639 |
+
mask = mask.clone()
|
| 640 |
+
mask[empty, top[empty]] = True
|
| 641 |
+
return p, mask
|
| 642 |
+
|
| 643 |
+
def _experts_sparse(self, x_flat, mask, p, with_ste: bool) -> torch.Tensor:
|
| 644 |
+
"""稀疏执行: 只计算被激活的 (token, expert) 对.
|
| 645 |
+
前向: 激活专家求和 (权重 1, 激活参数量语义)
|
| 646 |
+
反向: STE —— 经 p 软通路把梯度送回 Router (with_ste=True 时)."""
|
| 647 |
+
out = torch.zeros_like(x_flat)
|
| 648 |
+
dtype = x_flat.dtype
|
| 649 |
+
active_ids = [i for i in range(len(self.experts)) if bool(mask[:, i].any())]
|
| 650 |
+
if (not self.training) and self.cache_hook is not None and active_ids:
|
| 651 |
+
self.cache_hook(active_ids) # SSD-LRU: 先搬入后执行 (GPU 上必须前置)
|
| 652 |
+
for i in active_ids:
|
| 653 |
+
ex = self.experts[i]
|
| 654 |
+
sel = mask[:, i].nonzero(as_tuple=True)[0]
|
| 655 |
+
e = ex(x_flat.index_select(0, sel))
|
| 656 |
+
if with_ste:
|
| 657 |
+
w = p.index_select(0, sel).select(1, i) # (S,)
|
| 658 |
+
w = (w - w.detach() + 1.0).to(dtype) # 前向=1, 反向 d/dp=1
|
| 659 |
+
else:
|
| 660 |
+
w = torch.ones((), device=e.device, dtype=dtype)
|
| 661 |
+
out = out.index_add(0, sel, e * w.unsqueeze(-1))
|
| 662 |
+
if (not self.training) and self.cache_post_hook is not None and active_ids:
|
| 663 |
+
self.cache_post_hook(active_ids) # 执行完毕: 超容部分按 LRU 驱逐 (含活跃)
|
| 664 |
+
return out
|
| 665 |
+
|
| 666 |
+
# ---------------- 前向 ---------------- #
|
| 667 |
+
def forward(self, x: torch.Tensor, route_mode: str = "fresh") -> torch.Tensor:
|
| 668 |
+
B, T, H = x.shape
|
| 669 |
+
x_flat = x.reshape(B * T, H)
|
| 670 |
+
|
| 671 |
+
if route_mode == "all": # 调试/等价验证: 全专家权重 1
|
| 672 |
+
if (not self.training) and self.cache_hook is not None:
|
| 673 |
+
self.cache_hook(list(range(len(self.experts))))
|
| 674 |
+
out = torch.zeros_like(x_flat)
|
| 675 |
+
for ex in self.experts:
|
| 676 |
+
out = out + ex(x_flat)
|
| 677 |
+
if (not self.training) and self.cache_post_hook is not None:
|
| 678 |
+
self.cache_post_hook(list(range(len(self.experts))))
|
| 679 |
+
return out.view(B, T, H)
|
| 680 |
+
|
| 681 |
+
if route_mode == "frozen" and self._last_mask is not None:
|
| 682 |
+
# 单次路由铁律: 复用最近一次 fresh 路由 (不触碰 Router, 不产生新路由)
|
| 683 |
+
mask = self._last_mask[:, -T:, :].reshape(B * T, -1)
|
| 684 |
+
out = self._experts_sparse(x_flat, mask, None, with_ste=False)
|
| 685 |
+
return out.view(B, T, H)
|
| 686 |
+
|
| 687 |
+
# ---- fresh 路由 ----
|
| 688 |
+
self.n_route_calls += 1
|
| 689 |
+
p, mask = self._route(x_flat)
|
| 690 |
+
self._last_mask = mask.detach().view(B, T, -1)
|
| 691 |
+
self._last_p = p.detach().view(B, T, -1)
|
| 692 |
+
out = self._experts_sparse(x_flat, mask, p, with_ste=self.training or p.requires_grad)
|
| 693 |
+
|
| 694 |
+
# 负载均衡 (Switch 式): aux = N * Σ f_i * P_i
|
| 695 |
+
with torch.no_grad():
|
| 696 |
+
f = mask.float().mean(0) # (N,) 激活频率
|
| 697 |
+
P = p.float().mean(0) # (N,) 平均概率
|
| 698 |
+
self.last_aux = self.n_experts * (f * P).sum()
|
| 699 |
+
self.last_active = mask.float().sum(-1).mean()
|
| 700 |
+
return out.view(B, T, H)
|
| 701 |
+
|
| 702 |
+
|
| 703 |
+
# --------------------------------------------------------------------------
|
| 704 |
+
# 自定义 DecoderLayer: 与 LlamaDecoderLayer 完全同构, 仅把 mlp 换成 PonderMoEMLP
|
| 705 |
+
# 并支持 moe_route_mode 透传 (fresh=新路由 / frozen=复用 / all=全激活)
|
| 706 |
+
# --------------------------------------------------------------------------
|
| 707 |
+
class PonderMoEDecoderLayer(nn.Module):
|
| 708 |
+
def __init__(self, config, layer_idx: int | None, moe_mlp: PonderMoEMLP):
|
| 709 |
+
super().__init__()
|
| 710 |
+
self.hidden_size = config.hidden_size
|
| 711 |
+
self.layer_idx = layer_idx
|
| 712 |
+
# 注意: 只注册 self.mlp 一个名字 (注册两个会产出重复 state_dict 键)
|
| 713 |
+
self.mlp = moe_mlp
|
| 714 |
+
|
| 715 |
+
@classmethod
|
| 716 |
+
def convert(cls, old_layer: nn.Module, config, n_experts: int, tau: float) -> "PonderMoEDecoderLayer":
|
| 717 |
+
"""接管一个现成 LlamaDecoderLayer 的全部子模块, 仅替换 mlp."""
|
| 718 |
+
new = cls(config, getattr(old_layer, "layer_idx", None),
|
| 719 |
+
PonderMoEMLP.from_dense(old_layer.mlp, n_experts, tau))
|
| 720 |
+
new.self_attn = old_layer.self_attn
|
| 721 |
+
new.input_layernorm = old_layer.input_layernorm
|
| 722 |
+
new.post_attention_layernorm = old_layer.post_attention_layernorm
|
| 723 |
+
return new
|
| 724 |
+
|
| 725 |
+
def forward(
|
| 726 |
+
self,
|
| 727 |
+
hidden_states: torch.Tensor,
|
| 728 |
+
attention_mask: torch.Tensor | None = None,
|
| 729 |
+
position_ids: torch.LongTensor | None = None,
|
| 730 |
+
past_key_values=None,
|
| 731 |
+
use_cache: bool = False,
|
| 732 |
+
position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
|
| 733 |
+
moe_route_mode: str = "fresh",
|
| 734 |
+
**kwargs,
|
| 735 |
+
) -> torch.Tensor:
|
| 736 |
+
residual = hidden_states
|
| 737 |
+
hidden_states = self.input_layernorm(hidden_states)
|
| 738 |
+
hidden_states, _ = self.self_attn(
|
| 739 |
+
hidden_states=hidden_states,
|
| 740 |
+
attention_mask=attention_mask,
|
| 741 |
+
position_ids=position_ids,
|
| 742 |
+
past_key_values=past_key_values,
|
| 743 |
+
use_cache=use_cache,
|
| 744 |
+
position_embeddings=position_embeddings,
|
| 745 |
+
**kwargs,
|
| 746 |
+
)
|
| 747 |
+
hidden_states = residual + hidden_states
|
| 748 |
+
|
| 749 |
+
residual = hidden_states
|
| 750 |
+
hidden_states = self.post_attention_layernorm(hidden_states)
|
| 751 |
+
hidden_states = self.mlp(hidden_states, route_mode=moe_route_mode)
|
| 752 |
+
hidden_states = residual + hidden_states
|
| 753 |
+
return hidden_states
|
| 754 |
+
|
| 755 |
+
|
| 756 |
+
# --------------------------------------------------------------------------
|
| 757 |
+
# MTP 多 Token 预测头 (宪法 §三.4, 训练期增强; 推理自动旁路)
|
| 758 |
+
# 深度 k: 输入 [h_t, e_{t+k}] -> 预测 token t+k+1 (共享主模型 lm_head)
|
| 759 |
+
# --------------------------------------------------------------------------
|
| 760 |
+
class PonderMTPHead(nn.Module):
|
| 761 |
+
def __init__(self, config, depth: int):
|
| 762 |
+
super().__init__()
|
| 763 |
+
H = config.hidden_size
|
| 764 |
+
self.depth = depth # 该头负责预测 t+depth+1
|
| 765 |
+
self.norm_h = LlamaRMSNorm(H, eps=config.rms_norm_eps)
|
| 766 |
+
self.norm_e = LlamaRMSNorm(H, eps=config.rms_norm_eps)
|
| 767 |
+
self.proj = nn.Linear(2 * H, H, bias=False)
|
| 768 |
+
self.act = nn.GELU()
|
| 769 |
+
self.out = nn.Linear(H, H, bias=False)
|
| 770 |
+
self.norm_out = LlamaRMSNorm(H, eps=config.rms_norm_eps)
|
| 771 |
+
|
| 772 |
+
def forward(self, h: torch.Tensor, e: torch.Tensor) -> torch.Tensor:
|
| 773 |
+
"""h, e: (B, L', H) 已对齐 —— 位置 t 处为 h_t 与 e_{t+depth}."""
|
| 774 |
+
x = self.proj(torch.cat([self.norm_h(h), self.norm_e(e)], dim=-1))
|
| 775 |
+
return self.norm_out(self.out(self.act(x)))
|
| 776 |
+
|
| 777 |
+
|
| 778 |
+
# --------------------------------------------------------------------------
|
| 779 |
+
# SSD-LRU 专家缓存调度原型 (宪法 §二.4 推理策略层)
|
| 780 |
+
# --------------------------------------------------------------------------
|
| 781 |
+
class SSDLRUExpertManager:
|
| 782 |
+
"""专家权重驻留 storage (默认 CPU, 模拟 SSD), 被路由激活时搬入计算设备,
|
| 783 |
+
容量满按 LRU 驱逐. 统计验证"单次路由 => 每 token 每层恰 1 次 miss,
|
| 784 |
+
Ponder K 轮迭代与 commit 全命中" —— 缓存不击穿.
|
| 785 |
+
仅推理期可用 (训练期交换 .data 会破坏 autograd, install 时自动排除)."""
|
| 786 |
+
|
| 787 |
+
def __init__(self, model: "PonderMoELlamaForCausalLM", capacity_per_layer: int | None = None,
|
| 788 |
+
storage_device: str = "cpu"):
|
| 789 |
+
self.model = model
|
| 790 |
+
self.storage_device = storage_device
|
| 791 |
+
self.device = next(model.parameters()).device # 捕获主设备 (搬移前)
|
| 792 |
+
self.layers: dict[int, PonderMoEMLP] = {}
|
| 793 |
+
self.order: dict[int, OrderedDict] = {}
|
| 794 |
+
self.capacity: dict[int, int] = {}
|
| 795 |
+
self.stats: dict[int, dict] = {}
|
| 796 |
+
cfg = model.config
|
| 797 |
+
N = int(cfg.moe_num_experts)
|
| 798 |
+
for li, layer in enumerate(model.model.layers):
|
| 799 |
+
if isinstance(layer, PonderMoEDecoderLayer):
|
| 800 |
+
self.layers[li] = layer.mlp
|
| 801 |
+
self.order[li] = OrderedDict()
|
| 802 |
+
self.capacity[li] = int(capacity_per_layer or N)
|
| 803 |
+
self.stats[li] = {"hit": 0, "miss": 0, "evict": 0, "fetch": 0, "calls": 0,
|
| 804 |
+
"miss_calls": 0}
|
| 805 |
+
|
| 806 |
+
@classmethod
|
| 807 |
+
def install(cls, model, capacity_per_layer: int | None = None,
|
| 808 |
+
storage_device: str = "cpu") -> "SSDLRUExpertManager":
|
| 809 |
+
assert not model.training, "SSD-LRU 仅用于推理 (训练期禁止权重搬运)"
|
| 810 |
+
mgr = cls(model, capacity_per_layer, storage_device)
|
| 811 |
+
for li, ml in mgr.layers.items():
|
| 812 |
+
ml.cache_hook = mgr._make_hook(li)
|
| 813 |
+
ml.cache_post_hook = mgr._make_post_hook(li)
|
| 814 |
+
ml.experts.to(storage_device) # 全部专家下放 storage
|
| 815 |
+
return mgr
|
| 816 |
+
|
| 817 |
+
def _make_hook(self, li: int):
|
| 818 |
+
def hook(active_ids: list[int]):
|
| 819 |
+
st, order = self.stats[li], self.order[li]
|
| 820 |
+
st["calls"] += 1 # 访问批次 (含纯命中批)
|
| 821 |
+
had_miss = False
|
| 822 |
+
# 第一遍: 本次活跃专家全部搬入计算设备 (先保证执行正确性)
|
| 823 |
+
for eid in active_ids:
|
| 824 |
+
if eid in order:
|
| 825 |
+
st["hit"] += 1
|
| 826 |
+
order.move_to_end(eid)
|
| 827 |
+
else:
|
| 828 |
+
st["miss"] += 1
|
| 829 |
+
had_miss = True
|
| 830 |
+
order[eid] = True
|
| 831 |
+
self._move(li, eid, self.device)
|
| 832 |
+
st["fetch"] += 1
|
| 833 |
+
# 第二遍: 超容时按 LRU 驱逐, 但绝不驱逐本次活跃专家
|
| 834 |
+
# (工作集 > 容量时允许缓存暂时超容, 执行正确性优先于驱逐压力)
|
| 835 |
+
excess = len(order) - self.capacity[li]
|
| 836 |
+
for k in list(order.keys()):
|
| 837 |
+
if excess <= 0:
|
| 838 |
+
break
|
| 839 |
+
if k in active_ids:
|
| 840 |
+
continue
|
| 841 |
+
order.pop(k)
|
| 842 |
+
self._move(li, k, self.storage_device)
|
| 843 |
+
st["evict"] += 1
|
| 844 |
+
excess -= 1
|
| 845 |
+
if had_miss:
|
| 846 |
+
st["miss_calls"] += 1 # 击穿判据: 实际产生 IO 的批次
|
| 847 |
+
return hook
|
| 848 |
+
|
| 849 |
+
def _make_post_hook(self, li: int):
|
| 850 |
+
def post_hook(active_ids: list[int]):
|
| 851 |
+
st, order = self.stats[li], self.order[li]
|
| 852 |
+
# 物理容量硬上界: 执行完毕后超容部分按 LRU 驱逐
|
| 853 |
+
# (此时专家已用完, 驱逐含本次活跃专家也安全; 欠配 => 下批重新 miss = 真实 thrash)
|
| 854 |
+
while len(order) > self.capacity[li]:
|
| 855 |
+
old, _ = order.popitem(last=False)
|
| 856 |
+
self._move(li, old, self.storage_device)
|
| 857 |
+
st["evict"] += 1
|
| 858 |
+
return post_hook
|
| 859 |
+
|
| 860 |
+
def _move(self, li: int, eid: int, device: str):
|
| 861 |
+
ex = self.layers[li].experts[eid]
|
| 862 |
+
for p in ex.parameters():
|
| 863 |
+
p.data = p.data.to(device)
|
| 864 |
+
|
| 865 |
+
def summary(self) -> dict:
|
| 866 |
+
hit = sum(s["hit"] for s in self.stats.values())
|
| 867 |
+
miss = sum(s["miss"] for s in self.stats.values())
|
| 868 |
+
return {
|
| 869 |
+
"per_layer": {li: dict(s) for li, s in self.stats.items()},
|
| 870 |
+
"hit": hit, "miss": miss,
|
| 871 |
+
"evict": sum(s["evict"] for s in self.stats.values()),
|
| 872 |
+
"fetch": sum(s["fetch"] for s in self.stats.values()),
|
| 873 |
+
"calls": sum(s["calls"] for s in self.stats.values()),
|
| 874 |
+
"miss_calls": sum(s["miss_calls"] for s in self.stats.values()),
|
| 875 |
+
"hit_rate": hit / max(1, hit + miss),
|
| 876 |
+
"capacity_per_layer": dict(self.capacity),
|
| 877 |
+
}
|
| 878 |
+
|
| 879 |
+
def remove(self):
|
| 880 |
+
"""卸载挂钩并把专家搬回主设备 (恢复常驻)."""
|
| 881 |
+
for li, ml in self.layers.items():
|
| 882 |
+
ml.cache_hook = None
|
| 883 |
+
ml.cache_post_hook = None
|
| 884 |
+
ml.experts.to(self.device)
|
| 885 |
+
|
| 886 |
+
|
| 887 |
+
# --------------------------------------------------------------------------
|
| 888 |
+
# Model: Ponder (质量) + 动态 MoE (效率) + MTP (吞吐, 训练期)
|
| 889 |
+
# --------------------------------------------------------------------------
|
| 890 |
+
class PonderMoELlamaForCausalLM(PonderLlamaForCausalLM):
|
| 891 |
+
"""
|
| 892 |
+
用法 — 从稠密基座一行构造 (FFN 自动拆专家):
|
| 893 |
+
base = LlamaConfig.from_pretrained("tchbcb/MiniCPM5-2B-cpu")
|
| 894 |
+
cfg = PonderMoELlamaConfig.from_llama(
|
| 895 |
+
base, moe_num_experts=8, mtp_num_tokens=2, ponder_signal="learned")
|
| 896 |
+
model = PonderMoELlamaForCausalLM.from_pretrained(..., config=cfg)
|
| 897 |
+
|
| 898 |
+
moe_num_experts=0 且 mtp_num_tokens=0 时与 PonderLlamaForCausalLM 行为一致。
|
| 899 |
+
"""
|
| 900 |
+
|
| 901 |
+
def __init__(self, config: PonderMoELlamaConfig):
|
| 902 |
+
super().__init__(config)
|
| 903 |
+
cfg = self.config
|
| 904 |
+
self._moe_ready = False
|
| 905 |
+
if int(getattr(cfg, "moe_num_experts", 0) or 0) > 0:
|
| 906 |
+
self._convert_ffn_to_moe() # 在 post_init 之后: Router 零初始化不被覆盖
|
| 907 |
+
D = int(getattr(cfg, "mtp_num_tokens", 0) or 0)
|
| 908 |
+
self.mtp_heads = (nn.ModuleList([PonderMTPHead(cfg, k + 1) for k in range(D)])
|
| 909 |
+
if D > 0 else None)
|
| 910 |
+
self._moe_ready = True
|
| 911 |
+
|
| 912 |
+
# ------------------------------------------------------------------ #
|
| 913 |
+
# 结构改造
|
| 914 |
+
# ------------------------------------------------------------------ #
|
| 915 |
+
def _convert_ffn_to_moe(self):
|
| 916 |
+
cfg = self.config
|
| 917 |
+
N = int(cfg.moe_num_experts)
|
| 918 |
+
assert cfg.intermediate_size % N == 0, \
|
| 919 |
+
f"intermediate_size({cfg.intermediate_size}) 须被 moe_num_experts({N}) 整除"
|
| 920 |
+
tau = cfg.moe_tau
|
| 921 |
+
s = cfg.ponder_start_layer
|
| 922 |
+
ids = (range(s, cfg.num_hidden_layers) if cfg.moe_scope == "ponder"
|
| 923 |
+
else range(cfg.num_hidden_layers))
|
| 924 |
+
for li in ids:
|
| 925 |
+
self.model.layers[li] = PonderMoEDecoderLayer.convert(
|
| 926 |
+
self.model.layers[li], cfg, N, tau)
|
| 927 |
+
|
| 928 |
+
def _moe_mlps(self) -> list[PonderMoEMLP]:
|
| 929 |
+
return ([l.mlp for l in self.model.layers if isinstance(l, PonderMoEDecoderLayer)]
|
| 930 |
+
if self._moe_ready else [])
|
| 931 |
+
|
| 932 |
+
def _reset_moe_stats(self):
|
| 933 |
+
for ml in self._moe_mlps():
|
| 934 |
+
ml.last_aux = None
|
| 935 |
+
ml.last_active = None
|
| 936 |
+
|
| 937 |
+
def _collect_moe_stats(self):
|
| 938 |
+
auxs = [ml.last_aux for ml in self._moe_mlps() if ml.last_aux is not None]
|
| 939 |
+
acts = [ml.last_active for ml in self._moe_mlps() if ml.last_active is not None]
|
| 940 |
+
aux = torch.stack(auxs).mean() if auxs else None
|
| 941 |
+
act = torch.stack(acts).mean() if acts else None
|
| 942 |
+
return aux, act
|
| 943 |
+
|
| 944 |
+
# ------------------------------------------------------------------ #
|
| 945 |
+
# MTP (训练/评估且给定 labels 时; generate 自动旁路)
|
| 946 |
+
# ------------------------------------------------------------------ #
|
| 947 |
+
def _mtp_forward(self, h_final, input_ids, labels):
|
| 948 |
+
m = self.model
|
| 949 |
+
B, L, _ = h_final.shape
|
| 950 |
+
emb = m.embed_tokens(input_ids)
|
| 951 |
+
losses, accs = [], []
|
| 952 |
+
for head in self.mtp_heads:
|
| 953 |
+
k = head.depth
|
| 954 |
+
if L - k < 2:
|
| 955 |
+
continue
|
| 956 |
+
h = h_final[:, :L - k, :] # h_t, t=0..L-k-1
|
| 957 |
+
e = emb[:, k:, :] # e_{t+k}
|
| 958 |
+
hh = head(h, e) # (B, L-k, H)
|
| 959 |
+
lg = self.lm_head(hh)[:, :-1, :] # 预测 t+k+1
|
| 960 |
+
tg = labels[:, k + 1:]
|
| 961 |
+
ce = F.cross_entropy(lg.reshape(-1, lg.size(-1)).float(),
|
| 962 |
+
tg.reshape(-1), ignore_index=-100)
|
| 963 |
+
if torch.isfinite(ce):
|
| 964 |
+
losses.append(ce)
|
| 965 |
+
with torch.no_grad():
|
| 966 |
+
valid = tg != -100
|
| 967 |
+
if bool(valid.any()):
|
| 968 |
+
accs.append((((lg.argmax(-1) == tg) & valid).sum().float()
|
| 969 |
+
/ valid.sum()).detach())
|
| 970 |
+
mtp_loss = torch.stack(losses).mean() if losses else None
|
| 971 |
+
mtp_acc = torch.stack(accs).mean() if accs else None
|
| 972 |
+
return mtp_loss, mtp_acc
|
| 973 |
+
|
| 974 |
+
# ------------------------------------------------------------------ #
|
| 975 |
+
# 思考块执行 (带路由模式: 第 1 轮 fresh, 之后 frozen —— 单次路由铁律)
|
| 976 |
+
# ------------------------------------------------------------------ #
|
| 977 |
+
def _run_block(self, h, attention_mask, position_ids, position_embeddings,
|
| 978 |
+
cache, kwargs, route_mode: str = "fresh"):
|
| 979 |
+
ckpt_fn = getattr(self.model, "_gradient_checkpointing_func", None)
|
| 980 |
+
use_ckpt = (self.training and cache is None
|
| 981 |
+
and getattr(self.model, "gradient_checkpointing", False)
|
| 982 |
+
and ckpt_fn is not None)
|
| 983 |
+
for layer in self._ponder_layers:
|
| 984 |
+
lkw = dict(kwargs)
|
| 985 |
+
if isinstance(layer, PonderMoEDecoderLayer):
|
| 986 |
+
lkw["moe_route_mode"] = route_mode
|
| 987 |
+
if use_ckpt:
|
| 988 |
+
h = ckpt_fn(
|
| 989 |
+
layer.__call__, h,
|
| 990 |
+
attention_mask=attention_mask,
|
| 991 |
+
position_ids=position_ids,
|
| 992 |
+
position_embeddings=position_embeddings,
|
| 993 |
+
past_key_values=None,
|
| 994 |
+
use_cache=False,
|
| 995 |
+
**lkw,
|
| 996 |
+
)
|
| 997 |
+
else:
|
| 998 |
+
h = layer(
|
| 999 |
+
h,
|
| 1000 |
+
attention_mask=attention_mask,
|
| 1001 |
+
position_ids=position_ids,
|
| 1002 |
+
position_embeddings=position_embeddings,
|
| 1003 |
+
past_key_values=cache,
|
| 1004 |
+
use_cache=cache is not None,
|
| 1005 |
+
**lkw,
|
| 1006 |
+
)
|
| 1007 |
+
return h
|
| 1008 |
+
|
| 1009 |
+
# ------------------------------------------------------------------ #
|
| 1010 |
+
# 主前向 (父类结构 + 路由模式 + MoE/MTP 损失)
|
| 1011 |
+
# ------------------------------------------------------------------ #
|
| 1012 |
+
@can_return_tuple
|
| 1013 |
+
def forward(
|
| 1014 |
+
self,
|
| 1015 |
+
input_ids: torch.LongTensor | None = None,
|
| 1016 |
+
attention_mask: torch.Tensor | None = None,
|
| 1017 |
+
position_ids: torch.LongTensor | None = None,
|
| 1018 |
+
past_key_values: DynamicCache | None = None,
|
| 1019 |
+
inputs_embeds: torch.FloatTensor | None = None,
|
| 1020 |
+
labels: torch.LongTensor | None = None,
|
| 1021 |
+
use_cache: bool | None = None,
|
| 1022 |
+
logits_to_keep: int | torch.Tensor = 0,
|
| 1023 |
+
output_ponder: bool = False,
|
| 1024 |
+
**kwargs,
|
| 1025 |
+
) -> PonderMoECausalLMOutputWithPast:
|
| 1026 |
+
cfg = self.config
|
| 1027 |
+
m = self.model
|
| 1028 |
+
self._reset_moe_stats()
|
| 1029 |
+
|
| 1030 |
+
# ---- 输入 & 位置 ------------------------------------------------
|
| 1031 |
+
if (input_ids is None) ^ (inputs_embeds is not None):
|
| 1032 |
+
raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
|
| 1033 |
+
if inputs_embeds is None:
|
| 1034 |
+
inputs_embeds = m.embed_tokens(input_ids)
|
| 1035 |
+
B, Q = inputs_embeds.shape[:2]
|
| 1036 |
+
|
| 1037 |
+
if use_cache is None:
|
| 1038 |
+
use_cache = False if self.training else cfg.use_cache
|
| 1039 |
+
if use_cache and past_key_values is None:
|
| 1040 |
+
past_key_values = DynamicCache(config=cfg)
|
| 1041 |
+
|
| 1042 |
+
cache_position = kwargs.pop("cache_position", None)
|
| 1043 |
+
past_seen = past_key_values.get_seq_length() if past_key_values is not None else 0
|
| 1044 |
+
if position_ids is None:
|
| 1045 |
+
base = (cache_position if cache_position is not None
|
| 1046 |
+
else torch.arange(Q, device=inputs_embeds.device) + past_seen)
|
| 1047 |
+
position_ids = base.reshape(1, -1).expand(B, -1)[:, :Q].to(inputs_embeds.device)
|
| 1048 |
+
|
| 1049 |
+
# ---- mask / rope -------------------------------------------------
|
| 1050 |
+
causal_mask = create_causal_mask(
|
| 1051 |
+
config=cfg, inputs_embeds=inputs_embeds, attention_mask=attention_mask,
|
| 1052 |
+
past_key_values=past_key_values, position_ids=position_ids,
|
| 1053 |
+
)
|
| 1054 |
+
position_embeddings = m.rotary_emb(inputs_embeds, position_ids=position_ids)
|
| 1055 |
+
|
| 1056 |
+
# ---- Stage A: 前段层单遍 (scope=all 时 MoE 层每次调用都是新 token -> fresh) --
|
| 1057 |
+
h = inputs_embeds
|
| 1058 |
+
for layer in m.layers[: cfg.ponder_start_layer]:
|
| 1059 |
+
if isinstance(layer, PonderMoEDecoderLayer):
|
| 1060 |
+
h = layer(h, attention_mask=causal_mask, position_ids=position_ids,
|
| 1061 |
+
position_embeddings=position_embeddings,
|
| 1062 |
+
past_key_values=past_key_values, use_cache=use_cache,
|
| 1063 |
+
moe_route_mode="fresh", **kwargs)
|
| 1064 |
+
else:
|
| 1065 |
+
h = layer(h, attention_mask=causal_mask, position_ids=position_ids,
|
| 1066 |
+
position_embeddings=position_embeddings,
|
| 1067 |
+
past_key_values=past_key_values, use_cache=use_cache, **kwargs)
|
| 1068 |
+
|
| 1069 |
+
# ---- 思考窗口 W --------------------------------------------------
|
| 1070 |
+
if Q == 1 or self.training or cfg.ponder_all_positions:
|
| 1071 |
+
W = Q
|
| 1072 |
+
else:
|
| 1073 |
+
W = max(1, min(cfg.ponder_window, Q))
|
| 1074 |
+
|
| 1075 |
+
eps = cfg.ponder_epsilon
|
| 1076 |
+
K = cfg.max_ponder_steps
|
| 1077 |
+
device, dtype = inputs_embeds.device, inputs_embeds.dtype
|
| 1078 |
+
|
| 1079 |
+
remaining = torch.ones(B, W, device=device, dtype=dtype)
|
| 1080 |
+
cum = torch.zeros(B, W, device=device, dtype=dtype)
|
| 1081 |
+
active = torch.ones(B, W, device=device, dtype=torch.bool)
|
| 1082 |
+
mix = torch.zeros(B, W, h.size(-1), device=device, dtype=dtype)
|
| 1083 |
+
w_list: list[torch.Tensor] = []
|
| 1084 |
+
h_prev_win = None
|
| 1085 |
+
h_full = None
|
| 1086 |
+
h_win = h[:, -W:, :] if W < Q else h
|
| 1087 |
+
executed = 0
|
| 1088 |
+
win_mask_built = False
|
| 1089 |
+
|
| 1090 |
+
forced = int(getattr(cfg, "ponder_forced_steps", 0) or 0)
|
| 1091 |
+
step_ce_weight = float(getattr(cfg, "ponder_step_ce_weight", 0.0) or 0.0)
|
| 1092 |
+
use_step_ce = (step_ce_weight > 0.0 and labels is not None and self.training)
|
| 1093 |
+
step_pairs: list[tuple[torch.Tensor, torch.Tensor]] = []
|
| 1094 |
+
loop_K = forced if forced > 0 else K
|
| 1095 |
+
|
| 1096 |
+
for k in range(1, loop_K + 1):
|
| 1097 |
+
executed = k
|
| 1098 |
+
if k > 1:
|
| 1099 |
+
self._crop_ponder_kv(past_key_values, W)
|
| 1100 |
+
use_window = (k > 1) and (past_key_values is not None) and (W < Q)
|
| 1101 |
+
route_mode = "fresh" if k == 1 else "frozen" # 单次路由铁律
|
| 1102 |
+
if not use_window:
|
| 1103 |
+
h = self._run_block(h, causal_mask, position_ids,
|
| 1104 |
+
position_embeddings, past_key_values, kwargs,
|
| 1105 |
+
route_mode=route_mode)
|
| 1106 |
+
h_win = h if W == Q else h[:, -W:, :]
|
| 1107 |
+
if W < Q:
|
| 1108 |
+
h_full = h
|
| 1109 |
+
else:
|
| 1110 |
+
if not win_mask_built:
|
| 1111 |
+
past_len = int(past_key_values.layers[cfg.ponder_start_layer].get_seq_length())
|
| 1112 |
+
mask_win = self._win_mask(W, past_len, device, dtype, attention_mask)
|
| 1113 |
+
pos_emb_win = (position_embeddings[0][:, -W:],
|
| 1114 |
+
position_embeddings[1][:, -W:])
|
| 1115 |
+
win_mask_built = True
|
| 1116 |
+
h_win = self._run_block(h_win, mask_win, position_ids[:, -W:],
|
| 1117 |
+
pos_emb_win, past_key_values, kwargs,
|
| 1118 |
+
route_mode=route_mode)
|
| 1119 |
+
|
| 1120 |
+
if forced > 0:
|
| 1121 |
+
continue
|
| 1122 |
+
|
| 1123 |
+
lam = self._halting_prob(h_win, h_prev_win)
|
| 1124 |
+
lam_eff = lam * active.to(lam.dtype)
|
| 1125 |
+
w_k = remaining * lam_eff
|
| 1126 |
+
mix = mix + w_k.unsqueeze(-1) * h_win
|
| 1127 |
+
w_list.append(w_k)
|
| 1128 |
+
remaining = remaining * (1.0 - lam_eff)
|
| 1129 |
+
cum = cum + w_k
|
| 1130 |
+
active = active & (cum < 1.0 - eps)
|
| 1131 |
+
h_prev_win = h_win
|
| 1132 |
+
if use_step_ce:
|
| 1133 |
+
ce_k = self._step_ce(h_win, labels)
|
| 1134 |
+
step_pairs.append((w_k.detach(), ce_k))
|
| 1135 |
+
if not bool(active.any()):
|
| 1136 |
+
break
|
| 1137 |
+
|
| 1138 |
+
if forced > 0:
|
| 1139 |
+
if W < Q:
|
| 1140 |
+
h_final = h_full.clone()
|
| 1141 |
+
h_final[:, -W:, :] = h_win
|
| 1142 |
+
else:
|
| 1143 |
+
h_final = h_win
|
| 1144 |
+
w_stack = F.one_hot(
|
| 1145 |
+
torch.full((B, W), forced - 1, device=device), forced
|
| 1146 |
+
).to(dtype)
|
| 1147 |
+
else:
|
| 1148 |
+
mix = mix + remaining.unsqueeze(-1) * h_win
|
| 1149 |
+
w_stack = torch.stack(w_list, dim=-1).to(dtype)
|
| 1150 |
+
w_stack = torch.cat([w_stack[..., :-1],
|
| 1151 |
+
w_stack[..., -1:] + remaining.unsqueeze(-1)], dim=-1)
|
| 1152 |
+
if W < Q:
|
| 1153 |
+
h_final = h_full
|
| 1154 |
+
h_final[:, -W:, :] = mix
|
| 1155 |
+
else:
|
| 1156 |
+
h_final = mix
|
| 1157 |
+
|
| 1158 |
+
# ---- commit pass: 复用 frozen 路由写最终 KV -----------------------
|
| 1159 |
+
if (use_cache and cfg.commit_kv and past_key_values is not None
|
| 1160 |
+
and executed > 1 and forced == 0):
|
| 1161 |
+
self._crop_ponder_kv(past_key_values, W)
|
| 1162 |
+
if not win_mask_built:
|
| 1163 |
+
self._run_block(h_final, causal_mask, position_ids,
|
| 1164 |
+
position_embeddings, past_key_values, kwargs,
|
| 1165 |
+
route_mode="frozen")
|
| 1166 |
+
else:
|
| 1167 |
+
self._run_block(h_final[:, -W:, :], mask_win, position_ids[:, -W:],
|
| 1168 |
+
pos_emb_win, past_key_values, kwargs,
|
| 1169 |
+
route_mode="frozen")
|
| 1170 |
+
|
| 1171 |
+
# ---- logits / loss ----------------------------------------------
|
| 1172 |
+
slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
|
| 1173 |
+
logits = self.lm_head(m.norm(h_final[:, slice_indices, :]))
|
| 1174 |
+
|
| 1175 |
+
loss, ponder_kl, ponder_step_ce = None, None, None
|
| 1176 |
+
if labels is not None:
|
| 1177 |
+
ce_logits = logits.float()
|
| 1178 |
+
loss = F.cross_entropy(
|
| 1179 |
+
ce_logits[..., :-1, :].reshape(-1, ce_logits.size(-1)),
|
| 1180 |
+
labels[..., 1:].reshape(-1), ignore_index=-100)
|
| 1181 |
+
|
| 1182 |
+
# PonderNet KL 正则 (与父类一致, fp32 域防下溢)
|
| 1183 |
+
p = (w_stack / w_stack.sum(-1, keepdim=True).clamp_min(1e-8)).float()
|
| 1184 |
+
Kd = p.size(-1)
|
| 1185 |
+
n_idx = torch.arange(1, Kd + 1, device=p.device, dtype=p.dtype)
|
| 1186 |
+
pg = cfg.ponder_prior_p
|
| 1187 |
+
geom = (1 - pg) ** (n_idx - 1) * pg
|
| 1188 |
+
geom = torch.cat([geom[..., :-1],
|
| 1189 |
+
((1 - pg) ** (Kd - 1) * torch.ones_like(geom[..., -1:]))],
|
| 1190 |
+
dim=-1)
|
| 1191 |
+
geom = geom / geom.sum()
|
| 1192 |
+
kl = (p * (torch.log(p.clamp_min(1e-8))
|
| 1193 |
+
- torch.log(geom.expand_as(p)))).sum(-1)
|
| 1194 |
+
ponder_kl = kl.mean()
|
| 1195 |
+
loss = loss + cfg.ponder_loss_beta * ponder_kl
|
| 1196 |
+
|
| 1197 |
+
if use_step_ce and step_pairs:
|
| 1198 |
+
ws = torch.stack([pp[0] for pp in step_pairs], dim=0)
|
| 1199 |
+
ces = torch.stack([pp[1] for pp in step_pairs], dim=0)
|
| 1200 |
+
wsum = ws.sum(0).clamp_min(1e-8)
|
| 1201 |
+
per_pos = (ws * ces.view(-1, 1, 1)).sum(0) / wsum
|
| 1202 |
+
ponder_step_ce = per_pos.mean()
|
| 1203 |
+
loss = loss + step_ce_weight * ponder_step_ce
|
| 1204 |
+
|
| 1205 |
+
# MoE 负载均衡 (宪法 §六). checkpointing 下 last_aux 无梯度 -> 跳过加项仍报告
|
| 1206 |
+
moe_aux, moe_active = self._collect_moe_stats()
|
| 1207 |
+
if (moe_aux is not None and cfg.moe_aux_weight > 0
|
| 1208 |
+
and torch.is_tensor(moe_aux) and moe_aux.requires_grad):
|
| 1209 |
+
loss = loss + cfg.moe_aux_weight * moe_aux
|
| 1210 |
+
else:
|
| 1211 |
+
moe_aux, moe_active = self._collect_moe_stats()
|
| 1212 |
+
|
| 1213 |
+
# MTP 多 Token 预测损失 (宪法 §六; 训练增强, 推理旁路)
|
| 1214 |
+
mtp_loss = mtp_acc = None
|
| 1215 |
+
if (labels is not None and self.mtp_heads is not None
|
| 1216 |
+
and input_ids is not None):
|
| 1217 |
+
mtp_loss, mtp_acc = self._mtp_forward(h_final, input_ids, labels)
|
| 1218 |
+
if mtp_loss is not None and cfg.mtp_loss_weight > 0:
|
| 1219 |
+
loss = loss + cfg.mtp_loss_weight * mtp_loss
|
| 1220 |
+
|
| 1221 |
+
# ---- 诊断 --------------------------------------------------------
|
| 1222 |
+
cum_w = torch.cumsum(w_stack, dim=-1)
|
| 1223 |
+
steps_used = ((cum_w < 1.0 - eps).sum(-1) + 1).clamp(max=w_stack.size(-1))
|
| 1224 |
+
self._log({
|
| 1225 |
+
"mode": "train" if self.training else ("decode" if Q == 1 else "prefill"),
|
| 1226 |
+
"signal": cfg.ponder_signal,
|
| 1227 |
+
"executed": executed,
|
| 1228 |
+
"steps_max": int(steps_used.max()),
|
| 1229 |
+
"steps_mean": float(steps_used.float().mean()),
|
| 1230 |
+
"moe_active": (float(moe_active) if torch.is_tensor(moe_active) else None),
|
| 1231 |
+
})
|
| 1232 |
+
|
| 1233 |
+
return PonderMoECausalLMOutputWithPast(
|
| 1234 |
+
loss=loss,
|
| 1235 |
+
logits=logits,
|
| 1236 |
+
past_key_values=past_key_values,
|
| 1237 |
+
ponder_weights=w_stack if output_ponder else None,
|
| 1238 |
+
ponder_steps=steps_used if output_ponder else None,
|
| 1239 |
+
ponder_kl=ponder_kl,
|
| 1240 |
+
ponder_step_ce=ponder_step_ce,
|
| 1241 |
+
moe_aux_loss=moe_aux,
|
| 1242 |
+
moe_active_mean=moe_active,
|
| 1243 |
+
mtp_loss=mtp_loss,
|
| 1244 |
+
mtp_acc=mtp_acc,
|
| 1245 |
+
)
|
| 1246 |
+
|
| 1247 |
+
# --------------------------------------------------------------------------
|
| 1248 |
+
# 发布版统一注册与别名 (auto_map 指向别名; 原类名保留以兼容旧代码)
|
| 1249 |
+
# --------------------------------------------------------------------------
|
| 1250 |
+
SamaiPnetConfig = PonderMoELlamaConfig
|
| 1251 |
+
SamaiPnetForCausalLM = PonderMoELlamaForCausalLM
|
| 1252 |
+
|
| 1253 |
+
# transformers 5.x 严格校验: AutoModelForCausalLM.register 要求
|
| 1254 |
+
# model_class.config_class 与实际 config 类一致 (否则 ValueError)
|
| 1255 |
+
PonderLlamaForCausalLM.config_class = PonderLlamaConfig
|
| 1256 |
+
PonderMoELlamaForCausalLM.config_class = PonderMoELlamaConfig
|
| 1257 |
+
|
| 1258 |
+
try: # 重复 import 安全
|
| 1259 |
+
AutoConfig.register("llama_ponder", PonderLlamaConfig)
|
| 1260 |
+
AutoConfig.register("llama_ponder_moe", PonderMoELlamaConfig)
|
| 1261 |
+
AutoModelForCausalLM.register(PonderLlamaConfig, PonderLlamaForCausalLM)
|
| 1262 |
+
AutoModelForCausalLM.register(PonderMoELlamaConfig, PonderMoELlamaForCausalLM)
|
| 1263 |
+
except Exception:
|
| 1264 |
+
pass
|
serve_contract.json
ADDED
|
@@ -0,0 +1,105 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"contract_version": "r5b-2-integrated",
|
| 3 |
+
"declared_at": "2026-09-09",
|
| 4 |
+
"supersedes": "r5b-1",
|
| 5 |
+
"declared_repo_id": "tchbcn/samai-pnet-dmoe-2b",
|
| 6 |
+
"actual_repo_id": "tchbcb/samai-pnet-dmoe-2b",
|
| 7 |
+
"note": "用户指定仓名 tchbcn/samai-pnet-dmoe-2b; 所持 token 实际属主为 tchbcb (whoami 实证, tchbcn 命名空间 403 无权创建), 故以 tchbcb/ 落库, 仓内容与命名意图一致",
|
| 8 |
+
"release": {
|
| 9 |
+
"model_file": "model.safetensors (全量合并: 基座+LoRA+halting头+Router/MTP)",
|
| 10 |
+
"dtype_policy": "主干/专家 fp16; ponder_head fp32 (round4 教训)",
|
| 11 |
+
"trust_remote_code": true,
|
| 12 |
+
"requirements": "torch>=2.4, transformers>=5.0, peft 仅训练侧需要"
|
| 13 |
+
},
|
| 14 |
+
"lineage": {
|
| 15 |
+
"base": "tchbcb/MiniCPM5-2B-cpu (42L/H2048/I6144/GQA16+2/V130560)",
|
| 16 |
+
"convert": "pnet2b_moe: 思考块[34,42) 8层x8同构专家, 参数守恒",
|
| 17 |
+
"round7": "ckpt_2b (宪法四损失基线)",
|
| 18 |
+
"round5B": "ckpt_2b_r5b: per-step CE 0.5 + difficulty prior, init-from ckpt_2b"
|
| 19 |
+
},
|
| 20 |
+
"frozen_structure": {
|
| 21 |
+
"ponder_start_layer": 34,
|
| 22 |
+
"max_ponder_steps": 8,
|
| 23 |
+
"ponder_window": 1,
|
| 24 |
+
"moe_num_experts": 8,
|
| 25 |
+
"moe_scope": "ponder",
|
| 26 |
+
"mtp_num_tokens": 2
|
| 27 |
+
},
|
| 28 |
+
"inference_semantics": {
|
| 29 |
+
"halting": "learned head (fp32, sigmoid), bias 已训练",
|
| 30 |
+
"stop_rule": "cum(w) >= 1 - eps 即停, 残余质量记给最后执行步",
|
| 31 |
+
"epsilon": 0.01,
|
| 32 |
+
"tau": 0.125,
|
| 33 |
+
"route_rule": "每 token 仅第 1 轮迭代 fresh 路由一次, 循环/commit 全程 frozen 复用",
|
| 34 |
+
"moe_activation": "p_i >= tau 激活, 激活数 1~8 自适应, 无一过阈 top-1 保底",
|
| 35 |
+
"kv": "commit_kv=true: 混合表示经 commit pass 写最终 KV",
|
| 36 |
+
"mtp": "训练期增强, 推理 generate() 自动旁路"
|
| 37 |
+
},
|
| 38 |
+
"eval_anchors_90rows_zh_cot": {
|
| 39 |
+
"round7_baseline": {
|
| 40 |
+
"hard": {
|
| 41 |
+
"ce": 4.073070367177327,
|
| 42 |
+
"active": 1.6183046340942382,
|
| 43 |
+
"steps": 2.066666666666667,
|
| 44 |
+
"mtp": 0.5716088751951853,
|
| 45 |
+
"n": 15
|
| 46 |
+
},
|
| 47 |
+
"easy": {
|
| 48 |
+
"ce": 4.018902349472046,
|
| 49 |
+
"active": 1.6024614969889324,
|
| 50 |
+
"steps": 2.0,
|
| 51 |
+
"mtp": 0.5493664860725402,
|
| 52 |
+
"n": 15
|
| 53 |
+
},
|
| 54 |
+
"medium": {
|
| 55 |
+
"ce": 4.228601948420207,
|
| 56 |
+
"active": 1.614480972290039,
|
| 57 |
+
"steps": 2.033333333333333,
|
| 58 |
+
"mtp": 0.5261600712935129,
|
| 59 |
+
"n": 15
|
| 60 |
+
}
|
| 61 |
+
},
|
| 62 |
+
"round5B_live_reference": {
|
| 63 |
+
"hard": {
|
| 64 |
+
"ce": 3.3185231844584147,
|
| 65 |
+
"active": 1.5587466398874918,
|
| 66 |
+
"steps": 2.8666666666666667,
|
| 67 |
+
"mtp": 0.6124512155850729,
|
| 68 |
+
"n": 15
|
| 69 |
+
},
|
| 70 |
+
"easy": {
|
| 71 |
+
"ce": 3.264301141103109,
|
| 72 |
+
"active": 1.5368041038513183,
|
| 73 |
+
"steps": 2.433333333333333,
|
| 74 |
+
"mtp": 0.6108810146649678,
|
| 75 |
+
"n": 15
|
| 76 |
+
},
|
| 77 |
+
"medium": {
|
| 78 |
+
"ce": 3.5550480524698895,
|
| 79 |
+
"active": 1.5593219121297202,
|
| 80 |
+
"steps": 2.6,
|
| 81 |
+
"mtp": 0.5710210422674815,
|
| 82 |
+
"n": 15
|
| 83 |
+
}
|
| 84 |
+
},
|
| 85 |
+
"round5B_merged_release": {
|
| 86 |
+
"hard": {
|
| 87 |
+
"ce": 3.3181373119354247,
|
| 88 |
+
"steps": 2.8666666666666667,
|
| 89 |
+
"n": 15
|
| 90 |
+
},
|
| 91 |
+
"easy": {
|
| 92 |
+
"ce": 3.2644861777623495,
|
| 93 |
+
"steps": 2.433333333333333,
|
| 94 |
+
"n": 15
|
| 95 |
+
},
|
| 96 |
+
"medium": {
|
| 97 |
+
"ce": 3.555168342590332,
|
| 98 |
+
"steps": 2.6,
|
| 99 |
+
"n": 15
|
| 100 |
+
}
|
| 101 |
+
},
|
| 102 |
+
"eval_protocol": "eval_ponder_moe.py 口径: use_cache=False, batch_size=2, fp16, 90 条 zh_cot",
|
| 103 |
+
"merge_note": "干净进程终验 (verify_release.json, trust_remote_code): ΔCE≤0.001, steps 三桶与 live 参考一致 (2.433/2.600/2.867), 分化 +0.433 完整保留; 同进程对分探针曾见 ≤0.065 CE 交叉污染, 已排除"
|
| 104 |
+
}
|
| 105 |
+
}
|
tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"add_prefix_space": null,
|
| 3 |
+
"backend": "tokenizers",
|
| 4 |
+
"bos_token": "<s>",
|
| 5 |
+
"clean_up_tokenization_spaces": false,
|
| 6 |
+
"eos_token": "</s>",
|
| 7 |
+
"is_local": true,
|
| 8 |
+
"legacy": true,
|
| 9 |
+
"local_files_only": false,
|
| 10 |
+
"model_max_length": 131072,
|
| 11 |
+
"pad_token": "</s>",
|
| 12 |
+
"sp_model_kwargs": {},
|
| 13 |
+
"spaces_between_special_tokens": false,
|
| 14 |
+
"tokenizer_class": "TokenizersBackend",
|
| 15 |
+
"unk_token": "<unk>",
|
| 16 |
+
"use_default_system_prompt": false
|
| 17 |
+
}
|