--- license: mit language: - zh - en pipeline_tag: text-generation library_name: transformers tags: - custom_code - safetensors - ymodel - ymodel32 --- *this file is generated by Qwen3.8-27B under human supervision* ## 模型描述 `YModel3.2-fm-1B` 是 YModel3.2 家族的 1B 参数**基础权重模型**(foundation weights)。 ## 模型细节 | 项目 | 值 | | --- | --- | | 架构 | YModel3.2 | | Scale level | 1 | | 参数量 | 1,120,343,553(1.12B,含嵌入)| | Hidden size | 1024 | | 块数 | 31(6 个 FullBlock + 25 个 GroupBlock)| | layer pattern | `fgggggfgggggfgggggfgggggfgggggf` | | 注意力头 / KV 头 | 8 / 2(GQA)| | 头维度 | 256 | | Group size | 8(固定分组)| | FFN | SwiGLU + SEBlock,intermediate 3072 | | DCP | dynamic compression probe(probe hidden 1024);use_dcp=false,分组固定,probe 权重保留 | | Sengram | bucket 16384 / topk 12 | | 词表 | 151,669(Qwen3 tokenizer,tied embeddings)| | 上下文 | 8192 | | 精度 | bfloat16 | ### Group Attn 与 Token Superposition YModel3.2-fm 的 group attn 主要借鉴了论文 **Efficient Pre-Training with Token Superposition**(Bowen Peng, Théo Gigant, Jeffrey Quesnelle, arXiv:2605.06546)的思想:将连续 token 分组(superpose)为一个整体进行高效处理。论文提出的 Token-Superposition Training(TST)把多个连续 token 合并为一个 bag、以 multi-hot 交叉熵(MCE)目标训练,再经 recovery 阶段回到标准训练,在 10B A1B 规模上报告了至多 2.5x 的预训练时间缩减。本模型的 group attn 是其架构侧的延伸:GroupBlock 以 group_size=8 的分组处理 token,FullBlock / GroupBlock 按 f/g pattern 交错排布,并用 F2G / G2F 层完成组与完整 token 流之间的信息转换。 ## 训练细节 训练分四个阶段完成(bf16,AdamW 与 Muon 双优化器): 1. **预训练**:causal LM,1024 上下文,**8.52B Tokens**,冻结嵌入与 lm_head。 2. **SFT**:UltraData SFT + Instant 混合语料,8192 上下文(预训练为 1024,此阶段首次扩展到 8192),Muon 5e-6 / AdamW 5e-7,嵌入与 lm_head 冻结,**4.14B Tokens**。 3. **蒸馏**:teacher 为 Qwen3-0.6B(CE=0.5 / KL=0.5,temperature=1.0),固定 group_size=4,DCP probe 冻结,哈希辅助监督加强至 1e-1,Muon 1e-4 / AdamW 1e-5,**0.98B Tokens**。 ## Usage ```python from transformers import AutoTokenizer, AutoModelForCausalLM model_id = "SnifferCaptain/YModel3.2-fm-1B" tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=True, torch_dtype="auto", ) ``` ### 对话格式 模型使用 ChatML 风格的对话边界。完整的输入可以写成: ```text <|im_start|>system {system prompt}<|im_end|> <|im_start|>user {user content}<|im_end|> <|im_start|>assistant {thinking_definitions} {reasoning content} {assistant content}<|im_end|> ``` 其中,`{thinking_definitions}` 是紧跟在 `` 后面的**第一行控制信息**,不是思考正文。它可以是: - `juice = 1.23`:思考预算提示,可以使用小数。 - `structured = true`:表示结构化思考。 - `juice = 1.23; structured = true`:同时指定预算和结构化思考。 控制行结束后换行,再写入 `{reasoning content}`;思考结束使用 ``,之后空一行写最终的 `{assistant content}`。自由形式思考可以省略 `structured`,也可以显式写成 `structured = false`。如果没有控制字段,保留空的控制行即可: ```text {reasoning content} ``` 每个角色都使用 `<|im_start|>role\n...<|im_end|>` 边界;`system` 和 `user` 内容不会放入 ``,思考与最终回答只属于 `assistant` 消息。 注意:该架构的 `generate()` 为模型自定义实现,不使用 KV cache(每步重算完整前缀),长序列生成会相应变慢。 ## Notes - 包含自定义代码:`modeling_ymodel32.py`(自包含:配置类 + 完整架构)、`ymodel32_eval.py`(参考实现),加载需 `trust_remote_code=True`。 - 权重为单个 `model.safetensors`(bfloat16,约 2.55GB)。 - 基准分数与示例输出待补充。 --- ## Model Description `YModel3.2-fm-1B` is the 1B-parameter **foundation-weights model** of the YModel3.2 family. ## Model Details | Item | Value | | --- | --- | | Architecture | YModel3.2 | | Scale level | 1 | | Parameters | 1,120,343,553 (1.12B, incl. embeddings) | | Hidden size | 1024 | | Blocks | 31 (6 FullBlocks + 25 GroupBlocks) | | Layer pattern | `fgggggfgggggfgggggfgggggfgggggf` | | Attention heads / KV heads | 8 / 2 (GQA) | | Head dim | 256 | | Group size | 8 (fixed grouping) | | FFN | SwiGLU + SEBlock, intermediate 3072 | | DCP | dynamic compression probe (probe hidden 1024); use_dcp=false, fixed grouping, probe weights retained | | Sengram | bucket 16384 / topk 12 | | Vocab | 151,669 (Qwen3 tokenizer, tied embeddings) | | Context | 8192 | | Dtype | bfloat16 | ### Group Attn and Token Superposition The group attn of YModel3.2-fm is inspired primarily by **Efficient Pre-Training with Token Superposition** (Bowen Peng, Théo Gigant, Jeffrey Quesnelle, arXiv:2605.06546): superposing contiguous tokens into a group and processing them as a single unit. The paper's Token-Superposition Training (TST) merges many contiguous tokens into one bag and trains with a multi-hot cross-entropy (MCE) objective, then recovers to standard training, reporting up to a 2.5x pre-training time reduction at 10B A1B scale. This model's group attn is the architectural extension of that idea: GroupBlocks process tokens in groups of 8, FullBlocks and GroupBlocks are interleaved per the f/g pattern, and F2G / G2F layers transfer information between the group and full-token streams. ## Training Details Training ran in four stages (bf16, AdamW + Muon dual optimizers): 1. **Pre-training**: causal LM, 1024 context, **8.52B Tokens**, embeddings and lm_head frozen. 2. **SFT**: mixed UltraData SFT + Instant corpus, 8192 context (pre-training used 1024, so 8192 is new context here), Muon 5e-6 / AdamW 5e-7, embeddings and lm_head frozen, **4.14B Tokens**. 3. **Distillation**: teacher Qwen3-0.6B (CE=0.5 / KL=0.5, temperature 1.0), fixed group_size=4, DCP probe frozen, hash auxiliary supervision raised to 1e-1, Muon 1e-4 / AdamW 1e-5, **0.98B Tokens**. ## Usage ```python from transformers import AutoTokenizer, AutoModelForCausalLM model_id = "SnifferCaptain/YModel3.2-fm-1B" tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=True, torch_dtype="auto", ) ``` ### Chat Format The model uses ChatML-style turn boundaries. A complete input can be written as: ```text <|im_start|>system {system prompt}<|im_end|> <|im_start|>user {user content}<|im_end|> <|im_start|>assistant {thinking_definitions} {reasoning content} {assistant content}<|im_end|> ``` `{thinking_definitions}` is the **first control line** immediately after ``, not the reasoning body. It may be: - `juice = 1.23` for a numeric reasoning-budget hint; - `structured = true` to mark structured reasoning; or - `juice = 1.23; structured = true` to provide both fields. Terminate the control line with a newline, then place the reasoning in `{reasoning content}`. Close it with ``, leave one blank line, and put the final answer in `{assistant content}`. For free-form reasoning, `structured` may be omitted or written as `structured = false`. If no control field is needed, keep an empty control line: ```text {reasoning content} ``` Note: this architecture ships a custom `generate()` implementation without KV caching (the full prefix is recomputed each step), so long-sequence generation is correspondingly slower. ## Notes - Custom code included: `modeling_ymodel32.py` (self-contained: config class + full architecture) and `ymodel32_eval.py` (reference implementation); load with `trust_remote_code=True`. - Weights: single `model.safetensors` (bfloat16, ~2.55GB). - Benchmark scores and sample outputs: to be added.