---
license: mit
language:
- zh
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- custom_code
- safetensors
- ymodel
- ymodel32
---
*this file is generated by Qwen3.8-27B under human supervision*
## 模型描述
`YModel3.2-fm-1B` 是 YModel3.2 家族的 1B 参数**基础权重模型**(foundation weights)。
## 模型细节
| 项目 | 值 |
| --- | --- |
| 架构 | YModel3.2 |
| Scale level | 1 |
| 参数量 | 1,120,343,553(1.12B,含嵌入)|
| Hidden size | 1024 |
| 块数 | 31(6 个 FullBlock + 25 个 GroupBlock)|
| layer pattern | `fgggggfgggggfgggggfgggggfgggggf` |
| 注意力头 / KV 头 | 8 / 2(GQA)|
| 头维度 | 256 |
| Group size | 8(固定分组)|
| FFN | SwiGLU + SEBlock,intermediate 3072 |
| DCP | dynamic compression probe(probe hidden 1024);use_dcp=false,分组固定,probe 权重保留 |
| Sengram | bucket 16384 / topk 12 |
| 词表 | 151,669(Qwen3 tokenizer,tied embeddings)|
| 上下文 | 8192 |
| 精度 | bfloat16 |
### Group Attn 与 Token Superposition
YModel3.2-fm 的 group attn 主要借鉴了论文 **Efficient Pre-Training with Token Superposition**(Bowen Peng, Théo Gigant, Jeffrey Quesnelle, arXiv:2605.06546)的思想:将连续 token 分组(superpose)为一个整体进行高效处理。论文提出的 Token-Superposition Training(TST)把多个连续 token 合并为一个 bag、以 multi-hot 交叉熵(MCE)目标训练,再经 recovery 阶段回到标准训练,在 10B A1B 规模上报告了至多 2.5x 的预训练时间缩减。本模型的 group attn 是其架构侧的延伸:GroupBlock 以 group_size=8 的分组处理 token,FullBlock / GroupBlock 按 f/g pattern 交错排布,并用 F2G / G2F 层完成组与完整 token 流之间的信息转换。
## 训练细节
训练分四个阶段完成(bf16,AdamW 与 Muon 双优化器):
1. **预训练**:causal LM,1024 上下文,**8.52B Tokens**,冻结嵌入与 lm_head。
2. **SFT**:UltraData SFT + Instant 混合语料,8192 上下文(预训练为 1024,此阶段首次扩展到 8192),Muon 5e-6 / AdamW 5e-7,嵌入与 lm_head 冻结,**4.14B Tokens**。
3. **蒸馏**:teacher 为 Qwen3-0.6B(CE=0.5 / KL=0.5,temperature=1.0),固定 group_size=4,DCP probe 冻结,哈希辅助监督加强至 1e-1,Muon 1e-4 / AdamW 1e-5,**0.98B Tokens**。
## Usage
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "SnifferCaptain/YModel3.2-fm-1B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype="auto",
)
```
### 对话格式
模型使用 ChatML 风格的对话边界。完整的输入可以写成:
```text
<|im_start|>system
{system prompt}<|im_end|>
<|im_start|>user
{user content}<|im_end|>
<|im_start|>assistant
{thinking_definitions}
{reasoning content}
{assistant content}<|im_end|>
```
其中,`{thinking_definitions}` 是紧跟在 `` 后面的**第一行控制信息**,不是思考正文。它可以是:
- `juice = 1.23`:思考预算提示,可以使用小数。
- `structured = true`:表示结构化思考。
- `juice = 1.23; structured = true`:同时指定预算和结构化思考。
控制行结束后换行,再写入 `{reasoning content}`;思考结束使用 ``,之后空一行写最终的 `{assistant content}`。自由形式思考可以省略 `structured`,也可以显式写成 `structured = false`。如果没有控制字段,保留空的控制行即可:
```text
{reasoning content}
```
每个角色都使用 `<|im_start|>role\n...<|im_end|>` 边界;`system` 和 `user` 内容不会放入 ``,思考与最终回答只属于 `assistant` 消息。
注意:该架构的 `generate()` 为模型自定义实现,不使用 KV cache(每步重算完整前缀),长序列生成会相应变慢。
## Notes
- 包含自定义代码:`modeling_ymodel32.py`(自包含:配置类 + 完整架构)、`ymodel32_eval.py`(参考实现),加载需 `trust_remote_code=True`。
- 权重为单个 `model.safetensors`(bfloat16,约 2.55GB)。
- 基准分数与示例输出待补充。
---
## Model Description
`YModel3.2-fm-1B` is the 1B-parameter **foundation-weights model** of the YModel3.2 family.
## Model Details
| Item | Value |
| --- | --- |
| Architecture | YModel3.2 |
| Scale level | 1 |
| Parameters | 1,120,343,553 (1.12B, incl. embeddings) |
| Hidden size | 1024 |
| Blocks | 31 (6 FullBlocks + 25 GroupBlocks) |
| Layer pattern | `fgggggfgggggfgggggfgggggfgggggf` |
| Attention heads / KV heads | 8 / 2 (GQA) |
| Head dim | 256 |
| Group size | 8 (fixed grouping) |
| FFN | SwiGLU + SEBlock, intermediate 3072 |
| DCP | dynamic compression probe (probe hidden 1024); use_dcp=false, fixed grouping, probe weights retained |
| Sengram | bucket 16384 / topk 12 |
| Vocab | 151,669 (Qwen3 tokenizer, tied embeddings) |
| Context | 8192 |
| Dtype | bfloat16 |
### Group Attn and Token Superposition
The group attn of YModel3.2-fm is inspired primarily by **Efficient Pre-Training with Token Superposition** (Bowen Peng, Théo Gigant, Jeffrey Quesnelle, arXiv:2605.06546): superposing contiguous tokens into a group and processing them as a single unit. The paper's Token-Superposition Training (TST) merges many contiguous tokens into one bag and trains with a multi-hot cross-entropy (MCE) objective, then recovers to standard training, reporting up to a 2.5x pre-training time reduction at 10B A1B scale. This model's group attn is the architectural extension of that idea: GroupBlocks process tokens in groups of 8, FullBlocks and GroupBlocks are interleaved per the f/g pattern, and F2G / G2F layers transfer information between the group and full-token streams.
## Training Details
Training ran in four stages (bf16, AdamW + Muon dual optimizers):
1. **Pre-training**: causal LM, 1024 context, **8.52B Tokens**, embeddings and lm_head frozen.
2. **SFT**: mixed UltraData SFT + Instant corpus, 8192 context (pre-training used 1024, so 8192 is new context here), Muon 5e-6 / AdamW 5e-7, embeddings and lm_head frozen, **4.14B Tokens**.
3. **Distillation**: teacher Qwen3-0.6B (CE=0.5 / KL=0.5, temperature 1.0), fixed group_size=4, DCP probe frozen, hash auxiliary supervision raised to 1e-1, Muon 1e-4 / AdamW 1e-5, **0.98B Tokens**.
## Usage
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "SnifferCaptain/YModel3.2-fm-1B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype="auto",
)
```
### Chat Format
The model uses ChatML-style turn boundaries. A complete input can be written as:
```text
<|im_start|>system
{system prompt}<|im_end|>
<|im_start|>user
{user content}<|im_end|>
<|im_start|>assistant
{thinking_definitions}
{reasoning content}
{assistant content}<|im_end|>
```
`{thinking_definitions}` is the **first control line** immediately after ``, not the reasoning body. It may be:
- `juice = 1.23` for a numeric reasoning-budget hint;
- `structured = true` to mark structured reasoning; or
- `juice = 1.23; structured = true` to provide both fields.
Terminate the control line with a newline, then place the reasoning in `{reasoning content}`. Close it with ``, leave one blank line, and put the final answer in `{assistant content}`. For free-form reasoning, `structured` may be omitted or written as `structured = false`. If no control field is needed, keep an empty control line:
```text
{reasoning content}
```
Note: this architecture ships a custom `generate()` implementation without KV caching (the full prefix is recomputed each step), so long-sequence generation is correspondingly slower.
## Notes
- Custom code included: `modeling_ymodel32.py` (self-contained: config class + full architecture) and `ymodel32_eval.py` (reference implementation); load with `trust_remote_code=True`.
- Weights: single `model.safetensors` (bfloat16, ~2.55GB).
- Benchmark scores and sample outputs: to be added.