metadata
language: en
tags:
- causal-lm
- gqa
- qk-norm
- rope
- swiglu
- refresh-gate
- rose-x1
license: apache-2.0
TestSubject-2
Rose X1.5 SLM (98.8M params).
Architecture
| Property | Value |
|---|---|
| Layers | 22 |
| Hidden | 576 |
| Heads | 9 (kv=3) |
| QK Norm | True |
| XSA | True |
| Refresh Gate | [6, 13] |
| Params | 98.766M |
Training
- Tokens: 19,284,377,600
- Val PPL: 10.85
- Optimizer: muon_adamw