metadata
language: en
tags:
- causal-lm
- gqa
- qk-norm
- rope
- swiglu
- refresh-gate
- rose-x1
license: apache-2.0
TestSubject-1
Rose X1.5 SLM (148.8M params).
Architecture
| Property | Value |
|---|---|
| Layers | 30 |
| Hidden | 640 |
| Heads | 10 (kv=5) |
| QK Norm | True |
| XSA | True |
| Refresh Gate | [9, 19] |
| Params | 148.826M |
Training
- Tokens: 4,611,481,600
- Val PPL: 10.75
- Optimizer: muon_adamw