| language: en | |
| tags: | |
| - causal-lm | |
| - gqa | |
| - qk-norm | |
| - rope | |
| - swiglu | |
| - refresh-gate | |
| - rose-x1 | |
| license: apache-2.0 | |
| # TestSubject-1 | |
| Rose X1.5 SLM (148.8M params). | |
| ## Architecture | |
| | Property | Value | | |
| |---|---| | |
| | Layers | 30 | | |
| | Hidden | 640 | | |
| | Heads | 10 (kv=5) | | |
| | QK Norm | True | | |
| | XSA | True | | |
| | Refresh Gate | [9, 19] | | |
| | Params | 148.826M | | |
| ## Training | |
| - Tokens: 6,917,222,400 | |
| - Val PPL: 10.46 | |
| - Optimizer: muon_adamw | |