| language: en | |
| tags: | |
| - causal-lm | |
| - gqa | |
| - qk-norm | |
| - rope | |
| - swiglu | |
| - refresh-gate | |
| - rose-x1 | |
| license: apache-2.0 | |
| # TestSubject-2 | |
| Rose X1.5 SLM (98.8M params). | |
| ## Architecture | |
| | Property | Value | | |
| |---|---| | |
| | Layers | 22 | | |
| | Hidden | 576 | | |
| | Heads | 9 (kv=3) | | |
| | QK Norm | True | | |
| | XSA | True | | |
| | Refresh Gate | [6, 13] | | |
| | Params | 98.766M | | |
| ## Training | |
| - Tokens: 22,218,956,800 | |
| - Val PPL: 10.79 | |
| - Optimizer: muon_adamw | |