metadata
language: en
tags:
- causal-lm
- gqa
- differential-attention
- logit-softcap
- qk-norm
- rope
- swiglu
- refresh-gate
- tst
license: apache-2.0
TS-A
Rose X2 SLM (167.3M params).
Training Progress
- Tokens Seen: 11,319,091,200
- Compute Tokens: 11,319,091,200
- Step: 21,600
- Val PPL: inf
Architecture
| Property | Value |
|---|---|
| Layers | 30 |
| Hidden | 640 |
| Heads | 10 (kv=5) |
| Diff Attention | True |
| Logit Softcap | True (cap=50.0) |
| Params | 167.258M |
Features
- GQA
- QK-Norm
- RoPE
- SwiGLU
- RMSNorm
- DifferentialAttention
- LogitSoftcap
- XSA
- RefreshGate
- TST