| --- |
| language: en |
| tags: |
| - causal-lm |
| - gqa |
| - differential-attention |
| - logit-softcap |
| - qk-norm |
| - rope |
| - swiglu |
| - refresh-gate |
| - tst |
| license: apache-2.0 |
| --- |
| |
| # TS-A |
|
|
| Rose X2 SLM (167.3M params). |
|
|
| ## Training Progress |
| - **Tokens Seen:** 11,319,091,200 |
| - **Compute Tokens:** 11,319,091,200 |
| - **Step:** 21,600 |
| - **Val PPL:** inf |
|
|
| ## Architecture |
| | Property | Value | |
| |---|---| |
| | Layers | 30 | |
| | Hidden | 640 | |
| | Heads | 10 (kv=5) | |
| | Diff Attention | True | |
| | Logit Softcap | True (cap=50.0) | |
| | Params | 167.258M | |
|
|
| ## Features |
| - GQA |
| - QK-Norm |
| - RoPE |
| - SwiGLU |
| - RMSNorm |
| - DifferentialAttention |
| - LogitSoftcap |
| - XSA |
| - RefreshGate |
| - TST |
|
|
