---
language: en
tags:
- causal-lm
- gqa
- differential-attention
- logit-softcap
- qk-norm
- rope
- swiglu
- refresh-gate
- tst
license: apache-2.0
---

# TS-A

Rose X2 SLM (167.3M params).

## Training Progress
- **Tokens Seen:** 11,319,091,200
- **Compute Tokens:** 11,319,091,200
- **Step:** 21,600
- **Val PPL:** inf

## Architecture
| Property | Value |
|---|---|
| Layers | 30 |
| Hidden | 640 |
| Heads | 10 (kv=5) |
| Diff Attention | True |
| Logit Softcap | True (cap=50.0) |
| Params | 167.258M |

## Features
- GQA
- QK-Norm
- RoPE
- SwiGLU
- RMSNorm
- DifferentialAttention
- LogitSoftcap
- XSA
- RefreshGate
- TST