Update README.md
Browse files
README.md
CHANGED
|
@@ -40,9 +40,7 @@ Sol Nano is a 2,895,188-parameter language model trained on 5 billion tokens. It
|
|
| 40 |
| Optimizer updates | 19,074 |
|
| 41 |
| Weights | FP32 safetensors |
|
| 42 |
|
| 43 |
-
The blocks use causal grouped-query attention, RoPE, QK normalization, and loop conditioning. Token embeddings share weights with the output head. TN-Gram provides factorized local memory for orders 2-5.
|
| 44 |
-
|
| 45 |
-
Attention runs through compiled Triton FlexAttention. The backend does not use PyTorch SDPA.
|
| 46 |
|
| 47 |
## Use
|
| 48 |
|
|
|
|
| 40 |
| Optimizer updates | 19,074 |
|
| 41 |
| Weights | FP32 safetensors |
|
| 42 |
|
| 43 |
+
The blocks use causal grouped-query attention, RoPE, QK normalization, and loop conditioning. Token embeddings share weights with the output head. TN-Gram provides factorized local memory for orders 2-5.
|
|
|
|
|
|
|
| 44 |
|
| 45 |
## Use
|
| 46 |
|