File size: 4,753 Bytes
5e6a2f0 33fce7e 5e6a2f0 33fce7e 53c7b5f 33fce7e 53c7b5f 33fce7e 53c7b5f 33fce7e 53c7b5f 33fce7e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 | ---
license: mit
library_name: pytorch
tags:
- language-modeling
- transformer
- attention
- ablation
- research
language:
- en
datasets:
- HuggingFaceFW/fineweb-edu
metrics:
- perplexity
pipeline_tag: text-generation
---
# SMAT — Semantic Attention
Trained checkpoints for **SMAT** (Semantic Attention), a transformer attention
variant with a learnable semantic-similarity bias and per-token value gate.
- **Code:** [github.com/OutrageouslyBad200/SMATest](https://github.com/OutrageouslyBad200/SMATest)
- **Architecture:** 24 layers × 384d × 6 heads, block size 256, ~64 M parameters
- **Tokenizer:** GPT-2 (`tiktoken`, vocab 50 257)
- **Training data:** FineWeb-Edu sample-10BT, 98 M tokens
- **Training compute:** 12 000 optimizer steps, batch 16 × grad_accum 2 (effective 32),
RTX 4060
## Equation
```
Attn(Q,K,V) = softmax(QK^T/sqrt(d_k) + λ·S + P + M) · (G ⊙ V)
```
- `S_ij = cos(W_s h_i, W_s h_j)` — cosine similarity in shared projection
- `c_j = (1/n) Σ_{l≤j} S_jl` — causal semantic centrality
- `G_j = σ(w_g^T h_j + μ·c_j + β)` — per-token value gate
- `λ = softplus(λ_raw)` — constrained positive scalar (per layer)
## Repository contents
This HuggingFace repo hosts 20 checkpoints from the 5-seed ablation in
Experiment 6 of the SMAT paper:
```
baseline_s0/final.pt s_only_s0/final.pt g_only_s0/final.pt full_s0/final.pt
baseline_s1/final.pt s_only_s1/final.pt g_only_s1/final.pt full_s1/final.pt
baseline_s2/final.pt s_only_s2/final.pt g_only_s2/final.pt full_s2/final.pt
baseline_s3/final.pt s_only_s3/final.pt g_only_s3/final.pt full_s3/final.pt
baseline_s4/final.pt s_only_s4/final.pt g_only_s4/final.pt full_s4/final.pt
```
Each variant directory also contains `config.json` and `metrics.jsonl`
(per-step training + eval logs).
| Variant | `use_S` | `use_G` | Description |
|---------|---------|---------|-------------|
| `baseline` | False | False | Standard attention |
| `s_only` | True | False | Semantic bias only |
| `g_only` | False | True | Value gate only |
| `full` | True | True | Full SMAT |
## Results
Validation perplexity on FineWeb-Edu, 5 seeds, 12 000 steps:
| Variant | Mean ppl | Std | Δ vs baseline | Seed wins |
|---------|----------|-----|---------------|-----------|
| Baseline | 79.75 | 1.69 | — | — |
| S-only | 79.47 | 1.71 | −0.35% | 4/5 |
| G-only | 79.02 | 1.65 | −0.90% | 5/5 |
| **Full SMAT** | **78.65** | 1.75 | **−1.37%** | **5/5** |
0 NaN failures across 240 000 optimizer steps.
## Usage
```bash
pip install torch numpy tiktoken huggingface_hub
git clone https://github.com/OutrageouslyBad200/SMATest.git
cd SMATest
```
Download a single checkpoint:
```python
from huggingface_hub import hf_hub_download
ckpt_path = hf_hub_download(
repo_id="OutrageouslyBad200/SMATest",
filename="full_s0/final.pt",
)
```
Load it into the SMAT model:
```python
import torch
from model import Config, SMATTransformer
state = torch.load(ckpt_path, map_location="cuda")
cfg = Config(**state["config"])
model = SMATTransformer(cfg).cuda()
model.load_state_dict(state["state_dict"])
model.eval()
```
Reproduce surgical ablations (Experiment 7):
```bash
python ablate.py --ckpt full_s0/final.pt --n_batches 80
```
## Surgical-ablation findings (Experiment 7)
Run on Full SMAT, val ppl 79.010:
| Ablation | val ppl | Δ |
|----------|---------|---|
| λ=0 (S still drives c) | 79.40 | +0.49% |
| S removed entirely | 80.48 | +1.85% |
| Random S (same norm) | 81.23 | +2.80% |
| G replaced by mean | 196.99 | +149% |
| G forced to 1.0 | 625 850 | catastrophic |
- The gate **G** is catastrophically essential.
- **S** routes mostly through `μ·c` in the gate (74 % of lift), not through
`λ·S` in attention (26 %).
- Per-token gate differentiation matters: replacing G with its mean costs 149 %.
## Limitations
- Small base model (~64 M params); larger-scale runs (100 M on FineWeb / FineMath)
show stronger lifts (−11 % to −17 %) but are not included as released checkpoints.
- Trained only on English FineWeb-Edu sample-10BT — generalization to other
domains untested at this scale.
- Not instruction-tuned, not RLHF'd, no safety filtering. Research artifact only.
## Citation
```bibtex
@misc{smat2026,
author = {OutrageouslyBad200},
title = {SMAT: Semantic Attention},
year = {2026},
howpublished = {\url{https://github.com/OutrageouslyBad200/SMATest}},
}
```
## Contact
For further information on training runs, intermediate experiments, or the
unpublished paper draft, please contact the creator via
[GitHub](https://github.com/OutrageouslyBad200) or HuggingFace.
## License
[MIT License](https://github.com/OutrageouslyBad200/smat/blob/main/LICENSE).
|