Upload 91 files

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+figures_seed0/lambda_per_layer.png filter=lfs diff=lfs merge=lfs -text

Forpaper.txt

@@ -0,0 +1,424 @@
+================================================================================
+SMAT (Semantic Attention) - Findings Summary for Paper
+================================================================================
+Compiled: 2026-06-03
+Source files used (all in this repo, all re-verified for this document):
+  runs/{baseline,s_only,g_only,full}_s{0..4}/metrics.jsonl    (20 training runs)
+  runs_4ksteps/{baseline,s_only,g_only,full}_s0/metrics.jsonl (4 partial runs)
+  figures_final/ablation_inference.json                        (surgical ablations)
+  figures_final/ablation_results.csv                           (final eval table)
+
+Every number in sections 2-6 below was recomputed directly from the raw
+metrics.jsonl files (not copied from any pre-existing CSV or README). The
+per-seed deltas, means, std devs, and lambda/G statistics were all
+re-derived from scratch and cross-checked twice.
+
+The cross-scale results in section 7 are from prior experiments outside
+this repo; they are cited as such and were not re-verified from raw data
+here. Treat them as historical context, not as primary evidence.
+
+
+================================================================================
+1. WHAT SMAT IS (one paragraph)
+================================================================================
+SMAT modifies standard transformer attention with two additions: (1) a
+learnable semantic-similarity bias S added to the QK^T logits, scaled by a
+per-layer positive scalar lambda; and (2) a per-token value gate G that
+multiplies V before attention aggregation. The gate is also influenced by
+a causal semantic-centrality term c derived from S. Equation:
+
+    Attn(Q,K,V) = softmax( QK^T / sqrt(d_k) + lambda * S + P + M ) . (G (*) V)
+
+where
+    S_ij = cos(W_s h_i, W_s h_j)            shared-projection cosine similarity
+    c_j  = (1/n) sum_{l<=j} S_jl            causal semantic centrality
+    G_j  = sigmoid(w_g^T h_j + mu*c_j + b)  per-token value gate
+    lambda = softplus(lambda_raw)           constrained positive, per layer
+    P                                       learnable [L,L] positional bias
+    M                                       causal mask
+
+W_s is shared across heads within a layer. G is per-token and broadcast
+across head channels. Four variants were trained:
+    baseline   use_S = False, use_G = False  (standard attention)
+    s_only     use_S = True,  use_G = False  (lambda*S only)
+    g_only     use_S = False, use_G = True   (gate only)
+    full       use_S = True,  use_G = True   (both components)
+
+
+================================================================================
+2. TRAINING SETUP (the 5-seed ablation)
+================================================================================
+Architecture:    24 layers, 384 dim, 6 heads, block size 256, vocab 50257
+Parameters:      63,376,512 (~64M) per variant
+Optimizer:       Fused AdamW, lr 6e-4, 200-step linear warmup, cosine to 6e-5
+Batch:           16, grad_accum 2 -> effective batch 32
+Tokens per step: 16 * 2 * 256 = 8,192
+Total steps:     12,000   -> 98,304,000 tokens per run (~98M)
+Eval:            every 500 steps on 40 val batches (val_ppl = exp(val_loss))
+Precision:       bf16 autocast, TF32, cudnn.benchmark, SDPA attention path
+Hardware:        single RTX 4060 (vram ~6 GB during training)
+Corpus:          FineWeb-Edu sample-10BT, GPT-2 tokenization
+Seeds:           {0, 1, 2, 3, 4} for each of the four variants -> 20 runs total
+
+Every variant uses an IDENTICAL data stream for the same seed (data_seed=1337
+plus run seed); this is what makes the per-seed comparison meaningful.
+
+
+================================================================================
+3. PRIMARY RESULT - 5-SEED ABLATION TABLE
+================================================================================
+Final val perplexity at step 11999 (98M tokens seen), per variant per seed:
+
+  seed   baseline    s_only            g_only            full
+  ----  ---------  ----------------  ----------------  ----------------
+    0    78.3244    78.7377 (+0.53%)  78.2175 (-0.14%)  77.8634 (-0.59%)
+    1    80.9320    80.8649 (-0.08%)  80.5771 (-0.44%)  80.0262 (-1.12%)
+    2    82.3365    82.0735 (-0.32%)  81.3363 (-1.21%)  81.2602 (-1.31%)
+    3    79.3884    77.9830 (-1.77%)  77.9834 (-1.77%)  77.7016 (-2.12%)
+    4    77.7449    77.6807 (-0.08%)  77.0092 (-0.95%)  76.3953 (-1.74%)
+
+Per-seed deltas are computed as (variant_ppl - baseline_ppl) / baseline_ppl
+within the same seed, NOT against the baseline mean.
+
+Aggregate (mean across 5 seeds; std is population std):
+
+  variant      n   mean_ppl     std    delta_mean    seed wins vs baseline
+  --------    --  ---------  -------  ------------  -----------------------
+  Baseline     5   79.7452   1.6894         -                  -
+  S-only       5   79.4680   1.7131       -0.348%             4 / 5
+  G-only       5   79.0247   1.6463       -0.904%             5 / 5
+  Full SMAT    5   78.6493   1.7498       -1.374%             5 / 5
+
+Key observations:
+  - Full SMAT beats baseline on EVERY seed (5/5).
+  - The two components are additive in direction: S contributes ~0.35%,
+    G contributes ~0.90%, and Full SMAT (both together) gives ~1.37%.
+    S + G individually sum to ~1.25%, slightly less than full's 1.37%;
+    the components are mostly additive, with a small interaction in their
+    favor.
+  - Variance ACROSS seeds (std ~1.7 ppl) is larger than the mean effect
+    (~1.1 ppl). The signal is real but small relative to seed noise; it
+    only becomes clear because every variant uses the SAME data stream per
+    seed, so the within-seed comparison cancels almost all of that noise.
+  - The smallest within-seed gain for Full SMAT is -0.59% (seed 0);
+    the largest is -2.12% (seed 3). Direction is consistent; magnitude
+    varies by roughly a factor of 4 across seeds.
+
+
+================================================================================
+4. DIAGNOSTICS (training dynamics)
+================================================================================
+4.1 Lambda (semantic-bias scale, per-layer, softplus-positive)
+    Init value (all layers, all seeds): 0.1269 ( = softplus(lambda_raw_init) )
+    End-of-training mean lambda (averaged across 24 layers), per seed:
+        Full SMAT:    s0 0.3069  s1 0.3075  s2 0.3031  s3 0.2954  s4 0.3176
+                      -> mean across seeds ~ 0.306
+        S-only:       s0 0.3140  s1 0.3188  s2 0.3221  s3 0.3183  s4 0.3261
+                      -> mean across seeds ~ 0.320
+
+    Per-layer pattern (Full SMAT seed 0 final checkpoint, 24 layers):
+        layers 0-2:    lambda ~ 0.14 - 0.18    (slightly below init)
+        layers 3-8:    lambda ~ 0.08 - 0.16    (DECREASES below init)
+        layers 9-17:   lambda ~ 0.32 - 0.68    (climbs sharply)
+        layers 18-23:  lambda ~ 0.18 - 0.58    (settles, still elevated)
+        global min:    0.0834 (layer 5)
+        global max:    0.6821 (layer 16)
+
+    Across all 10 SMAT-bearing runs (s_only + full, 5 seeds each),
+    the observed per-layer lambda range was [0.059, 0.715]. The shape -
+    suppressed in early layers, dominant in middle layers - is robust
+    across all 5 seeds.
+
+4.2 Gate G (per-token, sigmoid output, averaged over tokens for logging)
+    Initial G_mean per run is around 0.5 (sigmoid initialized near zero
+    pre-activation). Across training G_mean rises briefly toward ~0.58
+    in the first ~1000 steps, then decays.
+    Final G_mean at step 11999 (Full SMAT): s0 0.3506, s1 0.3333,
+        s2 0.3306, s3 0.3447, s4 0.3397   -> mean ~ 0.340
+    Final G_mean (G-only):                  s0 0.3484, s1 0.3359,
+        s2 0.3676, s3 0.3625, s4 0.3563   -> mean ~ 0.354
+    G_std (across tokens) at end of training is ~ 0.16, meaning the gate
+    is NOT collapsed: many tokens have G well above the mean, many have it
+    well below. Per-token differentiation is preserved.
+
+4.3 Stability
+    NaN failures across all 20 runs at 12k steps each (= 240,000 optimizer
+    steps total): ZERO.
+    No NaN-driven step skips were recorded in any run. Training is
+    numerically clean under bf16 + TF32 + fused AdamW on this hardware.
+
+4.4 Wall-clock cost
+    Baseline runs: ~4000 - 5300 sec per run (~ 1.1 - 1.5 hours)
+    S-only runs:   ~4700 - 6100 sec
+    G-only runs:   ~4300 - 5700 sec
+    Full runs:     ~5400 - 6600 sec  (~ 1.5 - 1.8 hours)
+    Cost overhead of Full SMAT vs baseline: roughly +20 to +35% wall time
+    at this size on a 4060. (Variation within each variant is hardware/
+    thermal noise across the days the runs were executed.)
+
+
+================================================================================
+5. INFERENCE-TIME SURGICAL ABLATIONS
+================================================================================
+Performed on the Full SMAT seed-0 final checkpoint (no retraining; the
+trained weights are loaded, individual components are zeroed/replaced
+during a forward pass, and val ppl is re-measured on 80 val batches).
+Source: figures_final/ablation_inference.json
+
+Intact Full SMAT (this measurement, 80 batches): val ppl = 79.010
+(The 40-batch eval at the end of training reported 77.863; the 80-batch
+slice used for the ablation is a different val subset, hence the small
+difference. What matters is the delta within this measurement context.)
+
+  ablation       val_ppl       delta vs intact   what was changed
+  ------------  -----------  -----------------  ------------------------
+  lambda_zero      79.40           +0.49%       lambda set to 0 in the
+                                                attention bias only;
+                                                S still flows through
+                                                mu*c into the gate.
+  S_off            80.48           +1.85%       S removed completely:
+                                                no lambda*S in attention
+                                                AND no mu*c in the gate.
+  random_S         81.22           +2.80%       S replaced by a random
+                                                matrix with the same
+                                                Frobenius norm. Tests
+                                                whether S's STRUCTURE
+                                                matters, or just its norm.
+  G_one         625,850            +792,016%    G forced to constant 1.0
+                                                (gate disabled).
+  G_mean          196.99           +149.32%     G replaced by its scalar
+                                                mean over tokens (per-
+                                                token differentiation
+                                                removed; mean value kept).
+
+Decomposition of S's contribution:
+  Total S contribution (S_off vs intact):       1.85%
+  Contribution via lambda*S in attention:       0.49%   ->  26% of S's lift
+  Contribution via mu*c in the gate:            1.36%   ->  74% of S's lift
+  Most of S's value is being read through the gate's centrality term,
+  NOT through the additive logit bias.
+
+Per-layer lambda=0 ablations (24 separate ablations, zeroing lambda in
+one layer at a time):
+  Maximum single-layer cost:    +0.0580%   (layer 14)
+  Sum of all 24 per-layer deltas: +0.219%
+  This is far less than the +0.49% you get from zeroing lambda in every
+  layer at once. Interpretation: the lambda*S contribution is REDUNDANT
+  across layers - no single layer is load-bearing. The benefit is small
+  and distributed.
+
+Structural vs scalar test:
+  random_S (+2.80%) hurts MORE than S_off (+1.85%). Replacing S with a
+  same-norm random matrix is worse than removing it entirely. This means
+  the model is using S's learned STRUCTURE, not just the presence of an
+  extra signal of a given magnitude. A random matrix with the right
+  energy actively damages the model; trained S is genuinely informative.
+
+Gate criticality:
+  G=1.0 collapses the model entirely (val ppl > 600,000). The gate is not
+  a small modulation - the trained network depends on it as a primary
+  routing mechanism.
+  G=mean costs +149%. Replacing per-token gating with a single scalar mean
+  destroys most of the gate's value. The gate's worth is in WHICH tokens
+  it suppresses or amplifies, not in its average level.
+
+
+================================================================================
+6. PARTIAL 4K-STEP RUN (1 seed, 32M tokens) - CONTEXT ONLY
+================================================================================
+A shorter run at 4,000 steps (32,768,000 tokens) was performed for ONE
+seed before the full 12k-step ablation was launched. It is not directly
+comparable to the 5-seed result above (different budget, no seed
+variance), but it shows the same direction and a larger relative gap
+because perplexity is higher overall at lower training budgets.
+
+  variant     val_ppl    delta vs baseline
+  ---------  ---------  ------------------
+  baseline    148.4449         -
+  s_only      147.8903       -0.37%
+  g_only      144.9733       -2.34%
+  full        141.5651       -4.63%
+
+This is consistent with the broader pattern: at smaller training budgets
+(or smaller scales), the relative effect of SMAT is larger. As training
+proceeds and the baseline gets closer to its asymptote, the absolute
+ppl gap and the relative gap both shrink. The 12k-step 5-seed result
+(section 3) is the rigorous number; the 4k-step result is a directional
+sanity check.
+
+DO NOT cite the 4k-step delta as if it were comparable to the 12k-step
+delta. The user has already flagged this as a "partial run, not directly
+comparable" in the repo README.
+
+
+================================================================================
+7. CROSS-SCALE CONTEXT (PRIOR EXPERIMENTS, NOT FROM THIS REPO)
+================================================================================
+The following are described in this repo's README as prior experiments
+done before this codebase. They are NOT re-verified from raw data here
+and should be cited as historical context, not as primary evidence of
+this paper's findings. If the paper depends on them, the underlying logs
+should be located and re-verified before they appear in any final claim.
+
+  scale   corpus              tokens     delta   seeds   note
+  -----  -------------------  --------  -------  -----   -------------------
+   7M    Shakespeare           short     mixed     1     directional crossover
+                                                         around step 150
+  100M   FineWeb (smoke)       short    -7%       1     100-step smoke run
+  100M   FineWeb              100M     -11%      3 of 3 strongest replicated
+  100M   FineMath             100M     -17%      3 of 3 strongest result
+   68M   FineWeb (SMAT+A90)    20M     -22.5%     1     SMAT combined with
+                                                         BlockAttnRes; out
+                                                         of scope for V1.
+   64M   FineWeb (THIS REPO)   98M     -1.37%    5 of 5 primary result
+
+The pattern across these (taking them at face value): direction is
+preserved across every scale, every corpus, every seed where data
+exists. Magnitude shrinks as the scale and training budget grow, with
+the exception of the 100M experiments where the gap is largest.
+
+If the paper's headline number is from the 100M FineWeb or FineMath
+experiments, the V1 evidence is the 5-seed run in section 3 of THIS
+document plus those prior experiments. The 5-seed result is the
+hardest-to-dispute piece because every variant trained on identical
+data and direction is consistent across 5/5 seeds.
+
+
+================================================================================
+8. WHAT'S LOAD-BEARING vs WHAT IS NOT
+================================================================================
+Load-bearing (do not remove):
+  - The gate G. Catastrophic if removed (+792,016%) and badly degraded
+    if reduced to its mean (+149%). The gate is the single most important
+    component SMAT adds.
+  - The structure of S (not just its norm). random_S hurts more than
+    S_off, so the learned S projection contains real information.
+  - The mu*c centrality channel from S into the gate. This accounts for
+    ~74% of S's total contribution. If S is used at all, it should be
+    used HERE.
+
+Not load-bearing in any individual layer (and largely redundant overall):
+  - The per-layer lambda*S term in the attention bias. Removing it in
+    one layer costs at most 0.06% ppl. Removing it in all layers costs
+    only 0.49%. The benefit exists but it is small and distributed; no
+    single layer's lambda is critical, and a uniform lambda across
+    layers is the wrong abstraction (early layers actively prefer it
+    near zero, middle layers want it near 0.5-0.7).
+
+Net implication for V2 (this is what the data argues for):
+  - Drop the per-layer lambda_raw parameters and the lambda*S term in
+    attention. Keep S's projection W_s and keep mu*c flowing into the
+    gate. You retain ~74% of S's lift while removing 24 scalar
+    parameters and one matrix add per layer.
+  - If you want lambda to stay, replace per-layer lambda with a
+    per-layer gate-mix scalar driven by the same per-layer pattern
+    observed here (suppressed in early layers, elevated in middle
+    layers).
+
+
+================================================================================
+9. CAVEATS, LIMITATIONS, THINGS NOT TO OVERCLAIM
+================================================================================
+  - The headline -1.37% at 64M / 98M tokens is real (5/5 seeds, identical
+    per-seed data streams; one-sided sign test gives p = 1/32 = 0.03125
+    under the null hypothesis of no effect), but it is SMALL in absolute
+    terms (about 1.1 perplexity points). The within-seed effect is what
+    makes it visible; do not present mean +/- std as if seed noise were
+    the only consideration.
+  - Std across seeds (1.65 - 1.75) is larger than the mean effect.
+    Always quote the WITHIN-SEED comparison or the SIGN TEST when making
+    the case, not just delta-of-means with overlapping error bars.
+  - The surgical ablation results in section 5 are from ONE checkpoint
+    (Full SMAT seed 0). The qualitative conclusions (G is critical, S
+    routes through mu*c, layer-wise lambda is redundant) are stark
+    enough that one checkpoint is informative, but it would be more
+    defensible to repeat on at least one more seed before publication.
+  - The cross-scale numbers in section 7 are not verified from raw data
+    in this repo. Do not put them in a results table without separately
+    locating and confirming the underlying logs.
+  - All training was on a single RTX 4060 with bf16 + TF32. Numbers may
+    shift slightly on different hardware; the direction should not.
+  - Tokenizer is GPT-2 byte-pair. Corpus is FineWeb-Edu sample-10BT.
+    Results on other corpora (FineMath in particular, see section 7)
+    may differ in magnitude.
+
+
+================================================================================
+10. SUGGESTED PAPER CLAIMS (ordered by strength of evidence in this repo)
+================================================================================
+STRONG (directly supported by 20-run ablation, this repo, this document):
+  1. Adding a per-token value gate to attention reduces val ppl by
+     ~0.9% on FineWeb at 64M params, 5/5 seeds.
+  2. Adding both the semantic-similarity bias AND the gate reduces
+     val ppl by ~1.4% on FineWeb at 64M params, 5/5 seeds.
+  3. Training is numerically stable (0 NaN failures over 240,000
+     optimizer steps across 20 runs).
+  4. lambda exhibits a robust per-layer pattern: suppressed in early
+     layers, elevated in middle layers, settled in late layers.
+     This pattern is consistent across all 5 seeds.
+
+STRONG (supported by surgical-ablation evidence, single checkpoint):
+  5. S's contribution flows ~74% through the gate's centrality term
+     and only ~26% through the additive logit bias.
+  6. The trained S has learned STRUCTURE; replacing it with a random
+     matrix of equal norm hurts more than removing it entirely.
+  7. The per-token gate is catastrophic to remove and very expensive
+     to replace with a mean. Per-token differentiation matters.
+
+WEAKER (need cross-scale / cross-seed re-verification before paper):
+  8. SMAT's relative effect is larger at smaller training budgets and
+     at certain corpora (FineMath in particular).
+  9. SMAT directionality is preserved across scales 7M to 100M.
+
+DO NOT CLAIM without further work:
+  - That SMAT is a uniformly large improvement (it is not at this scale).
+  - That SMAT helps at scales beyond 100M (no data yet).
+  - That per-layer lambda is essential (the surgical ablation shows it
+    is largely redundant).
+
+
+================================================================================
+11. RAW NUMBERS (machine-readable, for any tables in the paper)
+================================================================================
+12k-step 5-seed ablation, final val ppl:
+
+baseline = [78.32438343443285, 80.93203870134798, 82.33645607970243,
+            79.38843805716219, 77.74486565469935]
+s_only   = [78.73771250646860, 80.86492907631813, 82.07353893301419,
+            77.98296438017636, 77.68072112072338]
+g_only   = [78.21752726763506, 80.57706364462581, 81.33629330240751,
+            77.98339201086276, 77.00922636947199]
+full     = [77.86344065821494, 80.02616085611275, 81.26017429181104,
+            77.70156877103959, 76.39525023929815]
+
+Means:
+  baseline 79.74523638546896
+  s_only   79.46797320334013
+  g_only   79.02470051900063
+  full     78.64931896329529
+
+Population std:
+  baseline 1.6894...  s_only 1.7131...  g_only 1.6463...  full 1.7498...
+
+Sample (n-1) std (if you prefer):
+  baseline 1.8888  s_only 1.9153  g_only 1.8406  full 1.9563
+
+Delta of means vs baseline (%):
+  s_only -0.3477   g_only -0.9035   full -1.3743
+
+Per-layer trained lambda (Full SMAT seed 0, 24 values, from
+ablation_inference.json):
+  [0.1837, 0.1491, 0.1557, 0.1365, 0.1209, 0.0834, 0.1213, 0.0920,
+   0.1605, 0.3192, 0.3323, 0.4566, 0.3248, 0.5507, 0.5484, 0.4943,
+   0.6821, 0.4693, 0.3914, 0.5843, 0.1961, 0.3536, 0.2781, 0.1833]
+  (printed to 4 dp; full precision in figures_final/ablation_inference.json)
+
+Surgical ablation deltas (Full SMAT seed 0, 80-batch eval slice, intact
+ppl 79.010):
+  lambda_zero  +0.4903%   S_off  +1.8547%   random_S  +2.8032%
+  G_one    +792,015.9609%   G_mean  +149.3211%
+
+
+================================================================================
+END
+================================================================================

HUGGINGFACE_README.md

@@ -0,0 +1,159 @@
+---
+license: mit
+library_name: pytorch
+tags:
+  - language-modeling
+  - transformer
+  - attention
+  - ablation
+  - research
+language:
+  - en
+datasets:
+  - HuggingFaceFW/fineweb-edu
+metrics:
+  - perplexity
+pipeline_tag: text-generation
+---
+
+# SMAT — Semantic Attention
+
+Trained checkpoints for **SMAT** (Semantic Attention), a transformer attention
+variant with a learnable semantic-similarity bias and per-token value gate.
+
+- **Code:** [github.com/OutrageouslyBad200/smat](https://github.com/OutrageouslyBad200/smat)
+- **Architecture:** 24 layers × 384d × 6 heads, block size 256, ~64 M parameters
+- **Tokenizer:** GPT-2 (`tiktoken`, vocab 50 257)
+- **Training data:** FineWeb-Edu sample-10BT, 98 M tokens
+- **Training compute:** 12 000 optimizer steps, batch 16 × grad_accum 2 (effective 32),
+  RTX 4060
+
+## Equation
+
+```
+Attn(Q,K,V) = softmax(QK^T/sqrt(d_k) + λ·S + P + M) · (G ⊙ V)
+```
+
+- `S_ij = cos(W_s h_i, W_s h_j)` — cosine similarity in shared projection
+- `c_j = (1/n) Σ_{l≤j} S_jl` — causal semantic centrality
+- `G_j = σ(w_g^T h_j + μ·c_j + β)` — per-token value gate
+- `λ = softplus(λ_raw)` — constrained positive scalar (per layer)
+
+## Repository contents
+
+This HuggingFace repo hosts 20 checkpoints from the 5-seed ablation in
+Experiment 6 of the SMAT paper:
+
+```
+baseline_s0/final.pt   s_only_s0/final.pt   g_only_s0/final.pt   full_s0/final.pt
+baseline_s1/final.pt   s_only_s1/final.pt   g_only_s1/final.pt   full_s1/final.pt
+baseline_s2/final.pt   s_only_s2/final.pt   g_only_s2/final.pt   full_s2/final.pt
+baseline_s3/final.pt   s_only_s3/final.pt   g_only_s3/final.pt   full_s3/final.pt
+baseline_s4/final.pt   s_only_s4/final.pt   g_only_s4/final.pt   full_s4/final.pt
+```
+
+Each variant directory also contains `config.json` and `metrics.jsonl`
+(per-step training + eval logs).
+
+| Variant | `use_S` | `use_G` | Description |
+|---------|---------|---------|-------------|
+| `baseline` | False | False | Standard attention |
+| `s_only`   | True  | False | Semantic bias only |
+| `g_only`   | False | True  | Value gate only |
+| `full`     | True  | True  | Full SMAT |
+
+## Results
+
+Validation perplexity on FineWeb-Edu, 5 seeds, 12 000 steps:
+
+| Variant | Mean ppl | Std | Δ vs baseline | Seed wins |
+|---------|----------|-----|---------------|-----------|
+| Baseline | 79.75 | 1.69 | — | — |
+| S-only | 79.47 | 1.71 | −0.35% | 4/5 |
+| G-only | 79.02 | 1.65 | −0.90% | 5/5 |
+| **Full SMAT** | **78.65** | 1.75 | **−1.37%** | **5/5** |
+
+0 NaN failures across 240 000 optimizer steps.
+
+## Usage
+
+```bash
+pip install torch numpy tiktoken huggingface_hub
+git clone https://github.com/OutrageouslyBad200/smat.git
+cd smat
+```
+
+Download a single checkpoint:
+
+```python
+from huggingface_hub import hf_hub_download
+ckpt_path = hf_hub_download(
+    repo_id="OutrageouslyBad200/smat",
+    filename="full_s0/final.pt",
+)
+```
+
+Load it into the SMAT model:
+
+```python
+import torch
+from model import Config, SMATTransformer
+
+state = torch.load(ckpt_path, map_location="cuda")
+cfg = Config(**state["config"])
+model = SMATTransformer(cfg).cuda()
+model.load_state_dict(state["state_dict"])
+model.eval()
+```
+
+Reproduce surgical ablations (Experiment 7):
+
+```bash
+python ablate.py --ckpt full_s0/final.pt --n_batches 80
+```
+
+## Surgical-ablation findings (Experiment 7)
+
+Run on Full SMAT, val ppl 79.010:
+
+| Ablation | val ppl | Δ |
+|----------|---------|---|
+| λ=0 (S still drives c) | 79.40 | +0.49% |
+| S removed entirely | 80.48 | +1.85% |
+| Random S (same norm) | 81.23 | +2.80% |
+| G replaced by mean | 196.99 | +149% |
+| G forced to 1.0 | 625 850 | catastrophic |
+
+- The gate **G** is catastrophically essential.
+- **S** routes mostly through `μ·c` in the gate (74 % of lift), not through
+  `λ·S` in attention (26 %).
+- Per-token gate differentiation matters: replacing G with its mean costs 149 %.
+
+## Limitations
+
+- Small base model (~64 M params); larger-scale runs (100 M on FineWeb / FineMath)
+  show stronger lifts (−11 % to −17 %) but are not included as released checkpoints.
+- Trained only on English FineWeb-Edu sample-10BT — generalization to other
+  domains untested at this scale.
+- Not instruction-tuned, not RLHF'd, no safety filtering. Research artifact only.
+
+## Citation
+
+```bibtex
+@misc{smat2026,
+  author       = {OutrageouslyBad200},
+  title        = {SMAT: Semantic Attention},
+  year         = {2026},
+  howpublished = {\url{https://github.com/OutrageouslyBad200/smat}},
+}
+```
+
+## Contact
+
+For further information on training runs, intermediate experiments, or the
+unpublished paper draft, please contact the creator via
+[GitHub](https://github.com/OutrageouslyBad200) or HuggingFace.
+
+## License
+
+[MIT License](https://github.com/OutrageouslyBad200/smat/blob/main/LICENSE).

all_results.csv

@@ -0,0 +1,25 @@
+runset,variant,seed,params,final_step,tokens_seen,val_loss,val_ppl,lambda_start,lambda_end,lambda_per_layer_min,lambda_per_layer_max,G_start,G_end,elapsed_s,nan_failures
+runs,baseline,0,63376512,11999,98304000,4.360858964920044,78.32438343443285,,,,,,,5309.61736869812,0
+runs,baseline,1,63376512,11999,98304000,4.393609774112702,80.93203870134798,,,,,,,5335.801450252533,0
+runs,baseline,2,63376512,11999,98304000,4.410813975334167,82.33645607970243,,,,,,,4016.196511745453,0
+runs,baseline,3,63376512,11999,98304000,4.374352741241455,79.38843805716219,,,,,,,4001.651510953903,0
+runs,baseline,4,63376512,11999,98304000,4.353432512283325,77.74486565469935,,,,,,,3997.937122106552,0
+runs,full,0,63975624,11999,98304000,4.354956531524659,77.86344065821494,0.12692801654338837,0.3069125845407446,0.08336106687784195,0.6821404099464417,0.5030857535700003,0.3505655957075457,6544.294906139374,0
+runs,full,1,63975624,11999,98304000,4.382353591918945,80.02616085611275,0.12692801654338837,0.3074745563790202,0.09392528235912323,0.6946620941162109,0.4995858383675416,0.3332850156972806,6583.949243545532,0
+runs,full,2,63975624,11999,98304000,4.397656035423279,81.26017429181104,0.12692801654338837,0.30312894998739165,0.11483360826969147,0.6690832376480103,0.48413818205396336,0.33056403052372235,5372.456871032715,0
+runs,full,3,63975624,11999,98304000,4.352875447273254,77.70156877103959,0.12692801654338837,0.29537375799069804,0.09290912002325058,0.6339113712310791,0.4967380625506242,0.3447236562302957,5374.616672039032,0
+runs,full,4,63975624,11999,98304000,4.335920524597168,76.39525023929815,0.12692801654338837,0.3175949805105726,0.09670225530862808,0.6432420015335083,0.49705679963032406,0.3396819991370042,5370.658619880676,0
+runs,g_only,0,63385776,11999,98304000,4.359493756294251,78.21752726763506,,,,,0.4967447916666667,0.3484141031901042,5647.636380910873,0
+runs,g_only,1,63385776,11999,98304000,4.389214038848877,80.57706364462581,,,,,0.5144856770833334,0.3359120686848958,5649.687728404999,0
+runs,g_only,2,63385776,11999,98304000,4.398592329025268,81.33629330240751,,,,,0.5052897135416666,0.3676300048828125,4351.29425239563,0
+runs,g_only,3,63385776,11999,98304000,4.356495881080628,77.98339201086276,,,,,0.4998372395833333,0.362548828125,4340.425580263138,0
+runs,g_only,4,63385776,11999,98304000,4.34392523765564,77.00922636947199,,,,,0.5079752604166666,0.3562774658203125,4339.748939752579,0
+runs,s_only,0,63966360,11999,98304000,4.366122233867645,78.7377125064686,0.12692801654338837,0.31397200624148053,0.06856122612953186,0.7095168232917786,,,6095.362888336182,0
+runs,s_only,1,63966360,11999,98304000,4.392780220508575,80.86492907631813,0.12692801654338837,0.3187686977908015,0.058697257190942764,0.6693422198295593,,,6085.201226711273,0
+runs,s_only,2,63966360,11999,98304000,4.407615661621094,82.07353893301419,0.12692801654338837,0.32208347382644814,0.07567994296550751,0.7152900695800781,,,4696.141266584396,0
+runs,s_only,3,63966360,11999,98304000,4.3564903974533085,77.98296438017636,0.12692801654338837,0.3182733968521158,0.0908360481262207,0.6816989183425903,,,4680.379953622818,0
+runs,s_only,4,63966360,11999,98304000,4.352607107162475,77.68072112072338,0.12692801654338837,0.3260646794612209,0.07901874929666519,0.6664481163024902,,,4688.767492055893,0
+runs_4ksteps,baseline,0,63376512,3999,32768000,5.000214111804962,148.44493951411144,,,,,,,1804.3665130138397,0
+runs_4ksteps,full,0,63975624,3999,32768000,4.9527598261833194,141.5651204177534,0.12692801654338837,0.16720068734139204,,,0.5060585426787535,0.4214151619623105,2212.9264504909515,0
+runs_4ksteps,g_only,0,63385776,3999,32768000,4.976549911499023,144.973346966276,,,,,0.4969075520833333,0.4358113606770833,1914.1351833343506,0
+runs_4ksteps,s_only,0,63966360,3999,32768000,4.996470749378204,147.89029507035235,0.12692801654338837,0.16468850150704384,,,,,2049.8153450489044,0

baseline_s0/config.json

@@ -0,0 +1,21 @@
+{
+  "variant": "baseline",
+  "seed": 0,
+  "out": "C:\\Users\\Bhorn\\Desktop\\SMATest\\runs\\baseline_s0",
+  "max_steps": 12000,
+  "warmup": 200,
+  "lr_max": 0.0006,
+  "lr_min": 6e-05,
+  "wd": 0.1,
+  "batch_size": 16,
+  "grad_accum": 2,
+  "block_size": 256,
+  "n_layer": 24,
+  "n_head": 6,
+  "n_embd": 384,
+  "vocab_size": 50257,
+  "data_seed": 1337,
+  "eval_interval": 500,
+  "eval_batches": 40,
+  "log_interval": 20
+}

baseline_s0/final.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bce27c7b560e565c25258a6ea84e5173a531c8167830748f786bd4bb96b4541d
+size 253572039

baseline_s1/config.json

@@ -0,0 +1,21 @@
+{
+  "variant": "baseline",
+  "seed": 1,
+  "out": "C:\\Users\\Bhorn\\Desktop\\SMATest\\runs\\baseline_s1",
+  "max_steps": 12000,
+  "warmup": 200,
+  "lr_max": 0.0006,
+  "lr_min": 6e-05,
+  "wd": 0.1,
+  "batch_size": 16,
+  "grad_accum": 2,
+  "block_size": 256,
+  "n_layer": 24,
+  "n_head": 6,
+  "n_embd": 384,
+  "vocab_size": 50257,
+  "data_seed": 1001340,
+  "eval_interval": 500,
+  "eval_batches": 40,
+  "log_interval": 20
+}

baseline_s1/final.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a5d08149cc3d598b4b31279b1e19ed2a79854bed1e1212d1d3094bfd50d0c0d
+size 253572039

baseline_s2/config.json

@@ -0,0 +1,21 @@
+{
+  "variant": "baseline",
+  "seed": 2,
+  "out": "C:\\Users\\Bhorn\\Desktop\\SMATest\\runs\\baseline_s2",
+  "max_steps": 12000,
+  "warmup": 200,
+  "lr_max": 0.0006,
+  "lr_min": 6e-05,
+  "wd": 0.1,
+  "batch_size": 16,
+  "grad_accum": 2,
+  "block_size": 256,
+  "n_layer": 24,
+  "n_head": 6,
+  "n_embd": 384,
+  "vocab_size": 50257,
+  "data_seed": 2001343,
+  "eval_interval": 1000,
+  "eval_batches": 20,
+  "log_interval": 20
+}

baseline_s2/final.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b59023116afb5ef8093dfc4e71d3ee798e4ee2a88a5db75fa8c695bae1a795de
+size 253572039

baseline_s3/config.json

@@ -0,0 +1,21 @@
+{
+  "variant": "baseline",
+  "seed": 3,
+  "out": "C:\\Users\\Bhorn\\Desktop\\SMATest\\runs\\baseline_s3",
+  "max_steps": 12000,
+  "warmup": 200,
+  "lr_max": 0.0006,
+  "lr_min": 6e-05,
+  "wd": 0.1,
+  "batch_size": 16,
+  "grad_accum": 2,
+  "block_size": 256,
+  "n_layer": 24,
+  "n_head": 6,
+  "n_embd": 384,
+  "vocab_size": 50257,
+  "data_seed": 3001346,
+  "eval_interval": 1000,
+  "eval_batches": 20,
+  "log_interval": 20
+}

baseline_s3/final.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:22e76aba2e7217efe0857e02a66e135844cc511ee89d9cec34be1a6354b0f5ea
+size 253572039

baseline_s4/config.json

@@ -0,0 +1,21 @@
+{
+  "variant": "baseline",
+  "seed": 4,
+  "out": "C:\\Users\\Bhorn\\Desktop\\SMATest\\runs\\baseline_s4",
+  "max_steps": 12000,
+  "warmup": 200,
+  "lr_max": 0.0006,
+  "lr_min": 6e-05,
+  "wd": 0.1,
+  "batch_size": 16,
+  "grad_accum": 2,
+  "block_size": 256,
+  "n_layer": 24,
+  "n_head": 6,
+  "n_embd": 384,
+  "vocab_size": 50257,
+  "data_seed": 4001349,
+  "eval_interval": 1000,
+  "eval_batches": 20,
+  "log_interval": 20
+}

baseline_s4/final.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1653ae95a80f8c998ce1f4d3e1a8c1e6466df0324a40c29b930f3c62f5b7887
+size 253572039

figures_4ksteps/ablation_results.csv

@@ -0,0 +1,5 @@
+variant,seed,final_val_ppl,delta_pct_vs_baseline,final_val_loss,final_step,tokens_seen,lambda_start,lambda_end,G_start,G_end
+baseline,0,148.44493951411144,+0.00,5.000214111804962,3999,32768000,None,None,None,None
+full,0,141.5651204177534,-4.63,4.9527598261833194,3999,32768000,0.12692801654338837,0.16720068734139204,0.5060585426787535,0.4214151619623105
+g_only,0,144.973346966276,-2.34,4.976549911499023,3999,32768000,None,None,0.4969075520833333,0.4358113606770833
+s_only,0,147.89029507035235,-0.37,4.996470749378204,3999,32768000,0.12692801654338837,0.16468850150704384,None,None

figures_seed0/ablation_results.csv

@@ -0,0 +1,5 @@
+variant,seed,final_val_ppl,delta_pct_vs_baseline,final_val_loss,final_step,tokens_seen,lambda_start,lambda_end,G_start,G_end
+baseline,0,78.32438343443285,+0.00,4.360858964920044,11999,98304000,None,None,None,None
+full,0,77.86344065821494,-0.59,4.354956531524659,11999,98304000,0.12692801654338837,0.3069125845407446,0.5030857535700003,0.3505655957075457
+g_only,0,78.21752726763506,-0.14,4.359493756294251,11999,98304000,None,None,0.4967447916666667,0.3484141031901042
+s_only,0,78.7377125064686,+0.53,4.366122233867645,11999,98304000,0.12692801654338837,0.31397200624148053,None,None

full_s0/config.json

@@ -0,0 +1,21 @@
+{
+  "variant": "full",
+  "seed": 0,
+  "out": "C:\\Users\\Bhorn\\Desktop\\SMATest\\runs\\full_s0",
+  "max_steps": 12000,
+  "warmup": 200,
+  "lr_max": 0.0006,
+  "lr_min": 6e-05,
+  "wd": 0.1,
+  "batch_size": 16,
+  "grad_accum": 2,
+  "block_size": 256,
+  "n_layer": 24,
+  "n_head": 6,
+  "n_embd": 384,
+  "vocab_size": 50257,
+  "data_seed": 1337,
+  "eval_interval": 500,
+  "eval_batches": 40,
+  "log_interval": 20
+}

full_s0/final.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5a8d83d8725d3cfe8ed58aa8720b6a40900aed797f617addfea94d49a315bda
+size 256003111

full_s1/config.json

@@ -0,0 +1,21 @@
+{
+  "variant": "full",
+  "seed": 1,
+  "out": "C:\\Users\\Bhorn\\Desktop\\SMATest\\runs\\full_s1",
+  "max_steps": 12000,
+  "warmup": 200,
+  "lr_max": 0.0006,
+  "lr_min": 6e-05,
+  "wd": 0.1,
+  "batch_size": 16,
+  "grad_accum": 2,
+  "block_size": 256,
+  "n_layer": 24,
+  "n_head": 6,
+  "n_embd": 384,
+  "vocab_size": 50257,
+  "data_seed": 1001340,
+  "eval_interval": 500,
+  "eval_batches": 40,
+  "log_interval": 20
+}

full_s1/final.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e653be057f08c2c10bb1be3ee3633ef3c490f33dab83e9a3891339327c80f10
+size 256003111

full_s2/config.json

@@ -0,0 +1,21 @@
+{
+  "variant": "full",
+  "seed": 2,
+  "out": "C:\\Users\\Bhorn\\Desktop\\SMATest\\runs\\full_s2",
+  "max_steps": 12000,
+  "warmup": 200,
+  "lr_max": 0.0006,
+  "lr_min": 6e-05,
+  "wd": 0.1,
+  "batch_size": 16,
+  "grad_accum": 2,
+  "block_size": 256,
+  "n_layer": 24,
+  "n_head": 6,
+  "n_embd": 384,
+  "vocab_size": 50257,
+  "data_seed": 2001343,
+  "eval_interval": 1000,
+  "eval_batches": 20,
+  "log_interval": 20
+}

full_s2/final.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ffbdf35a45d7b455806ed6ef360340c606c1a9c1ca78eff6811b679009e180c0
+size 256003111

full_s3/config.json

@@ -0,0 +1,21 @@
+{
+  "variant": "full",
+  "seed": 3,
+  "out": "C:\\Users\\Bhorn\\Desktop\\SMATest\\runs\\full_s3",
+  "max_steps": 12000,
+  "warmup": 200,
+  "lr_max": 0.0006,
+  "lr_min": 6e-05,
+  "wd": 0.1,
+  "batch_size": 16,
+  "grad_accum": 2,
+  "block_size": 256,
+  "n_layer": 24,
+  "n_head": 6,
+  "n_embd": 384,
+  "vocab_size": 50257,
+  "data_seed": 3001346,
+  "eval_interval": 1000,
+  "eval_batches": 20,
+  "log_interval": 20
+}

full_s3/final.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:484ac9a39316cb184ca5e19e871ca977fa8529d4c3bafaa6613ee6e3aeef2eef
+size 256003111

full_s4/config.json

@@ -0,0 +1,21 @@
+{
+  "variant": "full",
+  "seed": 4,
+  "out": "C:\\Users\\Bhorn\\Desktop\\SMATest\\runs\\full_s4",
+  "max_steps": 12000,
+  "warmup": 200,
+  "lr_max": 0.0006,
+  "lr_min": 6e-05,
+  "wd": 0.1,
+  "batch_size": 16,
+  "grad_accum": 2,
+  "block_size": 256,
+  "n_layer": 24,
+  "n_head": 6,
+  "n_embd": 384,
+  "vocab_size": 50257,
+  "data_seed": 4001349,
+  "eval_interval": 1000,
+  "eval_batches": 20,
+  "log_interval": 20
+}

full_s4/final.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:324684aa157a690a58646c82d7c760e6278b709168dc4be9372d2c0778f0e999
+size 256003111

g_only_s0/config.json

@@ -0,0 +1,21 @@
+{
+  "variant": "g_only",
+  "seed": 0,
+  "out": "C:\\Users\\Bhorn\\Desktop\\SMATest\\runs\\g_only_s0",
+  "max_steps": 12000,
+  "warmup": 200,
+  "lr_max": 0.0006,
+  "lr_min": 6e-05,
+  "wd": 0.1,
+  "batch_size": 16,
+  "grad_accum": 2,
+  "block_size": 256,
+  "n_layer": 24,
+  "n_head": 6,
+  "n_embd": 384,
+  "vocab_size": 50257,
+  "data_seed": 1337,
+  "eval_interval": 500,
+  "eval_batches": 40,
+  "log_interval": 20
+}

g_only_s0/final.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0fb40abef9f7c1c7d9b7557e45a4bf5793c3ed6a875cbe9faef05d3b58c23b64
+size 253629735

g_only_s1/config.json

@@ -0,0 +1,21 @@
+{
+  "variant": "g_only",
+  "seed": 1,
+  "out": "C:\\Users\\Bhorn\\Desktop\\SMATest\\runs\\g_only_s1",
+  "max_steps": 12000,
+  "warmup": 200,
+  "lr_max": 0.0006,
+  "lr_min": 6e-05,
+  "wd": 0.1,
+  "batch_size": 16,
+  "grad_accum": 2,
+  "block_size": 256,
+  "n_layer": 24,
+  "n_head": 6,
+  "n_embd": 384,
+  "vocab_size": 50257,
+  "data_seed": 1001340,
+  "eval_interval": 500,
+  "eval_batches": 40,
+  "log_interval": 20
+}

g_only_s1/final.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0904b072959de61365838c737bf14d2ec72dbffbc5a02fcfa7fa49895a8fed85
+size 253629735