Training used bf16 MIXED precision (HF Trainer bf16=True), so the optimizer kept FP32
master weights and the checkpoint was saved in FP32. Casting the parameters to bfloat16
halves the download; verified argmax-identical on the mask probes, with confident
predictions unchanged to ~3 decimal places (Kamer 0.9716 -> 0.9718).
The RoPE buffer (model.freqs_cis_real) is deliberately KEPT in float32:
torch.view_as_complex accepts float32/float64 only and rejects bfloat16, so casting it
would break the forward pass. model.py's freqs_cis property now also calls .float()
before the complex view, so the model is safe even if a loader casts buffers to the
config dtype.
Model: swaps the run-4 backbone (~91B tokens, pool included a per-document CC-licensed
CommonCrawl subset and machine-translated FineWeb-Edu) for run-2 (~27B tokens, 13,000 steps),
whose pool has no web-crawl lineage of any kind. run-2 also scores higher on DUMB
(squadnl F1 69.6 vs 61.7, dbrd 93.3 vs 91.1) under an identical fine-tuning recipe.
Fixes shipped with the modeling code:
* RoPE: freqs_cis was a NON-persistent complex64 buffer, so it was absent from the checkpoint
and transformers>=5 materialised it uninitialised -> NaN logits for every input. It is now
persisted as a real-valued float32 [L, D/2, 2] buffer (transformers' safetensors loader
rejects complex dtypes: "Cannot load safetensors of unknown dtype C64") and exposed as a
complex tensor via a `freqs_cis` property. Verified: finite and correct under both
transformers 4.57.6 and 5.13.0.
* SwiGLU: xformers is now optional. The code falls back to a pure-torch SwiGLU with the
identical packed w12/w3 layout, verif…
