| # OxMini ablation results |
|
|
| Measured 2026-08-28T14:38:35.260789+00:00 on Apple M4 Pro (`macOS-26.5.2-arm64`) using CPU PyTorch. |
| All variants used the same character data, seed (1337), context length (256), width/depth, optimizer recipe, batch size (4), and update count (120). Parameter spread is 2.84%. |
|
|
| | Variant | Parameters | Steps | Final val loss | Test loss | Test PPL | Next-char acc. | Wall time | |
| |---|---:|---:|---:|---:|---:|---:|---:| |
| | Plain GPT (full attention, 1 stream) | 2,107,776 | 120 | 2.8228 | 2.8057 | 16.54 | 26.17% | 17.1s | |
| | Hybrid KDA-lite/MLA-lite (1 stream) | 2,168,984 | 120 | 2.4979 | 2.4867 | 12.02 | 37.35% | 150.2s | |
| | OxMini hybrid + mHC-lite (4 streams) | 2,169,388 | 120 | 2.5595 | 2.5477 | 12.78 | 35.21% | 153.5s | |
|
|
| ## Interpretation |
|
|
| The lowest held-out loss in this short run came from **Hybrid KDA-lite/MLA-lite (1 stream)**. This is a toy-scale, single-seed comparison, so small differences are descriptive, not evidence that an architecture is generally superior. In particular, the approximated KDA and mHC mechanisms are expected to earn their keep at scales and context lengths far beyond this experiment. |
|
|
| Unedited samples for every arm are preserved in [`ABLATION_SAMPLES.md`](ABLATION_SAMPLES.md) and beside local metrics under `checkpoints/ablations/<variant>/samples.md`. |
|
|
| The plain GPT arm was about nine times faster than either recurrent arm because KDA-lite is an explicit correctness-first Python scan. Samples from all three 120-step arms were mostly garbled; hybrid/full samples contained somewhat more word-like fragments, but no arm produced strong qualitative text. mHC-lite did not improve either held-out loss or next-character accuracy over hybrid-only in this run. |
|
|
| ## Machine-readable results |
|
|
| ```json |
| { |
| "batch_size": 4, |
| "parameter_spread": 0.028400636492872645, |
| "seed": 1337, |
| "steps": 120, |
| "variants": { |
| "baseline": { |
| "best_val_loss": 2.822754889726639, |
| "elapsed_seconds": 17.095340332998603, |
| "final_val_loss": 2.82275489, |
| "next_char_accuracy": 0.26171875, |
| "parameters": 2107776, |
| "steps": 120, |
| "test_loss": 2.805739849805832, |
| "test_perplexity": 16.539307984355137 |
| }, |
| "full": { |
| "best_val_loss": 2.5594610571861267, |
| "elapsed_seconds": 153.5054647920042, |
| "final_val_loss": 2.55946106, |
| "next_char_accuracy": 0.35205078125, |
| "parameters": 2169388, |
| "steps": 120, |
| "test_loss": 2.547688275575638, |
| "test_perplexity": 12.777531482717045 |
| }, |
| "hybrid": { |
| "best_val_loss": 2.497876286506653, |
| "elapsed_seconds": 150.20865466599935, |
| "final_val_loss": 2.49787629, |
| "next_char_accuracy": 0.37353515625, |
| "parameters": 2168984, |
| "steps": 120, |
| "test_loss": 2.486685276031494, |
| "test_perplexity": 12.021362507248224 |
| } |
| } |
| } |
| ``` |
|
|
