OxMini ablation results

Measured 2026-08-28T14:38:35.260789+00:00 on Apple M4 Pro (macOS-26.5.2-arm64) using CPU PyTorch. All variants used the same character data, seed (1337), context length (256), width/depth, optimizer recipe, batch size (4), and update count (120). Parameter spread is 2.84%.

Variant	Parameters	Steps	Final val loss	Test loss	Test PPL	Next-char acc.	Wall time
Plain GPT (full attention, 1 stream)	2,107,776	120	2.8228	2.8057	16.54	26.17%	17.1s
Hybrid KDA-lite/MLA-lite (1 stream)	2,168,984	120	2.4979	2.4867	12.02	37.35%	150.2s
OxMini hybrid + mHC-lite (4 streams)	2,169,388	120	2.5595	2.5477	12.78	35.21%	153.5s

Interpretation

The lowest held-out loss in this short run came from Hybrid KDA-lite/MLA-lite (1 stream). This is a toy-scale, single-seed comparison, so small differences are descriptive, not evidence that an architecture is generally superior. In particular, the approximated KDA and mHC mechanisms are expected to earn their keep at scales and context lengths far beyond this experiment.

Unedited samples for every arm are preserved in ABLATION_SAMPLES.md and beside local metrics under checkpoints/ablations/<variant>/samples.md.

The plain GPT arm was about nine times faster than either recurrent arm because KDA-lite is an explicit correctness-first Python scan. Samples from all three 120-step arms were mostly garbled; hybrid/full samples contained somewhat more word-like fragments, but no arm produced strong qualitative text. mHC-lite did not improve either held-out loss or next-character accuracy over hybrid-only in this run.

Machine-readable results

{
  "batch_size": 4,
  "parameter_spread": 0.028400636492872645,
  "seed": 1337,
  "steps": 120,
  "variants": {
    "baseline": {
      "best_val_loss": 2.822754889726639,
      "elapsed_seconds": 17.095340332998603,
      "final_val_loss": 2.82275489,
      "next_char_accuracy": 0.26171875,
      "parameters": 2107776,
      "steps": 120,
      "test_loss": 2.805739849805832,
      "test_perplexity": 16.539307984355137
    },
    "full": {
      "best_val_loss": 2.5594610571861267,
      "elapsed_seconds": 153.5054647920042,
      "final_val_loss": 2.55946106,
      "next_char_accuracy": 0.35205078125,
      "parameters": 2169388,
      "steps": 120,
      "test_loss": 2.547688275575638,
      "test_perplexity": 12.777531482717045
    },
    "hybrid": {
      "best_val_loss": 2.497876286506653,
      "elapsed_seconds": 150.20865466599935,
      "final_val_loss": 2.49787629,
      "next_char_accuracy": 0.37353515625,
      "parameters": 2168984,
      "steps": 120,
      "test_loss": 2.486685276031494,
      "test_perplexity": 12.021362507248224
    }
  }
}