Independent audit of the numbers in this release
This document records how the numbers in README.md and docs/ZeroPoint*.md were
checked. The audit was done independently of the project's own summaries: every
publishable number was re-derived from the saved checkpoints where possible, or checked
arithmetically against the reports. No training run was performed for the audit.
What was reproducible
Four trained ZeroPoint checkpoints from the comparison/diagnostic experiments were kept
(experiments/out/diag_* in the private source tree; not shipped here — they are not
needed, the experiment scripts retrain them in minutes). Loading them and re-running the
documented evaluation protocols reproduced, bit-for-bit:
| quantity | re-derived value |
|---|---|
| 64-char validation loss (3 seeds) | 1.5214 ± 0.0134 (per seed 1.5311 / 1.5307 / 1.5024) |
| 8-sequence length segments | 1.5550 / 1.6164 / 1.6410 / 1.7040 |
| 32-sequence length segments | 1.4372 / 1.4742 / 1.6617 / 1.6932; far−near +0.2190 ± 0.0630 |
| parameter counts (all ablation rows) | 115,240 base; 115,366 / 115,210 / 115,318 / 115,222 / 114,720 / 115,112 / 115,240 |
| inference state | 2,048 floats fixed; KV cache 16,384 / 262,144 / 1,048,576 at 64 / 1,024 / 4,096 |
| matmul FLOPs per char @ T=64 | ZeroPoint 232,448; Transformer 241,664 |
The Transformer baselines were not saved. Their reported rows (1.5452 ± 0.0100 and 1.5271 ± 0.0083) and the sinusoidal-PE extrapolation collapse (4.64) therefore rest on the original report and could not be re-derived; they are labelled accordingly in the card.
Claim-by-claim status
| claim | status | what the card says |
|---|---|---|
| ~110k parameters | corrected | exact 115,240 (Transformer 116,576; sin-PE 112,480) |
| validation loss "on par" | confirmed as a null result | 1.5214 vs 1.5452, diff 0.0238 < 2× std 0.0268 → not significant; no superiority claim |
| inference state 1/128 at the 1024th char | confirmed (with the position stated) | 2,048 vs 262,144 floats; 8× at 64, 512× at 4,096 |
| handles text 4× training length | confirmed, with degradation | +0.037 nats/char at 4×; +0.219 at 64× |
| training "one matmul over the whole sequence, as fast as attention" | corrected | chunked parallel (chunk 32, O(n/32) sequential steps); wall-clock not measured |
| forward compute 232K vs 242K FLOPs/char | confirmed (T=64 only) | stated together with the length dependence (Transformer 2,306,048 at 4,096) |
| length-extrapolation table (8 sequences) | superseded | replaced by 32-sequence values; old table kept only in the archived report with a note |
| Transformer(sin) collapse beyond training length | confirmed at 8 sequences | 4.64 vs 1.45, ≈ chance; N=8 caveat and "not evidence about attention in general" |
| "recurrent vs parallel algorithm" caused the 10-18 correction | corrected | the two forms agree to 1.4e-5 in float32; the real change was 8 → 32 validation sequences |
| "16 slots, only 3–5 used" (slot collapse) | corrected | metric bug (entropy on a non-normalised, gate-scaled write distribution); normalised: ~6–10 write / ~10 read slots out of 16, pooling-dependent — not used as a claim |
| ablation: hierarchy / learnable λ / no PE / heads | confirmed | +0.051 / +0.028 / +0.061 / +0.028, all above 2× max std; 8-head row marginal |
| ablation: 4/32 slots, write gate, write competition | confirmed as null | all below the decision line at this scale |
| precise recall is a "known weakness" | corrected | inconclusive: constant lr 3e-3 traps both architectures at loss ≈ ln 2; 1e-3 cosine solves N=2 |
| "32 switch combinations" in the self-check | corrected | 64 combinations (ema flag added later); parallel ≡ recurrent to < 1e-8 (float64) |
| exact recall / batch-size / calibration / long-run results | out of scope | those belong to the separate 1.4M model, which is not part of this release |
Why no "beats the Transformer" claim
Under the project's pre-registered rule (a difference counts only if it exceeds 2× the seed-to-seed standard deviation), the head-to-head gap is 0.89× the threshold. A standard- error criterion would put the gap at ~2.5σ in ZeroPoint's favour, but that criterion was not pre-registered, only 3 seeds were run, and the learning rate was selected on one of the same seeds. The card therefore says "statistically indistinguishable" and reports the raw numbers so readers can apply their own criterion.
Practical checks that ship with the code
tests/test_zerocg.py— numerical gradient checks for the framework's autograd.tests/test_zeropoint.py— all 64 switch combinations, parallel vs recurrent equivalence (< 1e-8 float64), and a fixed golden reference intests/golden/.tests/make_golden.py— regenerates the golden reference.