ZeroPoint v0 research prototype: new from-scratch NumPy framework, memory-slot architecture, audited Transformer comparison
5c6c847 verified | # Independent audit of the numbers in this release | |
| This document records how the numbers in `README.md` and `docs/ZeroPoint*.md` were | |
| checked. The audit was done independently of the project's own summaries: every | |
| publishable number was re-derived from the saved checkpoints where possible, or checked | |
| arithmetically against the reports. **No training run was performed for the audit.** | |
| ## What was reproducible | |
| Four trained ZeroPoint checkpoints from the comparison/diagnostic experiments were kept | |
| (`experiments/out/diag_*` in the private source tree; not shipped here — they are not | |
| needed, the experiment scripts retrain them in minutes). Loading them and re-running the | |
| documented evaluation protocols reproduced, **bit-for-bit**: | |
| | quantity | re-derived value | | |
| |---|---| | |
| | 64-char validation loss (3 seeds) | 1.5214 ± 0.0134 (per seed 1.5311 / 1.5307 / 1.5024) | | |
| | 8-sequence length segments | 1.5550 / 1.6164 / 1.6410 / 1.7040 | | |
| | 32-sequence length segments | 1.4372 / 1.4742 / 1.6617 / 1.6932; far−near +0.2190 ± 0.0630 | | |
| | parameter counts (all ablation rows) | 115,240 base; 115,366 / 115,210 / 115,318 / 115,222 / 114,720 / 115,112 / 115,240 | | |
| | inference state | 2,048 floats fixed; KV cache 16,384 / 262,144 / 1,048,576 at 64 / 1,024 / 4,096 | | |
| | matmul FLOPs per char @ T=64 | ZeroPoint 232,448; Transformer 241,664 | | |
| The Transformer baselines were **not** saved. Their reported rows (1.5452 ± 0.0100 and | |
| 1.5271 ± 0.0083) and the sinusoidal-PE extrapolation collapse (4.64) therefore rest on the | |
| original report and could not be re-derived; they are labelled accordingly in the card. | |
| ## Claim-by-claim status | |
| | claim | status | what the card says | | |
| |---|---|---| | |
| | ~110k parameters | corrected | exact 115,240 (Transformer 116,576; sin-PE 112,480) | | |
| | validation loss "on par" | confirmed as a null result | 1.5214 vs 1.5452, diff 0.0238 < 2× std 0.0268 → not significant; no superiority claim | | |
| | inference state 1/128 at the 1024th char | confirmed (with the position stated) | 2,048 vs 262,144 floats; 8× at 64, 512× at 4,096 | | |
| | handles text 4× training length | confirmed, with degradation | +0.037 nats/char at 4×; +0.219 at 64× | | |
| | training "one matmul over the whole sequence, as fast as attention" | corrected | chunked parallel (chunk 32, O(n/32) sequential steps); wall-clock not measured | | |
| | forward compute 232K vs 242K FLOPs/char | confirmed (T=64 only) | stated together with the length dependence (Transformer 2,306,048 at 4,096) | | |
| | length-extrapolation table (8 sequences) | superseded | replaced by 32-sequence values; old table kept only in the archived report with a note | | |
| | Transformer(sin) collapse beyond training length | confirmed at 8 sequences | 4.64 vs 1.45, ≈ chance; N=8 caveat and "not evidence about attention in general" | | |
| | "recurrent vs parallel algorithm" caused the 10-18 correction | corrected | the two forms agree to 1.4e-5 in float32; the real change was 8 → 32 validation sequences | | |
| | "16 slots, only 3–5 used" (slot collapse) | corrected | metric bug (entropy on a non-normalised, gate-scaled write distribution); normalised: ~6–10 write / ~10 read slots out of 16, pooling-dependent — not used as a claim | | |
| | ablation: hierarchy / learnable λ / no PE / heads | confirmed | +0.051 / +0.028 / +0.061 / +0.028, all above 2× max std; 8-head row marginal | | |
| | ablation: 4/32 slots, write gate, write competition | confirmed as null | all below the decision line at this scale | | |
| | precise recall is a "known weakness" | corrected | inconclusive: constant lr 3e-3 traps both architectures at loss ≈ ln 2; 1e-3 cosine solves N=2 | | |
| | "32 switch combinations" in the self-check | corrected | 64 combinations (ema flag added later); parallel ≡ recurrent to < 1e-8 (float64) | | |
| | exact recall / batch-size / calibration / long-run results | out of scope | those belong to the separate 1.4M model, which is not part of this release | | |
| ## Why no "beats the Transformer" claim | |
| Under the project's pre-registered rule (a difference counts only if it exceeds 2× the | |
| seed-to-seed standard deviation), the head-to-head gap is 0.89× the threshold. A standard- | |
| error criterion would put the gap at ~2.5σ in ZeroPoint's favour, but that criterion was | |
| not pre-registered, only 3 seeds were run, and the learning rate was selected on one of the | |
| same seeds. The card therefore says "statistically indistinguishable" and reports the raw | |
| numbers so readers can apply their own criterion. | |
| ## Practical checks that ship with the code | |
| - `tests/test_zerocg.py` — numerical gradient checks for the framework's autograd. | |
| - `tests/test_zeropoint.py` — all 64 switch combinations, parallel vs recurrent equivalence | |
| (< 1e-8 float64), and a fixed golden reference in `tests/golden/`. | |
| - `tests/make_golden.py` — regenerates the golden reference. | |