ZeroPoint — a new, from-scratch framework for sequence modeling

Status: v0 research prototype (November 2026). This is a brand-new framework: it is not a fine-tune, not a wrapper around PyTorch / transformers, and not a derivative of an existing released model. Both the framework and the architecture are implemented here from scratch, in pure Python + NumPy.

ZeroPoint is two things:

ZeroCG — a small deep-learning framework written from scratch: Tensor with reverse-mode automatic differentiation, nn layers, AdamW (checkpointable), and matrix-multiply FLOP counting. Dependencies: Python 3 + NumPy. Nothing else.
ZeroPoint — a sequence architecture that replaces self-attention with a set of memory slots that forget at different, learnable speeds. Training uses a chunked-parallel form (chunk = 32); inference uses a per-character recurrence whose state size is fixed and independent of text length, and it needs no positional encoding.

This repository contains no pretrained weights. It is an architecture + framework release, together with the full experiment suite that compares ZeroPoint against a GPT-style Transformer baseline at ~115k parameters and a character level. A separate 1.4M-parameter model trained on a private corpus (internal name ZeroX) is not part of this release, and neither are its weights, corpus, or training runs.

TL;DR — the measured comparison

On the only head-to-head experiment (same data, same parameter budget, same training recipe, same evaluation), ZeroPoint and the Transformer are statistically tied under the project's own "difference > 2× seed std" rule. We do not claim ZeroPoint is better.

model	params	val loss (nats/char, 3 seeds)	matmul FLOPs/char @T=64	inference state @64 / @1024 / @4096 (floats)
Transformer (learned PE)	116,576	1.5452 ± 0.0100	241,664	16,384 / 262,144 / 1,048,576
Transformer (sinusoidal PE)	112,480	1.5271 ± 0.0083	241,664	16,384 / 262,144 / 1,048,576
ZeroPoint	115,240	1.5214 ± 0.0134	232,448	2,048 / 2,048 / 2,048

ZeroPoint − Transformer = 0.0238; 2 × max seed std = 0.0268 → ratio 0.89, not significant. ZeroPoint − Transformer(sin) = 0.0057 → ratio 0.21, not significant.
The transformer's KV cache is 8× ZeroPoint's state at 64 characters, 128× at 1,024 characters and 512× at 4,096 characters. ZeroPoint's state is 2,048 floats (8 KiB in fp32) at any length.
The Transformer's per-character matmul cost grows with context (2,306,048 FLOPs/char at 4,096) while ZeroPoint's stays ~constant (234,464). Wall-clock speed was not measured — this release ships FLOP counts, not throughput numbers.

Read Comparison conditions before quoting any number, and Limitations before drawing conclusions.

What is actually new (and what is not)

New code. zerocg/ is a self-contained NumPy implementation: reverse-mode autograd, layers, AdamW, checkpointing, FLOP counting. models/zeropoint.py is the architecture implementation. Neither depends on PyTorch; the early PyTorch reference scripts of the project are not part of this release and were never used for the results.

New combination. The specific design — per-slot learnable decay λ with a geometric half-life initialisation (1.5 → 64 characters) + softmax competition when writing + a write-strength gate + chunked-parallel / recurrent dual form + no positional encoding + learnable initial slot state — is, to our knowledge, a new combination. It is the object of study here.

Not new. Every individual ingredient has prior art, and the design document says so: memory networks / Neural Turing Machines (soft read-write memory), linear attention (Katharopoulos et al., 2020 — parallel/recurrent dual form), RetNet (Sun et al., 2023 — decayed recurrence without positional encoding), and Mamba / state-space models (fixed-size recurrent state). This work makes no priority claim and is not "the first" anything. It is also not a claim that memory slots are generally better than attention: at this scale they tie.

Comparison conditions (read before quoting any number)

Everything below is what the two sides of the comparison actually shared, measured on 2026-10-11 (head-to-head and ablation) and 2026-10-18 (length re-evaluation).

item	setting
Corpus	ASCII-only Python standard-library `.py` source, 1,558,395 characters, vocabulary 96
Split	contiguous 90 % train / 10 % validation
Task	character-level next-character prediction
Models	width 64, 2 layers, 4 heads (D = 16), MLP hidden 256 (4×), Pre-LN, GELU, no dropout, no weight tying
ZeroPoint specifics	16 slots/head, initial half-lives geometric 1.5→64 chars, learnable λ and s0, chunk 32, no positional encoding
Context length	64 characters (training and short-context eval)
Batch	32 sequences × 64 characters = 2,048 chars/step
Steps	2,000 (about 2 minutes per model on one CPU core)
Optimizer	AdamW β = (0.9, 0.999), weight decay 0.01, gradient clip 1.0
LR schedule	100-step warmup, cosine decay to 10 %
LR selection	per model: sweep {1e-3, 3e-3, 6e-3} on seed 0, take the best for seeds 1–2; all three models selected 6e-3
Seeds	0, 1, 2 (seed controls initialisation and data order)
Evaluation	mean cross-entropy over 40 fixed validation batches × 32 sequences × 64 chars (seed 999); mean ± sample std over 3 seeds
Hardware	one Apple Silicon Mac, CPU only, NumPy float32 — no GPU
Data used	only the reports in `experiments/`; the numbers below were independently re-derived from saved checkpoints where possible

Not controlled / things this comparison does not tell you:

One corpus, one domain, one split. Source code is not natural language. No downstream tasks, no instruction following, no reasoning evals — only validation loss.
Tiny scale. 115k parameters, 2 layers, 64-char context, 2,000 steps. Nothing here extrapolates to large models or long training.
The baseline is a minimal GPT-style decoder with learned or sinusoidal positional encoding, Pre-LN and GELU. It is not tuned beyond a 3-value LR sweep; no RoPE/ALiBi, no dropout, no weight decay tuning.
3 seeds only, and the LR is selected on seed 0, which is also included in the 3-seed mean (a winner's-curse of ≤ ~0.01 nats/char; the same protocol was used for all models).
Transformer checkpoints were not saved, so the Transformer rows cannot be re-derived from artifacts — they rest on the original report. The ZeroPoint side was re-derived bit-for-bit (see Data verification).
No wall-clock or memory benchmark. FLOPs count matrix multiplications only; CPU wall-clock is dominated by Python overhead and would not be a meaningful comparison.
Length extrapolation is evaluated at two different sample sizes (32 sequences for ZeroPoint, 8 for the sinusoidal Transformer — the baseline checkpoints are gone).
Ablation numbers use a different fixed LR (3e-3) than the head-to-head (6e-3), so the two tables are only internally comparable. Do not rank across them.
Exact recall is not measured by this release: the recall experiment is inconclusive because of its learning-rate configuration (see below), and must not be cited as an architecture comparison.

Head-to-head results

Protocol: identical width/layer/head/MLP configuration; parameter counts within ~4 % (115,240 / 116,576 / 112,480); identical data, batches, seeds, optimiser, schedule and evaluation; the only difference is the token-mixing layer (memory slots vs self-attention) and the positional encoding.

validation loss (nats/char, lower is better; 3 seeds, 2000 steps, lr 6e-3)
ZeroPoint            1.5214 ± 0.0134
Transformer (sin)    1.5271 ± 0.0083
Transformer (learned) 1.5452 ± 0.0100

Per-seed ZeroPoint values: 1.5311 / 1.5307 / 1.5024. The largest gap (vs learned-PE Transformer) is 0.0238 nats/char, below the project's 2 × seed-std decision line of 0.0268. Under a stricter standard-error criterion the gap would be ~2.5σ in ZeroPoint's favour, but that is not the rule this project pre-registered, and 3 seeds is thin. The honest reading is: indistinguishable at this scale.

Inference state and compute

	@64 chars	@1,024 chars	@4,096 chars
Transformer KV cache (floats)	16,384	262,144 (128×)	1,048,576 (512×)
ZeroPoint state (floats)	2,048	2,048	2,048
Transformer matmul FLOPs/char	241,664	733,184	2,306,048
ZeroPoint matmul FLOPs/char	232,448	234,368	234,464

ZeroPoint's state is per-layer slots (2 layers × 4 heads × 16 slots × 16 dims) and is independent of sequence length. This is the property the architecture was designed for: a fixed-size recurrent state with no KV cache. It is not a claim that this is faster in wall-clock time — see the limitations.

Length behaviour (corrected 32-sequence evaluation)

Trained on 64 characters, then evaluated on 4,096-character sequences, in segments. 3 seeds, 32 validation sequences, same chunked-parallel algorithm for every model:

positions	ZeroPoint (trained at 64)	ZeroPoint (trained at 256)
0–63	1.4372 ± 0.0187	1.4741 ± 0.0117
64–255 (4× training length)	1.4742 ± 0.0341	1.4486 ± 0.0098
256–1023	1.6617 ± 0.0234	1.5098 ± 0.0022
1024–4095 (64× training length)	1.6932 ± 0.0305	1.5155 ± 0.0048
far − near gap	+0.2190 ± 0.0630	+0.0669 ± 0.0051

At 4× the training length the loss rises by 0.037 nats/char; at 64× by 0.219. Extrapolation degrades gracefully, but it does degrade — do not quote "handles 4× training length" without the +0.037.
Training at 256 characters reduces the far−near gap 3.3× (to +0.067) but costs +0.036 at 64 characters. That run also changed batch size 32 → 8 (to keep 2,048 chars/step), so the short-range cost is confounded with batch size; treat it as a hypothesis, not isolated.
Sinusoidal-PE Transformer collapses outside the training window: 1.4512 within 0–63, then 4.6359 at 64–255 and 4.6057 at 256–1023, versus ln 96 = 4.5642, i.e. chance level. This is an 8-sequence × 3-seed measurement, and the baseline checkpoints were not retained, so it cannot be re-run with 32 sequences. It is a real, large effect, but it is also a known weakness of plain sinusoidal absolute PE — it is not evidence that attention in general cannot extrapolate (relative-position and NoPE models are outside this comparison).

The earlier 8-sequence values (1.5550 / 1.6164 / 1.6410 / 1.7040) are kept only in the archived report and are superseded by the table above; the 8-sequence sample was noise-dominated (the same model scores 1.5550 vs 1.4372 on 0–63 depending on sample size).

Ablations (fixed LR 3e-3; internally comparable only)

Each row changes one thing; parameter counts are matched within 1 % by adjusting the MLP width; 2,000 steps, 3 seeds, same corpus. Base: 1.5500 ± 0.0073. Decision line per row: 2 × max(base std, row std). Δ is row − base (positive = worse).

variant	Δ (nats/char)	Δ / threshold	verdict
uniform half-life init (no hierarchy)	+0.0510	3.23×	clear: hierarchy matters
+ sinusoidal positional encoding	+0.0606	3.88×	clear: no PE is better here
λ frozen at initial values	+0.0284	1.46×	clear: learnable decay matters
1 head (64 dims, MLP widened to match)	+0.0276	1.89×	clear: multiple heads matter
8 heads (8 dims each, MLP narrowed)	+0.0173	1.19×	marginal / weak
4 slots/head	−0.0034	0.20×	no significant difference
32 slots/head	+0.0124	0.85×	no significant difference
no write-strength gate	+0.0022	0.15×	no significant difference
sigmoid write instead of softmax competition	−0.0013	0.09×	no significant difference

At this scale, the things that do matter are the hierarchical half-life initialisation, learnable λ, multiple heads and the absence of positional encoding. Slot count (4–32), the write-strength gate, and write competition do not show a significant effect here. Note the base loss differs from the head-to-head table because the ablation fixes lr = 3e-3 (base 1.5500) while the head-to-head uses the selected lr = 6e-3 (1.5214).

Exact recall: inconclusive — do not cite as an architecture comparison

The long-distance associative-recall experiment (k1 v1 … kN vN [query kq] → vq) was meant to test the expected weakness of a fixed slot state. It produced no usable conclusion:

With the original constant lr = 3e-3, both architectures plateau near 50 % on N=2 (chance is 6 % for 16 values) and near 16–23 % for larger N, after 20,000 steps. The plateau is a training-configuration artefact, not an architecture property: with a 1e-3 cosine schedule, a Transformer trained on N=2 alone reaches 100 % in 8,000 steps, and the diagnostic shows the loss ≈ ln 2 (50/50 hedge) trap.
The mixed-N setup also dilutes the easy examples (N=2 is ~1/15 of the data), so the experiment needs a curriculum and/or per-N weighting before it can compare architectures.

The 20,000-step table (Transformer 50/30/23/20/18 %; ZeroPoint-16 51/32/23/20/16 %; ZeroPoint-32 49/31/24/20/17 %) is therefore not evidence that ZeroPoint is worse at recall. It is a negative result about the experimental setup. A corrected re-run is future work.

Repository layout

zerocg/                 from-scratch NumPy framework (autograd, nn, AdamW, FLOP counter)
models/zeropoint.py     ZeroPoint architecture + the small char-LM harness used in the experiments
models/transformer.py   GPT-style Transformer baseline (learned or sinusoidal PE)
experiments/            compare / ablation / recall (+ recall_diag), length/ema/slot diagnostics
experiments/results/    dated archives of the reports (kept, annotated as historical)
tests/                  numerical gradient checks, parallel=recurrent equivalence, golden reference
docs/ZeroPoint.md       design document (Chinese; structured after "Attention Is All You Need")
docs/ZeroPoint_EN.md    English translation of the design document
docs/AUDIT.md           independent verification notes for the numbers in this card

Reproducing

pip install numpy
python tests/test_zerocg.py          # framework self-checks
python tests/test_zeropoint.py       # architecture + parallel/recurrent equivalence (~20 s)
cd experiments
python compare.py                    # ~35 min on one CPU core
python ablation.py                   # ~70 min
python recall.py                     # ~15 min; known inconclusive (see above)

Experiments are pure NumPy on CPU. compare.py / ablation.py write a dated archive to experiments/results/ and update the latest report in experiments/.

Data verification

The numbers in this card were independently audited, not merely copied from the project's own reports:

The ZeroPoint side was re-derived bit-for-bit from the saved diagnostic checkpoints (val64 = 1.5214 ± 0.0134, per-seed 1.5311 / 1.5307 / 1.5024; the 32-sequence segment values above; the ablation parameter counts). The Transformer checkpoints were never saved, so those rows could not be re-derived (stated as a limitation above).
Corrections applied after the audit: the 8-sequence length-extrapolation table was superseded by the 32-sequence re-evaluation; the "effective write slots 3.2 / 5.0" number was a metric bug (entropy over a non-normalised, gate-scaled write distribution — the normalised values are 6–10 write slots and ~10 read slots out of 16) and is not used; "precise recall is a known weakness" was downgraded to "inconclusive"; "one matrix multiply, as fast as attention" was corrected to "chunked parallel, one matmul per chunk"; "110k parameters" was corrected to the exact 115,240.
See docs/AUDIT.md for the claim-by-claim audit table.

Limitations (summary)

Small scale: 115k params, 2 layers, 64-char context, 2,000 steps, one source-code corpus.
Only validation loss was measured; no downstream tasks, no generation-quality evaluation, no scaling study.
The baseline is a minimal GPT-style decoder, not a tuned Transformer; the comparison is fair in configuration but does not represent "Transformers" in general.
3 seeds; the LR was selected on seed 0 (small selection bias).
No wall-clock or memory measurements; FLOP counts only.
Length extrapolation: ZeroPoint degrades +0.037 nats/char at 4× training length; the sinusoidal Transformer's collapse is a known property of absolute PE, measured with only 8 sequences.
Exact recall: inconclusive; needs a corrected training configuration.
The framework is a research prototype, not a production framework: no GPU kernels, no distributed training, no mixed precision.

Citation

@misc{zeropoint2026,
  title        = {ZeroPoint: a from-scratch NumPy framework and a hierarchical memory-slot
                  sequence architecture},
  author       = {Holmes2011},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Holmes2011/ZeroPoint}},
  note         = {v0 research prototype; architecture and framework release, no pretrained weights}
}

License

Apache-2.0 (code and documentation). This repository contains no model weights, no training corpus, and no training runs of the project's separate 1.4M-parameter model.

中文摘要

ZeroPoint 是一个全新的框架，不是已有模型的封装、微调或衍生品。 本仓库从零实现了两部分：

ZeroCG：纯 Python + NumPy 的深度学习框架 —— 反向自动求导、nn 层、AdamW、矩阵乘 FLOP 统计，不依赖 PyTorch。
ZeroPoint：用一组遗忘速度可学习、按存储层级初始化的记忆槽取代自注意力。训练用 分块并行（块长 32，块间串行传状态），推理用逐字递推，状态固定、与文本长度无关， 不需要位置编码。

对比条件（引用任何数字前请先看）：Python 标准库源码 1,558,395 字符、字符级、词表 96；三个模型同宽 64 / 2 层 / 4 头 / MLP 256 / Pre-LN / GELU；上下文 64、batch 32、2000 步、 AdamW、100 步预热 + 余弦、学习率在 {1e-3, 3e-3, 6e-3} 里按种子 0 选（三者都选 6e-3）、种子 0/1/2；验证 loss = 40 个固定 batch × 32 序列 × 64 字的平均交叉熵；单机 CPU、NumPy。

主要结果：

头对头验证 loss：ZeroPoint 1.5214 ± 0.0134，Transformer 1.5452 ± 0.0100，正弦位置编码 Transformer 1.5271 ± 0.0083。最大差 0.0238 < 2× 种子标准差 0.0268 → 统计上没有差别，不声称 ZeroPoint 更好。
推理状态：ZeroPoint 固定 2,048 个浮点数（8 KiB fp32）；Transformer KV 缓存在第 64 字时 16,384（8×）、第 1024 字时 262,144（128×）、第 4096 字时 1,048,576（512×）。
长度外推：训练 64 字，4 倍长度 loss 升高 0.037，64 倍长度升高 0.219 ± 0.063；正弦位置编码的 Transformer 出训练长度后崩到约 4.64（≈随机水平；只有 8 条序列的测量）。
消融：分层半衰期初始化（+0.051）、可学习 λ（+0.028）、多头（1 头 +0.028）、去掉位置编码（+0.061）有明确作用；槽数 4~32、写强度门、写入竞争在该规模下无显著差异。
精确回忆：无结论（学习率配置问题），不能用来比较架构。

本仓库不含训练权重：项目里另一个 1.4M 参数的模型（内部名 ZeroX）及其语料、训练运行都不在本次发布范围内。

许可证：Apache-2.0。设计文档中文原文见 docs/ZeroPoint.md，英文版见 docs/ZeroPoint_EN.md。

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support