Blame for README.md

---
license: apache-2.0
language:
- en
tags:
- zeropoint
- zerocg
- numpy
- from-scratch
- sequence-modeling
- memory-slots
- linear-attention
- research
pretty_name: ZeroPoint
---

# ZeroPoint — a new, from-scratch framework for sequence modeling

**Status: v0 research prototype (November 2026). This is a brand-new framework: it is not a
fine-tune, not a wrapper around PyTorch / `transformers`, and not a derivative of an existing
released model. Both the framework and the architecture are implemented here from scratch,
in pure Python + NumPy.**

ZeroPoint is two things:

1. **ZeroCG** — a small deep-learning framework written from scratch: `Tensor` with
   reverse-mode automatic differentiation, `nn` layers, AdamW (checkpointable), and
   matrix-multiply FLOP counting. Dependencies: Python 3 + NumPy. Nothing else.
2. **ZeroPoint** — a sequence architecture that replaces self-attention with a set of
   **memory slots that forget at different, learnable speeds**. Training uses a
   **chunked-parallel** form (chunk = 32); inference uses a **per-character recurrence**
   whose state size is fixed and independent of text length, and it needs **no positional
   encoding**.

**This repository contains no pretrained weights.** It is an architecture + framework
release, together with the full experiment suite that compares ZeroPoint against a
GPT-style Transformer baseline at ~115k parameters and a character level. A separate
1.4M-parameter model trained on a private corpus (internal name *ZeroX*) is **not** part
of this release, and neither are its weights, corpus, or training runs.

---

## TL;DR — the measured comparison

On the only head-to-head experiment (same data, same parameter budget, same training
recipe, same evaluation), **ZeroPoint and the Transformer are statistically tied** under
the project's own "difference > 2× seed std" rule. We do **not** claim ZeroPoint is better.

| model | params | val loss (nats/char, 3 seeds) | matmul FLOPs/char @T=64 | inference state @64 / @1024 / @4096 (floats) |
|---|---|---|---|---|
| Transformer (learned PE) | 116,576 | 1.5452 ± 0.0100 | 241,664 | 16,384 / 262,144 / 1,048,576 |
| Transformer (sinusoidal PE) | 112,480 | 1.5271 ± 0.0083 | 241,664 | 16,384 / 262,144 / 1,048,576 |
| **ZeroPoint** | **115,240** | **1.5214 ± 0.0134** | **232,448** | **2,048 / 2,048 / 2,048** |

- ZeroPoint − Transformer = 0.0238; 2 × max seed std = 0.0268 → **ratio 0.89, not
  significant**. ZeroPoint − Transformer(sin) = 0.0057 → ratio 0.21, not significant.
- The transformer's KV cache is **8×** ZeroPoint's state at 64 characters, **128×** at
  1,024 characters and **512×** at 4,096 characters. ZeroPoint's state is 2,048 floats
  (8 KiB in fp32) at any length.
- The Transformer's per-character matmul cost grows with context (2,306,048 FLOPs/char at
  4,096) while ZeroPoint's stays ~constant (234,464). **Wall-clock speed was not measured**
  — this release ships FLOP counts, not throughput numbers.

Read [**Comparison conditions**](#comparison-conditions-read-before-quoting-any-number)
before quoting any number, and [**Limitations**](#limitations) before drawing conclusions.

---

## What is actually new (and what is not)

**New code.** `zerocg/` is a self-contained NumPy implementation: reverse-mode autograd,
layers, AdamW, checkpointing, FLOP counting. `models/zeropoint.py` is the architecture
implementation. Neither depends on PyTorch; the early PyTorch reference scripts of the
project are not part of this release and were never used for the results.

**New combination.** The specific design — **per-slot learnable decay λ with a geometric
half-life initialisation (1.5 → 64 characters) + softmax competition when writing +
a write-strength gate + chunked-parallel / recurrent dual form + no positional encoding +
learnable initial slot state** — is, to our knowledge, a new combination. It is the object
of study here.

**Not new.** Every individual ingredient has prior art, and the design document says so:
memory networks / Neural Turing Machines (soft read-write memory), linear attention
(Katharopoulos et al., 2020 — parallel/recurrent dual form), RetNet (Sun et al., 2023 —
decayed recurrence without positional encoding), and Mamba / state-space models (fixed-size
recurrent state). This work makes **no priority claim** and is not "the first" anything.
It is also not a claim that memory slots are generally better than attention: at this scale
they tie.

---

## Comparison conditions (read before quoting any number)

Everything below is what the two sides of the comparison actually shared, measured on
**2026-10-11** (head-to-head and ablation) and **2026-10-18** (length re-evaluation).

| item | setting |
|---|---|
| Corpus | ASCII-only Python standard-library `.py` source, **1,558,395** characters, vocabulary **96** |
| Split | contiguous 90 % train / 10 % validation |
| Task | character-level next-character prediction |
| Models | width 64, 2 layers, 4 heads (D = 16), MLP hidden 256 (4×), Pre-LN, GELU, no dropout, no weight tying |
| ZeroPoint specifics | 16 slots/head, initial half-lives geometric 1.5→64 chars, learnable λ and s0, chunk 32, no positional encoding |
| Context length | 64 characters (training and short-context eval) |
| Batch | 32 sequences × 64 characters = 2,048 chars/step |
| Steps | 2,000 (about 2 minutes per model on one CPU core) |
| Optimizer | AdamW β = (0.9, 0.999), weight decay 0.01, gradient clip 1.0 |
| LR schedule | 100-step warmup, cosine decay to 10 % |
| LR selection | per model: sweep {1e-3, 3e-3, 6e-3} on seed 0, take the best for seeds 1–2; all three models selected 6e-3 |
| Seeds | 0, 1, 2 (seed controls initialisation and data order) |
| Evaluation | mean cross-entropy over 40 fixed validation batches × 32 sequences × 64 chars (seed 999); mean ± sample std over 3 seeds |
| Hardware | one Apple Silicon Mac, CPU only, NumPy float32 — no GPU |
| Data used | only the reports in `experiments/`; the numbers below were independently re-derived from saved checkpoints where possible |

**Not controlled / things this comparison does not tell you:**

- **One corpus, one domain, one split.** Source code is not natural language. No downstream
  tasks, no instruction following, no reasoning evals — only validation loss.
- **Tiny scale.** 115k parameters, 2 layers, 64-char context, 2,000 steps. Nothing here
  extrapolates to large models or long training.
- **The baseline is a minimal GPT-style decoder** with learned or sinusoidal positional
  encoding, Pre-LN and GELU. It is not tuned beyond a 3-value LR sweep; no RoPE/ALiBi, no
  dropout, no weight decay tuning.
- **3 seeds only**, and the LR is selected on seed 0, which is also included in the 3-seed
  mean (a winner's-curse of ≤ ~0.01 nats/char; the same protocol was used for all models).
- **Transformer checkpoints were not saved**, so the Transformer rows cannot be re-derived
  from artifacts — they rest on the original report. The ZeroPoint side *was* re-derived
  bit-for-bit (see [Data verification](#data-verification)).
- **No wall-clock or memory benchmark.** FLOPs count matrix multiplications only; CPU
  wall-clock is dominated by Python overhead and would not be a meaningful comparison.
- **Length extrapolation is evaluated at two different sample sizes** (32 sequences for
  ZeroPoint, 8 for the sinusoidal Transformer — the baseline checkpoints are gone).
- **Ablation numbers use a different fixed LR** (3e-3) than the head-to-head (6e-3), so the
  two tables are only internally comparable. Do not rank across them.
- **Exact recall is not measured** by this release: the recall experiment is inconclusive
  because of its learning-rate configuration (see below), and must not be cited as an
  architecture comparison.

---

## Head-to-head results

Protocol: identical width/layer/head/MLP configuration; parameter counts within ~4 %
(115,240 / 116,576 / 112,480); identical data, batches, seeds, optimiser, schedule and
evaluation; the only difference is the token-mixing layer (memory slots vs self-attention)
and the positional encoding.

```
validation loss (nats/char, lower is better; 3 seeds, 2000 steps, lr 6e-3)
ZeroPoint            1.5214 ± 0.0134
Transformer (sin)    1.5271 ± 0.0083
Transformer (learned) 1.5452 ± 0.0100
```

Per-seed ZeroPoint values: 1.5311 / 1.5307 / 1.5024. The largest gap (vs learned-PE
Transformer) is 0.0238 nats/char, **below the project's 2 × seed-std decision line of
0.0268**. Under a stricter standard-error criterion the gap would be ~2.5σ in ZeroPoint's
favour, but that is not the rule this project pre-registered, and 3 seeds is thin. **The
honest reading is: indistinguishable at this scale.**

### Inference state and compute

| | @64 chars | @1,024 chars | @4,096 chars |
|---|---|---|---|
| Transformer KV cache (floats) | 16,384 | 262,144 (128×) | 1,048,576 (512×) |
| ZeroPoint state (floats) | 2,048 | 2,048 | 2,048 |
| Transformer matmul FLOPs/char | 241,664 | 733,184 | 2,306,048 |
| ZeroPoint matmul FLOPs/char | 232,448 | 234,368 | 234,464 |

ZeroPoint's state is per-layer slots (2 layers × 4 heads × 16 slots × 16 dims) and is
independent of sequence length. This is the property the architecture was designed for: a
fixed-size recurrent state with no KV cache. It is **not** a claim that this is faster in
wall-clock time — see the limitations.

---

## Length behaviour (corrected 32-sequence evaluation)

Trained on 64 characters, then evaluated on 4,096-character sequences, in segments. 3
seeds, 32 validation sequences, same chunked-parallel algorithm for every model:

| positions | ZeroPoint (trained at 64) | ZeroPoint (trained at 256) |
|---|---|---|
| 0–63 | 1.4372 ± 0.0187 | 1.4741 ± 0.0117 |
| 64–255 (4× training length) | 1.4742 ± 0.0341 | 1.4486 ± 0.0098 |
| 256–1023 | 1.6617 ± 0.0234 | 1.5098 ± 0.0022 |
| 1024–4095 (64× training length) | 1.6932 ± 0.0305 | 1.5155 ± 0.0048 |
| far − near gap | **+0.2190 ± 0.0630** | **+0.0669 ± 0.0051** |

- At 4× the training length the loss rises by **0.037** nats/char; at 64× by **0.219**.
  Extrapolation degrades gracefully, but it **does** degrade — do not quote "handles 4×
  training length" without the +0.037.
- Training at 256 characters reduces the far−near gap 3.3× (to +0.067) but costs +0.036 at
  64 characters. That run also changed batch size 32 → 8 (to keep 2,048 chars/step), so the
  short-range cost is confounded with batch size; treat it as a hypothesis, not isolated.
- **Sinusoidal-PE Transformer collapses outside the training window:** 1.4512 within
  0–63, then **4.6359** at 64–255 and **4.6057** at 256–1023, versus ln 96 = 4.5642, i.e.
  chance level. This is an 8-sequence × 3-seed measurement, and the baseline checkpoints
  were not retained, so it cannot be re-run with 32 sequences. It is a real, large effect,
  but it is also a known weakness of plain sinusoidal absolute PE — **it is not evidence
  that attention in general cannot extrapolate** (relative-position and NoPE models are
  outside this comparison).

The earlier 8-sequence values (1.5550 / 1.6164 / 1.6410 / 1.7040) are kept only in the
archived report and are **superseded** by the table above; the 8-sequence sample was
noise-dominated (the same model scores 1.5550 vs 1.4372 on 0–63 depending on sample size).

---

## Ablations (fixed LR 3e-3; internally comparable only)

Each row changes one thing; parameter counts are matched within 1 % by adjusting the MLP
width; 2,000 steps, 3 seeds, same corpus. Base: **1.5500 ± 0.0073**. Decision line per
row: 2 × max(base std, row std). Δ is row − base (positive = worse).

| variant | Δ (nats/char) | Δ / threshold | verdict |
|---|---|---|---|
| uniform half-life init (no hierarchy) | **+0.0510** | 3.23× | clear: hierarchy matters |
| + sinusoidal positional encoding | **+0.0606** | 3.88× | clear: no PE is better here |
| λ frozen at initial values | **+0.0284** | 1.46× | clear: learnable decay matters |
| 1 head (64 dims, MLP widened to match) | **+0.0276** | 1.89× | clear: multiple heads matter |
| 8 heads (8 dims each, MLP narrowed) | +0.0173 | 1.19× | marginal / weak |
| 4 slots/head | −0.0034 | 0.20× | no significant difference |
| 32 slots/head | +0.0124 | 0.85× | no significant difference |
| no write-strength gate | +0.0022 | 0.15× | no significant difference |
| sigmoid write instead of softmax competition | −0.0013 | 0.09× | no significant difference |

At this scale, the things that **do** matter are the hierarchical half-life initialisation,
learnable λ, multiple heads and the absence of positional encoding. Slot count (4–32),
the write-strength gate, and write competition do **not** show a significant effect here.
Note the base loss differs from the head-to-head table because the ablation fixes
lr = 3e-3 (base 1.5500) while the head-to-head uses the selected lr = 6e-3 (1.5214).

---

## Exact recall: inconclusive — do not cite as an architecture comparison

The long-distance associative-recall experiment (`k1 v1 … kN vN [query kq] → vq`) was meant
to test the expected weakness of a fixed slot state. It produced **no usable conclusion**:

- With the original constant lr = 3e-3, both architectures plateau near 50 % on N=2
  (chance is 6 % for 16 values) and near 16–23 % for larger N, after 20,000 steps. The
  plateau is a **training-configuration** artefact, not an architecture property: with a
  1e-3 cosine schedule, a Transformer trained on N=2 alone reaches **100 %** in 8,000
  steps, and the diagnostic shows the loss ≈ ln 2 (50/50 hedge) trap.
- The mixed-N setup also dilutes the easy examples (N=2 is ~1/15 of the data), so the
  experiment needs a curriculum and/or per-N weighting before it can compare architectures.

The 20,000-step table (Transformer 50/30/23/20/18 %; ZeroPoint-16 51/32/23/20/16 %;
ZeroPoint-32 49/31/24/20/17 %) is therefore **not** evidence that ZeroPoint is worse at
recall. It is a negative result about the experimental setup. A corrected re-run is future
work.

---

## Repository layout

```
zerocg/                 from-scratch NumPy framework (autograd, nn, AdamW, FLOP counter)
models/zeropoint.py     ZeroPoint architecture + the small char-LM harness used in the experiments
models/transformer.py   GPT-style Transformer baseline (learned or sinusoidal PE)
experiments/            compare / ablation / recall (+ recall_diag), length/ema/slot diagnostics
experiments/results/    dated archives of the reports (kept, annotated as historical)
tests/                  numerical gradient checks, parallel=recurrent equivalence, golden reference
docs/ZeroPoint.md       design document (Chinese; structured after "Attention Is All You Need")
docs/ZeroPoint_EN.md    English translation of the design document
docs/AUDIT.md           independent verification notes for the numbers in this card
```

## Reproducing

```bash
pip install numpy
python tests/test_zerocg.py          # framework self-checks
python tests/test_zeropoint.py       # architecture + parallel/recurrent equivalence (~20 s)
cd experiments
python compare.py                    # ~35 min on one CPU core
python ablation.py                   # ~70 min
python recall.py                     # ~15 min; known inconclusive (see above)
```

Experiments are pure NumPy on CPU. `compare.py` / `ablation.py` write a dated archive to
`experiments/results/` and update the latest report in `experiments/`.

---

## Data verification

The numbers in this card were independently audited, not merely copied from the project's
own reports:

- The **ZeroPoint side was re-derived bit-for-bit** from the saved diagnostic checkpoints
  (`val64 = 1.5214 ± 0.0134`, per-seed 1.5311 / 1.5307 / 1.5024; the 32-sequence segment
  values above; the ablation parameter counts). The Transformer checkpoints were never
  saved, so those rows could not be re-derived (stated as a limitation above).
- Corrections applied after the audit: the 8-sequence length-extrapolation table was
  superseded by the 32-sequence re-evaluation; the "effective write slots 3.2 / 5.0"
  number was a metric bug (entropy over a non-normalised, gate-scaled write distribution —
  the normalised values are ~6–10 write slots and ~10 read slots out of 16) and is not
  used; "precise recall is a known weakness" was downgraded to "inconclusive"; "one matrix
  multiply, as fast as attention" was corrected to "chunked parallel, one matmul per chunk";
  "~110k parameters" was corrected to the exact 115,240.
- See [`docs/AUDIT.md`](docs/AUDIT.md) for the claim-by-claim audit table.

---

## Limitations (summary)

- Small scale: 115k params, 2 layers, 64-char context, 2,000 steps, one source-code corpus.
- Only validation loss was measured; no downstream tasks, no generation-quality evaluation,
  no scaling study.
- The baseline is a minimal GPT-style decoder, not a tuned Transformer; the comparison is
  fair in configuration but does not represent "Transformers" in general.
- 3 seeds; the LR was selected on seed 0 (small selection bias).
- No wall-clock or memory measurements; FLOP counts only.
- Length extrapolation: ZeroPoint degrades +0.037 nats/char at 4× training length; the
  sinusoidal Transformer's collapse is a known property of absolute PE, measured with only
  8 sequences.
- Exact recall: inconclusive; needs a corrected training configuration.
- The framework is a research prototype, not a production framework: no GPU kernels, no
  distributed training, no mixed precision.

---

## Citation

```bibtex
@misc{zeropoint2026,
  title        = {ZeroPoint: a from-scratch NumPy framework and a hierarchical memory-slot
                  sequence architecture},
  author       = {Holmes2011},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Holmes2011/ZeroPoint}},
  note         = {v0 research prototype; architecture and framework release, no pretrained weights}
}
```

## License

Apache-2.0 (code and documentation). This repository contains **no model weights, no
training corpus, and no training runs** of the project's separate 1.4M-parameter model.

---

## 中文摘要

**ZeroPoint 是一个全新的框架，不是已有模型的封装、微调或衍生品。** 本仓库从零实现了两部分：

1. **ZeroCG**：纯 Python + NumPy 的深度学习框架 —— 反向自动求导、`nn` 层、AdamW、矩阵乘
   FLOP 统计，不依赖 PyTorch。
2. **ZeroPoint**：用一组**遗忘速度可学习、按存储层级初始化**的记忆槽取代自注意力。训练用
   **分块并行**（块长 32，块间串行传状态），推理用**逐字递推**，状态固定、与文本长度无关，
   **不需要位置编码**。

**对比条件（引用任何数字前请先看）**：Python 标准库源码 1,558,395 字符、字符级、词表 96；
三个模型同宽 64 / 2 层 / 4 头 / MLP 256 / Pre-LN / GELU；上下文 64、batch 32、2000 步、
AdamW、100 步预热 + 余弦、学习率在 {1e-3, 3e-3, 6e-3} 里按种子 0 选（三者都选 6e-3）、
种子 0/1/2；验证 loss = 40 个固定 batch × 32 序列 × 64 字的平均交叉熵；单机 CPU、NumPy。

**主要结果**：

- 头对头验证 loss：ZeroPoint **1.5214 ± 0.0134**，Transformer **1.5452 ± 0.0100**，
  正弦位置编码 Transformer **1.5271 ± 0.0083**。最大差 0.0238 < 2× 种子标准差 0.0268
  → **统计上没有差别，不声称 ZeroPoint 更好**。
- 推理状态：ZeroPoint 固定 2,048 个浮点数（8 KiB fp32）；Transformer KV 缓存在第 64 字时
  16,384（8×）、第 1024 字时 262,144（**128×**）、第 4096 字时 1,048,576（**512×**）。
- 长度外推：训练 64 字，4 倍长度 loss 升高 **0.037**，64 倍长度升高 **0.219 ± 0.063**；
  正弦位置编码的 Transformer 出训练长度后崩到约 4.64（≈随机水平；只有 8 条序列的测量）。
- 消融：分层半衰期初始化（+0.051）、可学习 λ（+0.028）、多头（1 头 +0.028）、去掉位置编码
  （+0.061）有明确作用；槽数 4~32、写强度门、写入竞争在该规模下**无显著差异**。
- 精确回忆：**无结论**（学习率配置问题），不能用来比较架构。

**本仓库不含训练权重**：项目里另一个 1.4M 参数的模型（内部名 ZeroX）及其语料、训练运行
都不在本次发布范围内。

许可证：Apache-2.0。设计文档中文原文见 [`docs/ZeroPoint.md`](docs/ZeroPoint.md)，
英文版见 [`docs/ZeroPoint_EN.md`](docs/ZeroPoint_EN.md)。