ZeroPoint v0 research prototype: new from-scratch NumPy framework, memory-slot architecture, audited Transformer comparison

.gitignore

@@ -0,0 +1,45 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+dist/
+*.egg-info/
+.venv/
+venv/
+env/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.ipynb_checkpoints/
+
+# macOS
+.DS_Store
+
+# Model weights and checkpoints (this release ships no weights)
+*.npz
+*.npy
+!tests/golden/*.npz
+*.pt
+*.pth
+*.bin
+*.safetensors
+*.ckpt
+*.pkl
+*.pickle
+checkpoints/
+weights/
+runs/
+out/
+
+# Data and training artifacts
+corpus/
+data/
+datasets/
+*.log
+
+# Secrets
+.env
+.env.*

LICENSE

@@ -0,0 +1,202 @@
+Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "{}"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 ZeroPoint contributors
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+

README.md

@@ -0,0 +1,377 @@
+---
+license: apache-2.0
+language:
+- en
+tags:
+- zeropoint
+- zerocg
+- numpy
+- from-scratch
+- sequence-modeling
+- memory-slots
+- linear-attention
+- research
+pretty_name: ZeroPoint
+---
+
+# ZeroPoint — a new, from-scratch framework for sequence modeling
+
+**Status: v0 research prototype (November 2026). This is a brand-new framework: it is not a
+fine-tune, not a wrapper around PyTorch / `transformers`, and not a derivative of an existing
+released model. Both the framework and the architecture are implemented here from scratch,
+in pure Python + NumPy.**
+
+ZeroPoint is two things:
+
+1. **ZeroCG** — a small deep-learning framework written from scratch: `Tensor` with
+   reverse-mode automatic differentiation, `nn` layers, AdamW (checkpointable), and
+   matrix-multiply FLOP counting. Dependencies: Python 3 + NumPy. Nothing else.
+2. **ZeroPoint** — a sequence architecture that replaces self-attention with a set of
+   **memory slots that forget at different, learnable speeds**. Training uses a
+   **chunked-parallel** form (chunk = 32); inference uses a **per-character recurrence**
+   whose state size is fixed and independent of text length, and it needs **no positional
+   encoding**.
+
+**This repository contains no pretrained weights.** It is an architecture + framework
+release, together with the full experiment suite that compares ZeroPoint against a
+GPT-style Transformer baseline at ~115k parameters and a character level. A separate
+1.4M-parameter model trained on a private corpus (internal name *ZeroX*) is **not** part
+of this release, and neither are its weights, corpus, or training runs.
+
+---
+
+## TL;DR — the measured comparison
+
+On the only head-to-head experiment (same data, same parameter budget, same training
+recipe, same evaluation), **ZeroPoint and the Transformer are statistically tied** under
+the project's own "difference > 2× seed std" rule. We do **not** claim ZeroPoint is better.
+
+| model | params | val loss (nats/char, 3 seeds) | matmul FLOPs/char @T=64 | inference state @64 / @1024 / @4096 (floats) |
+|---|---|---|---|---|
+| Transformer (learned PE) | 116,576 | 1.5452 ± 0.0100 | 241,664 | 16,384 / 262,144 / 1,048,576 |
+| Transformer (sinusoidal PE) | 112,480 | 1.5271 ± 0.0083 | 241,664 | 16,384 / 262,144 / 1,048,576 |
+| **ZeroPoint** | **115,240** | **1.5214 ± 0.0134** | **232,448** | **2,048 / 2,048 / 2,048** |
+
+- ZeroPoint − Transformer = 0.0238; 2 × max seed std = 0.0268 → **ratio 0.89, not
+  significant**. ZeroPoint − Transformer(sin) = 0.0057 → ratio 0.21, not significant.
+- The transformer's KV cache is **8×** ZeroPoint's state at 64 characters, **128×** at
+  1,024 characters and **512×** at 4,096 characters. ZeroPoint's state is 2,048 floats
+  (8 KiB in fp32) at any length.
+- The Transformer's per-character matmul cost grows with context (2,306,048 FLOPs/char at
+  4,096) while ZeroPoint's stays ~constant (234,464). **Wall-clock speed was not measured**
+  — this release ships FLOP counts, not throughput numbers.
+
+Read [**Comparison conditions**](#comparison-conditions-read-before-quoting-any-number)
+before quoting any number, and [**Limitations**](#limitations) before drawing conclusions.
+
+---
+
+## What is actually new (and what is not)
+
+**New code.** `zerocg/` is a self-contained NumPy implementation: reverse-mode autograd,
+layers, AdamW, checkpointing, FLOP counting. `models/zeropoint.py` is the architecture
+implementation. Neither depends on PyTorch; the early PyTorch reference scripts of the
+project are not part of this release and were never used for the results.
+
+**New combination.** The specific design — **per-slot learnable decay λ with a geometric
+half-life initialisation (1.5 → 64 characters) + softmax competition when writing +
+a write-strength gate + chunked-parallel / recurrent dual form + no positional encoding +
+learnable initial slot state** — is, to our knowledge, a new combination. It is the object
+of study here.
+
+**Not new.** Every individual ingredient has prior art, and the design document says so:
+memory networks / Neural Turing Machines (soft read-write memory), linear attention
+(Katharopoulos et al., 2020 — parallel/recurrent dual form), RetNet (Sun et al., 2023 —
+decayed recurrence without positional encoding), and Mamba / state-space models (fixed-size
+recurrent state). This work makes **no priority claim** and is not "the first" anything.
+It is also not a claim that memory slots are generally better than attention: at this scale
+they tie.
+
+---
+
+## Comparison conditions (read before quoting any number)
+
+Everything below is what the two sides of the comparison actually shared, measured on
+**2026-10-11** (head-to-head and ablation) and **2026-10-18** (length re-evaluation).
+
+| item | setting |
+|---|---|
+| Corpus | ASCII-only Python standard-library `.py` source, **1,558,395** characters, vocabulary **96** |
+| Split | contiguous 90 % train / 10 % validation |
+| Task | character-level next-character prediction |
+| Models | width 64, 2 layers, 4 heads (D = 16), MLP hidden 256 (4×), Pre-LN, GELU, no dropout, no weight tying |
+| ZeroPoint specifics | 16 slots/head, initial half-lives geometric 1.5→64 chars, learnable λ and s0, chunk 32, no positional encoding |
+| Context length | 64 characters (training and short-context eval) |
+| Batch | 32 sequences × 64 characters = 2,048 chars/step |
+| Steps | 2,000 (about 2 minutes per model on one CPU core) |
+| Optimizer | AdamW β = (0.9, 0.999), weight decay 0.01, gradient clip 1.0 |
+| LR schedule | 100-step warmup, cosine decay to 10 % |
+| LR selection | per model: sweep {1e-3, 3e-3, 6e-3} on seed 0, take the best for seeds 1–2; all three models selected 6e-3 |
+| Seeds | 0, 1, 2 (seed controls initialisation and data order) |
+| Evaluation | mean cross-entropy over 40 fixed validation batches × 32 sequences × 64 chars (seed 999); mean ± sample std over 3 seeds |
+| Hardware | one Apple Silicon Mac, CPU only, NumPy float32 — no GPU |
+| Data used | only the reports in `experiments/`; the numbers below were independently re-derived from saved checkpoints where possible |
+
+**Not controlled / things this comparison does not tell you:**
+
+- **One corpus, one domain, one split.** Source code is not natural language. No downstream
+  tasks, no instruction following, no reasoning evals — only validation loss.
+- **Tiny scale.** 115k parameters, 2 layers, 64-char context, 2,000 steps. Nothing here
+  extrapolates to large models or long training.
+- **The baseline is a minimal GPT-style decoder** with learned or sinusoidal positional
+  encoding, Pre-LN and GELU. It is not tuned beyond a 3-value LR sweep; no RoPE/ALiBi, no
+  dropout, no weight decay tuning.
+- **3 seeds only**, and the LR is selected on seed 0, which is also included in the 3-seed
+  mean (a winner's-curse of ≤ ~0.01 nats/char; the same protocol was used for all models).
+- **Transformer checkpoints were not saved**, so the Transformer rows cannot be re-derived
+  from artifacts — they rest on the original report. The ZeroPoint side *was* re-derived
+  bit-for-bit (see [Data verification](#data-verification)).
+- **No wall-clock or memory benchmark.** FLOPs count matrix multiplications only; CPU
+  wall-clock is dominated by Python overhead and would not be a meaningful comparison.
+- **Length extrapolation is evaluated at two different sample sizes** (32 sequences for
+  ZeroPoint, 8 for the sinusoidal Transformer — the baseline checkpoints are gone).
+- **Ablation numbers use a different fixed LR** (3e-3) than the head-to-head (6e-3), so the
+  two tables are only internally comparable. Do not rank across them.
+- **Exact recall is not measured** by this release: the recall experiment is inconclusive
+  because of its learning-rate configuration (see below), and must not be cited as an
+  architecture comparison.
+
+---
+
+## Head-to-head results
+
+Protocol: identical width/layer/head/MLP configuration; parameter counts within ~4 %
+(115,240 / 116,576 / 112,480); identical data, batches, seeds, optimiser, schedule and
+evaluation; the only difference is the token-mixing layer (memory slots vs self-attention)
+and the positional encoding.
+
+```
+validation loss (nats/char, lower is better; 3 seeds, 2000 steps, lr 6e-3)
+ZeroPoint            1.5214 ± 0.0134
+Transformer (sin)    1.5271 ± 0.0083
+Transformer (learned) 1.5452 ± 0.0100
+```
+
+Per-seed ZeroPoint values: 1.5311 / 1.5307 / 1.5024. The largest gap (vs learned-PE
+Transformer) is 0.0238 nats/char, **below the project's 2 × seed-std decision line of
+0.0268**. Under a stricter standard-error criterion the gap would be ~2.5σ in ZeroPoint's
+favour, but that is not the rule this project pre-registered, and 3 seeds is thin. **The
+honest reading is: indistinguishable at this scale.**
+
+### Inference state and compute
+
+| | @64 chars | @1,024 chars | @4,096 chars |
+|---|---|---|---|
+| Transformer KV cache (floats) | 16,384 | 262,144 (128×) | 1,048,576 (512×) |
+| ZeroPoint state (floats) | 2,048 | 2,048 | 2,048 |
+| Transformer matmul FLOPs/char | 241,664 | 733,184 | 2,306,048 |
+| ZeroPoint matmul FLOPs/char | 232,448 | 234,368 | 234,464 |
+
+ZeroPoint's state is per-layer slots (2 layers × 4 heads × 16 slots × 16 dims) and is
+independent of sequence length. This is the property the architecture was designed for: a
+fixed-size recurrent state with no KV cache. It is **not** a claim that this is faster in
+wall-clock time — see the limitations.
+
+---
+
+## Length behaviour (corrected 32-sequence evaluation)
+
+Trained on 64 characters, then evaluated on 4,096-character sequences, in segments. 3
+seeds, 32 validation sequences, same chunked-parallel algorithm for every model:
+
+| positions | ZeroPoint (trained at 64) | ZeroPoint (trained at 256) |
+|---|---|---|
+| 0–63 | 1.4372 ± 0.0187 | 1.4741 ± 0.0117 |
+| 64–255 (4× training length) | 1.4742 ± 0.0341 | 1.4486 ± 0.0098 |
+| 256–1023 | 1.6617 ± 0.0234 | 1.5098 ± 0.0022 |
+| 1024–4095 (64× training length) | 1.6932 ± 0.0305 | 1.5155 ± 0.0048 |
+| far − near gap | **+0.2190 ± 0.0630** | **+0.0669 ± 0.0051** |
+
+- At 4× the training length the loss rises by **0.037** nats/char; at 64× by **0.219**.
+  Extrapolation degrades gracefully, but it **does** degrade — do not quote "handles 4×
+  training length" without the +0.037.
+- Training at 256 characters reduces the far−near gap 3.3× (to +0.067) but costs +0.036 at
+  64 characters. That run also changed batch size 32 → 8 (to keep 2,048 chars/step), so the
+  short-range cost is confounded with batch size; treat it as a hypothesis, not isolated.
+- **Sinusoidal-PE Transformer collapses outside the training window:** 1.4512 within
+  0–63, then **4.6359** at 64–255 and **4.6057** at 256–1023, versus ln 96 = 4.5642, i.e.
+  chance level. This is an 8-sequence × 3-seed measurement, and the baseline checkpoints
+  were not retained, so it cannot be re-run with 32 sequences. It is a real, large effect,
+  but it is also a known weakness of plain sinusoidal absolute PE — **it is not evidence
+  that attention in general cannot extrapolate** (relative-position and NoPE models are
+  outside this comparison).
+
+The earlier 8-sequence values (1.5550 / 1.6164 / 1.6410 / 1.7040) are kept only in the
+archived report and are **superseded** by the table above; the 8-sequence sample was
+noise-dominated (the same model scores 1.5550 vs 1.4372 on 0–63 depending on sample size).
+
+---
+
+## Ablations (fixed LR 3e-3; internally comparable only)
+
+Each row changes one thing; parameter counts are matched within 1 % by adjusting the MLP
+width; 2,000 steps, 3 seeds, same corpus. Base: **1.5500 ± 0.0073**. Decision line per
+row: 2 × max(base std, row std). Δ is row − base (positive = worse).
+
+| variant | Δ (nats/char) | Δ / threshold | verdict |
+|---|---|---|---|
+| uniform half-life init (no hierarchy) | **+0.0510** | 3.23× | clear: hierarchy matters |
+| + sinusoidal positional encoding | **+0.0606** | 3.88× | clear: no PE is better here |
+| λ frozen at initial values | **+0.0284** | 1.46× | clear: learnable decay matters |
+| 1 head (64 dims, MLP widened to match) | **+0.0276** | 1.89× | clear: multiple heads matter |
+| 8 heads (8 dims each, MLP narrowed) | +0.0173 | 1.19× | marginal / weak |
+| 4 slots/head | −0.0034 | 0.20× | no significant difference |
+| 32 slots/head | +0.0124 | 0.85× | no significant difference |
+| no write-strength gate | +0.0022 | 0.15× | no significant difference |
+| sigmoid write instead of softmax competition | −0.0013 | 0.09× | no significant difference |
+
+At this scale, the things that **do** matter are the hierarchical half-life initialisation,
+learnable λ, multiple heads and the absence of positional encoding. Slot count (4–32),
+the write-strength gate, and write competition do **not** show a significant effect here.
+Note the base loss differs from the head-to-head table because the ablation fixes
+lr = 3e-3 (base 1.5500) while the head-to-head uses the selected lr = 6e-3 (1.5214).
+
+---
+
+## Exact recall: inconclusive — do not cite as an architecture comparison
+
+The long-distance associative-recall experiment (`k1 v1 … kN vN [query kq] → vq`) was meant
+to test the expected weakness of a fixed slot state. It produced **no usable conclusion**:
+
+- With the original constant lr = 3e-3, both architectures plateau near 50 % on N=2
+  (chance is 6 % for 16 values) and near 16–23 % for larger N, after 20,000 steps. The
+  plateau is a **training-configuration** artefact, not an architecture property: with a
+  1e-3 cosine schedule, a Transformer trained on N=2 alone reaches **100 %** in 8,000
+  steps, and the diagnostic shows the loss ≈ ln 2 (50/50 hedge) trap.
+- The mixed-N setup also dilutes the easy examples (N=2 is ~1/15 of the data), so the
+  experiment needs a curriculum and/or per-N weighting before it can compare architectures.
+
+The 20,000-step table (Transformer 50/30/23/20/18 %; ZeroPoint-16 51/32/23/20/16 %;
+ZeroPoint-32 49/31/24/20/17 %) is therefore **not** evidence that ZeroPoint is worse at
+recall. It is a negative result about the experimental setup. A corrected re-run is future
+work.
+
+---
+
+## Repository layout
+
+```
+zerocg/                 from-scratch NumPy framework (autograd, nn, AdamW, FLOP counter)
+models/zeropoint.py     ZeroPoint architecture + the small char-LM harness used in the experiments
+models/transformer.py   GPT-style Transformer baseline (learned or sinusoidal PE)
+experiments/            compare / ablation / recall (+ recall_diag), length/ema/slot diagnostics
+experiments/results/    dated archives of the reports (kept, annotated as historical)
+tests/                  numerical gradient checks, parallel=recurrent equivalence, golden reference
+docs/ZeroPoint.md       design document (Chinese; structured after "Attention Is All You Need")
+docs/ZeroPoint_EN.md    English translation of the design document
+docs/AUDIT.md           independent verification notes for the numbers in this card
+```
+
+## Reproducing
+
+```bash
+pip install numpy
+python tests/test_zerocg.py          # framework self-checks
+python tests/test_zeropoint.py       # architecture + parallel/recurrent equivalence (~20 s)
+cd experiments
+python compare.py                    # ~35 min on one CPU core
+python ablation.py                   # ~70 min
+python recall.py                     # ~15 min; known inconclusive (see above)
+```
+
+Experiments are pure NumPy on CPU. `compare.py` / `ablation.py` write a dated archive to
+`experiments/results/` and update the latest report in `experiments/`.
+
+---
+
+## Data verification
+
+The numbers in this card were independently audited, not merely copied from the project's
+own reports:
+
+- The **ZeroPoint side was re-derived bit-for-bit** from the saved diagnostic checkpoints
+  (`val64 = 1.5214 ± 0.0134`, per-seed 1.5311 / 1.5307 / 1.5024; the 32-sequence segment
+  values above; the ablation parameter counts). The Transformer checkpoints were never
+  saved, so those rows could not be re-derived (stated as a limitation above).
+- Corrections applied after the audit: the 8-sequence length-extrapolation table was
+  superseded by the 32-sequence re-evaluation; the "effective write slots 3.2 / 5.0"
+  number was a metric bug (entropy over a non-normalised, gate-scaled write distribution —
+  the normalised values are ~6–10 write slots and ~10 read slots out of 16) and is not
+  used; "precise recall is a known weakness" was downgraded to "inconclusive"; "one matrix
+  multiply, as fast as attention" was corrected to "chunked parallel, one matmul per chunk";
+  "~110k parameters" was corrected to the exact 115,240.
+- See [`docs/AUDIT.md`](docs/AUDIT.md) for the claim-by-claim audit table.
+
+---
+
+## Limitations (summary)
+
+- Small scale: 115k params, 2 layers, 64-char context, 2,000 steps, one source-code corpus.
+- Only validation loss was measured; no downstream tasks, no generation-quality evaluation,
+  no scaling study.
+- The baseline is a minimal GPT-style decoder, not a tuned Transformer; the comparison is
+  fair in configuration but does not represent "Transformers" in general.
+- 3 seeds; the LR was selected on seed 0 (small selection bias).
+- No wall-clock or memory measurements; FLOP counts only.
+- Length extrapolation: ZeroPoint degrades +0.037 nats/char at 4× training length; the
+  sinusoidal Transformer's collapse is a known property of absolute PE, measured with only
+  8 sequences.
+- Exact recall: inconclusive; needs a corrected training configuration.
+- The framework is a research prototype, not a production framework: no GPU kernels, no
+  distributed training, no mixed precision.
+
+---
+
+## Citation
+
+```bibtex
+@misc{zeropoint2026,
+  title        = {ZeroPoint: a from-scratch NumPy framework and a hierarchical memory-slot
+                  sequence architecture},
+  author       = {Holmes2011},
+  year         = {2026},
+  howpublished = {\url{https://huggingface.co/Holmes2011/ZeroPoint}},
+  note         = {v0 research prototype; architecture and framework release, no pretrained weights}
+}
+```
+
+## License
+
+Apache-2.0 (code and documentation). This repository contains **no model weights, no
+training corpus, and no training runs** of the project's separate 1.4M-parameter model.
+
+---
+
+## 中文摘要
+
+**ZeroPoint 是一个全新的框架，不是已有模型的封装、微调或衍生品。** 本仓库从零实现了两部分：
+
+1. **ZeroCG**：纯 Python + NumPy 的深度学习框架 —— 反向自动求导、`nn` 层、AdamW、矩阵乘
+   FLOP 统计，不依赖 PyTorch。
+2. **ZeroPoint**：用一组**遗忘速度可学习、按存储层级初始化**的记忆槽取代自注意力。训练用
+   **分块并行**（块长 32，块间串行传状态），推理用**逐字递推**，状态固定、与文本长度无关，
+   **不需要位置编码**。
+
+**对比条件（引用任何数字前请先看）**：Python 标准库源码 1,558,395 字符、字符级、词表 96；
+三个模型同宽 64 / 2 层 / 4 头 / MLP 256 / Pre-LN / GELU；上下文 64、batch 32、2000 步、
+AdamW、100 步预热 + 余弦、学习率在 {1e-3, 3e-3, 6e-3} 里按种子 0 选（三者都选 6e-3）、
+种子 0/1/2；验证 loss = 40 个固定 batch × 32 序列 × 64 字的平均交叉熵；单机 CPU、NumPy。
+
+**主要结果**：
+
+- 头对头验证 loss：ZeroPoint **1.5214 ± 0.0134**，Transformer **1.5452 ± 0.0100**，
+  正弦位置编码 Transformer **1.5271 ± 0.0083**。最大差 0.0238 < 2× 种子标准差 0.0268
+  → **统计上没有差别，不声称 ZeroPoint 更好**。
+- 推理状态：ZeroPoint 固定 2,048 个浮点数（8 KiB fp32）；Transformer KV 缓存在第 64 字时
+  16,384（8×）、第 1024 字时 262,144（**128×**）、第 4096 字时 1,048,576（**512×**）。
+- 长度外推：训练 64 字，4 倍长度 loss 升高 **0.037**，64 倍长度升高 **0.219 ± 0.063**；
+  正弦位置编码的 Transformer 出训练长度后崩到约 4.64（≈随机水平；只有 8 条序列的测量）。
+- 消融：分层半衰期初始化（+0.051）、可学习 λ（+0.028）、多头（1 头 +0.028）、去掉位置编码
+  （+0.061）有明确作用；槽数 4~32、写强度门、写入竞争在该规模下**无显著差异**。
+- 精确回忆：**无结论**（学习率配置问题），不能用来比较架构。
+
+**本仓库不含训练权重**：项目里另一个 1.4M 参数的模型（内部名 ZeroX）及其语料、训练运行
+都不在本次发布范围内。
+
+许可证：Apache-2.0。设计文档中文原文见 [`docs/ZeroPoint.md`](docs/ZeroPoint.md)，
+英文版见 [`docs/ZeroPoint_EN.md`](docs/ZeroPoint_EN.md)。

docs/AUDIT.md

@@ -0,0 +1,62 @@
+# Independent audit of the numbers in this release
+
+This document records how the numbers in `README.md` and `docs/ZeroPoint*.md` were
+checked. The audit was done independently of the project's own summaries: every
+publishable number was re-derived from the saved checkpoints where possible, or checked
+arithmetically against the reports. **No training run was performed for the audit.**
+
+## What was reproducible
+
+Four trained ZeroPoint checkpoints from the comparison/diagnostic experiments were kept
+(`experiments/out/diag_*` in the private source tree; not shipped here — they are not
+needed, the experiment scripts retrain them in minutes). Loading them and re-running the
+documented evaluation protocols reproduced, **bit-for-bit**:
+
+| quantity | re-derived value |
+|---|---|
+| 64-char validation loss (3 seeds) | 1.5214 ± 0.0134 (per seed 1.5311 / 1.5307 / 1.5024) |
+| 8-sequence length segments | 1.5550 / 1.6164 / 1.6410 / 1.7040 |
+| 32-sequence length segments | 1.4372 / 1.4742 / 1.6617 / 1.6932; far−near +0.2190 ± 0.0630 |
+| parameter counts (all ablation rows) | 115,240 base; 115,366 / 115,210 / 115,318 / 115,222 / 114,720 / 115,112 / 115,240 |
+| inference state | 2,048 floats fixed; KV cache 16,384 / 262,144 / 1,048,576 at 64 / 1,024 / 4,096 |
+| matmul FLOPs per char @ T=64 | ZeroPoint 232,448; Transformer 241,664 |
+
+The Transformer baselines were **not** saved. Their reported rows (1.5452 ± 0.0100 and
+1.5271 ± 0.0083) and the sinusoidal-PE extrapolation collapse (4.64) therefore rest on the
+original report and could not be re-derived; they are labelled accordingly in the card.
+
+## Claim-by-claim status
+
+| claim | status | what the card says |
+|---|---|---|
+| ~110k parameters | corrected | exact 115,240 (Transformer 116,576; sin-PE 112,480) |
+| validation loss "on par" | confirmed as a null result | 1.5214 vs 1.5452, diff 0.0238 < 2× std 0.0268 → not significant; no superiority claim |
+| inference state 1/128 at the 1024th char | confirmed (with the position stated) | 2,048 vs 262,144 floats; 8× at 64, 512× at 4,096 |
+| handles text 4× training length | confirmed, with degradation | +0.037 nats/char at 4×; +0.219 at 64× |
+| training "one matmul over the whole sequence, as fast as attention" | corrected | chunked parallel (chunk 32, O(n/32) sequential steps); wall-clock not measured |
+| forward compute 232K vs 242K FLOPs/char | confirmed (T=64 only) | stated together with the length dependence (Transformer 2,306,048 at 4,096) |
+| length-extrapolation table (8 sequences) | superseded | replaced by 32-sequence values; old table kept only in the archived report with a note |
+| Transformer(sin) collapse beyond training length | confirmed at 8 sequences | 4.64 vs 1.45, ≈ chance; N=8 caveat and "not evidence about attention in general" |
+| "recurrent vs parallel algorithm" caused the 10-18 correction | corrected | the two forms agree to 1.4e-5 in float32; the real change was 8 → 32 validation sequences |
+| "16 slots, only 3–5 used" (slot collapse) | corrected | metric bug (entropy on a non-normalised, gate-scaled write distribution); normalised: ~6–10 write / ~10 read slots out of 16, pooling-dependent — not used as a claim |
+| ablation: hierarchy / learnable λ / no PE / heads | confirmed | +0.051 / +0.028 / +0.061 / +0.028, all above 2× max std; 8-head row marginal |
+| ablation: 4/32 slots, write gate, write competition | confirmed as null | all below the decision line at this scale |
+| precise recall is a "known weakness" | corrected | inconclusive: constant lr 3e-3 traps both architectures at loss ≈ ln 2; 1e-3 cosine solves N=2 |
+| "32 switch combinations" in the self-check | corrected | 64 combinations (ema flag added later); parallel ≡ recurrent to < 1e-8 (float64) |
+| exact recall / batch-size / calibration / long-run results | out of scope | those belong to the separate 1.4M model, which is not part of this release |
+
+## Why no "beats the Transformer" claim
+
+Under the project's pre-registered rule (a difference counts only if it exceeds 2× the
+seed-to-seed standard deviation), the head-to-head gap is 0.89× the threshold. A standard-
+error criterion would put the gap at ~2.5σ in ZeroPoint's favour, but that criterion was
+not pre-registered, only 3 seeds were run, and the learning rate was selected on one of the
+same seeds. The card therefore says "statistically indistinguishable" and reports the raw
+numbers so readers can apply their own criterion.
+
+## Practical checks that ship with the code
+
+- `tests/test_zerocg.py` — numerical gradient checks for the framework's autograd.
+- `tests/test_zeropoint.py` — all 64 switch combinations, parallel vs recurrent equivalence
+  (< 1e-8 float64), and a fixed golden reference in `tests/golden/`.
+- `tests/make_golden.py` — regenerates the golden reference.

docs/ZeroPoint.md

@@ -0,0 +1,247 @@
+# ZeroPoint：分层记忆槽网络
+
+> 为 zeroOS 设计的序列模型架构。用它搭的语言模型叫 **ZeroX**，训练框架是 **ZeroCG**。（原名 ZeroMem。）文档结构参照《Attention Is All You Need》（Vaswani et al., 2017），
+> 每节开头注明对应论文的哪一节，方便对照阅读。
+>
+> 代码：[`models/zeropoint.py`](../models/zeropoint.py) ·
+> 自检：[`tests/test_zeropoint.py`](../tests/test_zeropoint.py) ·
+> 对照实验：[`experiments/compare.py`](../experiments/compare.py) → [`compare.md`](../experiments/compare.md) ·
+> 消融实验：[`experiments/ablation.py`](../experiments/ablation.py) → [`ablation.md`](../experiments/ablation.md) ·
+> 精确回忆：[`experiments/recall.py`](../experiments/recall.py) → [`recall.md`](../experiments/recall.md) ·
+> 历次结果存档：[`experiments/results/`](../experiments/results/)
+>
+> 状态：**v0（研究原型）**。
+
+---
+
+## 摘要
+
+Transformer 用自注意力取代了循环，换来了训练时的完全并行；代价是推理时每生成一个字都要回看之前所有的字，
+KV 缓存随文本长度线性增长。这对 GPU 服务器不是问题，对**没有 GPU、内存按页精打细算的操作系统内核**却是。
+
+我们提出 **ZeroPoint**：用一组**带不同遗忘速度的记忆槽**取代自注意力。每读一个字，模型决定把它**写进**哪个槽、
+从哪些槽**读出**；槽的遗忘速度按"寄存器 → 缓存 → 内存"的存储层级初始化，并在训练中学习。
+ZeroPoint 有两种数学上等价的算法：训练时用**分块并行**（块长 32，每块一次矩阵乘，块间串行传递状态），
+推理时**逐字递推**，状态大小固定，与文本长度无关，也不需要位置编码。
+
+在 **115,240** 参数、1,558,395 字符的字符级语言建模上，ZeroPoint 与同规模 Transformer 的验证 loss
+在 2× 种子标准差内无显著差异（1.5214 ± 0.0134 vs 1.5452 ± 0.0100，差 0.0238，判定线 0.0268；**不声称更好**）。
+推理状态是后者在第 1024 个字时的 **1/128**（2,048 vs 262,144 个浮点数），
+4 倍训练长度的文本上 loss 升高 0.037 nats/字符（64 倍时为 0.219）。
+
+## 1 引言　〔论文 §1〕
+
+论文 §1 的出发点是：循环网络"天生是串行的"，第 t 步必须等第 t−1 步算完，没法在一条样本内并行。
+自注意力把任意两个位置之间的路径缩短到 O(1)，训练可以完全并行。
+
+但论文没有讨论的另一面是**推理**：生成第 t 个字时，自注意力要和前面 t−1 个字的 Key/Value 逐一比较。
+这些 Key/Value 必须一直存着（KV 缓存）。对 zeroOS 来说：
+
+| | GPU 服务器 | zeroOS 内核 |
+|---|---|---|
+| 内存 | 几十 GB 显存 | 内核堆，按 4KB 页分配 |
+| 算力 | 上万个并行单元 | 一个 CPU 核，x87 浮点 |
+| 运行方式 | 一次处理一大批 | 用户敲一个字处理一个字 |
+
+**我们想要的是两者的好处：训练几乎像 Transformer 一样并行（分块并行，串行步数是长度的 1/32），推理像 RNN 一样只带一个固定大小的状态。**
+
+## 2 背景　〔论文 §2〕
+
+论文 §2 提到 End-to-End Memory Networks [34] 用"循环的注意力"读外部记忆。ZeroPoint 和这条线以及
+后来的"线性注意力 / 带衰减的循环模型"家族是近亲，不是凭空出现的：
+
+| 已有工作 | 相同点 | ZeroPoint 的不同 |
+|---|---|---|
+| 记忆网络 / 神经图灵机 | 软寻址读写记忆槽 | 不需要逐步串行训练；每个槽有自己的遗忘速度 |
+| 线性注意力（Katharopoulos 2020） | 并行 / 递推双形式 | 状态是 M 个可寻址的槽，不是 D×D 的外积矩阵 |
+| RetNet（Sun 2023） | 衰减 λ^(t−u) 实现双形式、不需要位置编码 | λ 可学且按存储层级初始化；写入时 softmax 竞争选槽 |
+| Mamba / 状态空间模型 | 递推推理、状态固定 | 不需要专门的扫描算子，一次矩阵乘就能并行 |
+
+真正属于 ZeroPoint 的是这个组合：**多时间尺度的可学习遗忘 + 写时竞争选槽 + 写强度门 + 一次矩阵乘的并行形式**，
+以及"为一个没有 GPU 的内核设计"这个出发点。
+
+## 3 模型结构　〔论文 §3〕
+
+### 3.1 层的堆叠　〔论文 §3.1〕
+
+和论文的解码器一样：N 个相同的层堆叠，每层两个子层——**记忆混合层**（代替多头自注意力）和**逐位置前馈网络**，
+每个子层外面套残差连接和 LayerNorm。
+
+两处和论文不同：
+
+- **LayerNorm 放在子层前面（Pre-LN）**：`x + Sublayer(LayerNorm(x))`。论文用的是 Post-LN
+  `LayerNorm(x + Sublayer(x))`，后来的 GPT-2 等工作发现 Pre-LN 训练更稳，不太依赖学习率预热。
+  对照组 Transformer 用的也是 Pre-LN，两边一致。
+- **只有解码器（自回归）**。论文是做翻译的编码器-解码器；zeroOS 的任务（续写、意图识别）用不上编码器。
+
+### 3.2 记忆混合层　〔论文 §3.2 注意力���
+
+#### 3.2.1 读与写　〔对应 §3.2.1 缩放点积注意力〕
+
+论文的注意力是"拿查询去和所有的键比"。ZeroPoint 反过来：**每个字在写入时就决定自己存到哪里**，
+读的时候只和 M 个槽打交道，不和之前的字直接打交道。
+
+设第 t 个字的向量为 `x_t`，记忆槽为 `S[1..M]`（每个是 D 维向量），保留率为 `λ[1..M]`：
+
+```
+写地址   w_t = softmax(x_t · Ww) × sigmoid(x_t · Wg)    写到哪些槽 × 写多重
+写入值   v_t = x_t · Wv
+更新     S[m] ← λ[m] · S[m]  +  w_t[m] · v_t           所有槽先按自己的速度遗忘，再写入
+读地址   a_t = softmax(x_t · Wr)                        从哪些槽读
+读出     r_t = Σ_m a_t[m] · S[m]
+输出     LayerNorm(r_t) · Wo
+```
+
+- **写地址用 softmax（槽之间竞争）**：一个字要么写进短期槽，要么写进长期槽，模型得决定"这个字值不值得长期记住"。
+- **写强度门**：空格、换行这类没信息量的字可以几乎不写，不冲掉已有的记忆。
+- **读出后做 LayerNorm**：长期槽可能累积了几十个字，短期槽只有一两个，量级差别很大。
+  这和论文 §3.2.1 除以 √d_k 的动机相同——别让数值落到 softmax / 后续层不好处理的区间。
+- **槽的初始状态 S0 可学习**（不是全 0）：否则序列开头几个字读出来的向量接近 0，
+  后面的 LayerNorm 会把噪声放大。
+
+#### 3.2.2 多头　〔对应 §3.2.2 多头注意力〕
+
+和论文一样分 H 个头，每头 D = d/H 维，各自有一套读写地址和 M 个槽，最后拼起来乘 Wo。
+论文的理由同样适用：单头会把不同的"记什么"平均到一起。
+
+#### 3.2.3 保留率与存储层级　〔ZeroPoint 独有〕
+
+```
+λ = 0.5 + 0.5 · sigmoid(θ)       λ ∈ (0.5, 1)，θ 可学
+```
+
+初始化时，每头的 M 个槽的**半衰期**（旧内容衰减到一半要经过几个字）按等比数列从 1.5 铺到 64：
+
+```
+槽 0      λ ≈ 0.63   半衰期 1.5 个字    ← 像寄存器：记"刚才那个字"
+槽 1~14   λ 逐渐变大                    ← 像缓存：记"这个词""这一句"
+槽 15     λ ≈ 0.989  半衰期 64 个字     ← 像内存：记"这一段在讲什么"
+```
+
+论文 §3.5 的正弦位置编码"波长从 2π 到 10000·2π 成等比数列"，ZeroPoint 的半衰期等比铺开是同一个思路：
+用一组不同尺度的分量覆盖从很近到很远的距离。区别是论文把尺度编码进**输入**，ZeroPoint 把尺度编码进**记忆本身**。
+
+### 3.3 逐位置前馈网络　〔论文 §3.3〕
+
+与论文相同的结构 `FFN(x) = act(x·W1 + b1)·W2 + b2`，内层宽度 4d（论文 512 → 2048 也是 4 倍）。
+激活函数用 GELU（论文是 ReLU；GPT 系列之后普遍用 GELU），对照组也用 GELU。
+
+### 3.4 嵌入与输出　〔论文 §3.4〕
+
+可学习的字嵌入，输出前 LayerNorm + 线性层 + softmax。没有采用论文的"输入/输出嵌入共享权重"，对照组也没有。
+
+### 3.5 位置信息　〔论文 §3.5〕
+
+论文说："模型里没有循环也没有卷积，所以必须注入位置信息。"
+
+**ZeroPoint 不需要位置编码。** 它本质上是一个（可并行的）循环结构：第 t 个字读到的内容里，
+u 个字之前写入的信息被乘上了 λ^(t−u)。不同的槽衰减速度不同，所以"某条信息在哪些槽里还剩多少"
+本身就告诉了模型它是多久以前的。消融实验 (G) 专门检验了这一点。
+
+## 4 为什么用记忆槽　〔论文 §4，表 1〕
+
+论文表 1 比较了三个指标：每层计算量、最少串行步数、任意两位置之间的最长路径。
+我们加上对内核最重要的两列：推理时每个字的计算量、推理时要保存的状态。
+
+n = 序列长度，d = 宽度，M = 每头槽数（实验里 M = 16，d = 64）。
+
+| 层类型 | 每层计算量（训练） | 串行步数 | 最长路径 | 推理：每字计算 | 推理：状态大小 |
+|---|---|---|---|---|---|
+| 自注意力 | O(n²·d) | O(1) | O(1) | O(n·d)，越往后越慢 | O(n·d)，随长度增长 |
+| 循环网络 | O(n·d²) | O(n) | O(n) | O(d²) | O(d) |
+| **ZeroPoint** | O(n·c·(M+d))（分块并行，c = 32） | O(n/c) | **O(1)**，但信号按 λ^(t−u) 衰减 | **O(d²)** | **O(M·d)，固定** |
+
+关键在于 ZeroPoint 有两种等价算法：
+
+**并行形式（训练用）**。把递推展开：
+
+```
+r_t = Σ_{u ≤ t} A[t,u] · v_u,    A[t,u] = Σ_m a_t[m] · w_u[m] · λ[m]^(t−u)
+```
+
+利用 `λ^(t−u) = λ^t · λ^(−u)`：
+
+```
+A = ( a ⊙ λ^t ) @ ( w ⊙ λ^(−u) )ᵀ    再把 u > t 的部分清零（因果）
+```
+
+只要一次矩阵乘，和注意力的 `Q·Kᵀ` 同样能整段并行。但 `λ^(−u)` 最大是 2^u，float32 到 u ≈ 120 就溢出。
+
+所以实际用的是**分块并行**：序列切成每块 c = 32 个字，进入一块时槽的状态是 `S_in`：
+
+```
+块内    r_i  = Σ_{j ≤ i} A[i,j] · v_j            A 同上，但 i、j 是块内下标��λ^(−j) 最大 2^31
+跨块    r_i += (a_i ⊙ λ^(i+1)) @ S_in
+块尾    S_out = λ^c ⊙ S_in + (w ⊙ λ^(c−1−j))ᵀ @ v
+```
+
+块内是矩阵乘，块和块之间只传一个 M × d 的状态。训练时每层计算量 O(n·c·(M+d))，
+**序列多长都行**（自检里 float32 跑 1000 个字不溢出，和递推形式一致）。
+
+**递推形式（推理用）**。逐字更新 `S`，每层只存 M·d 个浮点数，跟文本多长无关。
+
+两种形式结果一致：自检在**所有 64 种开关组合**下、300 个字（跨多个块）上逐位比对，float64 误差 < 1e-8。
+另有一份固定的参考数据 `tests/golden/zeropoint_ref.npz`，以后任何实现（包括内核里的 C 版本）都拿它对答案。
+**训练好的模型可以直接用递推形式放进内核**：不用 KV 缓存，每层固定一块 M × d 的状态，不需要动态分配。
+
+关于"最长路径 O(1)"要说实话：u 处写入的信息确实一步就能被 t 处读到，但已经乘上了 λ^(t−u)。
+注意力能"原样找回"任意远处的某个字，ZeroPoint 只能找回**还没忘干净、而且没被其他写入冲淡**的信息。
+
+## 5 训练　〔论文 §5〕
+
+| | 论文 | ZeroPoint 实验 |
+|---|---|---|
+| 数据 | WMT 2014 英德，450 万句对，BPE 37000 | Python 标准库源码 150 万字符，字符级，词表 96 |
+| 批次 | 约 25000 个词元 | 32 × 64 = 2048 个字符 |
+| 硬件 | 8 块 P100 | 一台 Mac 的 CPU（numpy，自研框架 zerocg） |
+| 优化器 | Adam β=(0.9, 0.98)，公式 3（预热 4000 步后按 1/√步数衰减） | AdamW β=(0.9, 0.999)，权重衰减 0.01；预热 100 步后余弦退火到 10% |
+| 正则化 | 残差 dropout 0.1、标签平滑 0.1 | 无（模型小、训练步数少，没观察到过拟合） |
+| 步数 | 10 万步（12 小时） | 2000 步（每个模型约 2 分钟） |
+
+ZeroPoint 和对照组用**完全相同**的训练设置、相同的随机种子（决定初始化和数据顺序）。
+
+## 6 结果　〔论文 §6〕
+
+### 6.1 对照实验　〔论文 §6.1，表 2〕
+
+> 结果见 [`experiments/compare.md`](../experiments/compare.md)（`compare.py` 自动生成）。
+
+### 6.2 模型变体（消融）　〔论文 §6.2，表 3〕
+
+> 结果见 [`experiments/ablation.md`](../experiments/ablation.md)（`ablation.py` 自动生成）。
+
+### 6.3 长度外推　〔论文 §3.5 的猜想〕
+
+论文选正弦位置编码的理由是它"**可能**可以外推到比训练时更长的序列"，但没有给出实验。
+我们在 64 字上训练 ZeroPoint 和正弦位置编码的 Transformer，然后在 4,096 字的文本上分段测试
+（32 条验证序列 × 3 个种子，两种模型用同一种分块并行算法）。
+
+- ZeroPoint：64~255 字（训练长度的 4 倍）loss 为 1.4742 ± 0.0341，比训练长度内（0~63：1.4372）高 **+0.037**；
+  1024~4095 字（64 倍）为 1.6932 ± 0.0305，高 **+0.219 ± 0.063**。**外推会变差，但没有崩掉。**
+- 正弦位置编码的 Transformer：出了训练长度就升到约 4.64（≈ ln 96 = 4.56，即随机水平）。
+  这是 8 条序列 × 3 个种子的测量，基线权重未保留，无法用 32 条序列复测。
+
+> 早期 8 条序列的结果（1.5550 / 1.6164 / 1.6410 / 1.7040）抽样噪声大，已由上面的 32 条序列结果取代。
+> 原始数据见 [`experiments/results/`](../experiments/results/)。
+
+### 6.4 精确回忆　〔无结论，不能用来比较架构〕
+
+> 结果见 [`experiments/recall.md`](../experiments/recall.md)。
+
+`recall.py` 的训练配置（恒定 lr 3e-3）把两个架构都困在"两个候选值各猜一半"的局部解（loss ≈ ln 2）。
+诊断显示换成 1e-3 余弦退火后，只训 N=2 的 Transformer 在 8,000 步内到 100%。在按修正后的学习率重跑之前，
+这张 20,000 步的表只能说明训练配置有问题，**不能说明 ZeroPoint 或 Transformer 谁更擅长精确回忆**。
+
+## 7 结论与局限　〔论文 §7〕
+
+**局限**（都是实话）：
+
+- **规模很小**：只在约 11.5 万参数、150 万字符、单一语料上测过，结论不能外推到大模型。
+- **精确回忆**：只有 M 个槽，信息会被压缩、混合。**推测**（未验证）它不擅长原样复制很久以前的内容；现有 recall 实验因学习率配置问题无结论。
+- **没有编码器-解码器版本**：论文的核心应用是翻译；ZeroPoint 目前只做了解码器。
+
+**下一步**：
+
+1. 如果精确回忆确实弱：加一层"滑动窗口注意力"（只看最近 32 个字，状态仍然固定）做混合结构；
+2. 在 zeroOS 的意图识别任务上训练；
+3. 用 C 实现递推形式放进内核（x87 浮点）——这是 ZeroPoint 存在的原因。

docs/ZeroPoint_EN.md

@@ -0,0 +1,214 @@
+# ZeroPoint: Hierarchical Memory-Slot Network
+
+> A sequence-model architecture designed for zeroOS. The language model built on it is called **ZeroX**, and the training framework is **ZeroCG**. (Formerly named ZeroMem.) The document follows the structure of *Attention Is All You Need* (Vaswani et al., 2017); each section begins by noting which section of the paper it corresponds to, for easy cross-reading.
+>
+> Code: [`models/zeropoint.py`](../models/zeropoint.py) ·
+> Self-checks: [`tests/test_zeropoint.py`](../tests/test_zeropoint.py) ·
+> Comparison experiment: [`experiments/compare.py`](../experiments/compare.py) → [`compare.md`](../experiments/compare.md) ·
+> Ablation experiment: [`experiments/ablation.py`](../experiments/ablation.py) → [`ablation.md`](../experiments/ablation.md) ·
+> Exact recall: [`experiments/recall.py`](../experiments/recall.py) → [`recall.md`](../experiments/recall.md) ·
+> Archived results: [`experiments/results/`](../experiments/results/)
+>
+> Status: **v0 (research prototype)**.
+
+---
+
+## Abstract
+
+The Transformer replaced recurrence with self-attention, gaining full parallelism at training time; the price is that, at inference time, generating each character requires looking back at all previous characters, and the KV cache grows linearly with text length. This is not a problem for GPU servers, but it is for **an operating-system kernel with no GPU and memory meted out page by page**.
+
+We propose **ZeroPoint**: replacing self-attention with a set of **memory slots that forget at different speeds**. As it reads each character, the model decides **which slot to write it into** and **from which slots to read**; the forgetting speeds of the slots are initialized according to a "register → cache → memory" storage hierarchy and are learned during training. ZeroPoint has two mathematically equivalent algorithms: at training time it uses a **chunked-parallel** form (chunk 32: one matrix multiplication per chunk, with a serial state carry between chunks), and at inference time it **recurs character by character**, with a state size that is fixed and independent of text length, and it needs no positional encoding.
+
+On character-level language modeling with **115,240** parameters and 1,558,395 characters, ZeroPoint is statistically indistinguishable from a same-size Transformer (1.5214 ± 0.0134 vs 1.5452 ± 0.0100; difference 0.0238 < 2× seed std = 0.0268 — **we do not claim it is better**). Its inference state is **1/128** of the latter's at the 1024th character (2,048 vs 262,144 floats), and at 4× the training length the loss rises by 0.037 nats/character (0.219 at 64×).
+
+## 1 Introduction 〔Paper §1〕
+
+The starting point of the paper's §1 is: recurrent networks "are inherently sequential" — step t has to wait for step t−1 to finish computing, so there is no parallelism within a single sample. Self-attention shortens the path between any two positions to O(1), so training can be fully parallel.
+
+But the other side, which the paper does not discuss, is **inference**: when generating the t-th character, self-attention has to compare against the Key/Value of the previous t−1 characters one by one. These Key/Values must be kept all along (the KV cache). For zeroOS:
+
+| | GPU server | zeroOS kernel |
+|---|---|---|
+| Memory | tens of GB of VRAM | kernel heap, allocated in 4KB pages |
+| Compute | tens of thousands of parallel units | one CPU core, x87 floating point |
+| Execution mode | process a large batch at once | process one character as the user types it |
+
+**What we want is the best of both: training almost as parallel as a Transformer (chunked parallel; sequential steps = length/32), inference carrying only a single fixed-size state like an RNN.**
+
+## 2 Background 〔Paper §2〕
+
+The paper's §2 mentions that End-to-End Memory Networks [34] read external memory with "recurrent attention". ZeroPoint is a close relative of this line of work and of the later "linear attention / recurrent models with decay" family; it did not appear out of thin air:
+
+| Prior work | Similarity | How ZeroPoint differs |
+|---|---|---|
+| Memory networks / Neural Turing Machine | soft-addressable read and write of memory slots | no need for step-by-step serial training; each slot has its own forgetting speed |
+| Linear attention (Katharopoulos 2020) | parallel / recurrent dual form | the state is M addressable slots, not a D×D outer-product matrix |
+| RetNet (Sun 2023) | decay λ^(t−u) realizes the dual form and needs no positional encoding | λ is learnable and initialized by storage hierarchy; writes compete for slots via softmax |
+| Mamba / state-space models | recurrent inference, fixed state | no specialized scan operator is needed; one matrix multiplication suffices to parallelize |
+
+What genuinely belongs to ZeroPoint is this combination: **multi-timescale learnable forgetting + write-time competition among slots + a write-strength gate + a parallel form that uses one matrix multiplication**, together with the starting point of "designing for a kernel with no GPU".
+
+## 3 Model Architecture 〔Paper §3〕
+
+### 3.1 Stacking of Layers 〔Paper §3.1〕
+
+Same as the paper's decoder: N identical layers are stacked, and each layer has two sub-layers — the **memory mixing layer** (replacing multi-head self-attention) and the **position-wise feed-forward network** — with a residual connection and LayerNorm around each sub-layer.
+
+There are two differences from the paper:
+
+- **LayerNorm is placed before the sub-layer (Pre-LN)**: `x + Sublayer(LayerNorm(x))`. The paper uses Post-LN `LayerNorm(x + Sublayer(x))`; later work such as GPT-2 found that Pre-LN trains more stably and does not depend much on learning-rate warmup. The control Transformer also uses Pre-LN, so both sides are consistent.
+- **Decoder only (autoregressive)**. The paper is an encoder-decoder for translation; zeroOS's tasks (continuation, intent recognition) do not need an encoder.
+
+### 3.2 Memory Mixing Layer 〔Paper §3.2 Attention〕
+
+#### 3.2.1 Reading and Writing 〔Corresponds to §3.2.1 Scaled Dot-Product Attention〕
+
+The paper's attention is "take a query and compare it with all the keys". ZeroPoint does the reverse: **each character decides where to store itself at write time**, and at read time it deals only with the M slots, never directly with previous characters.
+
+Let the vector of the t-th character be `x_t`, the memory slots be `S[1..M]` (each a D-dimensional vector), and the retention rates be `λ[1..M]`:
+
+```
+write address  w_t = softmax(x_t · Ww) × sigmoid(x_t · Wg)   which slots to write × how strongly
+write value    v_t = x_t · Wv
+update         S[m] ← λ[m] · S[m]  +  w_t[m] · v_t          all slots first forget at their own speed, then are written
+read address   a_t = softmax(x_t · Wr)                       from which slots to read
+read out       r_t = Σ_m a_t[m] · S[m]
+output         LayerNorm(r_t) · Wo
+```
+
+- **Write address uses softmax (competition among slots)**: a character either goes into a short-term slot or into a long-term slot; the model has to decide whether "this character is worth remembering long-term".
+- **Write-strength gate**: uninformative characters such as spaces and newlines can be written almost not at all, without washing out existing memories.
+- **LayerNorm after reading**: long-term slots may have accumulated dozens of characters while short-term slots have only one or two, so their magnitudes differ greatly. This is the same motivation as dividing by √d_k in the paper's §3.2.1 — do not let the values fall into a range that softmax / subsequent layers handle poorly.
+- **The slots' initial state S0 is learnable** (not all zeros): otherwise the vectors read out at the beginning of the sequence are close to 0, and the subsequent LayerNorm would amplify the noise.
+
+#### 3.2.2 Multiple Heads 〔Corresponds to §3.2.2 Multi-Head Attention〕
+
+As in the paper, the model is split into H heads, each of dimension D = d/H, each with its own set of read/write addresses and M slots; finally they are concatenated and multiplied by Wo. The paper's reasoning applies equally: a single head would average the different "what to remember" together.
+
+#### 3.2.3 Retention Rates and the Storage Hierarchy 〔ZeroPoint-specific〕
+
+ZeroPoint's retention rates realize a hierarchical, learnable decay:
+
+```
+λ = 0.5 + 0.5 · sigmoid(θ)       λ ∈ (0.5, 1), θ learnable
+```
+
+At initialization, the **half-lives** of the M slots of each head (how many characters it takes for old content to decay to half) are spread geometrically from 1.5 to 64:
+
+```
+slot 0      λ ≈ 0.63   half-life 1.5 characters   ← like a register: remember "the character just now"
+slots 1~14  λ gradually larger                    ← like a cache: remember "this word" "this sentence"
+slot 15     λ ≈ 0.989  half-life 64 characters    ← like memory: remember "what this passage is about"
+```
+
+The paper's §3.5 sinusoidal positional encoding has "wavelengths in a geometric progression from 2π to 10000·2π"; ZeroPoint's geometric spread of half-lives is the same idea: use a set of components at different scales to cover distances from very near to very far. The difference is that the paper encodes scale into the **input**, whereas ZeroPoint encodes scale into the **memory itself**.
+
+### 3.3 Position-wise Feed-Forward Network 〔Paper §3.3〕
+
+Same structure as the paper, `FFN(x) = act(x·W1 + b1)·W2 + b2`, with inner width 4d (the paper's 512 → 2048 is also 4x). The activation is GELU (the paper uses ReLU; after the GPT series, GELU is common), and the control group uses GELU too.
+
+### 3.4 Embeddings and Output 〔Paper §3.4〕
+
+Learnable character embeddings; before the output there is LayerNorm + a linear layer + softmax. We do not adopt the paper's "share weights between input and output embeddings", and neither does the control group.
+
+### 3.5 Positional Information 〔Paper §3.5〕
+
+The paper says: "the model contains no recurrence and no convolution, so positional information must be injected."
+
+**ZeroPoint does not need positional encoding.** It is essentially a (parallelizable) recurrent structure: in what the t-th character reads, information written u characters earlier has been multiplied by λ^(t−u). Different slots decay at different speeds, so "how much of a piece of information is left in which slots" itself tells the model how long ago it was. Ablation experiment (G) specifically tests this.
+
+## 4 Why Memory Slots 〔Paper §4, Table 1〕
+
+The paper's Table 1 compares three metrics: computation per layer, minimum number of sequential steps, and maximum path length between any two positions. We add the two columns that matter most for a kernel: computation per character at inference, and state to keep at inference.
+
+n = sequence length, d = width, M = number of slots per head (in the experiments M = 16, d = 64).
+
+| Layer type | Computation per layer (training) | Sequential steps | Maximum path | Inference: computation per character | Inference: state size |
+|---|---|---|---|---|---|
+| Self-attention | O(n²·d) | O(1) | O(1) | O(n·d), slower toward the end | O(n·d), grows with length |
+| Recurrent network | O(n·d²) | O(n) | O(n) | O(d²) | O(d) |
+| **ZeroPoint** | O(n·c·(M+d)) (chunked parallel, c = 32) | O(n/c) | **O(1)**, but the signal decays as λ^(t−u) | **O(d²)** | **O(M·d), fixed** |
+
+The key is that ZeroPoint has two equivalent algorithms:
+
+**Parallel form (for training)**. Expand the recurrence:
+
+```
+r_t = Σ_{u ≤ t} A[t,u] · v_u,    A[t,u] = Σ_m a_t[m] · w_u[m] · λ[m]^(t−u)
+```
+
+Using `λ^(t−u) = λ^t · λ^(−u)`:
+
+```
+A = ( a ⊙ λ^t ) @ ( w ⊙ λ^(−u) )ᵀ    then zero out the part where u > t (causal)
+```
+
+Only one matrix multiplication is needed, and it can process the whole sequence in parallel just like attention's `Q·Kᵀ`. But `λ^(−u)` is at most 2^u, and float32 overflows at u ≈ 120.
+
+So what is actually used is **chunked parallel**: the sequence is split into blocks of c = 32 characters; when entering a block, the slot state is `S_in`:
+
+```
+within block  r_i  = Σ_{j ≤ i} A[i,j] · v_j          A as above, but i, j are within-block indices; λ^(−j) at most 2^31
+across blocks r_i += (a_i ⊙ λ^(i+1)) @ S_in
+block end     S_out = λ^c ⊙ S_in + (w ⊙ λ^(c−1−j))ᵀ @ v
+```
+
+Within a block it is matrix multiplication, and only an M × d state is passed between blocks. At training time the computation per layer is O(n·c·(M+d)), and **any sequence length works** (in the self-check, float32 runs 1000 characters without overflow, matching the recurrent form).
+
+**Recurrent form (for inference)**. Update `S` character by character; each layer stores only M·d floating-point numbers, regardless of how long the text is.
+
+The two forms agree: the self-check compares them bit by bit on **all 64 switch combinations** over 300 characters (crossing several blocks), with float64 error < 1e-8. There is also a fixed reference file `tests/golden/zeropoint_ref.npz`; any future implementation (including a C version in the kernel) can check its answers against it. **A trained model can be put directly into the kernel in recurrent form**: no KV cache, a fixed M × d state per layer, no dynamic allocation.
+
+About "maximum path O(1)", we must be honest: information written at u can indeed be read at t in one step, but it has already been multiplied by λ^(t−u). Attention can "retrieve verbatim" a character from arbitrarily far away; ZeroPoint can only retrieve information **that has not yet been forgotten and has not been washed out by other writes**.
+
+## 5 Training 〔Paper §5〕
+
+| | Paper | ZeroPoint experiments |
+|---|---|---|
+| Data | WMT 2014 English-German, 4.5M sentence pairs, BPE 37000 | Python standard library source, 1.5M characters, character-level, vocabulary 96 |
+| Batch | about 25,000 tokens | 32 × 64 = 2048 characters |
+| Hardware | 8 P100s | one Mac's CPU (numpy, the in-house framework zerocg) |
+| Optimizer | Adam β=(0.9, 0.98), Eq. 3 (4000-step warmup then decay ∝ 1/√step) | AdamW β=(0.9, 0.999), weight decay 0.01; 100-step warmup then cosine decay to 10% |
+| Regularization | residual dropout 0.1, label smoothing 0.1 | none (small model, few training steps, no overfitting observed) |
+| Steps | 100K steps (12 hours) | 2000 steps (about 2 minutes per model) |
+
+ZeroPoint and the control group use **exactly the same** training settings and the same random seeds (which determine initialization and data order).
+
+## 6 Results 〔Paper §6〕
+
+### 6.1 Comparison Experiment 〔Paper §6.1, Table 2〕
+
+> Results: [`experiments/compare.md`](../experiments/compare.md) (auto-generated by `compare.py`).
+
+### 6.2 Model Variants (Ablation) 〔Paper §6.2, Table 3〕
+
+> Results: [`experiments/ablation.md`](../experiments/ablation.md) (auto-generated by `ablation.py`).
+
+### 6.3 Length Extrapolation 〔Conjecture in Paper §3.5〕
+
+The paper's reason for choosing sinusoidal positional encoding is that it "may allow the model to extrapolate to sequence lengths longer than the ones encountered during training", but no experiment was given. We train ZeroPoint and the sinusoidal-PE Transformer on 64 characters, then evaluate on 4,096-character text in segments (32 validation sequences × 3 seeds, same chunked-parallel algorithm for both models).
+
+- ZeroPoint: loss at 64–255 characters (4× the training length) is 1.4742 ± 0.0341, i.e. **+0.037** above the in-training-length value (0–63: 1.4372); at 1,024–4,095 characters (64×) it is 1.6932 ± 0.0305, i.e. **+0.219 ± 0.063**. **Extrapolation degrades gracefully but does degrade.**
+- The sinusoidal-PE Transformer jumps to about 4.64 once outside the training length (≈ ln 96 = 4.56, i.e. chance level). This is an 8-sequence × 3-seed measurement; the baseline checkpoints were not retained, so it could not be re-run with 32 sequences.
+
+> The earlier 8-sequence values (1.5550 / 1.6164 / 1.6410 / 1.7040) are noise-dominated and are superseded by the 32-sequence numbers above. Raw data: [`experiments/results/`](../experiments/results/).
+
+### 6.4 Exact Recall 〔inconclusive; not an architecture comparison〕
+
+> Results: [`experiments/recall.md`](../experiments/recall.md).
+
+The training configuration used by `recall.py` (constant lr 3e-3) traps both architectures at the "guess one of two candidates" local solution (loss ≈ ln 2). Diagnostics show that with a 1e-3 cosine schedule, the Transformer trained on N=2 alone reaches 100% in 8,000 steps. Until the experiment is re-run with the corrected learning rate, the 20,000-step table only shows that the *training setup* was wrong; **it says nothing about whether ZeroPoint or Transformer is better at exact recall**.
+
+## 7 Conclusion and Limitations 〔Paper §7〕
+
+**Limitations** (all honest):
+
+- **Very small scale**: tested only on about 115K parameters, 1.5M characters, and a single corpus; the conclusions cannot be extrapolated to large models.
+- **Exact recall**: with only M slots, information is compressed and mixed. It is a **hypothesis (not measured)** that this hurts verbatim copying of long-ago content; the existing recall experiment is inconclusive because of its learning-rate configuration.
+- **No encoder-decoder version**: the paper's core application is translation; ZeroPoint currently implements only the decoder.
+
+**Next steps**:
+
+1. If exact recall is indeed weak: add a "sliding-window attention" layer (looking only at the most recent 32 characters, with the state still fixed) to make a hybrid structure;
+2. Train on zeroOS's intent-recognition task;
+3. Implement the recurrent form in C and put it in the kernel (x87 floating point) — this is why ZeroPoint exists.

experiments/ablation.md

@@ -0,0 +1,21 @@
+# ZeroPoint 消融实验（仿论文表 3；历史存档：原名 ZeroMem）
+
+每行只改一处，其余和 base 相同。2000 步，lr 0.003，种子 [0, 1, 2]，其余训练设置同 [`compare.md`](compare.md)。
+**注意**：本表所有行固定 lr 0.003；`compare.md` 用的是每个模型选出的最佳 lr 0.006（base loss 1.5500 vs 1.5214）。两张表只在各自内部可比，不能混用。
+参数量和 base 差超过 1% 的变体，自动调整 MLP 宽度对齐参数量。
+Δ = 该行验证 loss 均值 − base 均值（**正数表示变差**）。
+
+> 判断标准：每行的阈值 = 2 × max(base 种子间标准差 0.0073, 该行标准差)；例如 λ 固定行 0.0194、8 头行 0.0146。|Δ| 小于阈值的差别**不能算数**。
+
+| | 改动 | 参数量 | 验证 loss | Δ | 结论 |
+|---|---|---|---|---|---|
+| base | ZeroPoint：4 头 × 16 槽，写强度门，写入竞争，分层初始化，λ 可学，可学初始状态，无位置编码 | 115,240 | 1.5500 ± 0.0073 | — | — |
+| (A) | 1 个头（每头 64 维）（MLP 宽度调为 306 以对齐参数） | 115,366 | 1.5776 ± 0.0036 | +0.0276 | 变差：这个部件有用 |
+| (A) | 8 个头（每头 8 维）（MLP 宽度调为 189 以对齐参数） | 115,210 | 1.5673 ± 0.0067 | +0.0173 | 变差：这个部件有用 |
+| (B) | 每头 4 个槽（MLP 宽度调为 311 以对齐参数） | 115,318 | 1.5466 ± 0.0087 | -0.0034 | 差别不显著 |
+| (B) | 每头 32 个槽（MLP 宽度调为 183 以对齐参数） | 115,222 | 1.5624 ± 0.0055 | +0.0124 | 差别不显著 |
+| (C) | 去掉写强度门 | 114,720 | 1.5522 ± 0.0072 | +0.0022 | 差别不显著 |
+| (D) | 写入不竞争：softmax 换成每槽独立的 sigmoid | 115,240 | 1.5487 ± 0.0066 | -0.0013 | 差别不显著 |
+| (E) | 不分层：所有槽初始半衰期相同（约 9.8 个字） | 115,240 | 1.6010 ± 0.0079 | +0.0510 | 变差：这个部件有用 |
+| (F) | λ 固定在初始值，不学习 | 115,112 | 1.5784 ± 0.0097 | +0.0284 | 变差：这个部件有用 |
+| (G) | 加上论文 §3.5 的正弦位置编码 | 115,240 | 1.6106 ± 0.0078 | +0.0606 | 变差：这个部件有用 |

experiments/ablation.py

@@ -0,0 +1,86 @@
+"""ZeroPoint 消融实验，仿《Attention Is All You Need》表 3：每次只改一处，看那个部件到底有没有用。
+
+    python3 experiments/ablation.py
+
+- 每组 3 个种子，报告 均值 ± 标准差
+- 参数量对齐：某个改动让参数量和 base 差超过 1% 时（比如改头数、槽数），自动调整 MLP 宽度把参数量拉回来，
+  这样变好变差就不是"参数多了/少了"造成的
+- 学习率固定 3e-3（所有变体相同）
+结果：experiments/ablation.md（最新）+ experiments/results/<时间>-ablation.md（存档）
+"""
+import numpy as np
+
+from common import CFG, ITERS, prepare_data, save_result, train
+from models import ZeroX
+
+SEEDS = [0, 1, 2]
+LR = 3e-3
+BASE = dict(n_slot=16, **CFG)
+
+# (行号, 说明, 相对 base 的改动)
+VARIANTS = [
+    ("base", "ZeroPoint：4 头 × 16 槽，写强度门，写入竞争，分层初始化，λ 可学，可学初始状态，无位置编码", {}),
+    ("(A)", "1 个头（每头 64 维）", dict(n_head=1)),
+    ("(A)", "8 个头（每头 8 维）", dict(n_head=8)),
+    ("(B)", "每头 4 个槽", dict(n_slot=4)),
+    ("(B)", "每头 32 个槽", dict(n_slot=32)),
+    ("(C)", "去掉写强度门", dict(gate=False)),
+    ("(D)", "写入不竞争：softmax 换成每槽独立的 sigmoid", dict(write="sigmoid")),
+    ("(E)", "不分层：所有槽初始半衰期相同（约 9.8 个字）", dict(init="uniform")),
+    ("(F)", "λ 固定在初始值，不学习", dict(learn_lambda=False)),
+    ("(G)", "加上论文 §3.5 的正弦位置编码", dict(pos="sin")),
+]
+
+
+def matched(V, change, target):
+    """返回 (参数量对齐后的配置, 是否调整过 MLP)。"""
+    cfg = {**BASE, **change}
+    n = ZeroX(V, **cfg).num_parameters()
+    if abs(n - target) / target <= 0.01:
+        return cfg, False
+    C, L = cfg["n_embd"], cfg["n_layer"]
+    hidden = 4 * C + round((target - n) / (L * (2 * C + 1)))   # 每个隐藏单元每层 2C+1 个参数
+    return {**cfg, "mlp_hidden": hidden}, True
+
+
+def main():
+    text, V, train_data, val_data = prepare_data()
+    target = ZeroX(V, **BASE).num_parameters()
+    print(f"消融实验：{len(VARIANTS)} 组 × 种子 {SEEDS}，每次 {ITERS} 步，lr {LR}", flush=True)
+    results = []
+    for row, desc, change in VARIANTS:
+        cfg, adjusted = matched(V, change, target)
+        vals = []
+        for seed in SEEDS:
+            np.random.seed(seed)
+            vals.append(train(f"{row} {desc[:10]}", ZeroX(V, **cfg), train_data, val_data, seed, LR))
+        n = ZeroX(V, **cfg).num_parameters()
+        results.append((row, desc + (f"（MLP 宽度调为 {cfg['mlp_hidden']} 以对齐参数）" if adjusted else ""),
+                        n, np.mean(vals), np.std(vals), vals))
+        print(f"  => {row} {desc}: {np.mean(vals):.4f} ± {np.std(vals):.4f}", flush=True)
+
+    base, base_std = results[0][3], results[0][4]
+    lines = ["# ZeroPoint 消融实验（仿论文表 3）\n",
+             f"每行只改一处，其余和 base 相同。{ITERS} 步，lr {LR}，种子 {SEEDS}，其余训练设置同 [`compare.md`](compare.md)。",
+             "参数量和 base 差超过 1% 的变体，自动调整 MLP 宽度对齐参数量。",
+             "Δ = 该行验证 loss 均值 − base 均值（**正数表示变差**）。\n",
+             f"> 判断标准：base 的种子间标准差是 {base_std:.4f}。|Δ| 小于约 2 倍标准差（{2 * base_std:.4f}）的差别**不能算数**。\n",
+             "| | 改动 | 参数量 | 验证 loss | Δ | 结论 |", "|---|---|---|---|---|---|"]
+    for row, desc, n, mean, std, _ in results:
+        if row == "base":
+            lines.append(f"| {row} | {desc} | {n:,} | {mean:.4f} ± {std:.4f} | — | — |")
+            continue
+        d = mean - base
+        noise = 2 * max(base_std, std)
+        if noise == 0:
+            verdict = "只有 1 个种子，无法判断"
+        elif abs(d) < noise:
+            verdict = "差别不显著"
+        else:
+            verdict = "变差：这个部件有用" if d > 0 else "变好：这个部件可能多余"
+        lines.append(f"| {row} | {desc} | {n:,} | {mean:.4f} ± {std:.4f} | {d:+.4f} | {verdict} |")
+    save_result("ablation", lines)
+
+
+if __name__ == "__main__":
+    main()

experiments/common.py

@@ -0,0 +1,119 @@
+"""各个实验脚本共用的部分：语料、批次、训练循环、计算量统计、结果存档。"""
+import datetime
+import glob
+import os
+import sys
+import sysconfig
+import time
+
+import numpy as np
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.join(HERE, ".."))
+import zerocg as zc  # noqa: E402
+from models import ZeroX  # noqa: E402
+from zerocg import functional as F  # noqa: E402
+
+ITERS = int(os.environ.get("ITERS", 2000))
+T, B = 64, 32
+CFG = dict(n_embd=64, n_head=4, n_layer=2)
+
+
+def load_corpus(path=None):
+    """默认语料：Python 标准库源码（纯 ASCII 的 .py 文件，约 150 万字符）。"""
+    if path:
+        return open(path, encoding="utf-8").read()
+    parts, total = [], 0
+    for p in sorted(glob.glob(os.path.join(sysconfig.get_paths()["stdlib"], "*.py"))):
+        try:
+            s = open(p, encoding="utf-8").read()
+        except (UnicodeDecodeError, OSError):
+            continue
+        if s.isascii():
+            parts.append(s)
+            total += len(s)
+        if total > 1_500_000:
+            break
+    return "\n".join(parts)
+
+
+def prepare_data(path=None):
+    text = load_corpus(path)
+    chars = sorted(set(text))
+    stoi = {c: i for i, c in enumerate(chars)}
+    data = np.array([stoi[c] for c in text], dtype=np.int64)
+    split = int(0.9 * len(data))
+    return text, len(chars), data[:split], data[split:]
+
+
+def batches(data, n, length, seed, batch=B):
+    r = np.random.default_rng(seed)
+    for _ in range(n):
+        ix = r.integers(0, len(data) - length - 1, batch)
+        yield np.stack([data[i:i + length] for i in ix]), np.stack([data[i + 1:i + length + 1] for i in ix])
+
+
+def loss_of(logits, y):
+    return F.cross_entropy(logits.reshape(-1, logits.shape[-1]), y.reshape(-1)).item()
+
+
+def train(name, model, train_data, val_data, seed, lr=3e-3, iters=None, seq_len=None, batch=None):
+    """返回验证 loss（固定在 64 字上评估，便于各实验比较）。
+    学习率：100 步线性预热 + 余弦退火到 10%；AdamW，权重衰减 0.01，梯度裁剪 1.0。
+    seq_len / batch 默认 T / B；改训练长度时请同时改 batch，保持每步的字数不变。"""
+    iters = iters or ITERS
+    opt = zc.optim.AdamW(model.parameters(), lr=lr, weight_decay=0.01)
+    t0 = time.time()
+    for it, (x, y) in enumerate(batches(train_data, iters, seq_len or T, seed, batch or B)):
+        opt.lr = lr * min(1, (it + 1) / 100) * (0.1 + 0.9 * 0.5 * (1 + np.cos(np.pi * it / iters)))
+        logits = model(x)
+        loss = F.cross_entropy(logits.reshape(-1, logits.shape[-1]), y.reshape(-1))
+        opt.zero_grad()
+        loss.backward()
+        zc.optim.clip_grad_norm(model.parameters(), 1.0)
+        opt.step()
+        if it % 500 == 0:
+            print(f"  [{name} seed={seed} lr={lr:g}] step {it:5d}  train {loss.item():.3f}  "
+                  f"{time.time() - t0:.0f}s", flush=True)
+    return evaluate(model, val_data)
+
+
+def evaluate(model, val_data, n=40):
+    model.eval()
+    with zc.no_grad():
+        val = float(np.mean([loss_of(model(x), y) for x, y in batches(val_data, n, T, 999)]))
+    model.train()
+    return val
+
+
+def forward_flops_per_char(model, V):
+    """前向一次、平均每个字多少次浮点运算（只算矩阵乘）。训练一步约是它的 3 倍。"""
+    x = np.zeros((1, T), dtype=np.int64)
+    with zc.no_grad(), zc.count_flops() as c:
+        model(x)
+    return c.flops / T
+
+
+def slot_usage(model):
+    """每层"有效槽数"：最近一次 forward 的平均读/写地址分布的 exp(熵)，对头取平均。
+    16 个槽都用得一样多 = 16；全挤在一个槽 = 1。"""
+    out = []
+    for b in model.blocks:
+        read, write = b.mix.usage
+        eff = lambda p: float(np.mean(np.exp(-(p * np.log(p + 1e-12)).sum(-1))))  # noqa: E731
+        out.append((eff(read), eff(write)))
+    return out
+
+
+def save_result(name, lines):
+    """结果写两份：experiments/results/<时间>-<name>.md（存档，不覆盖）和 experiments/<name>.md（最新）。"""
+    text = "\n".join(lines) + "\n"
+    stamp = datetime.datetime.now().strftime("%Y%m%d-%H%M")
+    os.makedirs(os.path.join(HERE, "results"), exist_ok=True)
+    for path in (os.path.join(HERE, "results", f"{stamp}-{name}.md"), os.path.join(HERE, f"{name}.md")):
+        open(path, "w").write(text)
+    print("\n" + text)
+
+
+def is_zeropoint(model):
+    return isinstance(model, ZeroX)

experiments/compare.md

@@ -0,0 +1,53 @@
+# ZeroPoint vs Transformer 对照实验
+
+> **历史说明（2026-11-08 统一）**：本报告生成于 2026-10-11，当时架构名为 ZeroMem；除名称统一为 ZeroPoint 外，正文保留原样。两处已过时的内容在文末加了更正注：① 长度外推只用了 8 条验证序列（已被 32 条序列的复测取代）；② "有效槽数"的写列用了未归一化的门控分布，数值偏低。
+
+- 语料：Python 标准库源码，1,558,395 字符，词表 96；90% 训练 / 10% 验证
+- 三个模型：宽 64、2 层、4 头、Pre-LN、GELU MLP 4 倍；ZeroPoint 每头 16 个槽、可学习初始状态、分块并行（块长 32）、无位置编码
+- 训练：上下文 64、batch 32、2000 步、AdamW（权重衰减 0.01）、预热 100 步 + 余弦、梯度裁剪 1.0
+- 学习率：每个模型在 [0.001, 0.003, 0.006] 里各试一次（种子 0），取最好的再跑种子 [1, 2]
+- 验证 loss = 40 个固定验证 batch 的平均交叉熵（nats/字符，越低越好），均值 ± 标准差（3 个种子）
+- 计算量 = 前向时每个字的矩阵乘浮点运算次数（不受机器负载影响；训练一步约 3 倍）
+
+| 模型 | 参数量 | 最佳学习率 | 验证 loss | 前向计算量（每字） | 推理状态（第 64 字 / 第 4096 字，浮点数） |
+|---|---|---|---|---|---|
+| Transformer | 116,576 | 0.006 | 1.5452 ± 0.0100 | 242 K | 16,384 / 1,048,576（KV 缓存） |
+| Transformer (sin) | 112,480 | 0.006 | 1.5271 ± 0.0083 | 242 K | 16,384 / 1,048,576（KV 缓存） |
+| ZeroPoint | 115,240 | 0.006 | 1.5214 ± 0.0134 | 232 K | 2,048 / 2,048（固定） |
+
+## 学习率搜索（种子 0 的验证 loss）
+
+| 模型 | lr=0.001 | lr=0.003 | lr=0.006 |
+|---|---|---|---|
+| Transformer | 1.6908 | 1.5530 | 1.5520 |
+| Transformer (sin) | 1.8006 | 1.5939 | 1.5374 |
+| ZeroPoint | 1.6682 | 1.5433 | 1.5311 |
+
+## 长度外推（训练上下文 64；8 条验证序列 × 3 个种子平均）
+
+| 位置 | ZeroPoint | Transformer (sin) | Transformer（可学习位置编码） |
+|---|---|---|---|
+| 0 ~ 63（训练见过的长度） | 1.5550 | 1.4512 | 能跑 |
+| 64 ~ 255 | 1.6164 | 4.6359 | 跑不了（只有 64 个位置） |
+| 256 ~ 1023 | 1.6410 | 4.6057 | 跑不了（只有 64 个位置） |
+| 1024 ~ 4095 | 1.7040 | —（注意力矩阵太大，没测） | 跑不了（只有 64 个位置） |
+
+## ZeroPoint 槽的使用情况（有效槽数，满分 16）
+
+有效槽数 = 平均读/写地址分布的 exp(熵)：16 个槽用得一样多 = 16，全挤在一个槽 = 1。
+
+| 层 | 读 | 写 |
+|---|---|---|
+| 0 | 10.1 | 3.2 |
+| 1 | 9.9 | 5.0 |
+
+---
+
+## 更正（2026-11-08 追加，不改上表）
+
+1. **长度外推**：上表只用了 8 条验证序列 × 3 个种子，抽样噪声大（同一模型 0~63 段：8 条序列 1.5550，32 条序列 1.4372）。
+   32 条序列的复测：0~63 **1.4372 ± 0.0187**、64~255 **1.4742 ± 0.0341**、256~1023 **1.6617 ± 0.0234**、
+   1024~4095 **1.6932 ± 0.0305**（远端 − 近端 = **+0.2190 ± 0.0630**）。详见 `diagnose.md` / `ema_longrun.md`。
+   Transformer(sin) 的 4.6359 / 4.6057 仍是 8 条序列的测量（基线权重未保留），≈ ln 96 = 4.56，即随机水平。
+2. **有效槽数**：`slot_usage()` 的写列用的是未归一化的门控分布（每层质量只有 0.47 / 0.59），所以 3.2 / 5.0 偏低。
+   归一化后读列不变（10.1 / 9.9），写列约 6.0 / 10.4（每层）；按段池化 8.6~8.9 / 16。不能据此说"16 个槽只用 3~5 个"。

experiments/compare.py

@@ -0,0 +1,128 @@
+"""ZeroPoint vs Transformer 对照实验（字符级语言模型）。
+
+    python3 experiments/compare.py            # 默认语料：Python 标准库源码（离线可用）
+    python3 experiments/compare.py my.txt     # 换成自己的文本
+    ITERS=2000 python3 experiments/compare.py
+
+三个模型宽度、层数、头数、MLP 完全一样，唯一区别是"token 混合层 + 位置编码"：
+    Transformer        自注意力 + 可学习位置编码
+    Transformer (sin)  自注意力 + 论文 §3.5 的正弦位置编码（理论上能外推到更长序列）
+    ZeroPoint            记忆槽，无位置编码
+
+流程：
+    1. 学习率搜索：每个模型在 LRS 里各试一次（种子 0），取验证 loss 最低的
+    2. 用各自最好的学习率再跑种子 1、2，报告 3 个种子的 均值 ± 标准差
+    3. 长度外推：训练只见过 64 个字，测到 4096 个字
+结果：experiments/compare.md（最新）+ experiments/results/<时间>-compare.md（存档）
+"""
+import sys
+
+import numpy as np
+
+from common import (CFG, ITERS, T, batches, evaluate, forward_flops_per_char, is_zeropoint, prepare_data,
+                    save_result, slot_usage, train)
+from models import TransformerLM, ZeroX
+import zerocg as zc
+
+LRS = [1e-3, 3e-3, 6e-3]
+SEEDS = [0, 1, 2]
+MODELS = {
+    "Transformer": lambda V: TransformerLM(V, T, **CFG),
+    "Transformer (sin)": lambda V: TransformerLM(V, T, pos="sin", **CFG),
+    "ZeroPoint": lambda V: ZeroX(V, n_slot=16, **CFG),
+}
+# 长度外推的分段；Transformer 的注意力矩阵是 长度² 的，4096 字放不进内存，只测到 1024
+SEGMENTS = [(0, 64), (64, 256), (256, 1024), (1024, 4096)]
+LONG_MAX = {"Transformer (sin)": 1024, "ZeroPoint": 4096}
+LONG_N = 8   # 测试序列条数
+
+
+def long_context_loss(model, val_data, max_len):
+    x, y = next(batches(val_data, 1, max_len, 7, batch=LONG_N))
+    with zc.no_grad():
+        logits = model(x).data
+    logp = logits - logits.max(-1, keepdims=True)
+    logp = logp - np.log(np.exp(logp).sum(-1, keepdims=True))
+    nll = -np.take_along_axis(logp, y[..., None], -1)[..., 0]
+    return np.array([nll[:, a:b].mean() if b <= max_len else np.nan for a, b in SEGMENTS])
+
+
+def main():
+    path = sys.argv[1] if len(sys.argv) > 1 else None
+    text, V, train_data, val_data = prepare_data(path)
+    print(f"语料 {len(text):,} 字符，词表 {V}，每次训练 {ITERS} 步", flush=True)
+
+    sweep = {name: {} for name in MODELS}      # name -> lr -> 验证 loss（种子 0）
+    finals = {name: [] for name in MODELS}     # name -> [各种子的验证 loss]
+    long_loss = {name: [] for name in LONG_MAX}
+    usage = []
+
+    def run(name, seed, lr):
+        np.random.seed(seed)
+        model = MODELS[name](V)
+        val = train(name, model, train_data, val_data, seed, lr)
+        print(f"  => {name} seed={seed} lr={lr:g}: val {val:.4f}", flush=True)
+        return model, val
+
+    for name in MODELS:
+        best = None
+        for lr in LRS:
+            model, val = run(name, 0, lr)
+            sweep[name][lr] = val
+            if best is None or val < best[1]:
+                best = (model, val, lr)
+        best_lr = best[2]
+        trained = [best[0]] + [run(name, s, best_lr)[0] for s in SEEDS[1:]]
+        finals[name] = [best[1]] + [None] * (len(SEEDS) - 1)
+        for k, model in enumerate(trained):
+            if k > 0:
+                finals[name][k] = evaluate(model, val_data)
+            if name in LONG_MAX:
+                long_loss[name].append(long_context_loss(model, val_data, LONG_MAX[name]))
+            if is_zeropoint(model):
+                usage.append(slot_usage(model))
+
+    # ---- 写结果 ----
+    m0 = {name: make(V) for name, make in MODELS.items()}
+    lines = ["# ZeroPoint vs Transformer 对照实验\n",
+             f"- 语料：{'Python 标准库源码' if not path else path}，{len(text):,} 字符，词表 {V}；90% 训练 / 10% 验证",
+             f"- 三个模型：宽 {CFG['n_embd']}、{CFG['n_layer']} 层、{CFG['n_head']} 头、Pre-LN、GELU MLP 4 倍；"
+             "ZeroPoint 每头 16 个槽、可学习初始状态、分块并行（块长 32）、无位置编码",
+             f"- 训练：上下文 {T}、batch 32、{ITERS} 步、AdamW（权重衰减 0.01）、预热 100 步 + 余弦、梯度裁剪 1.0",
+             f"- 学习率：每个模型在 {LRS} 里各试一次（种子 0），取最好的再跑种子 {SEEDS[1:]}",
+             "- 验证 loss = 40 个固定验证 batch 的平均交叉熵（nats/字符，越低越好），均值 ± 标准差（3 个种子）",
+             "- 计算量 = 前向时每个字的矩阵乘浮点运算次数（不受机器负载影响；训练一步约 3 倍）\n",
+             "| 模型 | 参数量 | 最佳学习率 | 验证 loss | 前向计算量（每字） | 推理状态（第 64 字 / 第 4096 字，浮点数） |",
+             "|---|---|---|---|---|---|"]
+    for name, vals in finals.items():
+        m = m0[name]
+        best_lr = min(sweep[name], key=sweep[name].get)
+        state = (f"{m.state_floats():,} / {m.state_floats():,}（固定）" if is_zeropoint(m)
+                 else f"{m.state_floats(64):,} / {m.state_floats(4096):,}（KV 缓存）")
+        lines.append(f"| {name} | {m.num_parameters():,} | {best_lr:g} | {np.mean(vals):.4f} ± {np.std(vals):.4f} | "
+                     f"{forward_flops_per_char(m, V) / 1e3:.0f} K | {state} |")
+
+    lines += ["", "## 学习率搜索（种子 0 的验证 loss）\n",
+              "| 模型 | " + " | ".join(f"lr={lr:g}" for lr in LRS) + " |",
+              "|---|" + "---|" * len(LRS)]
+    for name, d in sweep.items():
+        lines.append(f"| {name} | " + " | ".join(f"{d[lr]:.4f}" for lr in LRS) + " |")
+
+    lines += ["", f"## 长度外推（训练上下文 {T}；{LONG_N} 条验证序列 × 3 个种子平均）\n",
+              "| 位置 | ZeroPoint | Transformer (sin) | Transformer（可学习位置编码） |", "|---|---|---|---|"]
+    lz, ls = np.nanmean(long_loss["ZeroPoint"], axis=0), np.nanmean(long_loss["Transformer (sin)"], axis=0)
+    for k, (a, b) in enumerate(SEGMENTS):
+        tag = "（训练见过的长度）" if b <= T else ""
+        s = "—（注意力矩阵太大，没测）" if np.isnan(ls[k]) else f"{ls[k]:.4f}"
+        lines.append(f"| {a} ~ {b - 1}{tag} | {lz[k]:.4f} | {s} | {'能跑' if b <= T else f'跑不了（只有 {T} 个位置）'} |")
+
+    u = np.mean(usage, axis=0)   # (层, 2)
+    lines += ["", "## ZeroPoint 槽的使用情况（有效槽数，满分 16）\n",
+              "有效槽数 = 平均读/写地址分布的 exp(熵)：16 个槽用得一样多 = 16，全挤在一个槽 = 1。\n",
+              "| 层 | 读 | 写 |", "|---|---|---|"]
+    lines += [f"| {i} | {r:.1f} | {w:.1f} |" for i, (r, w) in enumerate(u)]
+    save_result("compare", lines)
+
+
+if __name__ == "__main__":
+    main()

experiments/diagnose.md

@@ -0,0 +1,65 @@
+# 预注册实验：查三个问题
+
+> 写于实验之前（2026-10-18）。跑完只在下面追加结果，不改这一节。
+
+**背景**：ema_longrun.md 否定了"槽值无界累积"假设。无 ema 的模型在 1024~4095 字处 loss 比 64~255 字处高约 0.22；
+有 ema 的高约 1.77。compare.md 里 16 个槽只有 3~5 个被写入（**更正**：3~5 是按未归一化的门控分布算的偏低值；归一化后读约 10/16，写约 6~10/层）。
+
+## 假设和预测
+
+**H2（训练长度）**：长序列变差是因为训练只见过 64 字；模型在更长的位置上遇到的槽状态分布是训练时没见过的。
+- 预测：训练长度改为 256（每步字数不变、其余相同）后，1024~4095 与 64~255 的 loss 差 **缩小到 T64 的一半以下**。
+- 不成立时会看到：差距没有缩小（< 2 × 标准差的变化）。
+
+**H3（ema 为什么更差）**：ema 把长期槽的写入乘上 (1-λ)，训练时（64 字）长期槽几乎不起作用，模型学会只用短期槽；
+到了长序列，读地址仍指向短期槽，但短期槽里的内容和训练时的统计不同。
+- 预测 ③a：ema 模型的**写地址**集中在半衰期短的槽上：半衰期 > 16 字的槽，写入占比 < T64 模型的一半。
+- 预测 ③b：ema 模型学到的长期槽 λ 更接近 1（半衰期更长），或者长期槽的读占比在远处位置上升。
+- 不成立时会看到：两组的写入分布没有明显差别。
+
+**H4（槽塌缩）**：塌缩不是训练失败，而是写地址的 softmax 竞争让少数槽赢者通吃；在不同序列位置上"赢的槽"不同，
+所以整体统计（compare.md 的 3~5 个有效槽，见上面对该度量的更正）低估了实际用到的槽数。
+- 预测 ④：按位置分段（0~63 / 64~255 / 256~1023 / 1024~4095）分别统计有效写槽数，各段的并集 > 整体有效槽数 × 1.5；
+  同时各段的读/写分布之间有明显差别（任意两段的 JS 散度 > 0.1）。
+- 不成立时会看到：各段的分布几乎相同，用的始终是同几个槽。
+
+**判定**：每条按上面的数值阈值判，3 个种子均值；差别和 2 × 种子间标准差比。
+
+
+## 结果（实验之后写）
+
+### 验证 loss（64 字）和长度外推（32 条 × 4096 字；均值 ± 标准差，3 个种子）
+
+| 条件 | 64 字验证 | 0~63 | 64~255 | 256~1023 | 1024~4095 | 远−近 差 |
+|---|---|---|---|---|---|---|
+| T64 | 1.5214 ± 0.0134 | 1.4372 ± 0.0187 | 1.4742 ± 0.0341 | 1.6617 ± 0.0234 | 1.6932 ± 0.0305 | +0.2190 ± 0.0630 |
+| T64-ema | 1.5182 ± 0.0076 | 1.4308 ± 0.0039 | 1.6476 ± 0.0529 | 3.1084 ± 0.2157 | 3.4153 ± 0.2402 | +1.7677 ± 0.2188 |
+| T256 | 1.5572 ± 0.0060 | 1.4741 ± 0.0117 | 1.4486 ± 0.0098 | 1.5098 ± 0.0022 | 1.5155 ± 0.0048 | +0.0669 ± 0.0051 |
+
+### H3：写入落在半衰期 > 16 字的槽上的占比（0~63 段），和学到的最长半衰期
+
+| 条件 | 长期槽写入占比 | 最长半衰期（字） | 远处读长期槽占比的变化 |
+|---|---|---|---|
+| T64 | 0.061 ± 0.005 | 109 | +0.009 |
+| T64-ema | 0.014 ± 0.001 | 103 | +0.007 |
+
+### H4：槽的使用（T64，写地址；各层各头平均）
+
+| 段 | 有效写槽数 |
+|---|---|
+| 0~63 | 8.6 |
+| 64~255 | 8.7 |
+| 256~1023 | 8.5 |
+| 1024~4095 | 8.9 |
+| 整体 | 8.7 |
+
+各段用到的槽的并集 ÷ 整体有效槽数 = 0.68；各段分布之间最大的 JS 散度 = 0.010
+
+## 判定（按预注册的标准）
+
+- H2 训练长度：远−近差 T64 +0.2190 → T256 +0.0669 → **成立**
+- H3a ema 写入避开长期槽：占比 0.061 → 0.014 → **成立**
+- H3b ema 的长期槽更长或远处更依赖长期槽 → **不成立**
+- H4 槽塌缩是分段使用不同的槽：并集比 0.68，JS 0.010 → **不成立**
+
+模型已保存在 experiments/out/diag_*.npz，可以事后再分析。

experiments/diagnose.py

@@ -0,0 +1,216 @@
+"""预注册实验：查 ZeroPoint 的三个问题 —— 长序列变差的原因、ema 为什么更差、槽为什么塌缩。
+
+    python3 experiments/diagnose.py
+
+设计在跑之前写定（PREREG），跑完只追加结果。训练好的模型保存在 experiments/out/，供事后再分析。
+结果：experiments/diagnose.md + results/ 存档。
+"""
+import os
+
+import numpy as np
+
+from common import CFG, HERE, ITERS, T, B, batches, prepare_data, save_result, train
+from models import ZeroX
+import zerocg as zc
+
+SEEDS = [0, 1, 2]
+LR = 6e-3
+LONG_T, LONG_N = 4096, 32
+SEGMENTS = [(0, 64), (64, 256), (256, 1024), (1024, 4096)]
+OUT = os.path.join(HERE, "out")
+CONDITIONS = {   # 名字 -> (ema, 训练长度, batch)；每步的字数都是 2048
+    "T64":     (False, 64, 32),
+    "T64-ema": (True, 64, 32),
+    "T256":    (False, 256, 8),
+}
+
+PREREG = """# 预注册实验：查三个问题
+
+> 写于实验之前（2026-10-18）。跑完只在下面追加结果，不改这一节。
+
+**背景**：ema_longrun.md 否定了"槽值无界累积"假设。无 ema 的模型在 1024~4095 字处 loss 比 64~255 字处高约 0.22；
+有 ema 的高约 1.77。compare.md 里 16 个槽只有 3~5 个被写入。
+
+## 假设和预测
+
+**H2（训练长度）**：长序列变差是因为训练只见过 64 字；模型在更长的位置上遇到的槽状态分布是训练时没见过的。
+- 预测：训练长度改为 256（每步字数不变、其余相同）后，1024~4095 与 64~255 的 loss 差 **缩小到 T64 的一半以下**。
+- 不成立时会看到：差距没有缩小（< 2 × 标准差的变化）。
+
+**H3（ema 为什么更差）**：ema 把长期槽的写入乘上 (1-λ)，训练时（64 字）长期槽几乎不起作用，模型学会只用短期槽；
+到了长序列，读地址仍指向短期槽，但短期槽里的内容和训练时的统计不同。
+- 预测 ③a：ema 模型的**写地址**集中在半衰期短的槽上：半衰期 > 16 字的槽，写入占比 < T64 模型的一半。
+- 预测 ③b：ema 模型学到的长期槽 λ 更接近 1（半衰期更长），或者长期槽的读占比在远处位置上升。
+- 不成立时会看到：两组的写入分布没有明显差别。
+
+**H4（槽塌缩）**：塌缩不是训练失败，而是写地址的 softmax 竞争让少数槽赢者通吃；在不同序列位置上"赢的槽"不同，
+所以整体统计（compare.md 的 3~5 个有效槽）低估了实际用到的槽数。
+- 预测 ④：按位置分段（0~63 / 64~255 / 256~1023 / 1024~4095）分别统计有效写槽数，各段的并集 > 整体有效槽数 × 1.5；
+  同时各段的读/写分布之间有明显差别（任意两段的 JS 散度 > 0.1）。
+- 不成立时会看到：各段的分布几乎相同，用的始终是同几个槽。
+
+**判定**：每条按上面的数值阈值判，3 个种子均值；差别和 2 × 种子间标准差比。
+"""
+
+
+def long_nll(model, val_data):
+    out = []
+    for x, y in batches(val_data, 4, LONG_T, 7, batch=LONG_N // 4):
+        with zc.no_grad():
+            logits = model(x).data
+        logp = logits - logits.max(-1, keepdims=True)
+        logp = logp - np.log(np.exp(logp).sum(-1, keepdims=True))
+        out.append(-np.take_along_axis(logp, y[..., None], -1)[..., 0])
+    nll = np.concatenate(out)
+    return np.array([nll[:, a:b].mean() for a, b in SEGMENTS])
+
+
+def address_stats(model, val_data):
+    """递推跑 4096 字，按位置分段统计每层每头的平均读/写地址分布。返回 (层, 段, H, M) 两个数组。"""
+    x, _ = next(batches(val_data, 1, LONG_T, 7, batch=4))
+    states = model.init_state(4)
+    L = len(model.blocks)
+    H, M = model.blocks[0].mix.H, model.blocks[0].mix.M
+    reads = np.zeros((L, len(SEGMENTS), H, M))
+    writes = np.zeros((L, len(SEGMENTS), H, M))
+    for t in range(LONG_T):
+        _, states = model.step(x[:, t], states, t)
+        seg = next(i for i, (a, b) in enumerate(SEGMENTS) if a <= t < b)
+        for l, blk in enumerate(model.blocks):
+            a, w = blk.mix.last_addr
+            reads[l, seg] += a.mean(0)
+            writes[l, seg] += w.mean(0)
+    for i, (a, b) in enumerate(SEGMENTS):
+        reads[:, i] /= b - a
+        writes[:, i] /= b - a
+    return reads, writes
+
+
+def eff(p):
+    """exp(熵)：有效槽数。p 最后一维是槽。"""
+    p = p / p.sum(-1, keepdims=True)
+    return np.exp(-(p * np.log(p + 1e-12)).sum(-1))
+
+
+def js(p, q):
+    p, q = p / p.sum(), q / q.sum()
+    m = (p + q) / 2
+    kl = lambda a, b: (a * np.log((a + 1e-12) / (b + 1e-12))).sum()  # noqa: E731
+    return 0.5 * kl(p, m) + 0.5 * kl(q, m)
+
+
+def half_lives(model):
+    """每层每头每槽的半衰期（字）。"""
+    return np.array([np.log(0.5) / np.log(b.mix.retention().data) for b in model.blocks])
+
+
+def main():
+    text, V, train_data, val_data = prepare_data()
+    os.makedirs(OUT, exist_ok=True)
+    print(PREREG, flush=True)
+    res = {}
+    for name, (ema, seq_len, batch) in CONDITIONS.items():
+        rows = []
+        for seed in SEEDS:
+            np.random.seed(seed)
+            model = ZeroX(V, n_slot=16, ema=ema, **CFG)
+            val = train(name, model, train_data, val_data, seed, LR, seq_len=seq_len, batch=batch)
+            model.eval()
+            model.save(os.path.join(OUT, f"diag_{name}_s{seed}.npz"))
+            ll = long_nll(model, val_data)
+            reads, writes = address_stats(model, val_data)
+            hl = half_lives(model)
+            rows.append(dict(val=val, long=ll, reads=reads, writes=writes, hl=hl))
+            print(f"  => {name} seed={seed}: val {val:.4f}; long {np.round(ll, 4)}", flush=True)
+        res[name] = rows
+
+    def mean(name, key):
+        return np.mean([r[key] for r in res[name]], axis=0)
+
+    def std(name, key):
+        return np.std([r[key] for r in res[name]], axis=0)
+
+    lines = [PREREG, "\n## 结果（实验之后写）\n",
+             "### 验证 loss（64 字）和长度外推（32 条 × 4096 字；均值 ± 标准差，3 个种子）\n",
+             "| 条件 | 64 字验证 | " + " | ".join(f"{a}~{b - 1}" for a, b in SEGMENTS) + " | 远−近 差 |",
+             "|---|---|" + "---|" * (len(SEGMENTS) + 1)]
+    gaps = {}
+    for name in res:
+        lm, ls = mean(name, "long"), std(name, "long")
+        g = np.array([r["long"][3] - r["long"][1] for r in res[name]])
+        gaps[name] = g
+        lines.append(f"| {name} | {mean(name, 'val'):.4f} ± {std(name, 'val'):.4f} | " +
+                     " | ".join(f"{m:.4f} ± {s:.4f}" for m, s in zip(lm, ls)) + f" | {g.mean():+.4f} ± {g.std():.4f} |")
+
+    # ---- H2 ----
+    g64, g256 = gaps["T64"], gaps["T256"]
+    # T64 的远−近差必须本来就是正的（确实变差了），H2 才有意义
+    h2 = g64.mean() > 0 and g256.mean() < 0.5 * g64.mean() and (g64.mean() - g256.mean()) > 2 * max(g64.std(), g256.std())
+
+    # ---- H3：写入在长期槽上的占比；学到的半衰期 ----
+    def long_slot_write_share(name):
+        out = []
+        for r in res[name]:
+            share = []
+            for l in range(r["hl"].shape[0]):
+                mask = r["hl"][l] > 16                                   # (H, M)
+                w = r["writes"][l, 0]                                     # 训练长度内（0~63 段）的写分布 (H, M)
+                share.append((w * mask).sum() / w.sum())
+            out.append(np.mean(share))
+        return np.array(out)
+    s64, sema = long_slot_write_share("T64"), long_slot_write_share("T64-ema")
+    h3a = sema.mean() < 0.5 * s64.mean()
+    hl64, hlema = mean("T64", "hl"), mean("T64-ema", "hl")
+    far_read_long = {}
+    for name in ("T64", "T64-ema"):
+        vals = []
+        for r in res[name]:
+            v = []
+            for l in range(r["hl"].shape[0]):
+                mask = r["hl"][l] > 16
+                v.append((r["reads"][l, 3] * mask).sum() / r["reads"][l, 3].sum() -
+                         (r["reads"][l, 0] * mask).sum() / r["reads"][l, 0].sum())
+            vals.append(np.mean(v))
+        far_read_long[name] = np.array(vals)
+    h3b = np.max(hlema) > np.max(hl64) * 1.5 or far_read_long["T64-ema"].mean() > far_read_long["T64"].mean() + 0.1
+
+    lines += ["", "### H3：写入落在半衰期 > 16 字的槽上的占比（0~63 段），和学到的最长半衰期\n",
+              "| 条件 | 长期槽写入占比 | 最长半衰期（字） | 远处读长期槽占比的变化 |", "|---|---|---|---|"]
+    for name in ("T64", "T64-ema"):
+        sh = long_slot_write_share(name)
+        lines.append(f"| {name} | {sh.mean():.3f} ± {sh.std():.3f} | {np.max(mean(name, 'hl')):.0f} | "
+                     f"{far_read_long[name].mean():+.3f} |")
+
+    # ---- H4：分段的有效写槽数、并集、JS 散度 ----
+    lines += ["", "### H4：槽的使用（T64，写地址；各层各头平均）\n",
+              "| 段 | 有效写槽数 |", "|---|---|"]
+    union_ratio, js_min = [], []
+    for r in res["T64"]:
+        w = r["writes"]                                     # (L, seg, H, M)
+        overall = eff(w.mean(1))                            # (L, H)
+        per_seg = eff(w)                                    # (L, seg, H)
+        used = (w / w.sum(-1, keepdims=True)) > 1.0 / w.shape[-1]   # 用得比平均多的槽 (L, seg, H, M)
+        union = used.any(1).sum(-1)                         # (L, H)
+        union_ratio.append((union / overall).mean())
+        js_vals = [js(w[l, i, h], w[l, j, h]) for l in range(w.shape[0]) for h in range(w.shape[2])
+                   for i in range(len(SEGMENTS)) for j in range(i + 1, len(SEGMENTS))]
+        js_min.append(np.max(js_vals))
+    w_mean = mean("T64", "writes")
+    for i, (a, b) in enumerate(SEGMENTS):
+        lines.append(f"| {a}~{b - 1} | {eff(w_mean[:, i]).mean():.1f} |")
+    lines.append(f"| 整体 | {eff(w_mean.mean(1)).mean():.1f} |")
+    union_ratio, js_min = np.array(union_ratio), np.array(js_min)
+    h4 = union_ratio.mean() > 1.5 and js_min.mean() > 0.1
+    lines += [f"\n各段用到的槽的并集 ÷ 整体有效槽数 = {union_ratio.mean():.2f}；各段分布之间最大的 JS 散度 = {js_min.mean():.3f}"]
+
+    lines += ["", "## 判定（按预注册的标准）\n",
+              f"- H2 训练长度：远−近差 T64 {g64.mean():+.4f} → T256 {g256.mean():+.4f} → **{'成立' if h2 else '不成立'}**",
+              f"- H3a ema 写入避开长期槽：占比 {s64.mean():.3f} → {sema.mean():.3f} → **{'成立' if h3a else '不成立'}**",
+              f"- H3b ema 的长期槽更长或远处更依赖长期槽 → **{'成立' if h3b else '不成立'}**",
+              f"- H4 槽塌缩是分段使用不同的槽：并集比 {union_ratio.mean():.2f}，JS {js_min.mean():.3f} → **{'成立' if h4 else '不成立'}**",
+              "", "模型已保存在 experiments/out/diag_*.npz，可以事后再分析。"]
+    save_result("diagnose", lines)
+
+
+if __name__ == "__main__":
+    main()

experiments/ema_longrun.md

@@ -0,0 +1,55 @@
+# 预注册实验：长序列变差与 ema 写入
+
+> 写于实验之前（2026-10-18）。跑完后只在下面追加结果，不改这一节。
+
+**观察**：compare.md 里 ZeroPoint（无 ema）在 1024~4095 字处的 loss（1.70）高于 64~255 字处（1.62）。
+
+**假设 H1**：原因是长期槽（λ 接近 1）的数值随序列长度无界累积，超出训练时（64 字）见过的范围。
+
+**预测**：
+- ① 直接测：推理时槽值的最大绝对值，无 ema 的模型在 4000 字处明显大于 64 字处（比值 > 2）；有 ema 的模型比值接近 1（< 1.5）。
+- ② 间接测：开 ema 重新训练（其余完全相同），1024~4095 字处的 loss 不再明显高于 64~255 字处：差值 < 2 × 种子间标准差。
+
+**H1 不成立时会看到**：① 无 ema 的槽值没有明显增长；或 ② 开 ema 后远处 loss 照样升高。
+
+**判定**：① 和 ② 都成立 → 接受 H1，且 ema 是有效的修法；只有 ② 成立 → ema 有效但原因另有其他；
+都不成立 → 放弃 H1，下一步查学到的 λ 值。另外要求 ema 不损害短距离：64 字处验证 loss 的差别 < 2 × 标准差。
+
+**对照**：两组用同样的 3 个种子、同样 2000 步、学习率 0.006、同样语料；外推用 32 条 4096 字的验证序列，
+两组都用分块并行算法算 loss。（10-18 的修正是把验证序列从 8 条增加到 32 条；分块并行与递推形式数学等价，float32 下 logit 最大差 1.4e-5，不是算法差异。）
+
+
+## 结果（实验之后写）
+
+### 64 字处的验证 loss（均值 ± 标准差，3 个种子）
+
+| 模型 | 验证 loss |
+|---|---|
+| no-ema | 1.5214 ± 0.0134 |
+| ema | 1.5182 ± 0.0076 |
+
+### 长度外推（32 条 × 4096 字，分块并行算法；均值 ± 标准差）
+
+| 位置 | no-ema | ema |
+|---|---|---|
+| 0 ~ 63 | 1.4372 ± 0.0187 | 1.4308 ± 0.0039 |
+| 64 ~ 255 | 1.4742 ± 0.0341 | 1.6476 ± 0.0529 |
+| 256 ~ 1023 | 1.6617 ± 0.0234 | 3.1084 ± 0.2157 |
+| 1024 ~ 4095 | 1.6932 ± 0.0305 | 3.4153 ± 0.2402 |
+
+### 槽值的最大绝对值（递推推理，3 个种子平均）
+
+| 位置 | no-ema | ema |
+|---|---|---|
+| 64 | 9.08 | 1.60 |
+| 256 | 10.07 | 2.34 |
+| 1024 | 9.88 | 1.51 |
+| 4000 | 16.07 | 2.27 |
+
+## 判定（按预注册的标准）
+
+- 预测 ①（槽值比值 4000 字 / 64 字）：无 ema 1.93，有 ema 1.44 → **不成立**
+- 预测 ②（ema 模型 1024~4095 与 64~255 的 loss 差）：+1.7677，噪声阈值 0.4805 → **不成立**（对照：无 ema 的差是 +0.2190）
+- ema 不损害短距离：64 字处 loss 差 -0.0033 → **成立**
+
+**结论**：① ② 都不成立：**放弃 H1**，下一步查学到的 λ 值

experiments/ema_longrun.py

@@ -0,0 +1,128 @@
+"""预注册实验：ZeroPoint 长序列变差的原因是不是槽值无界累积？ema 写入能不能修好？
+
+    python3 experiments/ema_longrun.py
+
+实验设计在跑之前写定（见 PREREG），结果出来后不改判定标准。结果：experiments/ema_longrun.md + results/ 存档。
+"""
+import numpy as np
+
+from common import CFG, ITERS, T, batches, evaluate, prepare_data, save_result, train
+from models import ZeroX
+import zerocg as zc
+
+SEEDS = [0, 1, 2]
+LR = 6e-3                      # compare.md 里 ZeroPoint 的最佳学习率
+LONG_T = 4096
+LONG_N = 32                    # 外推测试序列条数（compare.py 只有 8 条，这里加到 32）
+SEGMENTS = [(0, 64), (64, 256), (256, 1024), (1024, 4096)]
+PROBE_POS = [64, 256, 1024, 4000]   # 测槽值范围的位置
+
+PREREG = """# 预注册实验：长序列变差与 ema 写入
+
+> 写于实验之前（2026-10-18）。跑完后只在下面追加结果，不改这一节。
+
+**观察**：compare.md 里 ZeroPoint（无 ema）在 1024~4095 字处的 loss（1.70）高于 64~255 字处（1.62）。
+
+**假设 H1**：原因是长期槽（λ 接近 1）的数值随序列长度无界累积，超出训练时（64 字）见过的范围。
+
+**预测**：
+- ① 直接测：推理时槽值的最大绝对值，无 ema 的模型在 4000 字处明显大于 64 字处（比值 > 2）；有 ema 的模型比值接近 1（< 1.5）。
+- ② 间接测：开 ema 重新训练（其余完全相同），1024~4095 字处的 loss 不再明显高于 64~255 字处：差值 < 2 × 种子间标准差。
+
+**H1 不成立时会看到**：① 无 ema 的槽值没有明显增长；或 ② 开 ema 后远处 loss 照样升高。
+
+**判定**：① 和 ② 都成立 → 接受 H1，且 ema 是有效的修法；只有 ② 成立 → ema 有效但原因另有其他；
+都不成立 → 放弃 H1，下一步查学到的 λ 值。另外要求 ema 不损害短距离：64 字处验证 loss 的差别 < 2 × 标准差。
+
+**对照**：两组用同样的 3 个种子、同样 {iters} 步、学习率 {lr}、同样语料；外推用 {n} 条 {lt} 字的验证序列，
+两组都用分块并行算法算 loss（compare.py 里 ZeroPoint 用的是递推算法，和 Transformer 不对等，这里统一）。
+"""
+
+
+def long_loss(model, val_data):
+    """(各段 loss 均值) ；分 4 批算，省内存。"""
+    nll_all = []
+    for x, y in batches(val_data, 4, LONG_T, 7, batch=LONG_N // 4):
+        with zc.no_grad():
+            logits = model(x).data
+        logp = logits - logits.max(-1, keepdims=True)
+        logp = logp - np.log(np.exp(logp).sum(-1, keepdims=True))
+        nll_all.append(-np.take_along_axis(logp, y[..., None], -1)[..., 0])
+    nll = np.concatenate(nll_all)
+    return np.array([nll[:, a:b].mean() for a, b in SEGMENTS])
+
+
+def slot_magnitude(model, val_data):
+    """递推跑 4096 字，记录各位置处所有层槽值的最大绝对值。返回 {位置: 最大|S|}。"""
+    x, _ = next(batches(val_data, 1, LONG_T, 7, batch=4))
+    states = model.init_state(4)
+    out = {}
+    for t in range(LONG_T):
+        _, states = model.step(x[:, t], states, t)
+        if t + 1 in PROBE_POS:
+            out[t + 1] = max(float(np.abs(S).max()) for S in states)
+    return out
+
+
+def main():
+    text, V, train_data, val_data = prepare_data()
+    print(PREREG.format(iters=ITERS, lr=LR, n=LONG_N, lt=LONG_T), flush=True)
+    res = {}
+    for ema in (False, True):
+        name = "ema" if ema else "no-ema"
+        vals, longs, mags = [], [], []
+        for seed in SEEDS:
+            np.random.seed(seed)
+            model = ZeroX(V, n_slot=16, ema=ema, **CFG)
+            vals.append(train(name, model, train_data, val_data, seed, LR))
+            model.eval()
+            longs.append(long_loss(model, val_data))
+            mags.append(slot_magnitude(model, val_data))
+            print(f"  => {name} seed={seed}: val {vals[-1]:.4f}; long {np.round(longs[-1], 4)}; "
+                  f"|S| {[round(mags[-1][p], 2) for p in PROBE_POS]}", flush=True)
+        res[name] = dict(val=np.array(vals), long=np.array(longs),
+                         mag=np.array([[m[p] for p in PROBE_POS] for m in mags]))
+
+    # ---- 结果 + 按预注册的标准判定 ----
+    lines = [PREREG.format(iters=ITERS, lr=LR, n=LONG_N, lt=LONG_T), "\n## 结果（实验之后写）\n",
+             "### 64 字处的验证 loss（均值 ± 标准差，3 个种子）\n", "| 模型 | 验证 loss |", "|---|---|"]
+    for name, r in res.items():
+        lines.append(f"| {name} | {r['val'].mean():.4f} ± {r['val'].std():.4f} |")
+    lines += ["", f"### 长度外推（{LONG_N} 条 × {LONG_T} 字，分块并行算法；均值 ± 标准差）\n",
+              "| 位置 | " + " | ".join(res) + " |", "|---|" + "---|" * len(res)]
+    for k, (a, b) in enumerate(SEGMENTS):
+        lines.append(f"| {a} ~ {b - 1} | " + " | ".join(f"{r['long'][:, k].mean():.4f} ± {r['long'][:, k].std():.4f}"
+                                                        for r in res.values()) + " |")
+    lines += ["", "### 槽值的最大绝对值（递推推理，3 个种子平均）\n",
+              "| 位置 | " + " | ".join(res) + " |", "|---|" + "---|" * len(res)]
+    for k, p in enumerate(PROBE_POS):
+        lines.append(f"| {p} | " + " | ".join(f"{r['mag'][:, k].mean():.2f}" for r in res.values()) + " |")
+
+    no, em = res["no-ema"], res["ema"]
+    ratio_no = (no["mag"][:, 3] / no["mag"][:, 0]).mean()
+    ratio_em = (em["mag"][:, 3] / em["mag"][:, 0]).mean()
+    p1 = ratio_no > 2 and ratio_em < 1.5
+    far, near = em["long"][:, 3], em["long"][:, 1]
+    gap, noise = (far - near).mean(), 2 * max(far.std(), near.std())
+    p2 = gap < noise
+    gap_no = (no["long"][:, 3] - no["long"][:, 1]).mean()
+    short_ok = abs(em["val"].mean() - no["val"].mean()) < 2 * max(em["val"].std(), no["val"].std())
+    if p1 and p2:
+        verdict = "① ② 都成立：**接受 H1**，ema 是有效的修法"
+    elif p2:
+        verdict = "只有 ② 成立：ema 有效，但长序列变差的原因不是（或不只是）槽值无界"
+    elif p1:
+        verdict = "只有 ① 成立：槽值确实无界，但 ema 没有修好远处的 loss，H1 不足以解释变差"
+    else:
+        verdict = "① ② 都不成立：**放弃 H1**，下一步查学到的 λ 值"
+    lines += ["", "## 判定（按预注册的标准）\n",
+              f"- 预测 ①（槽值比值 4000 字 / 64 字）：无 ema {ratio_no:.2f}，有 ema {ratio_em:.2f} → **{'成立' if p1 else '不成立'}**",
+              f"- 预测 ②（ema 模型 1024~4095 与 64~255 的 loss 差）：{gap:+.4f}，噪声阈值 {noise:.4f} → **{'成立' if p2 else '不成立'}**"
+              f"（对照：无 ema 的差是 {gap_no:+.4f}）",
+              f"- ema 不损害短距离：64 字处 loss 差 {em['val'].mean() - no['val'].mean():+.4f} → **{'成立' if short_ok else '不成立'}**",
+              f"\n**结论**：{verdict}" + ("" if short_ok else "；但 ema 损害了短距离效果，不能直接采用")]
+    save_result("ema_longrun", lines)
+
+
+if __name__ == "__main__":
+    main()

experiments/recall.md

@@ -0,0 +1,37 @@
+# 长距离精确回忆（联想检索）
+
+`k1 v1 k2 v2 … kN vN [问] kq` → 回答 kq 对应的值。16 个键、16 个值，随机猜的准确率是 1/16 ≈ 6%。
+三个模型宽 64、2 层、4 头；训练 20000 步，N 在 2~16 随机；每格是 512 道测试题 × 种子 [0, 1] 的平均准确率。
+
+| 模型 | N = 2 | N = 4 | N = 8 | N = 12 | N = 16 |
+|---|---|---|---|---|---|
+| Transformer | 50% | 30% | 23% | 20% | 18% |
+| ZeroPoint (16 槽) | 51% | 32% | 23% | 20% | 16% |
+| ZeroPoint (32 槽) | 49% | 31% | 24% | 20% | 17% |
+
+## 判定与解释（11-08 补）
+
+**预注册判定**（见 [`recall_prereg.md`](recall_prereg.md)）
+- P1（Transformer 在 N=16 准确率 ≥ 80%）**不成立**：18%。
+- P2（ZeroPoint 比 Transformer 低 ≥ 20 个百分点）**不成立**：N=16 时 16% / 17% vs 18%，差 1~2 点。
+- P3（32 槽比 16 槽高 ≥ 5 个百分点）**不成立**：17% vs 16%。
+
+**和 3000 步比**：几乎完全一样（Transformer N=2 52% → 50%、N=16 14% → 18%；ZeroPoint 16 槽 51% → 51%、17% → 16%）。训练量加长 6.7 倍没有任何可辨别的提升。
+
+## 诊断（11-08，`recall_diag.py`；诊断性实验，不参与上面的预注册判定）
+
+1. **固定 32 条 N=2 题反复训练**：600 步训练集准确率 100%，loss ≈ 0 → 框架、优化、反向传播没问题。
+2. **只训 N=1**（复制值，不需要匹配查询）：500 步 100% → 复制机制能学会。
+3. **只训 N=2，换学习率安排**（种子 0，各 8000 步）：
+   - 恒定 3e-3（本脚本原设置）：N=2 停在 53%，loss ≈ 0.72。
+   - 余弦 3e-3 → 3e-4：N=2 停在 53~59%，loss ≈ 0.72。
+   - **余弦 1e-3 → 1e-4：第 6000 步开始突破，第 8000 步 N=2 = 100%，loss 0.001。**
+   → 任务能学会；**原来的 3e-3 恒定学习率把模型困在"两个候选值各猜一半"的状态**（loss ≈ ln 2 ≈ 0.69）。另外，单独只用恒定 3e-3 训 N=2 到 20000 步也仍是 53%，所以不是步数问题。
+4. **混合 N（2~16）用同一个 1e-3 余弦跑 12000 步**：N=2 仍只有 56%。混合训练里 N=2 的样本只占约 1/15，梯度信号被稀释，12000 步对 N=2 只相当于 800 步左右——还不够。
+
+**结论（已测量）**：20000 步无结论不是"训练量差一点"，而是**训练配置问题**：
+- 恒定 3e-3 太高，模型停在 50/50 对冲的局部解，学不出"用查询键选值"的回路；
+- 混合 N 把简单样本（N 小）的信号稀释了大约 15 倍。
+- 只学 N=2 时，1e-3 余弦 8000 步就能到 100%，说明 Transformer 架构有能力学这个任务。
+
+**下一步**：按修正后的学习率（1e-3 余弦）重跑 recall，或者加课程（先 N=2，再逐步加长到 16）；需要重写预注册。在那之前，recall 的结论仍是"无结论"，**不能拿这张 20000 步的表去比架构**。

experiments/recall.py

@@ -0,0 +1,87 @@
+"""长距离精确回忆（联想检索）实验：专门测 ZeroPoint 的已知弱点。
+
+序列格式（每个数字是一个词元）：
+    k1 v1 k2 v2 ... kN vN  [问]  kq   →  模型要说出 kq 对应的 v
+键从 16 个里不重复地抽 N 个，值从另外 16 个里随机抽。要答对，模型必须把 N 对"键 → 值"原样记住。
+注意力天生擅长这个（用 kq 当查询去找那个键）；ZeroPoint 只有 M 个槽，对数一多就会混。
+
+    python3 experiments/recall.py
+训练时 N 在 2~16 之间随机，测试时分别报告 N = 2, 4, 8, 12, 16 的准确率。
+结果：experiments/recall.md（最新）+ experiments/results/<时间>-recall.md（存档）
+"""
+import os
+
+import numpy as np
+
+from common import CFG, save_result
+from models import TransformerLM, ZeroX
+import zerocg as zc
+from zerocg import functional as F
+
+ITERS = int(os.environ.get("RECALL_ITERS", 3000))
+NK, NV = 16, 16
+QUERY = NK + NV          # "问" 这个词元
+VOCAB = NK + NV + 1
+MAX_N = 16
+MAX_LEN = 2 * MAX_N + 2
+TEST_N = [2, 4, 8, 12, 16]
+SEEDS = [0, 1]
+MODELS = {
+    "Transformer": lambda: TransformerLM(VOCAB, MAX_LEN, **CFG),
+    "ZeroPoint (16 槽)": lambda: ZeroX(VOCAB, n_slot=16, **CFG),
+    "ZeroPoint (32 槽)": lambda: ZeroX(VOCAB, n_slot=32, **CFG),
+}
+
+
+def make_batch(rng, B, n):
+    keys = np.stack([rng.permutation(NK)[:n] for _ in range(B)])
+    vals = rng.integers(NK, NK + NV, size=(B, n))
+    seq = np.empty((B, 2 * n + 2), dtype=np.int64)
+    seq[:, 0:2 * n:2], seq[:, 1:2 * n:2] = keys, vals
+    q = rng.integers(0, n, B)
+    seq[:, -2], seq[:, -1] = QUERY, keys[np.arange(B), q]
+    return seq, vals[np.arange(B), q]
+
+
+def train_and_eval(make, seed):
+    np.random.seed(seed)
+    rng = np.random.default_rng(seed)
+    model = make()
+    opt = zc.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=0.01)
+    for it in range(ITERS):
+        opt.lr = 3e-3 * min(1, (it + 1) / 100)
+        x, y = make_batch(rng, 32, int(rng.integers(2, MAX_N + 1)))
+        loss = F.cross_entropy(model(x)[:, -1], y)
+        opt.zero_grad()
+        loss.backward()
+        zc.optim.clip_grad_norm(model.parameters(), 1.0)
+        opt.step()
+    model.eval()
+    test = np.random.default_rng(1000 + seed)
+    acc = {}
+    with zc.no_grad():
+        for n in TEST_N:
+            x, y = make_batch(test, 512, n)
+            acc[n] = float((model(x).data[:, -1].argmax(-1) == y).mean())
+    return acc
+
+
+def main():
+    print(f"联想检索：{len(MODELS)} 个模型 × 种子 {SEEDS}，每次 {ITERS} 步", flush=True)
+    res = {}
+    for name, make in MODELS.items():
+        runs = [train_and_eval(make, s) for s in SEEDS]
+        res[name] = {n: np.mean([r[n] for r in runs]) for n in TEST_N}
+        print(f"  => {name}: " + "  ".join(f"N={n}:{res[name][n]:.0%}" for n in TEST_N), flush=True)
+    lines = ["# 长距离精确回忆（联想检索）\n",
+             "`k1 v1 k2 v2 … kN vN [问] kq` → 回答 kq 对应的值。16 个键、16 个值，随机猜的准确率是 1/16 ≈ 6%。",
+             f"三个模型宽 {CFG['n_embd']}、{CFG['n_layer']} 层、{CFG['n_head']} 头；训练 {ITERS} 步，N 在 2~{MAX_N} 随机；"
+             f"每格是 512 道测试题 × 种子 {SEEDS} 的平均准确率。\n",
+             "| 模型 | " + " | ".join(f"N = {n}" for n in TEST_N) + " |", "|---|" + "---|" * len(TEST_N)]
+    for name, acc in res.items():
+        lines.append(f"| {name} | " + " | ".join(f"{acc[n]:.0%}" for n in TEST_N) + " |")
+    save_result("recall", lines)
+
+
+if __name__ == "__main__":
+    main()

experiments/recall_diag.py

@@ -0,0 +1,118 @@
+"""精确回忆实验的诊断脚本（诊断性，不参与预注册判定）。
+
+回答的问题：为什么 recall.py 跑 3000 步和 20000 步结果几乎一样（Transformer N=16 只有 14~18%）？
+
+    python3 experiments/recall_diag.py
+
+四段，都对固定测试题评估、种子 0：
+  A 固定 32 条 N=2 题反复训练：能不能背下来（检验框架/优化）。
+  B 只训 N=1（复制值，不需要匹配）：能不能学会复制。
+  C 只训 N=2，比较学习率安排：恒定 3e-3 / 余弦 3e-3→3e-4 / 余弦 1e-3→1e-4，各 8000 步。
+  D 混合 N（2~16）用 1e-3 余弦跑 12000 步：看混合训练能不能学会。
+
+结论（11-08，见 recall.md 的诊断一节）：A、B 都能 100%；C 里只有 1e-3 余弦能学会 N=2
+（第 8000 步 100%，loss 0.001），恒定 3e-3 跑 20000 步也停在 53%（loss ≈ ln2）；D 12000 步仍 56%，
+因为混合数据里 N=2 只占约 1/15，信号被稀释。
+"""
+import sys
+import os
+import time
+
+import numpy as np
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+
+import zerocg as zc  # noqa: E402
+from zerocg import functional as F  # noqa: E402
+from models import TransformerLM  # noqa: E402
+
+NK, NV = 16, 16
+QUERY = NK + NV
+CFG = dict(n_embd=64, n_head=4, n_layer=2)
+MAX_N = 16
+TEST_N = [2, 4, 8, 12, 16]
+
+
+def batch(rng, B, n):
+    keys = np.stack([rng.permutation(NK)[:n] for _ in range(B)])
+    vals = rng.integers(NK, NK + NV, size=(B, n))
+    seq = np.empty((B, 2 * n + 2), dtype=np.int64)
+    seq[:, 0:2 * n:2], seq[:, 1:2 * n:2] = keys, vals
+    q = rng.integers(0, n, B)
+    seq[:, -2], seq[:, -1] = QUERY, keys[np.arange(B), q]
+    return seq, vals[np.arange(B), q]
+
+
+def model():
+    return TransformerLM(NK + NV + 1, 2 * MAX_N + 2, **CFG)
+
+
+def acc(model, tests):
+    with zc.no_grad():
+        return {n: round(float((model(tests[n][0]).data[:, -1].argmax(-1) == tests[n][1]).mean()), 3)
+                for n in tests}
+
+
+def tests():
+    rng = np.random.default_rng(7000)
+    return {n: batch(rng, 512, n) for n in TEST_N}
+
+
+def run(label, iters, lr0, lr1, warmup, n_fixed=None, seed=0, eval_every=2000):
+    np.random.seed(seed)
+    rng = np.random.default_rng(seed)
+    m = model()
+    opt = zc.optim.AdamW(m.parameters(), lr=lr0, weight_decay=0.01)
+    t = tests()
+    t0 = time.time()
+    for it in range(iters):
+        frac = it / max(1, iters - 1)
+        opt.lr = lr0 * min(1, (it + 1) / max(1, warmup)) * (lr1 / lr0) ** frac
+        n = n_fixed if n_fixed else int(rng.integers(2, MAX_N + 1))
+        x, y = batch(rng, 32, n)
+        loss = F.cross_entropy(m(x)[:, -1], y)
+        opt.zero_grad()
+        loss.backward()
+        zc.optim.clip_grad_norm(m.parameters(), 1.0)
+        opt.step()
+        if (it + 1) % eval_every == 0:
+            a = acc(m, t)
+            print(f"  {label} step {it+1:5d} loss {float(loss.data):.3f} "
+                  + "  ".join(f"N={k}:{a[k]}" for k in TEST_N) + f"  ({time.time()-t0:.0f}s)", flush=True)
+    return a
+
+
+def main():
+    print("A 固定 32 条 N=2 反复训练（记忆）", flush=True)
+    np.random.seed(0)
+    m = model()
+    rng = np.random.default_rng(0)
+    x, y = batch(rng, 32, 2)
+    opt = zc.optim.AdamW(m.parameters(), lr=1e-3, weight_decay=0.01)
+    for it in range(600):
+        opt.lr = 3e-3 * min(1, (it + 1) / 100)
+        loss = F.cross_entropy(m(x)[:, -1], y)
+        opt.zero_grad()
+        loss.backward()
+        opt.step()
+        if (it + 1) % 200 == 0:
+            with zc.no_grad():
+                a = float((m(x).data[:, -1].argmax(-1) == y).mean())
+            print(f"  mem step {it+1} loss {float(loss.data):.4f} acc {a}", flush=True)
+
+    print("B 只训 N=1（复制）", flush=True)
+    run("N1", 1500, 3e-3, 3e-3, 100, n_fixed=1, eval_every=500)
+
+    print("C1 只训 N=2，恒定 3e-3（recall.py 的原设置）", flush=True)
+    run("C1", 8000, 3e-3, 3e-3, 100, n_fixed=2)
+    print("C2 只训 N=2，余弦 3e-3→3e-4", flush=True)
+    run("C2", 8000, 3e-3, 3e-4, 200, n_fixed=2)
+    print("C3 只训 N=2，余弦 1e-3→1e-4", flush=True)
+    run("C3", 8000, 1e-3, 1e-4, 200, n_fixed=2)
+
+    print("D 混合 N，余弦 1e-3→1e-4，12000 步", flush=True)
+    run("D", 12000, 1e-3, 1e-4, 200, n_fixed=None)
+
+
+if __name__ == "__main__":
+    main()

experiments/recall_prereg.md

@@ -0,0 +1,13 @@
+# 预注册实验：精确回忆（联想检索），训练加长到 2 万步
+
+> 写于实验之前（2026-11-01）。跑完在 recall.md 里看结果，这里不改。
+
+**背景**：10-11 的 recall 实验只训 3000 步，Transformer 和 ZeroPoint 都接近瞎猜（N=16 时 14~18%，瞎猜 6%），无结论。
+
+**做法**：同一脚本（recall.py），RECALL_ITERS=20000，其余不变（3 个模型 × 2 个种子，宽 64 / 2 层 / 4 头；N 在 2~16 随机；测 N = 2, 4, 8, 12, 16）。
+
+**预测**
+- P1 Transformer 学会：N=16 准确率 ≥ 80%（来源：注意力做键值检索是已知能力，3000 步不够是训练量问题）。
+- P2 ZeroPoint 落后：N=16 时比 Transformer 低 ≥ 20 个百分点（来源：16 个槽、每头 16 维，要原样存 16 对键值超出容量；无依据的具体数字，仅作记录）。
+- P3 槽数有用：32 槽比 16 槽在 N=16 时高 ≥ 5 个百分点（无依据，仅作记录）。
+- 判定：2 个种子的均值；差别小于两个种子之间的差距时不算数。

experiments/results/20261011-1223-compare.md

@@ -0,0 +1,42 @@
+> **历史存档**（2026-10-11 生成，原名 ZeroMem）。本文件保留生成时原样，未按 2026-11-08 的复测更正；最新版本与更正见 [`../compare.md`](../compare.md) 和 [`../diagnose.md`](../diagnose.md)。
+
+# ZeroMem vs Transformer 对照实验
+
+- 语料：Python 标准库源码，1,558,395 字符，词表 96；90% 训练 / 10% 验证
+- 三个模型：宽 64、2 层、4 头、Pre-LN、GELU MLP 4 倍；ZeroMem 每头 16 个槽、可学习初始状态、分块并行（块长 32）、无位置编码
+- 训练：上下文 64、batch 32、2000 步、AdamW（权重衰减 0.01）、预热 100 步 + 余弦、梯度裁剪 1.0
+- 学习率：每个模型在 [0.001, 0.003, 0.006] 里各试一次（种子 0），取最好的再跑种子 [1, 2]
+- 验证 loss = 40 个固定验证 batch 的平均交叉熵（nats/字符，越低越好），均值 ± 标准差（3 个种子）
+- 计算量 = 前向时每个字的矩阵乘浮点运算次数（不受机器负载影响；训练一步约 3 倍）
+
+| 模型 | 参数量 | 最佳学习率 | 验证 loss | 前向计算量（每字） | 推理状态（第 64 字 / 第 4096 字，浮点数） |
+|---|---|---|---|---|---|
+| Transformer | 116,576 | 0.006 | 1.5452 ± 0.0100 | 242 K | 16,384 / 1,048,576（KV 缓存） |
+| Transformer (sin) | 112,480 | 0.006 | 1.5271 ± 0.0083 | 242 K | 16,384 / 1,048,576（KV 缓存） |
+| ZeroMem | 115,240 | 0.006 | 1.5214 ± 0.0134 | 232 K | 2,048 / 2,048（固定） |
+
+## 学习率搜索（种子 0 的验证 loss）
+
+| 模型 | lr=0.001 | lr=0.003 | lr=0.006 |
+|---|---|---|---|
+| Transformer | 1.6908 | 1.5530 | 1.5520 |
+| Transformer (sin) | 1.8006 | 1.5939 | 1.5374 |
+| ZeroMem | 1.6682 | 1.5433 | 1.5311 |
+
+## 长度外推（训练上下文 64；8 条验证序列 × 3 个种子平均）
+
+| 位置 | ZeroMem | Transformer (sin) | Transformer（可学习位置编码） |
+|---|---|---|---|
+| 0 ~ 63（训练见过的长度） | 1.5550 | 1.4512 | 能跑 |
+| 64 ~ 255 | 1.6164 | 4.6359 | 跑不了（只有 64 个位置） |
+| 256 ~ 1023 | 1.6410 | 4.6057 | 跑不了（只有 64 个位置） |
+| 1024 ~ 4095 | 1.7040 | —（注意力矩阵太大，没测） | 跑不了（只有 64 个位置） |
+
+## ZeroMem 槽的使用情况（有效槽数，满分 16）
+
+有效槽数 = 平均读/写地址分布的 exp(熵)：16 个槽用得一样多 = 16，全挤在一个槽 = 1。
+
+| 层 | 读 | 写 |
+|---|---|---|
+| 0 | 10.1 | 3.2 |
+| 1 | 9.9 | 5.0 |

experiments/results/20261011-1258-ablation.md

@@ -0,0 +1,22 @@
+> **历史存档**（2026-10-11 生成，原名 ZeroMem）。本文件保留生成时原样，未按 2026-11-08 的复测更正；最新版本见 [`../ablation.md`](../ablation.md)。
+
+# ZeroMem 消融实验（仿论文表 3）
+
+每行只改一处，其余和 base 相同。2000 步，lr 0.003，种子 [0, 1, 2]，其余训练设置同 [`compare.md`](../compare.md)。
+参数量和 base 差超过 1% 的变体，自动调整 MLP 宽度对齐参数量。
+Δ = 该行验证 loss 均值 − base 均值（**正数表示变差**）。
+
+> 判断标准：base 的种子间标准差是 0.0073。|Δ| 小于约 2 倍标准差（0.0145）的差别**不能算数**。
+
+| | 改动 | 参数量 | 验证 loss | Δ | 结论 |
+|---|---|---|---|---|---|
+| base | ZeroMem：4 头 × 16 槽，写强度门，写入竞争，分层初始化，λ 可学，可学初始状态，无位置编码 | 115,240 | 1.5500 ± 0.0073 | — | — |
+| (A) | 1 个头（每头 64 维）（MLP 宽度调为 306 以对齐参数） | 115,366 | 1.5776 ± 0.0036 | +0.0276 | 变差：这个部件有用 |
+| (A) | 8 个头（每头 8 维）（MLP 宽度调为 189 以对齐参数） | 115,210 | 1.5673 ± 0.0067 | +0.0173 | 变差：这个部件有用 |
+| (B) | 每头 4 个槽（MLP 宽度调为 311 以对齐参数） | 115,318 | 1.5466 ± 0.0087 | -0.0034 | 差别不显著 |
+| (B) | 每头 32 个槽（MLP 宽度调为 183 以对齐参数） | 115,222 | 1.5624 ± 0.0055 | +0.0124 | 差别不显著 |
+| (C) | 去掉写强度门 | 114,720 | 1.5522 ± 0.0072 | +0.0022 | 差别不显著 |
+| (D) | 写入不竞争：softmax 换成每槽独立的 sigmoid | 115,240 | 1.5487 ± 0.0066 | -0.0013 | 差别不显著 |
+| (E) | 不分层：所有槽初始半衰期相同（约 9.8 个字） | 115,240 | 1.6010 ± 0.0079 | +0.0510 | 变差：这个部件有用 |
+| (F) | λ 固定在初始值，不学习 | 115,112 | 1.5784 ± 0.0097 | +0.0284 | 变差：这个部件有用 |
+| (G) | 加上论文 §3.5 的正弦位置编码 | 115,240 | 1.6106 ± 0.0078 | +0.0606 | 变差：这个部件有用 |

experiments/results/20261011-1305-recall.md

@@ -0,0 +1,12 @@
+> **历史存档**（3,000 步版本）。结论见 [`../recall.md`](../recall.md)：学习率配置导致**无结论**。
+
+# 长距离精确回忆（联想检索）
+
+`k1 v1 k2 v2 … kN vN [问] kq` → 回答 kq 对应的值。16 个键、16 个值，随机猜的准确率是 1/16 ≈ 6%。
+三个模型宽 64、2 层、4 头；训练 3000 步，N 在 2~16 随机；每格是 512 道测试题 × 种子 [0, 1] 的平均准确率。
+
+| 模型 | N = 2 | N = 4 | N = 8 | N = 12 | N = 16 |
+|---|---|---|---|---|---|
+| Transformer | 52% | 32% | 23% | 18% | 14% |
+| ZeroPoint (16 槽) | 51% | 32% | 24% | 21% | 17% |
+| ZeroPoint (32 槽) | 51% | 31% | 25% | 20% | 18% |

experiments/results/20261018-0838-ema_longrun.md

@@ -0,0 +1,57 @@
+> **历史存档**。其中"compare.py 里 ZeroPoint 用的是递推算法、和 Transformer 不对等"的说法有误：分块并行与递推形式数学等价（float32 差 1.4e-5），10-18 的修正实际是把 8 条验证序列增加到 32 条。
+
+# 预注册实验：长序列变差与 ema 写入
+
+> 写于实验之前（2026-10-18）。跑完后只在下面追加结果，不改这一节。
+
+**观察**：compare.md 里 ZeroPoint（无 ema）在 1024~4095 字处的 loss（1.70）高于 64~255 字处（1.62）。
+
+**假设 H1**：原因是长期槽（λ 接近 1）的数值随序列长度无界累积，超出训练时（64 字）见过的范围。
+
+**预测**：
+- ① 直接测：推理时槽值的最大绝对值，无 ema 的模型在 4000 字处明显大于 64 字处（比值 > 2）；有 ema 的模型比值接近 1（< 1.5）。
+- ② 间接测：开 ema 重新训练（其余完全相同），1024~4095 字处的 loss 不再明显高于 64~255 字处：差值 < 2 × 种子间标准差。
+
+**H1 不成立时会看到**：① 无 ema 的槽值没有明显增长；或 ② 开 ema 后远处 loss 照样升高。
+
+**判定**：① 和 ② 都成立 → 接受 H1，且 ema 是有效的修法；只有 ② 成立 → ema 有效但原因另有其他；
+都不成立 → 放弃 H1，下一步查学到的 λ 值。另外要求 ema 不损害短距离：64 字处验证 loss 的差别 < 2 × 标准差。
+
+**对照**：两组用同样的 3 个种子、同样 2000 步、学习率 0.006、同样语料；外推用 32 条 4096 字的验证序列，
+两组都用分块并行算法算 loss（compare.py 里 ZeroPoint 用的是递推算法，和 Transformer 不对等，这里统一）。
+
+
+## 结果（实验之后写）
+
+### 64 字处的验证 loss（均值 ± 标准差，3 个种子）
+
+| 模型 | 验证 loss |
+|---|---|
+| no-ema | 1.5214 ± 0.0134 |
+| ema | 1.5182 ± 0.0076 |
+
+### 长度外推（32 条 × 4096 字，分块并行算法；均值 ± 标准差）
+
+| 位置 | no-ema | ema |
+|---|---|---|
+| 0 ~ 63 | 1.4372 ± 0.0187 | 1.4308 ± 0.0039 |
+| 64 ~ 255 | 1.4742 ± 0.0341 | 1.6476 ± 0.0529 |
+| 256 ~ 1023 | 1.6617 ± 0.0234 | 3.1084 ± 0.2157 |
+| 1024 ~ 4095 | 1.6932 ± 0.0305 | 3.4153 ± 0.2402 |
+
+### 槽值的最大绝对值（递推推理，3 个种子平均）
+
+| 位置 | no-ema | ema |
+|---|---|---|
+| 64 | 9.08 | 1.60 |
+| 256 | 10.07 | 2.34 |
+| 1024 | 9.88 | 1.51 |
+| 4000 | 16.07 | 2.27 |
+
+## 判定（按预注册的标准）
+
+- 预测 ①（槽值比值 4000 字 / 64 字）：无 ema 1.93，有 ema 1.44 → **不成立**
+- 预测 ②（ema 模型 1024~4095 与 64~255 的 loss 差）：+1.7677，噪声阈值 0.4805 → **不成立**（对照：无 ema 的差是 +0.2190）
+- ema 不损害短距离：64 字处 loss 差 -0.0033 → **成立**
+
+**结论**：① ② 都不成立：**放弃 H1**，下一步查学到的 λ 值

experiments/results/20261018-0901-diagnose.md

@@ -0,0 +1,67 @@
+> **历史存档**。其中"有效槽数 3~5"是未归一化门控分布的偏低值；归一化后读约 10/16、写约 6~10/层（见 [`../diagnose.md`](../diagnose.md)）。
+
+# 预注册实验：查三个问题
+
+> 写于实验之前（2026-10-18）。跑完只在下面追加结果，不改这一节。
+
+**背景**：ema_longrun.md 否定了"槽值无界累积"假设。无 ema 的模型在 1024~4095 字处 loss 比 64~255 字处高约 0.22；
+有 ema 的高约 1.77。compare.md 里 16 个槽只有 3~5 个被写入。
+
+## 假设和预测
+
+**H2（训练长度）**：长序列变差是因为训练只见过 64 字；模型在更长的位置上遇到的槽状态分布是训练时没见过的。
+- 预测：训练长度改为 256（每步字数不变、其余相同）后，1024~4095 与 64~255 的 loss 差 **缩小到 T64 的一半以下**。
+- 不成立时会看到：差距没有缩小（< 2 × 标准差的变化）。
+
+**H3（ema 为什么更差）**：ema 把长期槽的写入乘上 (1-λ)，训练时（64 字）长期槽几乎不起作用，模型学会只用短期槽；
+到了长序列，读地址仍指向短期槽，但短期槽里的内容和训练时的统计不同。
+- 预测 ③a：ema 模型的**写地址**集中在半衰期短的槽上：半衰期 > 16 字的槽，写入占比 < T64 模型的一半。
+- 预测 ③b：ema 模型学到的长期槽 λ 更接近 1（半衰期更长），或者长期槽的读占比在远处位置上升。
+- 不成立时会看到：两组的写入分布没有明显差别。
+
+**H4（槽塌缩）**：塌缩不是训练失败，而是写地址的 softmax 竞争让少数槽赢者通吃；在不同序列位置上"赢的槽"不同，
+所以整体统计（compare.md 的 3~5 个有效槽）低估了实际用到的槽数。
+- 预测 ④：按位置分段（0~63 / 64~255 / 256~1023 / 1024~4095）分别统计有效写槽数，各段的并集 > 整体有效槽数 × 1.5；
+  同时各段的读/写分布之间有明显差别（任意两段的 JS 散度 > 0.1）。
+- 不成立时会看到：各段的分布几乎相同，用的始终是同几个槽。
+
+**判定**：每条按上面的数值阈值判，3 个种子均值；差别和 2 × 种子间标准差比。
+
+
+## 结果（实验之后写）
+
+### 验证 loss（64 字）和长度外推（32 条 × 4096 字；均值 ± 标准差，3 个种子）
+
+| 条件 | 64 字验证 | 0~63 | 64~255 | 256~1023 | 1024~4095 | 远−近 差 |
+|---|---|---|---|---|---|---|
+| T64 | 1.5214 ± 0.0134 | 1.4372 ± 0.0187 | 1.4742 ± 0.0341 | 1.6617 ± 0.0234 | 1.6932 ± 0.0305 | +0.2190 ± 0.0630 |
+| T64-ema | 1.5182 ± 0.0076 | 1.4308 ± 0.0039 | 1.6476 ± 0.0529 | 3.1084 ± 0.2157 | 3.4153 ± 0.2402 | +1.7677 ± 0.2188 |
+| T256 | 1.5572 ± 0.0060 | 1.4741 ± 0.0117 | 1.4486 ± 0.0098 | 1.5098 ± 0.0022 | 1.5155 ± 0.0048 | +0.0669 ± 0.0051 |
+
+### H3：写入落在半衰期 > 16 字的槽上的占比（0~63 段），和学到的最长半衰期
+
+| 条件 | 长期槽写入占比 | 最长半衰期（字） | 远处读长期槽占比的变化 |
+|---|---|---|---|
+| T64 | 0.061 ± 0.005 | 109 | +0.009 |
+| T64-ema | 0.014 ± 0.001 | 103 | +0.007 |
+
+### H4：槽的使用（T64，写地址；各层各头平均）
+
+| 段 | 有效写槽数 |
+|---|---|
+| 0~63 | 8.6 |
+| 64~255 | 8.7 |
+| 256~1023 | 8.5 |
+| 1024~4095 | 8.9 |
+| 整体 | 8.7 |
+
+各段用到的槽的并集 ÷ 整体有效槽数 = 0.68；各段分布之间最大的 JS 散度 = 0.010
+
+## 判定（按预注册的标准）
+
+- H2 训练长度：远−近差 T64 +0.2190 → T256 +0.0669 → **成立**
+- H3a ema 写入避开长期槽：占比 0.061 → 0.014 → **成立**
+- H3b ema 的长期槽更长或远处更依赖长期槽 → **不成立**
+- H4 槽塌缩是分段使用不同的槽：并集比 0.68，JS 0.010 → **不成立**
+
+模型已保存在 experiments/out/diag_*.npz，可以事后再分析。

experiments/results/20261101-0814-slots.md

@@ -0,0 +1,37 @@
+> **历史存档**（与 [`../slots.md`](../slots.md) 内容相同）。
+
+# 预注册分析：长期槽在干什么
+
+> 写于分析之前（2026-11-01）。跑完只追加结果，不改这一节。用 experiments/out/diag_*.npz（T64 / T64-ema / T256 各 3 种子）。
+
+**矛盾**：T64 模型写入只有 6.1% 落在半衰期 > 16 字的槽；但消融里去掉分层初始化变差 0.051。
+
+**三个可区分的假设**
+- H9 读取端在用：长期槽虽然写得少，但**读地址**落在长期槽上的占比明显高于写占比（读占比 ≥ 2 × 写占比）。
+  → 长期槽像"慢变的背景"，少量写入、持续读取。
+- H10 中期槽才是关键：分层初始化的作用主要来自半衰期 4~64 字的槽，不是 > 64 的；读写都集中在 4~64 桶（该桶读 + 写占比合计 > 50%）。
+- H11 训练长度决定长期槽的使用：T256 模型在 > 16 字桶上的写占比 ≥ 2 × T64 的。
+  来源：diagnose 里 T256 远处 loss 明显更好，推测它学会了用更长的槽。
+
+**判定**：3 个种子平均；各桶占比按所有层、所有头的平均。H9 / H11 的"2 倍"是无依据的整数阈值，仅作记录；
+同时报告原始占比，让读者自己看。
+
+
+## 结果（分析之后写）
+
+验证集 4 条 × 2048 字递推；占比 = 该桶内所有槽的地址权重之和（读 / 写各自归一化），层和头平均，3 个种子平均。
+
+| 条件 | 量 | 半衰期 0~4 | 半衰期 4~16 | 半衰期 16~64 | 半衰期 64~∞ |
+|---|---|---|---|---|---|
+| T64 | 读 | 0.703 | 0.120 | 0.127 | 0.050 |
+| T64 | 写 | 0.815 | 0.122 | 0.058 | 0.006 |
+| T64-ema | 读 | 0.524 | 0.149 | 0.167 | 0.159 |
+| T64-ema | 写 | 0.925 | 0.060 | 0.014 | 0.001 |
+| T256 | 读 | 0.660 | 0.180 | 0.152 | 0.008 |
+| T256 | 写 | 0.696 | 0.167 | 0.135 | 0.001 |
+
+## 判定（按预注册的标准）
+
+- H9 读取端在用长期槽：> 16 字桶 读 0.177 vs 写 0.064 → **成立**（2 倍阈值无依据，仅作记录）
+- H10 中期槽是关键：4~64 字桶 读写平均占比 0.213 > 0.5 → **不成立**
+- H11 训练长度 256 更用长期槽：> 16 字桶写占比 T256 0.136 vs T64 0.064 → **成立**（2 倍阈值无依据，仅作记录）

experiments/results/20261101-0936-recall.md

@@ -0,0 +1,12 @@
+> **历史存档**（20,000 步版本）。结论见 [`../recall.md`](../recall.md)：学习率配置导致**无结论**。
+
+# 长距离精确回忆（联想检索）
+
+`k1 v1 k2 v2 … kN vN [问] kq` → 回答 kq 对应的值。16 个键、16 个值，随机猜的准确率是 1/16 ≈ 6%。
+三个模型宽 64、2 层、4 头；训练 20000 步，N 在 2~16 随机；每格是 512 道测试题 × 种子 [0, 1] 的平均准确率。
+
+| 模型 | N = 2 | N = 4 | N = 8 | N = 12 | N = 16 |
+|---|---|---|---|---|---|
+| Transformer | 50% | 30% | 23% | 20% | 18% |
+| ZeroPoint (16 槽) | 51% | 32% | 23% | 20% | 16% |
+| ZeroPoint (32 槽) | 49% | 31% | 24% | 20% | 17% |

experiments/slots.md

@@ -0,0 +1,35 @@
+# 预注册分析：长期槽在干什么
+
+> 写于分析之前（2026-11-01）。跑完只追加结果，不改这一节。用 experiments/out/diag_*.npz（T64 / T64-ema / T256 各 3 种子）。
+
+**矛盾**：T64 模型写入只有 6.1% 落在半衰期 > 16 字的槽；但消融里去掉分层初始化变差 0.051。
+
+**三个可区分的假设**
+- H9 读取端在用：长期槽虽然写得少，但**读地址**落在长期槽上的占比明显高于写占比（读占比 ≥ 2 × 写占比）。
+  → 长期槽像"慢变的背景"，少量写入、持续读取。
+- H10 中期槽才是关键：分层初始化的作用主要来自半衰期 4~64 字的槽，不是 > 64 的；读写都集中在 4~64 桶（该桶读 + 写占比合计 > 50%）。
+- H11 训练长度决定长期槽的使用：T256 模型在 > 16 字桶上的写占比 ≥ 2 × T64 的。
+  来源：diagnose 里 T256 远处 loss 明显更好，推测它学会了用更长的槽。
+
+**判定**：3 个种子平均；各桶占比按所有层、所有头的平均。H9 / H11 的"2 倍"是无依据的整数阈值，仅作记录；
+同时报告原始占比，让读者自己看。
+
+
+## 结果（分析之后写）
+
+验证集 4 条 × 2048 字递推；占比 = 该桶内所有槽的地址权重之和（读 / 写各自归一化），层和头平均，3 个种子平均。
+
+| 条件 | 量 | 半衰期 0~4 | 半衰期 4~16 | 半衰期 16~64 | 半衰期 64~∞ |
+|---|---|---|---|---|---|
+| T64 | 读 | 0.703 | 0.120 | 0.127 | 0.050 |
+| T64 | 写 | 0.815 | 0.122 | 0.058 | 0.006 |
+| T64-ema | 读 | 0.524 | 0.149 | 0.167 | 0.159 |
+| T64-ema | 写 | 0.925 | 0.060 | 0.014 | 0.001 |
+| T256 | 读 | 0.660 | 0.180 | 0.152 | 0.008 |
+| T256 | 写 | 0.696 | 0.167 | 0.135 | 0.001 |
+
+## 判定（按预注册的标准）
+
+- H9 读取端在用长期槽：> 16 字桶 读 0.177 vs 写 0.064 → **成立**（2 倍阈值无依据，仅作记录）
+- H10 中期槽是关键：4~64 字桶 读写平均占比 0.213 > 0.5 → **不成立**
+- H11 训练长度 256 更用长期槽：> 16 字桶写占比 T256 0.136 vs T64 0.064 → **成立**（2 倍阈值无依据，仅作记录）

experiments/slots.py

@@ -0,0 +1,88 @@
+"""预注册分析：长期槽到底在干什么？（用 diagnose 保存的 9 个模型，不重训）
+
+    python3 experiments/slots.py
+
+背景：diagnose 发现无 ema 的模型只有 6% 的写入落在半衰期 > 16 字的槽上，但消融说明"分层初始化"有用（去掉变差 0.051）。
+结果：experiments/slots.md + results/ 存档。
+"""
+import os
+
+import numpy as np
+
+from common import HERE, batches, prepare_data, save_result
+from models import ZeroX
+
+OUT = os.path.join(HERE, "out")
+SEEDS = [0, 1, 2]
+T_RUN = 2048
+BUCKETS = [(0, 4), (4, 16), (16, 64), (64, 1e9)]   # 半衰期分桶（字）
+
+PREREG = """# 预注册分析：长期槽在干什么
+
+> 写于分析之前（2026-11-01）。跑完只追加结果，不改这一节。用 experiments/out/diag_*.npz（T64 / T64-ema / T256 各 3 种子）。
+
+**矛盾**：T64 模型写入只有 6.1% 落在半衰期 > 16 字的槽；但消融里去掉分层初始化变差 0.051。
+
+**三个可区分的假设**
+- H9 读取端在用：长期槽虽然写得少，但**读地址**落在长期槽上的占比明显高于写占比（读占比 ≥ 2 × 写占比）。
+  → 长期槽像"慢变的背景"，少量写入、持续读取。
+- H10 中期槽才是关键：分层初始化的作用主要来自半衰期 4~64 字的槽，不是 > 64 的；读写都集中在 4~64 桶（该桶读 + 写占比合计 > 50%）。
+- H11 训练长度决定长期槽的使用：T256 模型在 > 16 字桶上的写占比 ≥ 2 × T64 的。
+  来源：diagnose 里 T256 远处 loss 明显更好，推测它学会了用更长的槽。
+
+**判定**：3 个种子平均；各桶占比按所有层、所有头的平均。H9 / H11 的"2 倍"是无依据的整数阈值，仅作记录；
+同时报告原始占比，让读者自己看。
+"""
+
+
+def main():
+    text, V, train_data, val_data = prepare_data()
+    x, _ = next(batches(val_data, 1, T_RUN, 7, batch=4))
+    cfg = dict(n_embd=64, n_head=4, n_layer=2, n_slot=16)
+    rows = {}
+    for cond in ("T64", "T64-ema", "T256"):
+        reads, writes = np.zeros(len(BUCKETS)), np.zeros(len(BUCKETS))
+        for seed in SEEDS:
+            model = ZeroX(V, ema=(cond == "T64-ema"), **cfg)
+            model.load(os.path.join(OUT, f"diag_{cond}_s{seed}.npz")).eval()
+            states = model.init_state(4)
+            r_acc = [np.zeros((b.mix.H, b.mix.M)) for b in model.blocks]
+            w_acc = [np.zeros((b.mix.H, b.mix.M)) for b in model.blocks]
+            for t in range(T_RUN):
+                _, states = model.step(x[:, t], states, t)
+                for l, b in enumerate(model.blocks):
+                    a, w = b.mix.last_addr
+                    r_acc[l] += a.mean(0)
+                    w_acc[l] += w.mean(0)
+            for l, b in enumerate(model.blocks):
+                hl = np.log(0.5) / np.log(b.mix.retention().data)        # (H, M)
+                r = r_acc[l] / r_acc[l].sum(-1, keepdims=True)
+                w = w_acc[l] / w_acc[l].sum(-1, keepdims=True)
+                for k, (lo, hi) in enumerate(BUCKETS):
+                    m = (hl >= lo) & (hl < hi)
+                    reads[k] += (r * m).sum() / (r.shape[0]) / len(model.blocks)
+                    writes[k] += (w * m).sum() / (w.shape[0]) / len(model.blocks)
+        rows[cond] = (reads / len(SEEDS), writes / len(SEEDS))
+
+    lines = [PREREG, "\n## 结果（分析之后写）\n",
+             f"验证集 4 条 × {T_RUN} 字递推；占比 = 该桶内所有槽的地址权重之和（读 / 写各自归一化），层和头平均，3 个种子平均。\n",
+             "| 条件 | 量 | " + " | ".join(f"半衰期 {lo}~{'∞' if hi > 1e8 else int(hi)}" for lo, hi in BUCKETS) + " |",
+             "|---|---|" + "---|" * len(BUCKETS)]
+    for cond, (r, w) in rows.items():
+        lines.append(f"| {cond} | 读 | " + " | ".join(f"{v:.3f}" for v in r) + " |")
+        lines.append(f"| {cond} | 写 | " + " | ".join(f"{v:.3f}" for v in w) + " |")
+    r64, w64 = rows["T64"]
+    r256, w256 = rows["T256"]
+    long_r, long_w = r64[2] + r64[3], w64[2] + w64[3]
+    h9 = long_r >= 2 * long_w
+    h10 = (r64[1] + r64[2] + w64[1] + w64[2]) / 2 > 0.5
+    h11 = (w256[2] + w256[3]) >= 2 * long_w
+    lines += ["", "## 判定（按预注册的标准）\n",
+              f"- H9 读取端在用长期槽：> 16 字桶 读 {long_r:.3f} vs 写 {long_w:.3f} → **{'成立' if h9 else '不成立'}**（2 倍阈值无依据，仅作记录）",
+              f"- H10 中期槽是关键：4~64 字桶 读写平均占比 {(r64[1] + r64[2] + w64[1] + w64[2]) / 2:.3f} > 0.5 → **{'成立' if h10 else '不成立'}**",
+              f"- H11 训练长度 256 更用长期槽：> 16 字桶写占比 T256 {w256[2] + w256[3]:.3f} vs T64 {long_w:.3f} → **{'成立' if h11 else '不成立'}**（2 倍阈值无依据，仅作记录）"]
+    save_result("slots", lines)
+
+
+if __name__ == "__main__":
+    main()

models/__init__.py

@@ -0,0 +1,4 @@
+from .transformer import TransformerLM
+from .zeropoint import MemoryMixer, ZeroX
+
+__all__ = ["ZeroX", "MemoryMixer", "TransformerLM"]

models/transformer.py

@@ -0,0 +1,63 @@
+"""对照组：标准的 GPT 式 Transformer（因果自注意力），结构上和 ZeroX 一一对应。
+
+位置编码两种可选（《Attention Is All You Need》§3.5 两种都试过，结果几乎一样）：
+    pos="learned"  可学习的位置向量，只有 block_size 个，处理不了更长的序列
+    pos="sin"      论文用的正弦编码，任意长度都能算（论文选它就是因为"可能可以外推"）
+"""
+import numpy as np
+
+from zerocg import Tensor
+from zerocg import functional as F
+from zerocg import nn
+
+
+def sinusoidal_encoding(T, d):
+    """PE(pos, 2i) = sin(pos / 10000^(2i/d))，PE(pos, 2i+1) = cos(...)。论文 §3.5。"""
+    pos = np.arange(T)[:, None]
+    div = 10000 ** (np.arange(0, d, 2) / d)
+    pe = np.zeros((T, d), dtype=np.float32)
+    pe[:, 0::2] = np.sin(pos / div)
+    pe[:, 1::2] = np.cos(pos / div)
+    return pe
+
+
+class Block(nn.Module):
+    def __init__(self, n_embd, n_head, dropout=0.0):
+        self.ln1 = nn.LayerNorm(n_embd)
+        self.attn = nn.CausalSelfAttention(n_embd, n_head, dropout)
+        self.ln2 = nn.LayerNorm(n_embd)
+        self.fc1 = nn.Linear(n_embd, 4 * n_embd)
+        self.fc2 = nn.Linear(4 * n_embd, n_embd)
+        self.drop = nn.Dropout(dropout)
+
+    def forward(self, x):
+        x = x + self.attn(self.ln1(x))
+        return x + self.drop(self.fc2(F.gelu(self.fc1(self.ln2(x)))))
+
+
+class TransformerLM(nn.Module):
+    def __init__(self, vocab_size, block_size, n_embd=64, n_head=4, n_layer=2, dropout=0.0, pos="learned"):
+        assert pos in ("learned", "sin")
+        self.block_size = block_size
+        self.pos_kind = pos
+        self.tok = nn.Embedding(vocab_size, n_embd)
+        if pos == "learned":
+            self.pos = nn.Embedding(block_size, n_embd)
+        self.blocks = [Block(n_embd, n_head, dropout) for _ in range(n_layer)]
+        self.ln_f = nn.LayerNorm(n_embd)
+        self.head = nn.Linear(n_embd, vocab_size)
+
+    def forward(self, idx):
+        T = idx.shape[1]
+        if self.pos_kind == "learned":
+            assert T <= self.block_size, f"位置编码只有 {self.block_size} 个，处理不了 {T} 个字"
+            x = self.tok(idx) + self.pos(np.arange(T))
+        else:
+            x = self.tok(idx) + Tensor(sinusoidal_encoding(T, self.tok.weight.shape[1]))
+        for b in self.blocks:
+            x = b(x)
+        return self.head(self.ln_f(x))
+
+    def state_floats(self, T):
+        """推理到第 T 个字时 KV 缓存的大小（浮点数个数），随 T 线性增长。"""
+        return sum(2 * T * b.ln1.weight.shape[0] for b in self.blocks)

models/zeropoint.py

@@ -0,0 +1,217 @@
+"""ZeroPoint：分层记忆槽网络。
+
+设计文档见 docs/ZeroPoint.md。核心是 MemoryMixer（代替 Transformer 里的自注意力）：
+
+    每层有 H 个头，每个头有 M 个记忆槽 S[m]（D 维向量），每个槽有自己的保留率 λ[m]。
+    读到第 t 个字时：
+        w_t = softmax(x_t Ww) * sigmoid(x_t Wg)    写地址（槽之间竞争）× 写强度
+        S[m] ← λ[m] · S[m] + w_t[m] · v_t           所有槽先按自己的速度遗忘，再写入
+        a_t = softmax(x_t Wr)                       读地址
+        r_t = Σ_m a_t[m] · S[m]                     从槽里读出
+
+两种等价算法：
+    forward()  分块并行，训练用。序列切成每块 CHUNK 个字：
+                 块内  r = (A ⊙ 因果掩码) @ v，A[i,j] = Σ_m a_i[m] w_j[m] λ[m]^(i-j)
+                       利用 λ^(i-j) = λ^i · λ^(-j)，一次矩阵乘
+                 跨块  r += (a ⊙ λ^(i+1)) @ S_in        S_in = 进入这一块时的槽状态
+                 块尾  S_out = λ^c ⊙ S_in + (w ⊙ λ^(c-1-j))ᵀ @ v
+               λ^(-j) 最大 2^(CHUNK-1)，不会溢出，所以序列多长都行
+    step()     递推形式，推理用：每个字 O(H·M·D)，状态大小固定，没有 KV 缓存
+
+槽的初始状态 S0 是可学习参数（不是全 0），这样序列开头读出来的向量不会接近 0、被 LayerNorm 放大噪声。
+
+ema=True（指数滑动平均写入）：S[m] ← λ[m]·S[m] + (1-λ[m])·w_t[m]·v_t
+    槽的值永远落在"写进去过的值"的范围里，不会随序列变长而越积越大。
+    系统直觉模型要连续跑几天，用这个；实现上只是把写地址 w 乘上 (1-λ)，并行和递推两种形式都不用改。
+"""
+import numpy as np
+
+from zerocg import Tensor, concat
+from zerocg import functional as F
+from zerocg import nn
+
+from .transformer import sinusoidal_encoding
+
+LAMBDA_MIN = 0.5   # λ = 0.5 + 0.5·sigmoid(θ) ∈ (0.5, 1)
+CHUNK = 32         # 分块并行的块长：块内 λ^(-j) 最大 2^31，float32 绰绰有余
+
+
+def _layernorm_np(x, ln):
+    mu = x.mean(-1, keepdims=True)
+    var = ((x - mu) ** 2).mean(-1, keepdims=True)
+    return (x - mu) / np.sqrt(var + ln.eps) * ln.weight.data + ln.bias.data
+
+
+def _softmax_np(x):
+    e = np.exp(x - x.max(-1, keepdims=True))
+    return e / e.sum(-1, keepdims=True)
+
+
+def _gelu_np(a):
+    return 0.5 * a * (1 + np.tanh(np.sqrt(2 / np.pi) * (a + 0.044715 * a ** 3)))
+
+
+class MemoryMixer(nn.Module):
+    """参数里的开关只给消融实验用（experiments/ablation.py），默认值就是 ZeroPoint 本身：
+        gate          写强度门
+        write         "softmax"：写入时槽之间竞争；"sigmoid"：每个槽独立决定写不写
+        init          "hier"：半衰期按存储层级铺开；"uniform"：所有槽一样
+        learn_lambda  保留率 λ 是否可学
+        ema           写入按 (1-λ) 缩放（指数滑动平均），长时间运行不漂移
+    """
+
+    def __init__(self, n_embd, n_head, n_slot, max_half_life=64.0,
+                 gate=True, write="softmax", init="hier", learn_lambda=True, ema=False):
+        assert n_embd % n_head == 0 and write in ("softmax", "sigmoid") and init in ("hier", "uniform")
+        self.H, self.M, self.D = n_head, n_slot, n_embd // n_head
+        self.write_mode = write
+        self.ema = ema
+        self.read = nn.Linear(n_embd, n_head * n_slot)
+        self.write = nn.Linear(n_embd, n_head * n_slot)
+        self.gate = nn.Linear(n_embd, n_head) if gate else None
+        self.value = nn.Linear(n_embd, n_embd)
+        self.out = nn.Linear(n_embd, n_embd)
+        self.norm = nn.LayerNorm(self.D)   # 每个头读出来的向量各自归一化（不同槽的量级差别很大）
+        # 初始化成"存储层级"：槽 0 的半衰期约 1.5 个字（像寄存器），最后一个约 max_half_life 个字（像内存）
+        if init == "hier":
+            half_life = np.geomspace(1.5, max_half_life, n_slot)
+        else:   # 对照：所有槽都取层级的几何中点
+            half_life = np.full(n_slot, np.sqrt(1.5 * max_half_life))
+        lam = 0.5 ** (1 / half_life)
+        theta = np.tile(np.log((2 * lam - 1) / (2 - 2 * lam)), (n_head, 1))   # sigmoid 的反函数，(H, M)
+        self.theta = nn.Parameter(theta) if learn_lambda else Tensor(theta)
+        self.s0 = nn.Parameter(np.random.randn(n_head, n_slot, self.D) * 0.1)   # 可学习的初始状态
+        self.usage = None   # 最近一次 forward 的 (读地址, 写地址) 平均值，各 (H, M)，用来查槽有没有塌缩
+
+    def retention(self):
+        """每个槽的保留率 λ，形状 (H, M)。"""
+        return F.sigmoid(self.theta) * (1 - LAMBDA_MIN) + LAMBDA_MIN
+
+    def forward(self, x):
+        B, T, C = x.shape
+        H, M, D = self.H, self.M, self.D
+        a = F.softmax(self.read(x).reshape(B, T, H, M), axis=-1).transpose(0, 2, 1, 3)   # (B,H,T,M)
+        w = self.write(x).reshape(B, T, H, M)
+        w = F.softmax(w, axis=-1) if self.write_mode == "softmax" else F.sigmoid(w)
+        if self.gate is not None:
+            w = w * F.sigmoid(self.gate(x)).reshape(B, T, H, 1)
+        w = w.transpose(0, 2, 1, 3)                                                      # (B,H,T,M)
+        v = self.value(x).reshape(B, T, H, D).transpose(0, 2, 1, 3)                       # (B,H,T,D)
+        self.usage = (a.data.mean(axis=(0, 2)), w.data.mean(axis=(0, 2)))
+
+        lam = self.retention()                    # (H, M)
+        if self.ema:
+            w = w * (1 - lam).reshape(1, H, 1, M)
+        log_lam = lam.log().reshape(H, 1, M)
+        S = self.s0.reshape(1, H, M, D)           # 进入第一块时的状态（对 batch 广播）
+        outs = []
+        for s in range(0, T, CHUNK):
+            e = min(s + CHUNK, T)
+            c = e - s
+            ac, wc, vc = a[:, :, s:e], w[:, :, s:e], v[:, :, s:e]
+            i = Tensor(np.arange(c, dtype=np.float64).reshape(1, c, 1))
+            decay = (log_lam * i).exp()                                                   # λ^i, (H,c,M)
+            A = (ac * decay) @ (wc * (log_lam * -i).exp()).transpose(0, 1, 3, 2)          # (B,H,c,c)
+            A = A.masked_fill(np.triu(np.ones((c, c), dtype=bool), 1), 0.0)
+            outs.append(A @ vc + (ac * decay * lam.reshape(H, 1, M)) @ S)                 # 块内 + 跨块
+            if e < T:
+                tail = (log_lam * Tensor(np.arange(c - 1, -1, -1, dtype=np.float64).reshape(1, c, 1))).exp()
+                S = (lam ** c).reshape(1, H, M, 1) * S + (wc * tail).transpose(0, 1, 3, 2) @ vc
+        r = outs[0] if len(outs) == 1 else concat(outs, axis=2)
+        return self.out(self.norm(r).transpose(0, 2, 1, 3).reshape(B, T, C))
+
+    # ---------- 递推形式（numpy，无梯度）----------
+    def init_state(self, B):
+        return np.broadcast_to(self.s0.data, (B, self.H, self.M, self.D)).copy()
+
+    def step(self, x, S):
+        """x: (B, C) 一个字的输入；S: (B, H, M, D) 记忆槽。返回 (输出, 新状态)。"""
+        B = x.shape[0]
+        H, M, D = self.H, self.M, self.D
+        lin = lambda layer, z: z @ layer.weight.data + layer.bias.data  # noqa: E731
+        a = _softmax_np(lin(self.read, x).reshape(B, H, M))
+        w = lin(self.write, x).reshape(B, H, M)
+        w = _softmax_np(w) if self.write_mode == "softmax" else F.sigmoid_np(w)
+        if self.gate is not None:
+            w = w * F.sigmoid_np(lin(self.gate, x))[:, :, None]
+        v = lin(self.value, x).reshape(B, H, D)
+        lam = self.retention().data
+        if self.ema:
+            w = w * (1 - lam)[None]
+        self.last_addr = (a, w)   # 诊断用：这一步的读地址和写地址，(B, H, M)
+        S = lam[None, :, :, None] * S + w[..., None] * v[:, :, None, :]
+        r = np.einsum("bhm,bhmd->bhd", a, S)
+        r = _layernorm_np(r, self.norm).reshape(B, H * D)
+        return lin(self.out, r), S
+
+
+class Block(nn.Module):
+    def __init__(self, n_embd, n_head, n_slot, dropout=0.0, mlp_hidden=None, **mixer_opts):
+        hidden = mlp_hidden or 4 * n_embd
+        self.ln1 = nn.LayerNorm(n_embd)
+        self.mix = MemoryMixer(n_embd, n_head, n_slot, **mixer_opts)
+        self.ln2 = nn.LayerNorm(n_embd)
+        self.fc1 = nn.Linear(n_embd, hidden)
+        self.fc2 = nn.Linear(hidden, n_embd)
+        self.drop = nn.Dropout(dropout)
+
+    def forward(self, x):
+        x = x + self.drop(self.mix(self.ln1(x)))
+        return x + self.drop(self.fc2(F.gelu(self.fc1(self.ln2(x)))))
+
+    def step(self, x, S):
+        h, S = self.mix.step(_layernorm_np(x, self.ln1), S)
+        x = x + h
+        h = _gelu_np(_layernorm_np(x, self.ln2) @ self.fc1.weight.data + self.fc1.bias.data)
+        return x + h @ self.fc2.weight.data + self.fc2.bias.data, S
+
+
+class ZeroX(nn.Module):
+    """字符级语言模型。默认没有位置编码：位置信息来自各个槽不同的遗忘速度。
+    pos="sin" 只给消融实验用：额外加上论文的正弦位置编码，看看还有没有用。
+    mlp_hidden 默认 4 × n_embd；消融实验用它把各变体的参数量对齐。"""
+
+    def __init__(self, vocab_size, n_embd=64, n_head=4, n_slot=16, n_layer=2, dropout=0.0,
+                 pos="none", mlp_hidden=None, **mixer_opts):
+        assert pos in ("none", "sin")
+        self.pos_kind = pos
+        self.tok = nn.Embedding(vocab_size, n_embd)
+        self.blocks = [Block(n_embd, n_head, n_slot, dropout, mlp_hidden, **mixer_opts) for _ in range(n_layer)]
+        self.ln_f = nn.LayerNorm(n_embd)
+        self.head = nn.Linear(n_embd, vocab_size)
+
+    def forward(self, idx):
+        x = self.tok(idx)
+        if self.pos_kind == "sin":
+            x = x + Tensor(sinusoidal_encoding(idx.shape[1], x.shape[-1]))
+        for b in self.blocks:
+            x = b(x)
+        return self.head(self.ln_f(x))
+
+    def init_state(self, B):
+        return [b.mix.init_state(B) for b in self.blocks]
+
+    def step(self, idx_t, states, t=0):
+        """idx_t: (B,) 第 t 个字。返回 (logits (B, V), 新状态)。"""
+        x = self.tok.weight.data[idx_t]
+        if self.pos_kind == "sin":
+            x = x + sinusoidal_encoding(t + 1, x.shape[-1])[t]
+        new = []
+        for b, S in zip(self.blocks, states):
+            x, S = b.step(x, S)
+            new.append(S)
+        return _layernorm_np(x, self.ln_f) @ self.head.weight.data + self.head.bias.data, new
+
+    def forward_recurrent(self, idx):
+        """逐字递推跑完整个序列，返回 (B, T, V) 的 logits。"""
+        idx = np.asarray(idx)
+        states = self.init_state(idx.shape[0])
+        out = []
+        for t in range(idx.shape[1]):
+            logits, states = self.step(idx[:, t], states, t)
+            out.append(logits)
+        return np.stack(out, axis=1)
+
+    def state_floats(self):
+        """推理时需要保存的状态大小（浮点数个数），和序列长度无关。"""
+        return sum(b.mix.H * b.mix.M * b.mix.D for b in self.blocks)

requirements.txt

@@ -0,0 +1 @@
+numpy

scripts/upload.md

@@ -0,0 +1,92 @@
+# Publishing ZeroPoint to the Hugging Face Hub
+
+This note records the exact steps used to publish this release tree as a **public** model
+repository. It is for the maintainer performing the upload; users of the release do not
+need it. Never commit a real access token, and never print one in a terminal that records
+its output.
+
+The commands below were checked against `huggingface_hub` 1.27.0 (`hf` CLI).
+
+## 0. Environment: the local proxy is mandatory here
+
+In this environment `huggingface.co` is only reachable through the local HTTP proxy at
+`http://127.0.0.1:7897`. Set both proxy variables in every shell that talks to the Hub,
+and pin the official endpoint:
+
+```bash
+export HTTPS_PROXY=http://127.0.0.1:7897
+export HTTP_PROXY=http://127.0.0.1:7897
+export HF_ENDPOINT=https://huggingface.co
+```
+
+If the proxy is not running on port 7897, the commands below hang and then fail with a
+connection or SSL error — start the proxy first. Do not leave a mirror or an offline
+`HF_ENDPOINT` in the environment.
+
+## 1. Authenticate (token required)
+
+Create a **write** token at <https://huggingface.co/settings/tokens>. Then either log in
+interactively (the token is not echoed):
+
+```bash
+hf auth login
+```
+
+or pass the token from an environment variable, without ever writing it to a file:
+
+```bash
+read -rs HF_TOKEN && export HF_TOKEN      # paste at the prompt; input is hidden
+hf auth login --token "$HF_TOKEN"
+```
+
+`hf auth login --token "$HF_TOKEN"` stores the token in the local Hugging Face config
+(`~/.cache/huggingface/`), so the later commands pick it up automatically. Verify:
+
+```bash
+hf auth whoami
+```
+
+## 2. Create the public model repository
+
+```bash
+hf repos create <your-username>/ZeroPoint --type model --public
+```
+
+Replace `<your-username>` with the account (or organization) that owns the release. If the
+repository already exists, skip this step and make sure its visibility is public in the
+repository settings.
+
+## 3. Upload the release tree
+
+Upload the **contents** of `repo/` so that `README.md`, `LICENSE`, `docs/`, `models/`,
+`experiments/`, `tests/`, etc. land at the repository root:
+
+```bash
+cd "<release-root>"   # the directory containing README.md, LICENSE, models/, docs/ (i.e. the parent of this scripts/ folder)
+
+hf upload <your-username>/ZeroPoint . --type model \
+  --commit-message "ZeroPoint v0 research prototype: architecture, ZeroCG framework, comparison experiments"
+```
+
+This is a single-commit upload of the whole tree. It includes
+`tests/golden/zeropoint_ref.npz` (the fixed reference data required by the self-checks)
+and must not include model weights, training runs, corpora, or tokens — see
+`.gitignore`; the release tree itself contains none of those.
+
+If the upload is interrupted, re-run the same command: files already uploaded are skipped
+and the commit is retried.
+
+## 4. Verify the published repository
+
+```bash
+hf repos ls
+hf auth whoami
+
+# Confirm the public API view, through the proxy:
+curl -sS --proxy http://127.0.0.1:7897 \
+  "https://huggingface.co/api/models/<your-username>/ZeroPoint" | head -c 2000
+```
+
+Then open <https://huggingface.co/<your-username>/ZeroPoint> in a browser and confirm
+that `README.md`, `LICENSE`, `docs/ZeroPoint.md`, `docs/ZeroPoint_EN.md`, and the
+`tests/golden/` file are all present.

tests/golden/zeropoint_ref.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a2b72d72aeac2cd51bc8987ba5e3abbc29ebf623e2f93f40ac5995d77336ca0
+size 85168

tests/make_golden.py

@@ -0,0 +1,32 @@
+"""生成 ZeroPoint 的参考数据 tests/golden/zeropoint_ref.npz：固定的参数 + 输入 + 标准输出。
+
+所有实现（zerocg 并行形式、numpy 递推形式、以后内核里的 C 版本）都拿它对答案，
+谁和它对不上谁就有 bug。**只有在有意修改架构时才重新生成**，并在提交说明里写清楚为什么。
+
+    python3 tests/make_golden.py
+"""
+import os
+import sys
+
+import numpy as np
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.join(HERE, ".."))
+import zerocg as zc  # noqa: E402
+from models import ZeroX  # noqa: E402
+
+CONFIG = dict(vocab_size=13, n_embd=16, n_head=2, n_slot=4, n_layer=2)
+
+if __name__ == "__main__":
+    zc.set_default_dtype(np.float64)
+    np.random.seed(2026)
+    model = ZeroX(**CONFIG).eval()
+    # 参数不用初始值：随机扰动一下，避免初始化时的对称性掩盖 bug
+    for p in model.parameters():
+        p.data = p.data + np.random.randn(*p.shape) * 0.3
+    idx = np.random.randint(0, CONFIG["vocab_size"], size=(2, 70))   # 跨 3 个块
+    logits = model(idx).data
+    out = os.path.join(HERE, "golden", "zeropoint_ref.npz")
+    np.savez(out, idx=idx, logits=logits, **{"param:" + k: v for k, v in model.state_dict().items()},
+             **{"config:" + k: v for k, v in CONFIG.items()})
+    print(f"写入 {out}：输入 {idx.shape}，输出 {logits.shape}，{len(model.state_dict())} 个参数张量")

tests/test_zerocg.py

@@ -0,0 +1,250 @@
+"""框架自检：用数值差分验证每个算子 / 层的反向传播，并跑两个小模型确认能学会。
+运行: python3 tests/test_zerocg.py
+"""
+import os
+import sys
+
+import numpy as np
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+import zerocg as zc  # noqa: E402
+from zerocg import functional as F  # noqa: E402
+from zerocg import nn  # noqa: E402
+
+zc.set_default_dtype(np.float64)   # 数值梯度检查需要 float64 的精度
+rng = np.random.default_rng(0)
+
+
+def grad_check(fn, *shapes, tol=1e-5, positive=False):
+    """fn(*tensors) -> Tensor。比较自动求导和中心差分的梯度（float64）。"""
+    arrays = [rng.standard_normal(s) for s in shapes]
+    if positive:
+        arrays = [np.abs(a) + 0.5 for a in arrays]
+    ts = [zc.Tensor(a.copy(), requires_grad=True) for a in arrays]
+    w = np.asarray(rng.standard_normal(fn(*ts).shape))  # 随机投影成标量，检查完整雅可比
+    (fn(*ts) * zc.Tensor(w)).sum().backward()
+    eps = 1e-6
+    for t, a in zip(ts, arrays):
+        num = np.zeros_like(a)
+        for i in np.ndindex(a.shape):
+            for sign in (1, -1):
+                b = a.copy()
+                b[i] += sign * eps
+                inputs = [zc.Tensor(b) if x is t else zc.Tensor(x.data) for x in ts]
+                with zc.no_grad():
+                    num[i] += sign * (fn(*inputs).data * w).sum()
+            num[i] /= 2 * eps
+        err = np.abs(num - t.grad).max() / max(1.0, np.abs(num).max())
+        assert err < tol, f"梯度误差 {err:.2e}"
+
+
+def module_grad_check(module, x):
+    for p in module.parameters():
+        p.data = p.data.astype(np.float64)
+    params = module.parameters()
+    # 直接对参数逐个做差分
+    xt = zc.Tensor(x)
+    w = np.asarray(rng.standard_normal(module(xt).shape))
+    for p in params:
+        p.grad = None
+    (module(xt) * zc.Tensor(w)).sum().backward()
+    eps = 1e-6
+    for p in params:
+        num = np.zeros_like(p.data)
+        for i in np.ndindex(p.shape):
+            old = p.data[i]
+            p.data[i] = old + eps
+            with zc.no_grad():
+                up = (module(xt).data * w).sum()
+            p.data[i] = old - eps
+            with zc.no_grad():
+                down = (module(xt).data * w).sum()
+            p.data[i] = old
+            num[i] = (up - down) / (2 * eps)
+        err = np.abs(num - p.grad).max() / max(1.0, np.abs(num).max())
+        assert err < 1e-5, f"参数梯度误差 {err:.2e}"
+
+
+# ---------------- 算子 ----------------
+def test_ops():
+    grad_check(lambda a, b: a + b, (3, 4), (4,))
+    grad_check(lambda a, b: a - b, (3, 4), (3, 1))
+    grad_check(lambda a, b: a * b, (2, 3), (2, 3))
+    grad_check(lambda a, b: a / b, (2, 3), (3,), positive=True)
+    grad_check(lambda a: a ** 3, (4,))
+    grad_check(lambda a: a.sqrt(), (4,), positive=True)
+    grad_check(lambda a: -a + 2.0 - a * 3.0, (4,))
+    grad_check(lambda a: 1.0 / a, (4,), positive=True)
+    grad_check(lambda a: a.exp(), (3, 2))
+    grad_check(lambda a: a.log(), (3, 2), positive=True)
+    grad_check(lambda a: a.tanh(), (5,))
+    grad_check(lambda a: a.relu(), (5,))
+    grad_check(lambda a: a.masked_fill(np.eye(3, dtype=bool), -5.0), (3, 3))
+
+
+def test_reductions_and_shapes():
+    grad_check(lambda a: a.sum(), (3, 4))
+    grad_check(lambda a: a.sum(axis=1), (3, 4))
+    grad_check(lambda a: a.mean(axis=0, keepdims=True), (3, 4))
+    grad_check(lambda a: a.max(axis=-1), (3, 4))
+    grad_check(lambda a: a.reshape(4, 3), (3, 4))
+    grad_check(lambda a: a.transpose(2, 0, 1), (2, 3, 4))
+    grad_check(lambda a: a[1:, ::2], (3, 4))
+    grad_check(lambda a: a[np.array([0, 2, 0])], (3, 4))  # 重复索引
+    grad_check(lambda a, b: zc.concat([a, b, a], axis=1), (2, 3), (2, 1))
+
+
+def test_matmul():
+    grad_check(lambda a, b: a @ b, (3, 4), (4, 5))
+    grad_check(lambda a, b: a @ b, (2, 3, 4), (4, 5))      # 批量 @ 共享权重
+    grad_check(lambda a, b: a @ b, (2, 2, 3, 4), (2, 2, 4, 3))
+
+
+def test_functional():
+    grad_check(lambda a: F.softmax(a, axis=-1), (3, 5))
+    grad_check(lambda a: F.log_softmax(a, axis=-1), (3, 5))
+    grad_check(lambda a: F.gelu(a), (6,))
+    grad_check(lambda a: F.sigmoid(a), (6,))
+    grad_check(lambda a: F.cross_entropy(a, np.array([1, 0, 4])), (3, 5))
+    # 和直接的数学定义对比数值
+    x = rng.standard_normal((4, 6))
+    t = np.array([0, 5, 2, 2])
+    ref = -np.log(np.exp(x) / np.exp(x).sum(1, keepdims=True))[np.arange(4), t].mean()
+    assert abs(F.cross_entropy(zc.Tensor(x), t).item() - ref) < 1e-10
+
+
+def test_layers():
+    module_grad_check(nn.Linear(4, 3), rng.standard_normal((2, 4)))
+    module_grad_check(nn.LayerNorm(5), rng.standard_normal((3, 5)))
+    module_grad_check(nn.Sequential(nn.Linear(3, 4), nn.GELU(), nn.Linear(4, 2)), rng.standard_normal((2, 3)))
+    module_grad_check(nn.CausalSelfAttention(8, 2), rng.standard_normal((2, 4, 8)))
+
+    emb = nn.Embedding(5, 3)
+    out = emb(np.array([[1, 1, 4]]))
+    out.sum().backward()
+    assert emb.weight.grad[1].tolist() == [2, 2, 2] and emb.weight.grad[0].tolist() == [0, 0, 0]
+
+
+def test_attention_is_causal():
+    attn = nn.CausalSelfAttention(8, 2).eval()
+    x = rng.standard_normal((1, 5, 8)).astype(np.float32)
+    y1 = attn(zc.Tensor(x)).data
+    x[0, 4] += 10  # 改最后一个位置，前面的输出不应变化
+    y2 = attn(zc.Tensor(x)).data
+    assert np.allclose(y1[0, :4], y2[0, :4]) and not np.allclose(y1[0, 4], y2[0, 4])
+
+
+def test_count_flops():
+    with zc.count_flops() as c:
+        zc.randn(2, 3, 4) @ zc.randn(4, 5)
+    assert c.flops == 2 * (2 * 3 * 5) * 4
+
+
+def test_default_dtype():
+    zc.set_default_dtype(np.float32)
+    try:
+        x = zc.Tensor(np.ones(3))                       # float64 数组进来也变成 float32
+        assert x.dtype == np.float32 and (x * 0.5).dtype == np.float32
+        assert nn.Linear(2, 2).weight.dtype == np.float32
+        assert zc.Tensor(np.arange(3)).dtype.kind == "i"   # 整数下标不动
+        assert np.isfinite(F.sigmoid(zc.Tensor([-1000.0, 0.0, 1000.0])).data).all()
+    finally:
+        zc.set_default_dtype(np.float64)
+
+
+def test_state_dict_roundtrip(tmp="/tmp"):
+    path = os.path.join(os.environ.get("TMPDIR", tmp), "zerocg_test.npz")
+    a = nn.Sequential(nn.Linear(3, 4), nn.LayerNorm(4))
+    a.save(path)
+    b = nn.Sequential(nn.Linear(3, 4), nn.LayerNorm(4)).load(path)
+    x = zc.randn(2, 3)
+    assert np.allclose(a(x).data, b(x).data)
+    assert [n for n, _ in a.named_parameters()] == ["layers.0.weight", "layers.0.bias", "layers.1.weight", "layers.1.bias"]
+
+
+def test_optimizer_resume():
+    """训练 10 步 == 训练 5 步、存档、新建模型和优化器、读档、再训练 5 步。"""
+    X, y = rng.standard_normal((8, 3)), np.array([0, 1, 1, 0, 1, 0, 0, 1])
+
+    def make():
+        np.random.seed(7)
+        m = nn.Sequential(nn.Linear(3, 4), nn.Tanh(), nn.Linear(4, 2))
+        return m, zc.optim.AdamW(m.parameters(), lr=0.05, weight_decay=0.01)
+
+    def train(m, opt, n):
+        for _ in range(n):
+            loss = F.cross_entropy(m(zc.Tensor(X)), y)
+            opt.zero_grad()
+            loss.backward()
+            opt.step()
+
+    a, opt_a = make()
+    train(a, opt_a, 10)
+    b, opt_b = make()
+    train(b, opt_b, 5)
+    model_state, opt_state = b.state_dict(), opt_b.state_dict()
+    c, opt_c = make()
+    c.load_state_dict(model_state)
+    opt_c.load_state_dict(opt_state)
+    train(c, opt_c, 5)
+    assert all(np.array_equal(p, q) for p, q in zip(a.state_dict().values(), c.state_dict().values()))
+
+
+# ---------------- 端到端学习 ----------------
+def test_learns_xor():
+    np.random.seed(0)
+    X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=np.float32)
+    y = np.array([0, 1, 1, 0])
+    model = nn.Sequential(nn.Linear(2, 16), nn.Tanh(), nn.Linear(16, 2))
+    opt = zc.optim.AdamW(model.parameters(), lr=0.05)
+    for _ in range(300):
+        loss = F.cross_entropy(model(zc.Tensor(X)), y)
+        opt.zero_grad()
+        loss.backward()
+        opt.step()
+    assert (model(zc.Tensor(X)).data.argmax(1) == y).all(), f"XOR 没学会, loss={loss.item()}"
+
+
+def test_attention_memorizes_sequence():
+    """一层注意力 + 嵌入，记住一个短序列的下一个字符。"""
+    np.random.seed(0)
+    seq = np.array([0, 1, 2, 3, 2, 1, 0, 3])
+    x, y = seq[None, :-1], seq[1:]
+
+    class Tiny(nn.Module):
+        def __init__(self):
+            self.tok = nn.Embedding(4, 16)
+            self.pos = nn.Embedding(8, 16)
+            self.attn = nn.CausalSelfAttention(16, 2)
+            self.ln = nn.LayerNorm(16)
+            self.head = nn.Linear(16, 4)
+
+        def forward(self, idx):
+            T = idx.shape[1]
+            h = self.tok(idx) + self.pos(np.arange(T))
+            h = h + self.attn(self.ln(h))
+            return self.head(h)
+
+    model = Tiny()
+    opt = zc.optim.AdamW(model.parameters(), lr=0.02)
+    for _ in range(200):
+        loss = F.cross_entropy(model(x).reshape(-1, 4), y)
+        opt.zero_grad()
+        loss.backward()
+        zc.optim.clip_grad_norm(model.parameters(), 1.0)
+        opt.step()
+    assert loss.item() < 0.05, f"loss={loss.item()}"
+
+
+if __name__ == "__main__":
+    tests = [(k, v) for k, v in list(globals().items()) if k.startswith("test_")]
+    failed = 0
+    for name, fn in tests:
+        try:
+            fn()
+            print(f"  ok   {name}")
+        except Exception as e:  # noqa: BLE001
+            failed += 1
+            print(f"  FAIL {name}: {type(e).__name__}: {e}")
+    print(f"\n{len(tests) - failed}/{len(tests)} 通过")
+    sys.exit(1 if failed else 0)

tests/test_zeropoint.py

@@ -0,0 +1,153 @@
+"""ZeroPoint 架构自检。运行: python3 tests/test_zeropoint.py"""
+import itertools
+import os
+import sys
+
+import numpy as np
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.join(HERE, ".."))
+sys.path.insert(0, HERE)
+import zerocg as zc  # noqa: E402
+from models import ZeroX  # noqa: E402
+from models.zeropoint import CHUNK, MemoryMixer  # noqa: E402
+from test_zerocg import module_grad_check  # noqa: E402
+
+zc.set_default_dtype(np.float64)
+rng = np.random.default_rng(0)
+
+# 所有开关的组合（消融实验会用到的每一种都要验证两种算法一致）
+OPTIONS = [dict(zip(("gate", "write", "init", "learn_lambda", "pos", "ema"), combo))
+           for combo in itertools.product((True, False), ("softmax", "sigmoid"), ("hier", "uniform"),
+                                          (True, False), ("none", "sin"), (False, True))]
+
+
+def test_parallel_equals_recurrent():
+    """训练用的分块并行形式和推理用的递推形式，在所有开关组合下都必须一致。
+    长度 300：跨好几个块，也远超旧版并行形式 120 字的上限。"""
+    idx = rng.integers(0, 11, size=(2, 300))
+    worst = 0.0
+    for opts in OPTIONS:
+        np.random.seed(1)
+        model = ZeroX(vocab_size=11, n_embd=16, n_head=2, n_slot=5, n_layer=2, **opts).eval()
+        worst = max(worst, np.abs(model(idx).data - model.forward_recurrent(idx)).max())
+    assert worst < 1e-8, worst
+
+
+def test_matches_golden_reference():
+    """和 tests/golden/zeropoint_ref.npz 对答案：防止以后改代码时并行或递推形式悄悄变了。"""
+    ref = np.load(os.path.join(HERE, "golden", "zeropoint_ref.npz"))
+    cfg = {k[7:]: int(ref[k]) for k in ref.files if k.startswith("config:")}
+    model = ZeroX(**cfg).eval()
+    model.load_state_dict({k[6:]: ref[k] for k in ref.files if k.startswith("param:")})
+    assert np.abs(model(ref["idx"]).data - ref["logits"]).max() < 1e-10
+    assert np.abs(model.forward_recurrent(ref["idx"]) - ref["logits"]).max() < 1e-8
+
+
+def test_causal_across_chunks():
+    np.random.seed(2)
+    model = ZeroX(vocab_size=7, n_embd=16, n_head=2, n_slot=4).eval()
+    idx = rng.integers(0, 7, size=(1, 3 * CHUNK))
+    y1 = model(idx).data
+    p = CHUNK + 5            # 改第二块里的一个字
+    idx[0, p] = (idx[0, p] + 1) % 7
+    y2 = model(idx).data
+    assert np.allclose(y1[0, :p], y2[0, :p]) and not np.allclose(y1[0, p:], y2[0, p:])
+
+
+def test_ema_state_is_bounded():
+    """ema=True：写入恒为常数 v 时，槽会收敛到 v 本身，不会随长度无限变大（ema=False 时会涨到约 v/(1-λ)）。"""
+    np.random.seed(6)
+    for ema, bounded in ((True, True), (False, False)):
+        mixer = MemoryMixer(8, 1, 4, ema=ema)
+        S = mixer.init_state(1) * 0
+        x = np.ones((1, 8))
+        for _ in range(3000):
+            _, S = mixer.step(x, S)
+        v = np.abs(x @ mixer.value.weight.data + mixer.value.bias.data).max()
+        assert (np.abs(S).max() <= v * 1.0001) == bounded, (ema, np.abs(S).max(), v)
+
+
+def test_mixer_gradients():
+    np.random.seed(3)
+    module_grad_check(MemoryMixer(8, 2, 3), rng.standard_normal((2, 6, 8)))
+
+
+def test_mixer_gradients_across_chunks():
+    """梯度要能穿过块与块之间传递的状态，也要传到可学习的初始状态 s0。"""
+    np.random.seed(3)
+    mixer = MemoryMixer(4, 1, 2)
+    module_grad_check(mixer, rng.standard_normal((1, CHUNK + 3, 4)))
+    assert np.abs(mixer.s0.grad).max() > 0
+
+
+def test_initial_state_is_learned():
+    model = ZeroX(vocab_size=5, n_embd=8, n_head=2, n_slot=3, n_layer=1)
+    assert np.allclose(model.init_state(3)[0][2], model.blocks[0].mix.s0.data)
+    assert "blocks.0.mix.s0" in dict(model.named_parameters())
+
+
+def test_retention_init_is_hierarchical():
+    lam = MemoryMixer(16, 2, 6, max_half_life=64).retention().data
+    half_life = np.log(0.5) / np.log(lam[0])
+    assert np.all(np.diff(lam[0]) > 0) and abs(half_life[0] - 1.5) < 1e-3 and abs(half_life[-1] - 64) < 1e-2
+
+
+def test_usage_recorded():
+    model = ZeroX(vocab_size=5, n_embd=8, n_head=2, n_slot=3, n_layer=1)
+    model(rng.integers(0, 5, size=(2, 10)))
+    read, write = model.blocks[0].mix.usage
+    assert read.shape == write.shape == (2, 3) and np.allclose(read.sum(-1), 1)
+
+
+def test_state_is_constant_size():
+    model = ZeroX(vocab_size=5, n_embd=16, n_head=2, n_slot=4, n_layer=3)
+    states = model.init_state(1)
+    for t in range(300):
+        _, states = model.step(np.array([t % 5]), states, t)
+    assert sum(s.size for s in states) == model.state_floats() == 3 * 2 * 4 * 8
+    assert all(np.isfinite(s).all() for s in states)
+
+
+def test_float32_long_sequence_is_finite():
+    """float32 下 1000 个字的分块并行不溢出，并且和递推形式一致。"""
+    zc.set_default_dtype(np.float32)
+    try:
+        np.random.seed(5)
+        model = ZeroX(vocab_size=9, n_embd=16, n_head=2, n_slot=6).eval()
+        idx = rng.integers(0, 9, size=(1, 1000))
+        par = model(idx).data
+        assert par.dtype == np.float32 and np.isfinite(par).all()
+        assert np.abs(par - model.forward_recurrent(idx)).max() < 1e-3
+    finally:
+        zc.set_default_dtype(np.float64)
+
+
+def test_learns_to_copy_from_far_back():
+    """记忆任务：开头一个字，中间 40 个干扰字（跨块），最后复述开头那个字。"""
+    np.random.seed(4)
+    V, gap = 6, 40
+    model = ZeroX(vocab_size=V + 1, n_embd=32, n_head=2, n_slot=8, n_layer=1)
+    opt = zc.optim.AdamW(model.parameters(), lr=0.01)
+    for _ in range(300):
+        first = np.random.randint(0, V, 32)
+        idx = np.concatenate([first[:, None], np.full((32, gap), V)], axis=1)
+        loss = zc.functional.cross_entropy(model(idx)[np.arange(32), gap], first)
+        opt.zero_grad()
+        loss.backward()
+        opt.step()
+    assert loss.item() < 0.05, loss.item()
+
+
+if __name__ == "__main__":
+    tests = [(k, v) for k, v in list(globals().items()) if k.startswith("test_")]
+    failed = 0
+    for name, fn in tests:
+        try:
+            fn()
+            print(f"  ok   {name}")
+        except Exception as e:  # noqa: BLE001
+            failed += 1
+            print(f"  FAIL {name}: {type(e).__name__}: {e}")
+    print(f"\n{len(tests) - failed}/{len(tests)} 通过")
+    sys.exit(1 if failed else 0)

zerocg/__init__.py

@@ -0,0 +1,11 @@
+"""zerocg: 只用 numpy 实现的迷你深度学习框架。
+
+    import zerocg as zc
+    x = zc.randn(4, 3, requires_grad=True)
+    loss = (x * x).sum()
+    loss.backward()        # x.grad == 2 * x
+"""
+from . import functional, nn, optim
+from .tensor import Tensor, concat, count_flops, get_default_dtype, no_grad, ones, randn, set_default_dtype, tensor, zeros
+
+__all__ = ["Tensor", "concat", "count_flops", "no_grad", "set_default_dtype", "get_default_dtype", "tensor", "zeros", "ones", "randn", "nn", "optim", "functional"]

zerocg/functional.py

@@ -0,0 +1,69 @@
+"""常用函数。softmax / log_softmax / gelu 写成融合算子（手写反向），更快也更数值稳定。"""
+import numpy as np
+
+from .tensor import Tensor
+
+
+def relu(x):
+    return x.relu()
+
+
+def tanh(x):
+    return x.tanh()
+
+
+def sigmoid_np(a):
+    """数值稳定：只对 -|a| 求 exp，永远不会溢出。"""
+    e = np.exp(-np.abs(a))
+    return np.where(a >= 0, 1 / (1 + e), e / (1 + e))
+
+
+def sigmoid(x):
+    out = sigmoid_np(x.data)
+    return Tensor._op(out, (x,), lambda g: (g * out * (1 - out),))
+
+
+def gelu(x):
+    """GELU（tanh 近似，GPT-2 同款）。"""
+    c = np.sqrt(2 / np.pi)
+    a = x.data
+    inner = c * (a + 0.044715 * a ** 3)
+    t = np.tanh(inner)
+    out = 0.5 * a * (1 + t)
+
+    def backward(g):
+        dinner = c * (1 + 3 * 0.044715 * a ** 2)
+        return (g * (0.5 * (1 + t) + 0.5 * a * (1 - t ** 2) * dinner),)
+
+    return Tensor._op(out.astype(a.dtype), (x,), backward)
+
+
+def softmax(x, axis=-1):
+    e = np.exp(x.data - x.data.max(axis=axis, keepdims=True))
+    s = e / e.sum(axis=axis, keepdims=True)
+    return Tensor._op(s, (x,), lambda g: (s * (g - (g * s).sum(axis=axis, keepdims=True)),))
+
+
+def log_softmax(x, axis=-1):
+    shifted = x.data - x.data.max(axis=axis, keepdims=True)
+    out = shifted - np.log(np.exp(shifted).sum(axis=axis, keepdims=True))
+    s = np.exp(out)
+    return Tensor._op(out, (x,), lambda g: (g - s * g.sum(axis=axis, keepdims=True),))
+
+
+def cross_entropy(logits, targets):
+    """logits: (N, C)；targets: (N,) 整数类别。返回平均负对数似然。"""
+    targets = targets.data if isinstance(targets, Tensor) else np.asarray(targets)
+    logp = log_softmax(logits, axis=-1)
+    return -logp[np.arange(len(targets)), targets].mean()
+
+
+def mse_loss(pred, target):
+    return ((pred - target) ** 2).mean()
+
+
+def dropout(x, p, training=True):
+    if not training or p == 0:
+        return x
+    mask = (np.random.rand(*x.shape) >= p).astype(x.dtype) / (1 - p)
+    return x * Tensor(mask)

zerocg/nn.py

@@ -0,0 +1,162 @@
+"""神经网络层。Module 会自动收集属性里的 Parameter / 子 Module。"""
+import numpy as np
+
+from . import functional as F
+from .tensor import Tensor, get_default_dtype
+
+
+class Parameter(Tensor):
+    def __init__(self, data):
+        super().__init__(np.asarray(data, dtype=get_default_dtype()), requires_grad=True)
+
+
+class Module:
+    training = True
+
+    def forward(self, *args, **kwargs):
+        raise NotImplementedError
+
+    def __call__(self, *args, **kwargs):
+        return self.forward(*args, **kwargs)
+
+    def named_parameters(self, prefix=""):
+        for name, v in vars(self).items():
+            items = [(name, v)] if not isinstance(v, (list, tuple)) else [(f"{name}.{i}", x) for i, x in enumerate(v)]
+            for n, x in items:
+                if isinstance(x, Parameter):
+                    yield prefix + n, x
+                elif isinstance(x, Module):
+                    yield from x.named_parameters(prefix + n + ".")
+
+    def parameters(self):
+        return [p for _, p in self.named_parameters()]
+
+    def modules(self):
+        yield self
+        for v in vars(self).values():
+            for x in (v if isinstance(v, (list, tuple)) else [v]):
+                if isinstance(x, Module):
+                    yield from x.modules()
+
+    def train(self, mode=True):
+        for m in self.modules():
+            m.training = mode
+        return self
+
+    def eval(self):
+        return self.train(False)
+
+    def zero_grad(self):
+        for p in self.parameters():
+            p.grad = None
+
+    def num_parameters(self):
+        return sum(p.data.size for p in self.parameters())
+
+    # ---------- 保存 / 加载 ----------
+    def state_dict(self):
+        return {n: p.data.copy() for n, p in self.named_parameters()}
+
+    def load_state_dict(self, state):
+        for n, p in self.named_parameters():
+            assert state[n].shape == p.shape, f"{n} 形状不匹配: {state[n].shape} vs {p.shape}"
+            p.data = state[n].astype(p.dtype)
+
+    def save(self, path):
+        np.savez(path, **self.state_dict())
+
+    def load(self, path):
+        with np.load(path) as f:
+            self.load_state_dict(dict(f))
+        return self
+
+
+class Linear(Module):
+    def __init__(self, in_features, out_features, bias=True):
+        bound = 1 / np.sqrt(in_features)
+        self.weight = Parameter(np.random.uniform(-bound, bound, (in_features, out_features)))
+        self.bias = Parameter(np.zeros(out_features)) if bias else None
+
+    def forward(self, x):
+        y = x @ self.weight
+        return y + self.bias if self.bias is not None else y
+
+
+class Embedding(Module):
+    def __init__(self, num_embeddings, dim):
+        self.weight = Parameter(np.random.randn(num_embeddings, dim) * 0.02)
+
+    def forward(self, idx):
+        return self.weight[idx.data if isinstance(idx, Tensor) else np.asarray(idx)]
+
+
+class LayerNorm(Module):
+    def __init__(self, dim, eps=1e-5):
+        self.eps = eps
+        self.weight = Parameter(np.ones(dim))
+        self.bias = Parameter(np.zeros(dim))
+
+    def forward(self, x):
+        mu = x.mean(axis=-1, keepdims=True)
+        xc = x - mu
+        var = (xc * xc).mean(axis=-1, keepdims=True)
+        return xc / (var + self.eps).sqrt() * self.weight + self.bias
+
+
+class Dropout(Module):
+    def __init__(self, p=0.1):
+        self.p = p
+
+    def forward(self, x):
+        return F.dropout(x, self.p, self.training)
+
+
+class ReLU(Module):
+    def forward(self, x):
+        return x.relu()
+
+
+class Tanh(Module):
+    def forward(self, x):
+        return x.tanh()
+
+
+class GELU(Module):
+    def forward(self, x):
+        return F.gelu(x)
+
+
+class Sequential(Module):
+    def __init__(self, *layers):
+        self.layers = list(layers)
+
+    def forward(self, x):
+        for layer in self.layers:
+            x = layer(x)
+        return x
+
+    def __getitem__(self, i):
+        return self.layers[i]
+
+
+class CausalSelfAttention(Module):
+    """多头因果自注意力：位置 t 只能看到 <= t 的位置。输入/输出形状 (B, T, C)。"""
+
+    def __init__(self, n_embd, n_head, dropout=0.0):
+        assert n_embd % n_head == 0
+        self.n_head = n_head
+        self.qkv = Linear(n_embd, 3 * n_embd)
+        self.proj = Linear(n_embd, n_embd)
+        self.attn_drop = Dropout(dropout)
+        self.resid_drop = Dropout(dropout)
+
+    def forward(self, x):
+        B, T, C = x.shape
+        H, D = self.n_head, C // self.n_head
+        qkv = self.qkv(x).reshape(B, T, 3, H, D).transpose(2, 0, 3, 1, 4)  # (3, B, H, T, D)
+        q, k, v = qkv[0], qkv[1], qkv[2]
+        att = (q @ k.transpose(0, 1, 3, 2)) * (1.0 / np.sqrt(D))           # (B, H, T, T)
+        att = att.masked_fill(np.triu(np.ones((T, T), dtype=bool), 1), -1e9)
+        att = self.attn_drop(F.softmax(att, axis=-1))
+        y = (att @ v).transpose(0, 2, 1, 3).reshape(B, T, C)
+        return self.resid_drop(self.proj(y))

zerocg/optim.py

@@ -0,0 +1,87 @@
+"""优化器。"""
+import numpy as np
+
+
+class Optimizer:
+    def __init__(self, params, lr):
+        self.params = list(params)
+        self.lr = lr
+
+    def zero_grad(self):
+        for p in self.params:
+            p.grad = None
+
+    def step(self):
+        raise NotImplementedError
+
+
+class SGD(Optimizer):
+    def __init__(self, params, lr=0.01, momentum=0.0):
+        super().__init__(params, lr)
+        self.momentum = momentum
+        self.v = [np.zeros_like(p.data) for p in self.params]
+
+    def step(self):
+        for p, v in zip(self.params, self.v):
+            if p.grad is None:
+                continue
+            v *= self.momentum
+            v += p.grad
+            p.data -= self.lr * v
+
+
+class AdamW(Optimizer):
+    """Adam + 解耦权重衰减。weight_decay=0 时就是普通 Adam。"""
+
+    def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0.0):
+        super().__init__(params, lr)
+        self.b1, self.b2 = betas
+        self.eps = eps
+        self.weight_decay = weight_decay
+        self.t = 0
+        self.m = [np.zeros_like(p.data) for p in self.params]
+        self.v = [np.zeros_like(p.data) for p in self.params]
+
+    def step(self):
+        self.t += 1
+        c1, c2 = 1 - self.b1 ** self.t, 1 - self.b2 ** self.t
+        for p, m, v in zip(self.params, self.m, self.v):
+            if p.grad is None:
+                continue
+            g = p.grad
+            m *= self.b1
+            m += (1 - self.b1) * g
+            v *= self.b2
+            v += (1 - self.b2) * g * g
+            if self.weight_decay:
+                p.data -= self.lr * self.weight_decay * p.data
+            p.data -= self.lr * (m / c1) / (np.sqrt(v / c2) + self.eps)
+
+    def state_dict(self):
+        """优化器状态（步数 + 一阶/二阶动量）。和 model.state_dict() 一起存，才能精确续训。"""
+        state = {"t": np.array(self.t), "lr": np.array(self.lr)}
+        for i, (m, v) in enumerate(zip(self.m, self.v)):
+            state[f"m.{i}"], state[f"v.{i}"] = m.copy(), v.copy()
+        return state
+
+    def load_state_dict(self, state):
+        assert sum(k.startswith("m.") for k in state) == len(self.params), "参数个数和存档对不上"
+        self.t, self.lr = int(state["t"]), float(state["lr"])
+        for i in range(len(self.params)):
+            self.m[i][...] = state[f"m.{i}"]
+            self.v[i][...] = state[f"v.{i}"]
+
+
+Adam = AdamW
+
+
+def clip_grad_norm(params, max_norm):
+    """按全局 L2 范数裁剪梯度，返回裁剪前的范数。"""
+    grads = [p.grad for p in params if p.grad is not None]
+    total = float(np.sqrt(sum((g.astype(np.float64) ** 2).sum() for g in grads)))
+    if total > max_norm:
+        scale = max_norm / (total + 1e-6)
+        for p in params:
+            if p.grad is not None:
+                p.grad = p.grad * scale
+    return total

zerocg/tensor.py

@@ -0,0 +1,320 @@
+"""Tensor + 反向模式自动求导（只依赖 numpy）。
+
+每个运算产出一个新 Tensor，并记录:
+  _parents  : 参与运算的输入 Tensor
+  _backward : 函数 g -> (对每个 parent 的梯度, ...)
+backward() 先拓扑排序，再从输出往回逐个调用 _backward，把梯度累加到 .grad。
+"""
+import numpy as np
+
+_grad_enabled = True
+_default_dtype = np.dtype(np.float32)
+
+
+def set_default_dtype(dtype):
+    """所有浮点数据进入框架时统一转成这个类型（默认 float32）。
+    只有测试里的数值梯度检查需要切到 float64。整数数组（下标）不受影响。"""
+    global _default_dtype
+    _default_dtype = np.dtype(dtype)
+
+
+def get_default_dtype():
+    return _default_dtype
+
+
+_flops = None   # count_flops() 打开时是一个 [计数]，否则 None
+
+
+class count_flops:
+    """with count_flops() as c: ...  之后 c.flops = 期间所有矩阵乘的浮点运算次数（乘加算 2 次）。
+    只统计矩阵乘（模型里绝大部分计算量），不统计逐元素运算；不受机器负载影响，比计时可靠。"""
+
+    def __enter__(self):
+        global _flops
+        self._prev, _flops = _flops, [0]
+        return self
+
+    def __exit__(self, *exc):
+        global _flops
+        self.flops = _flops[0]
+        _flops = self._prev
+
+
+class no_grad:
+    """with no_grad(): 内部的运算不建计算图（推理/评估时用，省内存）。"""
+
+    def __enter__(self):
+        global _grad_enabled
+        self._prev, _grad_enabled = _grad_enabled, False
+
+    def __exit__(self, *exc):
+        global _grad_enabled
+        _grad_enabled = self._prev
+
+
+def _unbroadcast(grad, shape):
+    """把广播后的梯度求和还原成原始形状。"""
+    while grad.ndim > len(shape):
+        grad = grad.sum(axis=0)
+    for i, s in enumerate(shape):
+        if s == 1 and grad.shape[i] != 1:
+            grad = grad.sum(axis=i, keepdims=True)
+    return grad
+
+
+def _to_array(x):
+    if isinstance(x, Tensor):
+        return x.data
+    arr = np.asarray(x)
+    # 浮点一律转成默认类型，避免 float32 和 float64 混在一起被 numpy 悄悄升成 float64
+    if arr.dtype.kind == "f" and arr.dtype != _default_dtype:
+        arr = arr.astype(_default_dtype)
+    return arr
+
+
+class Tensor:
+    __array_priority__ = 100  # 让 ndarray + Tensor 走 Tensor.__radd__
+
+    def __init__(self, data, requires_grad=False):
+        self.data = _to_array(data)
+        self.requires_grad = requires_grad
+        self.grad = None
+        self._parents = ()
+        self._backward = None
+
+    # ---------- 基础属性 ----------
+    @property
+    def shape(self):
+        return self.data.shape
+
+    @property
+    def ndim(self):
+        return self.data.ndim
+
+    @property
+    def dtype(self):
+        return self.data.dtype
+
+    def __len__(self):
+        return len(self.data)
+
+    def __repr__(self):
+        return f"Tensor({self.data}, requires_grad={self.requires_grad})"
+
+    def numpy(self):
+        return self.data
+
+    def item(self):
+        return self.data.item()
+
+    def detach(self):
+        return Tensor(self.data)
+
+    # ---------- 建图 ----------
+    @staticmethod
+    def _op(data, parents, backward):
+        """创建运算结果。backward(g) 返回与 parents 一一对应的梯度（可为 None）。"""
+        out = Tensor(data)
+        if _grad_enabled and any(p.requires_grad for p in parents):
+            out.requires_grad = True
+            out._parents = parents
+            out._backward = backward
+        return out
+
+    def backward(self, grad=None):
+        if grad is None:
+            if self.data.size != 1:
+                raise RuntimeError("非标量输出调用 backward() 需要显式传入 grad")
+            grad = np.ones_like(self.data)
+
+        # 迭代式拓扑排序（避免深图递归爆栈）
+        topo, visited, stack = [], set(), [(self, False)]
+        while stack:
+            node, expanded = stack.pop()
+            if expanded:
+                topo.append(node)
+                continue
+            if id(node) in visited:
+                continue
+            visited.add(id(node))
+            stack.append((node, True))
+            for p in node._parents:
+                if p.requires_grad and id(p) not in visited:
+                    stack.append((p, False))
+
+        self.grad = grad if self.grad is None else self.grad + grad
+        for node in reversed(topo):
+            if node._backward is None or node.grad is None:
+                continue
+            for p, g in zip(node._parents, node._backward(node.grad)):
+                if g is not None and p.requires_grad:
+                    p.grad = g if p.grad is None else p.grad + g
+            # 释放计算图，中间节点的梯度也不再需要
+            node._parents, node._backward, node.grad = (), None, None
+
+    # ---------- 逐元素运算 ----------
+    def __add__(self, other):
+        other = other if isinstance(other, Tensor) else Tensor(other)
+        a, b = self, other
+        return Tensor._op(a.data + b.data, (a, b),
+                          lambda g: (_unbroadcast(g, a.shape), _unbroadcast(g, b.shape)))
+
+    def __mul__(self, other):
+        other = other if isinstance(other, Tensor) else Tensor(other)
+        a, b = self, other
+        return Tensor._op(a.data * b.data, (a, b),
+                          lambda g: (_unbroadcast(g * b.data, a.shape), _unbroadcast(g * a.data, b.shape)))
+
+    def __truediv__(self, other):
+        other = other if isinstance(other, Tensor) else Tensor(other)
+        a, b = self, other
+        return Tensor._op(a.data / b.data, (a, b),
+                          lambda g: (_unbroadcast(g / b.data, a.shape),
+                                     _unbroadcast(-g * a.data / (b.data ** 2), b.shape)))
+
+    def __pow__(self, p):
+        assert isinstance(p, (int, float)), "只支持标量指数"
+        a = self
+        return Tensor._op(a.data ** p, (a,), lambda g: (g * p * a.data ** (p - 1),))
+
+    def __neg__(self):
+        return Tensor._op(-self.data, (self,), lambda g: (-g,))
+
+    def __sub__(self, other):
+        return self + (-other if isinstance(other, Tensor) else Tensor(-_to_array(other)))
+
+    def __radd__(self, other):
+        return self + other
+
+    def __rmul__(self, other):
+        return self * other
+
+    def __rsub__(self, other):
+        return (-self) + other
+
+    def __rtruediv__(self, other):
+        return Tensor(other) / self
+
+    def exp(self):
+        out = np.exp(self.data)
+        return Tensor._op(out, (self,), lambda g: (g * out,))
+
+    def log(self):
+        a = self
+        return Tensor._op(np.log(a.data), (a,), lambda g: (g / a.data,))
+
+    def sqrt(self):
+        return self ** 0.5
+
+    def tanh(self):
+        out = np.tanh(self.data)
+        return Tensor._op(out, (self,), lambda g: (g * (1 - out ** 2),))
+
+    def relu(self):
+        mask = self.data > 0
+        return Tensor._op(self.data * mask, (self,), lambda g: (g * mask,))
+
+    def masked_fill(self, mask, value):
+        mask = _to_array(mask).astype(bool)
+        return Tensor._op(np.where(mask, value, self.data).astype(self.dtype), (self,),
+                          lambda g: (np.where(mask, 0, g),))
+
+    # ---------- 归约 ----------
+    def sum(self, axis=None, keepdims=False):
+        a = self
+
+        def backward(g):
+            if axis is not None and not keepdims:
+                g = np.expand_dims(g, axis)
+            return (np.broadcast_to(g, a.shape).copy(),)
+
+        return Tensor._op(a.data.sum(axis=axis, keepdims=keepdims), (a,), backward)
+
+    def mean(self, axis=None, keepdims=False):
+        n = self.data.size if axis is None else np.prod([self.shape[i] for i in np.atleast_1d(axis)])
+        return self.sum(axis, keepdims) * (1.0 / n)
+
+    def max(self, axis=None, keepdims=False):
+        a = self
+        out = a.data.max(axis=axis, keepdims=True)
+
+        def backward(g):
+            if axis is not None and not keepdims:
+                g = np.expand_dims(g, axis)
+            mask = a.data == out
+            return (mask * g / mask.sum(axis=axis, keepdims=True),)  # 并列最大值平分梯度
+
+        return Tensor._op(out if keepdims else out.squeeze(axis), (a,), backward)
+
+    # ---------- 形状 ----------
+    def reshape(self, *shape):
+        a = self
+        return Tensor._op(a.data.reshape(*shape), (a,), lambda g: (g.reshape(a.shape),))
+
+    def transpose(self, *axes):
+        axes = axes or tuple(reversed(range(self.ndim)))
+        inv = np.argsort(axes)
+        return Tensor._op(self.data.transpose(axes), (self,), lambda g: (g.transpose(inv),))
+
+    @property
+    def T(self):
+        return self.transpose()
+
+    def __getitem__(self, idx):
+        a = self
+        if isinstance(idx, Tensor):
+            idx = idx.data
+
+        parts = idx if isinstance(idx, tuple) else (idx,)
+        fancy = any(isinstance(i, (np.ndarray, list)) for i in parts)
+
+        def backward(g):
+            grad = np.zeros_like(a.data)
+            if fancy:
+                np.add.at(grad, idx, g)  # 数组下标可能重复（Embedding 查表），必须累加
+            else:
+                grad[idx] = g            # 普通切片不会重复，直接写，比 add.at 快得多
+            return (grad,)
+
+        return Tensor._op(a.data[idx], (a,), backward)
+
+    # ---------- 矩阵乘 ----------
+    def __matmul__(self, other):
+        other = other if isinstance(other, Tensor) else Tensor(other)
+        a, b = self, other
+        assert a.ndim >= 2 and b.ndim >= 2, "matmul 需要至少 2 维"
+        out = a.data @ b.data
+        if _flops is not None:
+            _flops[0] += 2 * out.size * a.shape[-1]
+
+        def backward(g):
+            ga = g @ np.swapaxes(b.data, -1, -2)
+            gb = np.swapaxes(a.data, -1, -2) @ g
+            return _unbroadcast(ga, a.shape), _unbroadcast(gb, b.shape)
+
+        return Tensor._op(out, (a, b), backward)
+
+
+def concat(tensors, axis=0):
+    """沿 axis 拼接。反向时把梯度按原来的大小切回去。"""
+    tensors = tuple(t if isinstance(t, Tensor) else Tensor(t) for t in tensors)
+    splits = np.cumsum([t.shape[axis] for t in tensors])[:-1]
+    return Tensor._op(np.concatenate([t.data for t in tensors], axis=axis), tensors,
+                      lambda g: tuple(np.split(g, splits, axis=axis)))
+
+
+# ---------- 构造函数 ----------
+def tensor(data, requires_grad=False):
+    return Tensor(data, requires_grad)
+
+
+def zeros(*shape, requires_grad=False):
+    return Tensor(np.zeros(shape, dtype=_default_dtype), requires_grad)
+
+
+def ones(*shape, requires_grad=False):
+    return Tensor(np.ones(shape, dtype=_default_dtype), requires_grad)
+
+
+def randn(*shape, requires_grad=False):
+    return Tensor(np.random.randn(*shape).astype(_default_dtype), requires_grad)