Portable golden reference (JSON instead of LFS npz); doc links updated
Browse files- .gitignore +1 -1
- docs/ZeroPoint.md +1 -1
- docs/ZeroPoint_EN.md +1 -1
- scripts/upload.md +1 -1
- tests/golden/zeropoint_ref.json +0 -0
- tests/make_golden.py +9 -4
- tests/test_zeropoint.py +10 -6
.gitignore
CHANGED
|
@@ -21,7 +21,7 @@ env/
|
|
| 21 |
# Model weights and checkpoints (this release ships no weights)
|
| 22 |
*.npz
|
| 23 |
*.npy
|
| 24 |
-
!tests/golden/*.
|
| 25 |
*.pt
|
| 26 |
*.pth
|
| 27 |
*.bin
|
|
|
|
| 21 |
# Model weights and checkpoints (this release ships no weights)
|
| 22 |
*.npz
|
| 23 |
*.npy
|
| 24 |
+
!tests/golden/*.json
|
| 25 |
*.pt
|
| 26 |
*.pth
|
| 27 |
*.bin
|
docs/ZeroPoint.md
CHANGED
|
@@ -181,7 +181,7 @@ A = ( a ⊙ λ^t ) @ ( w ⊙ λ^(−u) )ᵀ 再把 u > t 的部分清零(
|
|
| 181 |
**递推形式(推理用)**。逐字更新 `S`,每层只存 M·d 个浮点数,跟文本多长无关。
|
| 182 |
|
| 183 |
两种形式结果一致:自检在**所有 64 种开关组合**下、300 个字(跨多个块)上逐位比对,float64 误差 < 1e-8。
|
| 184 |
-
另有一份固定的参考数据 `tests/golden/zeropoint_ref.
|
| 185 |
**训练好的模型可以直接用递推形式放进内核**:不用 KV 缓存,每层固定一块 M × d 的状态,不需要动态分配。
|
| 186 |
|
| 187 |
关于"最长路径 O(1)"要说实话:u 处写入的信息确实一步就能被 t 处读到,但已经乘上了 λ^(t−u)。
|
|
|
|
| 181 |
**递推形式(推理用)**。逐字更新 `S`,每层只存 M·d 个浮点数,跟文本多长无关。
|
| 182 |
|
| 183 |
两种形式结果一致:自检在**所有 64 种开关组合**下、300 个字(跨多个块)上逐位比对,float64 误差 < 1e-8。
|
| 184 |
+
另有一份固定的参考数据 `tests/golden/zeropoint_ref.json`,以后任何实现(包括内核里的 C 版本)都拿它对答案。
|
| 185 |
**训练好的模型可以直接用递推形式放进内核**:不用 KV 缓存,每层固定一块 M × d 的状态,不需要动态分配。
|
| 186 |
|
| 187 |
关于"最长路径 O(1)"要说实话:u 处写入的信息确实一步就能被 t 处读到,但已经乘上了 λ^(t−u)。
|
docs/ZeroPoint_EN.md
CHANGED
|
@@ -157,7 +157,7 @@ Within a block it is matrix multiplication, and only an M × d state is passed b
|
|
| 157 |
|
| 158 |
**Recurrent form (for inference)**. Update `S` character by character; each layer stores only M·d floating-point numbers, regardless of how long the text is.
|
| 159 |
|
| 160 |
-
The two forms agree: the self-check compares them bit by bit on **all 64 switch combinations** over 300 characters (crossing several blocks), with float64 error < 1e-8. There is also a fixed reference file `tests/golden/zeropoint_ref.
|
| 161 |
|
| 162 |
About "maximum path O(1)", we must be honest: information written at u can indeed be read at t in one step, but it has already been multiplied by λ^(t−u). Attention can "retrieve verbatim" a character from arbitrarily far away; ZeroPoint can only retrieve information **that has not yet been forgotten and has not been washed out by other writes**.
|
| 163 |
|
|
|
|
| 157 |
|
| 158 |
**Recurrent form (for inference)**. Update `S` character by character; each layer stores only M·d floating-point numbers, regardless of how long the text is.
|
| 159 |
|
| 160 |
+
The two forms agree: the self-check compares them bit by bit on **all 64 switch combinations** over 300 characters (crossing several blocks), with float64 error < 1e-8. There is also a fixed reference file `tests/golden/zeropoint_ref.json`; any future implementation (including a C version in the kernel) can check its answers against it. **A trained model can be put directly into the kernel in recurrent form**: no KV cache, a fixed M × d state per layer, no dynamic allocation.
|
| 161 |
|
| 162 |
About "maximum path O(1)", we must be honest: information written at u can indeed be read at t in one step, but it has already been multiplied by λ^(t−u). Attention can "retrieve verbatim" a character from arbitrarily far away; ZeroPoint can only retrieve information **that has not yet been forgotten and has not been washed out by other writes**.
|
| 163 |
|
scripts/upload.md
CHANGED
|
@@ -69,7 +69,7 @@ hf upload <your-username>/ZeroPoint . --type model \
|
|
| 69 |
```
|
| 70 |
|
| 71 |
This is a single-commit upload of the whole tree. It includes
|
| 72 |
-
`tests/golden/zeropoint_ref.
|
| 73 |
and must not include model weights, training runs, corpora, or tokens — see
|
| 74 |
`.gitignore`; the release tree itself contains none of those.
|
| 75 |
|
|
|
|
| 69 |
```
|
| 70 |
|
| 71 |
This is a single-commit upload of the whole tree. It includes
|
| 72 |
+
`tests/golden/zeropoint_ref.json` (the fixed reference data required by the self-checks)
|
| 73 |
and must not include model weights, training runs, corpora, or tokens — see
|
| 74 |
`.gitignore`; the release tree itself contains none of those.
|
| 75 |
|
tests/golden/zeropoint_ref.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tests/make_golden.py
CHANGED
|
@@ -1,10 +1,11 @@
|
|
| 1 |
-
"""生成 ZeroPoint 的参考数据 tests/golden/zeropoint_ref.
|
| 2 |
|
| 3 |
所有实现(zerocg 并行形式、numpy 递推形式、以后内核里的 C 版本)都拿它对答案,
|
| 4 |
谁和它对不上谁就有 bug。**只有在有意修改架构时才重新生成**,并在提交说明里写清楚为什么。
|
| 5 |
|
| 6 |
python3 tests/make_golden.py
|
| 7 |
"""
|
|
|
|
| 8 |
import os
|
| 9 |
import sys
|
| 10 |
|
|
@@ -26,7 +27,11 @@ if __name__ == "__main__":
|
|
| 26 |
p.data = p.data + np.random.randn(*p.shape) * 0.3
|
| 27 |
idx = np.random.randint(0, CONFIG["vocab_size"], size=(2, 70)) # 跨 3 个块
|
| 28 |
logits = model(idx).data
|
| 29 |
-
out = os.path.join(HERE, "golden", "zeropoint_ref.
|
| 30 |
-
|
| 31 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
print(f"写入 {out}:输入 {idx.shape},输出 {logits.shape},{len(model.state_dict())} 个参数张量")
|
|
|
|
| 1 |
+
"""生成 ZeroPoint 的参考数据 tests/golden/zeropoint_ref.json:固定的参数 + 输入 + 标准输出。
|
| 2 |
|
| 3 |
所有实现(zerocg 并行形式、numpy 递推形式、以后内核里的 C 版本)都拿它对答案,
|
| 4 |
谁和它对不上谁就有 bug。**只有在有意修改架构时才重新生成**,并在提交说明里写清楚为什么。
|
| 5 |
|
| 6 |
python3 tests/make_golden.py
|
| 7 |
"""
|
| 8 |
+
import json
|
| 9 |
import os
|
| 10 |
import sys
|
| 11 |
|
|
|
|
| 27 |
p.data = p.data + np.random.randn(*p.shape) * 0.3
|
| 28 |
idx = np.random.randint(0, CONFIG["vocab_size"], size=(2, 70)) # 跨 3 个块
|
| 29 |
logits = model(idx).data
|
| 30 |
+
out = os.path.join(HERE, "golden", "zeropoint_ref.json")
|
| 31 |
+
data = {"idx": idx.tolist(), "logits": logits.tolist()}
|
| 32 |
+
data.update({"param:" + k: v.tolist() for k, v in model.state_dict().items()})
|
| 33 |
+
data.update({"config:" + k: int(v) for k, v in CONFIG.items()})
|
| 34 |
+
with open(out, "w", encoding="utf-8") as f:
|
| 35 |
+
json.dump(data, f, separators=(",", ":"), ensure_ascii=False)
|
| 36 |
+
f.write("\n")
|
| 37 |
print(f"写入 {out}:输入 {idx.shape},输出 {logits.shape},{len(model.state_dict())} 个参数张量")
|
tests/test_zeropoint.py
CHANGED
|
@@ -1,5 +1,6 @@
|
|
| 1 |
"""ZeroPoint 架构自检。运行: python3 tests/test_zeropoint.py"""
|
| 2 |
import itertools
|
|
|
|
| 3 |
import os
|
| 4 |
import sys
|
| 5 |
|
|
@@ -35,13 +36,16 @@ def test_parallel_equals_recurrent():
|
|
| 35 |
|
| 36 |
|
| 37 |
def test_matches_golden_reference():
|
| 38 |
-
"""和 tests/golden/zeropoint_ref.
|
| 39 |
-
|
| 40 |
-
|
|
|
|
| 41 |
model = ZeroX(**cfg).eval()
|
| 42 |
-
model.load_state_dict({k[6:]:
|
| 43 |
-
|
| 44 |
-
|
|
|
|
|
|
|
| 45 |
|
| 46 |
|
| 47 |
def test_causal_across_chunks():
|
|
|
|
| 1 |
"""ZeroPoint 架构自检。运行: python3 tests/test_zeropoint.py"""
|
| 2 |
import itertools
|
| 3 |
+
import json
|
| 4 |
import os
|
| 5 |
import sys
|
| 6 |
|
|
|
|
| 36 |
|
| 37 |
|
| 38 |
def test_matches_golden_reference():
|
| 39 |
+
"""和 tests/golden/zeropoint_ref.json 对答案:防止以后改代码时并行或递推形式悄悄变了。"""
|
| 40 |
+
with open(os.path.join(HERE, "golden", "zeropoint_ref.json"), encoding="utf-8") as f:
|
| 41 |
+
ref = json.load(f)
|
| 42 |
+
cfg = {k[7:]: int(v) for k, v in ref.items() if k.startswith("config:")}
|
| 43 |
model = ZeroX(**cfg).eval()
|
| 44 |
+
model.load_state_dict({k[6:]: np.array(v, dtype=np.float64) for k, v in ref.items()
|
| 45 |
+
if k.startswith("param:")})
|
| 46 |
+
idx, logits = np.array(ref["idx"]), np.array(ref["logits"])
|
| 47 |
+
assert np.abs(model(idx).data - logits).max() < 1e-10
|
| 48 |
+
assert np.abs(model.forward_recurrent(idx) - logits).max() < 1e-8
|
| 49 |
|
| 50 |
|
| 51 |
def test_causal_across_chunks():
|