Holmes2011 commited on
Commit
cf9d631
·
verified ·
1 Parent(s): 5c6c847

Portable golden reference (JSON instead of LFS npz); doc links updated

Browse files
.gitignore CHANGED
@@ -21,7 +21,7 @@ env/
21
  # Model weights and checkpoints (this release ships no weights)
22
  *.npz
23
  *.npy
24
- !tests/golden/*.npz
25
  *.pt
26
  *.pth
27
  *.bin
 
21
  # Model weights and checkpoints (this release ships no weights)
22
  *.npz
23
  *.npy
24
+ !tests/golden/*.json
25
  *.pt
26
  *.pth
27
  *.bin
docs/ZeroPoint.md CHANGED
@@ -181,7 +181,7 @@ A = ( a ⊙ λ^t ) @ ( w ⊙ λ^(−u) )ᵀ 再把 u > t 的部分清零(
181
  **递推形式(推理用)**。逐字更新 `S`,每层只存 M·d 个浮点数,跟文本多长无关。
182
 
183
  两种形式结果一致:自检在**所有 64 种开关组合**下、300 个字(跨多个块)上逐位比对,float64 误差 < 1e-8。
184
- 另有一份固定的参考数据 `tests/golden/zeropoint_ref.npz`,以后任何实现(包括内核里的 C 版本)都拿它对答案。
185
  **训练好的模型可以直接用递推形式放进内核**:不用 KV 缓存,每层固定一块 M × d 的状态,不需要动态分配。
186
 
187
  关于"最长路径 O(1)"要说实话:u 处写入的信息确实一步就能被 t 处读到,但已经乘上了 λ^(t−u)。
 
181
  **递推形式(推理用)**。逐字更新 `S`,每层只存 M·d 个浮点数,跟文本多长无关。
182
 
183
  两种形式结果一致:自检在**所有 64 种开关组合**下、300 个字(跨多个块)上逐位比对,float64 误差 < 1e-8。
184
+ 另有一份固定的参考数据 `tests/golden/zeropoint_ref.json`,以后任何实现(包括内核里的 C 版本)都拿它对答案。
185
  **训练好的模型可以直接用递推形式放进内核**:不用 KV 缓存,每层固定一块 M × d 的状态,不需要动态分配。
186
 
187
  关于"最长路径 O(1)"要说实话:u 处写入的信息确实一步就能被 t 处读到,但已经乘上了 λ^(t−u)。
docs/ZeroPoint_EN.md CHANGED
@@ -157,7 +157,7 @@ Within a block it is matrix multiplication, and only an M × d state is passed b
157
 
158
  **Recurrent form (for inference)**. Update `S` character by character; each layer stores only M·d floating-point numbers, regardless of how long the text is.
159
 
160
- The two forms agree: the self-check compares them bit by bit on **all 64 switch combinations** over 300 characters (crossing several blocks), with float64 error < 1e-8. There is also a fixed reference file `tests/golden/zeropoint_ref.npz`; any future implementation (including a C version in the kernel) can check its answers against it. **A trained model can be put directly into the kernel in recurrent form**: no KV cache, a fixed M × d state per layer, no dynamic allocation.
161
 
162
  About "maximum path O(1)", we must be honest: information written at u can indeed be read at t in one step, but it has already been multiplied by λ^(t−u). Attention can "retrieve verbatim" a character from arbitrarily far away; ZeroPoint can only retrieve information **that has not yet been forgotten and has not been washed out by other writes**.
163
 
 
157
 
158
  **Recurrent form (for inference)**. Update `S` character by character; each layer stores only M·d floating-point numbers, regardless of how long the text is.
159
 
160
+ The two forms agree: the self-check compares them bit by bit on **all 64 switch combinations** over 300 characters (crossing several blocks), with float64 error < 1e-8. There is also a fixed reference file `tests/golden/zeropoint_ref.json`; any future implementation (including a C version in the kernel) can check its answers against it. **A trained model can be put directly into the kernel in recurrent form**: no KV cache, a fixed M × d state per layer, no dynamic allocation.
161
 
162
  About "maximum path O(1)", we must be honest: information written at u can indeed be read at t in one step, but it has already been multiplied by λ^(t−u). Attention can "retrieve verbatim" a character from arbitrarily far away; ZeroPoint can only retrieve information **that has not yet been forgotten and has not been washed out by other writes**.
163
 
scripts/upload.md CHANGED
@@ -69,7 +69,7 @@ hf upload <your-username>/ZeroPoint . --type model \
69
  ```
70
 
71
  This is a single-commit upload of the whole tree. It includes
72
- `tests/golden/zeropoint_ref.npz` (the fixed reference data required by the self-checks)
73
  and must not include model weights, training runs, corpora, or tokens — see
74
  `.gitignore`; the release tree itself contains none of those.
75
 
 
69
  ```
70
 
71
  This is a single-commit upload of the whole tree. It includes
72
+ `tests/golden/zeropoint_ref.json` (the fixed reference data required by the self-checks)
73
  and must not include model weights, training runs, corpora, or tokens — see
74
  `.gitignore`; the release tree itself contains none of those.
75
 
tests/golden/zeropoint_ref.json ADDED
The diff for this file is too large to render. See raw diff
 
tests/make_golden.py CHANGED
@@ -1,10 +1,11 @@
1
- """生成 ZeroPoint 的参考数据 tests/golden/zeropoint_ref.npz:固定的参数 + 输入 + 标准输出。
2
 
3
  所有实现(zerocg 并行形式、numpy 递推形式、以后内核里的 C 版本)都拿它对答案,
4
  谁和它对不上谁就有 bug。**只有在有意修改架构时才重新生成**,并在提交说明里写清楚为什么。
5
 
6
  python3 tests/make_golden.py
7
  """
 
8
  import os
9
  import sys
10
 
@@ -26,7 +27,11 @@ if __name__ == "__main__":
26
  p.data = p.data + np.random.randn(*p.shape) * 0.3
27
  idx = np.random.randint(0, CONFIG["vocab_size"], size=(2, 70)) # 跨 3 个块
28
  logits = model(idx).data
29
- out = os.path.join(HERE, "golden", "zeropoint_ref.npz")
30
- np.savez(out, idx=idx, logits=logits, **{"param:" + k: v for k, v in model.state_dict().items()},
31
- **{"config:" + k: v for k, v in CONFIG.items()})
 
 
 
 
32
  print(f"写入 {out}:输入 {idx.shape},输出 {logits.shape},{len(model.state_dict())} 个参数张量")
 
1
+ """生成 ZeroPoint 的参考数据 tests/golden/zeropoint_ref.json:固定的参数 + 输入 + 标准输出。
2
 
3
  所有实现(zerocg 并行形式、numpy 递推形式、以后内核里的 C 版本)都拿它对答案,
4
  谁和它对不上谁就有 bug。**只有在有意修改架构时才重新生成**,并在提交说明里写清楚为什么。
5
 
6
  python3 tests/make_golden.py
7
  """
8
+ import json
9
  import os
10
  import sys
11
 
 
27
  p.data = p.data + np.random.randn(*p.shape) * 0.3
28
  idx = np.random.randint(0, CONFIG["vocab_size"], size=(2, 70)) # 跨 3 个块
29
  logits = model(idx).data
30
+ out = os.path.join(HERE, "golden", "zeropoint_ref.json")
31
+ data = {"idx": idx.tolist(), "logits": logits.tolist()}
32
+ data.update({"param:" + k: v.tolist() for k, v in model.state_dict().items()})
33
+ data.update({"config:" + k: int(v) for k, v in CONFIG.items()})
34
+ with open(out, "w", encoding="utf-8") as f:
35
+ json.dump(data, f, separators=(",", ":"), ensure_ascii=False)
36
+ f.write("\n")
37
  print(f"写入 {out}:输入 {idx.shape},输出 {logits.shape},{len(model.state_dict())} 个参数张量")
tests/test_zeropoint.py CHANGED
@@ -1,5 +1,6 @@
1
  """ZeroPoint 架构自检。运行: python3 tests/test_zeropoint.py"""
2
  import itertools
 
3
  import os
4
  import sys
5
 
@@ -35,13 +36,16 @@ def test_parallel_equals_recurrent():
35
 
36
 
37
  def test_matches_golden_reference():
38
- """和 tests/golden/zeropoint_ref.npz 对答案:防止以后改代码时并行或递推形式悄悄变了。"""
39
- ref = np.load(os.path.join(HERE, "golden", "zeropoint_ref.npz"))
40
- cfg = {k[7:]: int(ref[k]) for k in ref.files if k.startswith("config:")}
 
41
  model = ZeroX(**cfg).eval()
42
- model.load_state_dict({k[6:]: ref[k] for k in ref.files if k.startswith("param:")})
43
- assert np.abs(model(ref["idx"]).data - ref["logits"]).max() < 1e-10
44
- assert np.abs(model.forward_recurrent(ref["idx"]) - ref["logits"]).max() < 1e-8
 
 
45
 
46
 
47
  def test_causal_across_chunks():
 
1
  """ZeroPoint 架构自检。运行: python3 tests/test_zeropoint.py"""
2
  import itertools
3
+ import json
4
  import os
5
  import sys
6
 
 
36
 
37
 
38
  def test_matches_golden_reference():
39
+ """和 tests/golden/zeropoint_ref.json 对答案:防止以后改代码时并行或递推形式悄悄变了。"""
40
+ with open(os.path.join(HERE, "golden", "zeropoint_ref.json"), encoding="utf-8") as f:
41
+ ref = json.load(f)
42
+ cfg = {k[7:]: int(v) for k, v in ref.items() if k.startswith("config:")}
43
  model = ZeroX(**cfg).eval()
44
+ model.load_state_dict({k[6:]: np.array(v, dtype=np.float64) for k, v in ref.items()
45
+ if k.startswith("param:")})
46
+ idx, logits = np.array(ref["idx"]), np.array(ref["logits"])
47
+ assert np.abs(model(idx).data - logits).max() < 1e-10
48
+ assert np.abs(model.forward_recurrent(idx) - logits).max() < 1e-8
49
 
50
 
51
  def test_causal_across_chunks():