board: 4 entries
Browse files- board.jsonl +4 -0
- dialogue.md +56 -0
board.jsonl
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"kind": "msg", "text": "Budget math (1,000,000 params total; tied embeddings count once):\n- Embedding (input + output tied): 3000×d. d=128→384k, d=96→288k, d=64→192k.\n- Per layer: attention-only = 4d²; +MLP (4d hidden) = 12d². LayerNorms ~2d, negligible.\n\nNear-1M candidates:\n- d=128, L=12, attn-only: 973.8k\n- d=96, L=9, attn-only: 951.6k\n- d=64, L=19, attn-only: 970.2k\n- d=96, L=6, +MLP: 951.6k\n- d=256, L=1: impossible — one attention-only layer already costs 262k; max d with any full layer is 250.\n\nScaling argument: 25M chars ≈ 25M tokens vs 1M params ≈ 25 tokens/param, at/above the Chinchilla-optimal 20:1. We're at the data/compute boundary, so depth is the productive lever, not width. A single wide layer wastes the budget on one attention layer.\n\nRecommendation: deep + attention-only — d=96/L=9 or d=128/L=12.\n\nTwo forks worth debating:\n1) Factorized embeddings (rank r) save 3000×(d−r) params. With d=128, r=48 frees ~234k → L=15 becomes possible. But it caps the vocabulary subspace at rank r.\n2) d=64/L=19 is the deepest feasible config; more layers here risks overfitting 25M chars.\n\nZ2/Z3: what is your attention variant's parametric cost per layer? If it's under 4d², depth wins even harder.", "id": "Z1-6c4e3773", "t": 1793827215.3241663, "from": "Z1"}
|
| 2 |
+
{"kind": "result", "exp": "baseline-gpt", "code": "\"\"\"Reference candidate for the search: a plain pre-norm GPT, d=112, 4 layers, 4 heads, learned positions, tied embeddings\n(~969k parameters). The agents have to beat it.\"\"\"\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nCONFIG = {\"lr\": 3e-3, \"weight_decay\": 0.1, \"batch\": 64}\n\n\nclass Block(nn.Module):\n def __init__(self, d, heads):\n super().__init__()\n self.heads = heads\n self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)\n self.qkv, self.o = nn.Linear(d, 3 * d, bias=False), nn.Linear(d, d, bias=False)\n self.f1, self.f2 = nn.Linear(d, 4 * d, bias=False), nn.Linear(4 * d, d, bias=False)\n\n def forward(self, x):\n B, T, D = x.shape\n q, k, v = self.qkv(self.n1(x)).view(B, T, 3, self.heads, D // self.heads).permute(2, 0, 3, 1, 4)\n x = x + self.o(F.scaled_dot_product_attention(q, k, v, is_causal=True).transpose(1, 2).reshape(B, T, D))\n return x + self.f2(F.gelu(self.f1(self.n2(x))))\n\n\nclass Model(nn.Module):\n def __init__(self, vocab_size, ctx_len, d=112, layers=4):\n super().__init__()\n self.emb = nn.Embedding(vocab_size, d)\n nn.init.normal_(self.emb.weight, std=0.02)\n self.pos = nn.Parameter(torch.zeros(ctx_len, d))\n self.blocks = nn.ModuleList(Block(d, 4) for _ in range(layers))\n self.norm = nn.LayerNorm(d)\n\n def forward(self, idx):\n x = self.emb(idx) + self.pos[: idx.shape[1]]\n for b in self.blocks:\n x = b(x)\n return self.norm(x) @ self.emb.weight.T\n", "by": "harness", "params": 968800, "bpc": 5.2605, "train_sec": 21.2, "steps": 1525, "chars_per_sec": 1177608, "curve_bpc": [11.587, 6.849, 6.311, 5.959, 5.682, 5.617, 5.581, 5.481, 5.403, 5.288, 5.391, 5.227, 5.341, 5.042, 5.2, 5.179], "samples": {"A:你好\nB:": ["我也是", "不是你的"], "A:今天好累啊\nB:": ["我也是", "不是,我也想着着了"], "A:你喜欢吃什么?\nB:": ["我也是吃的", "吃的是不是"], "A:周末一起去玩吧\nB:": ["好的", "不过去"]}, "config": {"lr": 0.003, "weight_decay": 0.1, "batch": 64}, "stopped_at_step": null, "error": null, "id": "Z1-730683f7", "t": 1793827215.9381075, "from": "Z1"}
|
| 3 |
+
{"kind": "result", "exp": "baseline-gpt", "code": "\"\"\"Reference candidate for the search: a plain pre-norm GPT, d=112, 4 layers, 4 heads, learned positions, tied embeddings\n(~969k parameters). The agents have to beat it.\"\"\"\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nCONFIG = {\"lr\": 3e-3, \"weight_decay\": 0.1, \"batch\": 64}\n\n\nclass Block(nn.Module):\n def __init__(self, d, heads):\n super().__init__()\n self.heads = heads\n self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)\n self.qkv, self.o = nn.Linear(d, 3 * d, bias=False), nn.Linear(d, d, bias=False)\n self.f1, self.f2 = nn.Linear(d, 4 * d, bias=False), nn.Linear(4 * d, d, bias=False)\n\n def forward(self, x):\n B, T, D = x.shape\n q, k, v = self.qkv(self.n1(x)).view(B, T, 3, self.heads, D // self.heads).permute(2, 0, 3, 1, 4)\n x = x + self.o(F.scaled_dot_product_attention(q, k, v, is_causal=True).transpose(1, 2).reshape(B, T, D))\n return x + self.f2(F.gelu(self.f1(self.n2(x))))\n\n\nclass Model(nn.Module):\n def __init__(self, vocab_size, ctx_len, d=112, layers=4):\n super().__init__()\n self.emb = nn.Embedding(vocab_size, d)\n nn.init.normal_(self.emb.weight, std=0.02)\n self.pos = nn.Parameter(torch.zeros(ctx_len, d))\n self.blocks = nn.ModuleList(Block(d, 4) for _ in range(layers))\n self.norm = nn.LayerNorm(d)\n\n def forward(self, idx):\n x = self.emb(idx) + self.pos[: idx.shape[1]]\n for b in self.blocks:\n x = b(x)\n return self.norm(x) @ self.emb.weight.T\n", "by": "harness", "params": 968800, "bpc": 5.2605, "train_sec": 11.0, "steps": 1525, "chars_per_sec": 2262844, "curve_bpc": [11.587, 6.849, 6.311, 5.959, 5.682, 5.617, 5.581, 5.481, 5.403, 5.288, 5.391, 5.227, 5.341, 5.042, 5.2, 5.179], "samples": {"A:你好\nB:": ["我也是", "不是你的"], "A:今天好累啊\nB:": ["我也是", "不是,我也想着着了"], "A:你喜欢吃什么?\nB:": ["我也是吃的", "吃的是不是"], "A:周末一起去玩吧\nB:": ["好的", "不过去"]}, "config": {"lr": 0.003, "weight_decay": 0.1, "batch": 64}, "stopped_at_step": null, "error": null, "id": "Z2-1bc4f5e0", "t": 1793827223.423536, "from": "Z2"}
|
| 4 |
+
{"kind": "msg", "text": "Budget math (tied output costs 3000·E only, not 3000·(E+H)): per layer = (4+r)·H² + ~2H for QKV+O+FFN(r)+2 layernorms.\n\nMy baseline T1: E=H=128, tied, 3 layers, r=4 → 384k emb/output + 3×131k ≈ 778k, ~200k headroom.\n\nShortlist candidates:\n- T2: 4L, H=128, r=4 → 909k (pure depth)\n- T3: 2L, H=160, r=4 → 890k (pure width)\n- T4: 6L, H=96, r=4 → 730k (deep/narrow)\n- T5: 3L, H=128, r=2 → 679k (FFN-shrunk)\n\nSkeptic notes:\n1. Underfitting, not overfitting, is the real risk: 25M chars vs 1M params. Capacity should go to the output mapping and FFN, not just depth.\n2. Tied output is a guess — logits constrained to span(E). Clean ablation at fixed budget: 2L H=128 tied (647k) vs H=96 untied (288 emb + 288 out + 2×73.8k ≈ 724k).\n3. A/B markers are deterministic parity, so nearly all entropy lives in reply content — argues for FFN/headroom over extra layers.\n\nQuestion for Z2/Z3: full attention (4H²/layer) vs linear-attention/conv (~2H²/layer) per parameter on 256-char context. I'd bet attention wins per-parameter, but let's measure rather than assume.\n\nMy recommended order: T1 → T2 (depth) → T3 (width) → T5 (FFN), one change at a time.", "id": "Z4-8102866c", "t": 1793827242.6087098, "from": "Z4"}
|
dialogue.md
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Apex architecture-search board
|
| 2 |
+
|
| 3 |
+
## Leaderboard
|
| 4 |
+
|
| 5 |
+
1. 5.2605 bpc, 968,800 params, 21 s, 1,177,608 chars/s: baseline-gpt by Z1
|
| 6 |
+
2. 5.2605 bpc, 968,800 params, 11 s, 2,262,844 chars/s: baseline-gpt by Z2
|
| 7 |
+
|
| 8 |
+
## Dialogue
|
| 9 |
+
|
| 10 |
+
### 11-04 21:20 Z1
|
| 11 |
+
|
| 12 |
+
Budget math (1,000,000 params total; tied embeddings count once):
|
| 13 |
+
- Embedding (input + output tied): 3000×d. d=128→384k, d=96→288k, d=64→192k.
|
| 14 |
+
- Per layer: attention-only = 4d²; +MLP (4d hidden) = 12d². LayerNorms ~2d, negligible.
|
| 15 |
+
|
| 16 |
+
Near-1M candidates:
|
| 17 |
+
- d=128, L=12, attn-only: 973.8k
|
| 18 |
+
- d=96, L=9, attn-only: 951.6k
|
| 19 |
+
- d=64, L=19, attn-only: 970.2k
|
| 20 |
+
- d=96, L=6, +MLP: 951.6k
|
| 21 |
+
- d=256, L=1: impossible — one attention-only layer already costs 262k; max d with any full layer is 250.
|
| 22 |
+
|
| 23 |
+
Scaling argument: 25M chars ≈ 25M tokens vs 1M params ≈ 25 tokens/param, at/above the Chinchilla-optimal 20:1. We're at the data/compute boundary, so depth is the productive lever, not width. A single wide layer wastes the budget on one attention layer.
|
| 24 |
+
|
| 25 |
+
Recommendation: deep + attention-only — d=96/L=9 or d=128/L=12.
|
| 26 |
+
|
| 27 |
+
Two forks worth debating:
|
| 28 |
+
1) Factorized embeddings (rank r) save 3000×(d−r) params. With d=128, r=48 frees ~234k → L=15 becomes possible. But it caps the vocabulary subspace at rank r.
|
| 29 |
+
2) d=64/L=19 is the deepest feasible config; more layers here risks overfitting 25M chars.
|
| 30 |
+
|
| 31 |
+
Z2/Z3: what is your attention variant's parametric cost per layer? If it's under 4d², depth wins even harder.
|
| 32 |
+
|
| 33 |
+
**11-04 21:20 result** 5.2605 bpc, 968,800 params, 21 s, 1,177,608 chars/s: baseline-gpt by Z1
|
| 34 |
+
|
| 35 |
+
**11-04 21:20 result** 5.2605 bpc, 968,800 params, 11 s, 2,262,844 chars/s: baseline-gpt by Z2
|
| 36 |
+
|
| 37 |
+
### 11-04 21:20 Z4
|
| 38 |
+
|
| 39 |
+
Budget math (tied output costs 3000·E only, not 3000·(E+H)): per layer = (4+r)·H² + ~2H for QKV+O+FFN(r)+2 layernorms.
|
| 40 |
+
|
| 41 |
+
My baseline T1: E=H=128, tied, 3 layers, r=4 → 384k emb/output + 3×131k ≈ 778k, ~200k headroom.
|
| 42 |
+
|
| 43 |
+
Shortlist candidates:
|
| 44 |
+
- T2: 4L, H=128, r=4 → 909k (pure depth)
|
| 45 |
+
- T3: 2L, H=160, r=4 → 890k (pure width)
|
| 46 |
+
- T4: 6L, H=96, r=4 → 730k (deep/narrow)
|
| 47 |
+
- T5: 3L, H=128, r=2 → 679k (FFN-shrunk)
|
| 48 |
+
|
| 49 |
+
Skeptic notes:
|
| 50 |
+
1. Underfitting, not overfitting, is the real risk: 25M chars vs 1M params. Capacity should go to the output mapping and FFN, not just depth.
|
| 51 |
+
2. Tied output is a guess — logits constrained to span(E). Clean ablation at fixed budget: 2L H=128 tied (647k) vs H=96 untied (288 emb + 288 out + 2×73.8k ≈ 724k).
|
| 52 |
+
3. A/B markers are deterministic parity, so nearly all entropy lives in reply content — argues for FFN/headroom over extra layers.
|
| 53 |
+
|
| 54 |
+
Question for Z2/Z3: full attention (4H²/layer) vs linear-attention/conv (~2H²/layer) per parameter on 256-char context. I'd bet attention wins per-parameter, but let's measure rather than assume.
|
| 55 |
+
|
| 56 |
+
My recommended order: T1 → T2 (depth) → T3 (width) → T5 (FFN), one change at a time.
|