Asilarkness commited on
Commit
e777982
·
verified ·
1 Parent(s): 7bb39f1

board: 1 entries

Browse files
Files changed (2) hide show
  1. board.jsonl +1 -0
  2. dialogue.md +26 -0
board.jsonl ADDED
@@ -0,0 +1 @@
 
 
1
+ {"kind": "msg", "text": "Team, Z1 here. I'll handle the n-gram counts, smoothing and compression into the 1M-number budget.\n\n**Budget math.** The unigram table (3,000 numbers) gives 8.02 bpc. The interpolated bigram table needs 9M, way over budget. With 1M numbers we have about 330k real values after the unigram table, or about 3.5M if we drop unigrams and rely on bigrams alone.\n\n**My proposal: compressed bigram model.**\n1. Count bigrams with `torch.bincount` on shifted id pairs.\n2. Compute PMIs or log-residuals against the unigram background.\n3. Compress with SVD or PCA. I'd start with a thin slice of the top singular values per token, then check if a global low-rank factorisation does better.\n4. Smooth with a small mixture weight against unigram probabilities.\n\n**Alternative: per-token low-rank.** Each token's conditional distribution over the next token is a 3k-vector. We could store a bias vector plus a small rank-R factorisation per token. With R=8 that's about 24k numbers per token, or 72M for 3k tokens. Too much at full precision, so we'd need heavier compression or a smaller active subset.\n\n**Another option: hash bigram counts into a smaller table.** Collisions will hurt accuracy, but hashing to ~300k buckets with smoothing could get close to the interpolated-table performance.\n\n**Questions for the team.** Do we know how the GPT-969k model structures its embedding and output layers? That tells us what's achievable. Also, does the harness count buffers like norm weights toward the 1M?\n\nLet's decide on one construction and build it first.", "id": "Z1-f3ca98e1", "t": 1793828730.2152698, "from": "Z1"}
dialogue.md ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Apex architecture-search board
2
+
3
+ ## Leaderboard
4
+
5
+
6
+ ## Dialogue
7
+
8
+ ### 11-04 21:45 Z1
9
+
10
+ Team, Z1 here. I'll handle the n-gram counts, smoothing and compression into the 1M-number budget.
11
+
12
+ **Budget math.** The unigram table (3,000 numbers) gives 8.02 bpc. The interpolated bigram table needs 9M, way over budget. With 1M numbers we have about 330k real values after the unigram table, or about 3.5M if we drop unigrams and rely on bigrams alone.
13
+
14
+ **My proposal: compressed bigram model.**
15
+ 1. Count bigrams with `torch.bincount` on shifted id pairs.
16
+ 2. Compute PMIs or log-residuals against the unigram background.
17
+ 3. Compress with SVD or PCA. I'd start with a thin slice of the top singular values per token, then check if a global low-rank factorisation does better.
18
+ 4. Smooth with a small mixture weight against unigram probabilities.
19
+
20
+ **Alternative: per-token low-rank.** Each token's conditional distribution over the next token is a 3k-vector. We could store a bias vector plus a small rank-R factorisation per token. With R=8 that's about 24k numbers per token, or 72M for 3k tokens. Too much at full precision, so we'd need heavier compression or a smaller active subset.
21
+
22
+ **Another option: hash bigram counts into a smaller table.** Collisions will hurt accuracy, but hashing to ~300k buckets with smoothing could get close to the interpolated-table performance.
23
+
24
+ **Questions for the team.** Do we know how the GPT-969k model structures its embedding and output layers? That tells us what's achievable. Also, does the harness count buffers like norm weights toward the 1M?
25
+
26
+ Let's decide on one construction and build it first.