gustavecortal commited on
Commit
c538c43
·
verified ·
1 Parent(s): a743ed7

Model card: highlights and model overview, parameters counted as for n-gram models

Browse files
Files changed (1) hide show
  1. README.md +24 -12
README.md CHANGED
@@ -35,18 +35,30 @@ The problem of personal identity is the problem of how we can be both identical
35
  other selves.
36
  ```
37
 
38
- ## Model
39
-
40
- | | |
41
- |---|---|
42
- | Parameters (non-embedding) | 61.4M, in the 16 transformer blocks |
43
- | Parameters (total) | 111.7M, with the tied token embedding (18.9M) and a hashed bigram table (31.5M): both are lookups, read and not multiplied |
44
- | Blocks | 16, width 576, 9 query heads of 64 dimensions, SwiGLU feed-forward of 1,512 |
45
- | Attention | local (a window of 256 tokens) in 12 blocks, global in every fourth block with 3 key-value heads (grouped-query attention), QK-norm, a sigmoid gate on each head's output |
46
- | Positions | rotary embeddings on 32 of each head's 64 dimensions (base 10,000). In global blocks the other 32 key dimensions come from the previous token |
47
- | Bigram table | 245,760 hashed rows of 128 values with a fixed ±1 sign pattern, added to the first 128 residual dimensions before each block with a learned weight per block |
48
- | Context | 4,096 tokens. Documents are separated by the end-of-text token, and each attends only to itself |
49
- | Tokenizer | byte-level BPE, 32,768 tokens, trained on the training text |
 
 
 
 
 
 
 
 
 
 
 
 
50
 
51
  ## Training
52
 
 
35
  other selves.
36
  ```
37
 
38
+ ## Highlights
39
+
40
+ - **N-gram embedding**: a hashed table of 245,760 bigram rows adds what the previous token says about the current one, before every block. It holds 31M parameters that are read, not multiplied, so they cost almost no compute.
41
+ - **Hybrid attention**: three local blocks (a window of 256 tokens) for each global block, which uses grouped-query attention with 3 key-value heads.
42
+ - **Training recipe**: Muon for hidden matrices and AdamW for the rest, four passes with each document read at most four times, and the final weights blended with their moving average.
43
+ - **Runs in the browser**: an 8-bit WebAssembly engine on up to four threads, at 0.2% bits per byte above float32.
44
+
45
+ ## Model overview
46
+
47
+ - Type: causal language model (base, not instruction-tuned)
48
+ - Training stage: pre-training
49
+ - Number of parameters: 80M (61M non-embedding), plus 31M n-gram embedding
50
+ - Hidden dimension: 576
51
+ - Token embedding: 32,768, tied with the output layer
52
+ - N-gram embedding: 245,760 rows of 128 values (bigrams, with a fixed ±1 sign pattern), added to the first 128 hidden dimensions before every block
53
+ - Number of layers: 16
54
+ - Hidden layout: 4 × (3 × (local attention → SwiGLU) → 1 × (global attention → SwiGLU))
55
+ - Local attention: 9 heads of 64 dimensions, a window of 256 tokens
56
+ - Global attention: 9 query heads and 3 key-value heads of 64 dimensions. The 32 key dimensions without rotary embedding come from the previous token
57
+ - Rotary position embedding dimension: 32 of 64 (base 10,000)
58
+ - Attention: QK-norm and a sigmoid gate on each head's output
59
+ - SwiGLU intermediate dimension: 1,512
60
+ - Context length: 4,096 tokens. Documents are separated by the end-of-text token, and each attends only to itself
61
+ - Tokenizer: byte-level BPE trained on the training text
62
 
63
  ## Training
64