gustavecortal commited on
Commit
45d5cc1
·
verified ·
1 Parent(s): 7b8537a

Model card: the two-panel architecture figure, without the paragraph it replaces

Browse files
Files changed (2) hide show
  1. README.md +1 -3
  2. architecture.png +2 -2
README.md CHANGED
@@ -22,9 +22,7 @@ PhiloLM is a language model with 112M parameters, trained from scratch on open p
22
 
23
  ## Architecture
24
 
25
- ![PhiloLM architecture: token embedding, four groups of three local-attention blocks and one global block, a hashed bigram embedding added to every block, and the decoder block](architecture.png)
26
-
27
- Three of every four layers attend within a 256-token window, and every fourth attends over the whole context with grouped-query attention. A hashed table of token pairs (bigrams) adds 31.5M parameters that cost almost no computation, since each token only looks up its row.
28
 
29
  ## Training
30
 
 
22
 
23
  ## Architecture
24
 
25
+ ![PhiloLM architecture in two panels. Model: token IDs feed a token embedding and a bigram embedding, then four groups of four blocks, each with three local attention blocks (256-token window) and one global attention block (9 query heads, 3 key-value heads), a final RMSNorm and an LM head tied to the token embedding. Decoder block: the bigram vector, times a learned scale, is added to the first 128 of 576 dimensions of the residual stream, then RMSNorm and causal attention (QK norm, partial RoPE), then RMSNorm and a SwiGLU feed-forward network, each with a residual connection](architecture.png)
 
 
26
 
27
  ## Training
28
 
architecture.png CHANGED

Git LFS Details

  • SHA256: 3080c2b407d18856978f03bc7ebcf7d0a6dc3982aa262e49f07e3da51d69ea7e
  • Pointer size: 131 Bytes
  • Size of remote file: 375 kB

Git LFS Details

  • SHA256: 023bf626d3ce7ffeef40b0e012893f71bcc26dbc308e20570f39e9cb8eec85a0
  • Pointer size: 131 Bytes
  • Size of remote file: 358 kB