Fix config params to the true unique count (100,936). 106,568 is the file sum that double-counts the tied head; the README explains this.
b3bc31f verified | { | |
| "architectures": ["GPT"], | |
| "vocab_size": 128, | |
| "n_layer": 4, | |
| "n_embd": 44, | |
| "n_head": 2, | |
| "block_size": 512, | |
| "rope_theta": 100000.0, | |
| "ffn": "gelu", | |
| "ffn_mult": 2.667, | |
| "tie_word_embeddings": true, | |
| "params": 100936, | |
| "params_note": "unique param count (tied head counted once); the safetensors file stores the head twice, so its raw numel sum is 106568" | |
| } |