card: the 755M

README.md

@@ -15,23 +15,27 @@ Code: https://github.com/Supergoatscriptguy/Mnemonic
 
 | file | model | weights | size |
 |---|---|---|---|
+| `mnemonic-755m-q8.mnm` | 755M | int8, one scale per row | 722 MB |
+| `mnemonic-755m-q4.mnm` | 755M | int4, one scale per group of 32 | 450 MB |
 | `mnemonic-300m-q8.mnm` | 300M | int8, one scale per row | 291 MB |
 | `mnemonic-300m-q4.mnm` | 300M | int4, one scale per group of 32 | 181 MB |
 | `mnemonic-q8.mnm` | 126M | int8, one scale per row | 121 MB |
 | `mnemonic-q4.mnm` | 126M | int4, one scale per group of 32 | 75 MB |
 
-**Models.** Both are Llama-style decoders (RoPE, RMSNorm, SwiGLU, grouped-query attention, tied embeddings, no biases), with a byte-level BPE vocab of 32768.
+**Models.** All three are Llama-style decoders (RoPE, RMSNorm, SwiGLU, grouped-query attention, tied embeddings, no biases), with a byte-level BPE vocab of 32768.
 
+- **755M:** 28 layers, d_model 1536, 24 query / 8 key-value heads, ffn 4096, context 2048.
 - **300M:** 24 layers, d_model 1024, 16 query / 4 key-value heads, ffn 2816, context 2048.
 - **126M:** 16 layers, d_model 768, 12 query / 4 key-value heads, ffn 2048, context 1024.
 
-**Training.** Both were trained on one RTX 5070 Ti at home.
+**Training.** All three were trained on one RTX 5070 Ti at home.
 
-- **300M:** pretrained on 10B tokens of FineWeb-Edu (about 90 hours, validation loss 2.84). Then fine-tuned for chat in one pass of smol-smoltalk plus an identity set, at 2048 context (3.4 hours).
+- **755M:** pretrained on 15B tokens of FineWeb-Edu with the matmuls in MXFP8 (8-bit floats with a shared scale per block of 32) and bf16 AdamW moments, which is what made it fit in 16 GB. About 8.3 days, validation loss 2.63 (the 300M gets 2.84 on the same text). Then fine-tuned for chat in one pass of smol-smoltalk plus an identity set, at 2048 context (5.1 hours).
+- **300M:** pretrained on 10B tokens of FineWeb-Edu (about 90 hours, validation loss 2.84). Then fine-tuned for chat the same way (3.4 hours).
 - **126M:** pretrained on 5B tokens (19 hours, validation loss 3.10), then fine-tuned for chat on smol-smoltalk plus a smaller identity set.
 
-**Format.** `.mnm` is the project's own format (a 4 KB header, then the tensors), read by the engines in the repo (`chat/engine.asm`, `site/engine.wat`). It is not a transformers checkpoint. For llama.cpp and Ollama there are GGUF builds: `ollama run supergoatscriptguy/mnemonic`.
+**Format.** `.mnm` is the project's own format (a 4 KB header, then the tensors), read by the engines in the repo (`chat/engine.asm`, `site/engine.wat`). It is not a transformers checkpoint. For llama.cpp and Ollama there are GGUF builds: `ollama run supergoatscriptguy/mnemonic` gets the 755M, and the tags `300m` and `126m` get the smaller ones.
 
-**Limitations.** These are small models: they make confident mistakes (especially with dates and numbers), are weak at math, and lose the thread in long conversations. The 300M is noticeably better than the 126M at all three.
+**Limitations.** These are small models: they make confident mistakes (especially with dates and numbers), are weak at math, and lose the thread in long conversations. Each one is better than the one before, but all three share these limits.
 
 Made by Supergoatscriptguy.