SuperGoatScriptGuy commited on
Commit
678223f
·
verified ·
1 Parent(s): ab7789f

card: the math model now has 38 kinds and an Ollama tag

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -36,10 +36,10 @@ Code: https://github.com/Supergoatscriptguy/Mnemonic
36
  - **300M:** pretrained on 10B tokens of FineWeb-Edu (about 90 hours, validation loss 2.84). Then fine-tuned for chat the same way (3.4 hours).
37
  - **126M:** pretrained on 5B tokens (19 hours, validation loss 3.10), then fine-tuned for chat on smol-smoltalk plus a smaller identity set.
38
 
39
- **Mnemonic Math.** A separate model that only does math: the 300M, fine-tuned on about 15 million generated problems (3B tokens, about 26 hours) that are worked out one small step at a time, in 31 kinds: column arithmetic and long division, fractions, decimals, percents, ratios, powers, square roots, prime factorization, gcd and lcm, order of operations, linear and quadratic equations, systems, inequalities, absolute value, expanding and factoring, polynomial division, lines, sequences, statistics, geometry and word problems. It gets 99.7% of 3,085 held-out problems right (the misses are wording, not math) and all 66 of a set of questions typed the way people type them. The int4 file scores the same as the int8 on all of it. It knows only those kinds of problems, and answers anything else wrongly. Run it greedy: `bin\chat model=models\mnemonic-math-q8.mnm temp=0 max=2000`. There's no GGUF of it yet, because llama.cpp splits numbers into tokens differently from this tokenizer.
40
 
41
- **Format.** `.mnm` is the project's own format (a 4 KB header, then the tensors), read by the engines in the repo (`chat/engine.asm`, `site/engine.wat`). It is not a transformers checkpoint. For llama.cpp and Ollama there are GGUF builds: `ollama run supergoatscriptguy/mnemonic` gets the 755M, and the tags `300m` and `126m` get the smaller ones.
42
 
43
- **Limitations.** The chat models are small: they make confident mistakes (especially with dates and numbers), are weak at math, and lose the thread in long conversations. Each one is better than the one before, but all three share these limits. The math model is the other way around: good at its 31 kinds of problems and no use for anything else.
44
 
45
  Made by Supergoatscriptguy.
 
36
  - **300M:** pretrained on 10B tokens of FineWeb-Edu (about 90 hours, validation loss 2.84). Then fine-tuned for chat the same way (3.4 hours).
37
  - **126M:** pretrained on 5B tokens (19 hours, validation loss 3.10), then fine-tuned for chat on smol-smoltalk plus a smaller identity set.
38
 
39
+ **Mnemonic Math.** A separate model that only does math: the 300M, fine-tuned on about 22 million generated problems (4.6B tokens, about 40 hours) that are worked out one small step at a time, in 38 kinds: column arithmetic and long division, fractions, decimals, percents, ratios, powers, square roots, prime factorization, gcd and lcm, order of operations, linear and quadratic equations, systems, inequalities, absolute value, expanding and factoring, polynomial division, lines, sequences, statistics, geometry, unit conversions, money (discounts, tax, tips, change, interest), percent change, probability, equations like x² = 49, logarithms, derivatives of polynomials and word problems. It gets 99.6% of 3,085 held-out problems right (all but one of the misses are wording, not math), 697 of 700 of the 7 newest kinds, and 196 of 201 questions typed the way people type them. The int4 file scores the same on the held-out problems and gets 198 of the typed ones. Run it greedy: `bin\chat model=models\mnemonic-math-q8.mnm temp=0 max=2000`, or `ollama run supergoatscriptguy/mnemonic:math`.
40
 
41
+ **Format.** `.mnm` is the project's own format (a 4 KB header, then the tensors), read by the engines in the repo (`chat/engine.asm`, `site/engine.wat`). It is not a transformers checkpoint. For llama.cpp and Ollama there are GGUF builds: `ollama run supergoatscriptguy/mnemonic` gets the 755M, the tags `300m` and `126m` get the smaller ones, and `math` gets the math model.
42
 
43
+ **Limitations.** The chat models are small: they make confident mistakes (especially with dates and numbers), are weak at math, and lose the thread in long conversations. Each one is better than the one before, but all three share these limits. The math model is the other way around: good at its 38 kinds of problems and no use for anything else.
44
 
45
  Made by Supergoatscriptguy.