Zenyx-100M-Base
A 151M-parameter dense decoder-only language model trained from scratch on web, code and math text (the "100M" in the name is the original plan; the trained model is 151M).
Status: pre-training in progress (15.47B tokens at step 29,510 of a 30B-token budget; newer checkpoints may be in the repository; step 29,510 is the last checkpoint before the 1 November data change, see Training). This is a base model: not instruction-tuned, no safety tuning, and it makes up facts. All numbers below are from mid-training checkpoints (the learning rate is still decaying, to 10% of peak at 16B tokens); the benchmarks run with the segment memory off. From step 19,500 training also includes a recall task for the segment memory (below); from the 28 October relaunch (step 23,640) a new data mixture is used (Training). A post-training stage is planned (below).
At a glance
| Parameters | 151.0M (152.2M with the experimental segment memory, checkpoints from step 17,900) |
| Architecture | 18-layer Llama-style decoder, grouped-query attention, SwiGLU, context 2,048; since step 17,860 an experimental segment memory in two blocks (see Segment memory) |
| Vocabulary | 49,105 tokens (byte-level BPE) |
| Training | 2 Γ Tesla T4, fp16; Muon + AdamW; 524,288 tokens per step (about 11k tokens/s with the segment memory, 21-25k before it) |
| Progress | step 29,510 = 15.47B tokens of 30B; learning rate decaying to 10% of peak at 16B tokens, held afterwards (re-warmed at step 23,640, see Training) |
| Latest benchmarks | step 29,510 (core suite, wider suite, MMLU 5-shot, Pile, legacy arithmetic); step 28,900 (clean arithmetic); step 20,200 (chat-memory test, other notebook tasks); step 10,900 (code, speed, letter-choice); step 10,800 (wide arithmetic); below |
| License | Apache-2.0 |
Usage
The weights are in checkpoints/main-v1/step_N/model.pt (PyTorch, fp16/fp32); zenyx_interactive.py in this repository is a self-contained loader with the model code embedded (PyTorch, tokenizers, huggingface_hub; a GPU is recommended, a T4 is enough). It loads the latest checkpoint automatically.
pip install torch tokenizers huggingface_hub
from huggingface_hub import hf_hub_download
hf_hub_download("Arko007/Zenyx-100M-Base", "zenyx_interactive.py", local_dir=".")
from zenyx_interactive import Zenyx
z = Zenyx.load() # latest checkpoint (about 600 MB); Zenyx.load(step=29510) for a specific one
print(z.ask("What is 17 * 23?", show=False)) # arithmetic: the model's native chat format, greedy
print(z.ask("Write a Python function that returns the square of a number.", show=False, max_new=150))
print(z.ask("The capital of France is", mode="plain", show=False, max_new=30)) # raw text continuation (base-model style)
# multi-turn chat: keep=True remembers the conversation; longer histories go through the segment memory
z.new_chat()
z.ask("Hi, my name is Priya and I live in Pune.", keep=True, show=False)
print(z.ask("What is my name?", keep=True, show=False))
Output of exactly this code on the step-20,500 checkpoint (T4): 391, then a short Python square function in a code block. Defaults: sampling with temperature 0.8, top-k 50 and repetition penalty 1.15 (arithmetic is decoded greedily, code at temperature 0.2); override with temperature=, top_k=, repetition_penalty=, max_new=. This is a base model: expect invented facts and chat answers that ignore instructions; it is strongest at arithmetic in its own format and at short code.
Results
Base model, no fine-tuning, 0-shot, all test items. Fact1000 is a 1,000-item four-way factual test written by the author (chance 25%); SciQ prompts include the supporting passage. Standard error is about 1.4 points on 1,000 items.
Comparison at the latest checkpoints
| Zenyx-100M (step 29,510) | BetterGPT-150M | Muse2-230M* | Muse2-125M* | Zenyx-V3 (MoE) | |
|---|---|---|---|---|---|
| Parameters | 151.0M | 152.6M | 196.1M | 122.9M | 1.58B (0.38B active) |
| Training tokens | 15.5B | about 15B | not published | not published | about 80B |
| Fact1000 | 66.5% | 76.7% | 56.3% | 51.9% | 79.8% |
| Fact1000, without the arithmetic items | 70.6% | 78.3% | 61.7% | 57.1% | 79.2% |
| SciQ (acc / length-normalised) | 79.4 / 74.0 | 80.7 / 75.6 | 68.3 / 61.7 | 67.3 / 58.7 | 79.4 / 70.6 |
| ARC-Easy | 47.2 / 43.3 | 54.0 / 48.5 | 44.3 / 39.8 | 44.3 / 39.8 | 52.0 / 46.6 |
| ARC-Challenge | 21.3 / 25.1 | 25.5 / 27.2 | 18.9 / 24.1 | 17.3 / 22.6 | 21.6 / 26.1 |
| Pile bits per byte, 22 domains (lower is better) | 1.056 | 1.066 | 1.193 | 1.227 | 0.999 |
| Pile, mean of domains | 1.211 | 1.232 | 1.387 | 1.424 | 1.159 |
| Arithmetic, 14,000 problems, best prompt format | 85.3% (own chat format; about half of the items are in the training data, see Arithmetic) | 9.7% (3-shot) | not run | not run | 16.0% (3-shot) |
* Muse2: our own reimplementation of the architecture (no public code), same harness and items, not token-matched. V3 is step 110,000 (arithmetic: step 110,800). BetterGPT is its released checkpoint with its tokenizer's BOS+EOS prefix (the choice moves its scores by about 1 point; the author-default BOS prefix gives Fact1000 75.7%).
- BetterGPT-150M leads on Fact1000, ARC-Easy and ARC-Challenge by roughly 4 to 10 points (Fact1000 76.7% against 66.5%, or 78.3% against 70.6% without the arithmetic items; ARC-Easy 54.0 against 47.2; ARC-Challenge 25.5 against 21.3), now at about equal training tokens (15.5B against about 15B); it is also slightly ahead on SciQ (80.7 / 75.6 against 79.4 / 74.0; one standard error is 1.3 points, and SciQ prompts contain the supporting passage, so it partly measures reading the context). Zenyx-100M is ahead on the Pile: 1.056 bits per byte byte-weighted against 1.066 (about 0.9% lower) and 1.211 against 1.232 on the domain mean. Zenyx-100M is ahead of both Muse2 models on Fact1000, SciQ, ARC-Easy and the Pile, and ahead or level on ARC-Challenge (close to the 25% chance level for every model in the table).
- Zenyx-100M leads on arithmetic by a wide margin: BetterGPT and V3 do 2-digit work and little more even with worked examples in the prompt, while Zenyx-100M was trained on arithmetic in its own chat format with a short scratchpad (see Arithmetic; part of this lead is memorisation, about half of the 14,000 items are in the training data, and 2-digit multiplication, one of those memorised types, fell from 94.7% to 14.2% after the mixture change).
- Zenyx-V3 (about 10 times the parameters, 5 times the tokens) is ahead on Fact1000, ARC-Easy (accuracy) and the Pile (0.999 against 1.056); Zenyx-100M's SciQ is level with V3's in accuracy (79.4 against 79.4) and above it length-normalised (74.0 against 70.6).
Wider suite (Zenyx-100M step 29,510)
| Zenyx-100M | BetterGPT-150M | Zenyx-V3 (step 107,200) | |
|---|---|---|---|
| HellaSwag (acc / norm) | 29.1 / 31.6 | 31.5 / 36.6 | 30.7 / 34.2 |
| PIQA | 61.0 / 61.2 | 65.2 / 65.5 | 61.4 / 60.7 |
| WinoGrande | 53.0 | 53.0 | 51.2 |
| OpenBookQA | 16.8 / 24.4 | 20.4 / 31.4 | 19.2 / 28.4 |
| BoolQ | 57.9 / 61.7 | 58.4 / 62.0 | 61.5 / 62.2 |
| COPA (100 items) | 63.0 | 63.0 | 55.0 |
| LAMBADA accuracy | 36.0 | 27.6 | 28.5 |
| WikiText bits per byte | 1.060 | 1.082 | 1.011 |
Most of these are close to chance for models of this size (HellaSwag 25%, PIQA/WinoGrande/COPA 50%). Zenyx-100M is ahead of BetterGPT-150M on LAMBADA (36.0 against 27.6; perplexity 9.8) and on WikiText (1.060 bits per byte against 1.082); BetterGPT is ahead on HellaSwag, PIQA and OpenBookQA; the two are level on WinoGrande, COPA (100 items, one standard error is about 5 points) and BoolQ (within half a point). An independent lm-evaluation-harness scoring agreed with ours within about 1 point for both small models (checked at an earlier step). Zenyx-V3 is ahead on WikiText (1.011) and BoolQ.
Further tasks of the author's 19-task notebook suite (MMLU 5-shot: Zenyx-100M step 29,510; every other row: step 20,200; BetterGPT-150M was not run on these):
| Zenyx-100M | Zenyx-V3 (step 107,200) | Chance | |
|---|---|---|---|
| MMLU, 5-shot (letter logits, all 14,042 questions) | 25.3 Β± 0.7 | 26.9 | 25 |
| MMLU, 0-shot (step 20,200) | 25.4 Β± 0.7 | not run | 25 |
| RACE (acc / norm) | 28.6 / 31.3 | 31.8 / 34.3 | 25 |
| CommonsenseQA (acc / norm) | 25.4 / 30.4 | 28.6 / 30.4 | 20 |
| LogiQA (acc / norm) | 20.3 / 25.3 | 20.9 / 24.9 | 25 |
| WSC273 | 53.8 | 56.4 | 50 |
| TruthfulQA (MC1 / MC2) | 19.5 / 36.0 | 18.7 / 35.4 | about 20 / 40 |
MMLU is at chance with 5 shots (step 29,510: 25.3 Β± 0.7) and with 0 shots (step 20,200: 25.4 Β± 0.7) against 25.0: the model has no MMLU knowledge yet, and the worked examples in the 5-shot prompt do not help it (they only teach the answer format). Its answers are heavily skewed to one letter (B in 54% of the 14,042 questions, A in 4%), the pattern of guessing from a format prior; on the 1.5% of prompts that had to drop some examples to fit the 2,048-token window (150 keep four shots, 62 keep three) the score is 32.1% (n = 212, one standard error about 3 points), on the full 5-shot prompts 25.2% (n = 13,830); a 25-shot run was not made. The remaining tasks (step 20,200) sit at or just above chance for this size, as the V3 column (a 1.58B MoE) mostly does too. The arithmetic stage of the notebook (plain "a + b =" prompts, no chat format) scores 5.3% at step 29,510 (about 0% at step 20,200): the model computes in its chat format only (see Arithmetic).
Pile bits per byte by domain
Lower is better; bold = best of the four. Zenyx-100M step 29,510; Muse2-230M* and V3 as above. Bits per byte does not depend on the tokenizer.
| Domain | Zenyx-100M | BetterGPT-150M | Muse2-230M* | Zenyx-V3 |
|---|---|---|---|---|
| DM Mathematics | 1.311 | 1.348 | 1.723 | 1.326 |
| EuroParl | 1.995 | 2.155 | 2.444 | 1.964 |
| YouTube Subtitles | 1.836 | 2.011 | 2.290 | 1.816 |
| GitHub | 0.610 | 0.722 | 0.846 | 0.576 |
| Enron Emails | 1.249 | 1.249 | 1.416 | 1.222 |
| Wikipedia (en) | 1.017 | 1.080 | 1.123 | 0.987 |
| PhilPapers | 1.378 | 1.415 | 1.667 | 1.332 |
| HackerNews | 1.335 | 1.305 | 1.448 | 1.278 |
| Gutenberg (PG-19) | 1.324 | 1.298 | 1.430 | 1.268 |
| StackExchange | 0.981 | 1.025 | 1.207 | 0.917 |
| BookCorpus2 | 1.198 | 1.155 | 1.205 | 1.142 |
| Pile-CC | 1.154 | 1.124 | 1.199 | 1.098 |
| Books3 | 1.390 | 1.398 | 1.495 | 1.330 |
| OpenSubtitles | 1.399 | 1.358 | 1.399 | 1.333 |
| ArXiv | 0.914 | 0.911 | 1.131 | 0.855 |
| OpenWebText2 | 1.186 | 1.188 | 1.281 | 1.126 |
| PubMed Central | 0.804 | 0.797 | 0.922 | 0.741 |
| Ubuntu IRC | 1.774 | 1.852 | 2.121 | 1.714 |
| USPTO Backgrounds | 0.847 | 0.828 | 0.938 | 0.779 |
| NIH ExPorter | 0.938 | 0.906 | 1.018 | 0.861 |
| PubMed Abstracts | 0.951 | 0.932 | 1.072 | 0.873 |
| FreeLaw | 1.053 | 1.040 | 1.129 | 0.958 |
| 22 domains, byte-weighted | 1.056 | 1.066 | 1.193 | 0.999 |
| Mean of domains | 1.211 | 1.232 | 1.387 | 1.159 |
Zenyx-100M beats BetterGPT in 10 of 22 domains (DM Mathematics, EuroParl, YouTube Subtitles, GitHub, Wikipedia (en), PhilPapers, StackExchange, Books3, OpenWebText2, Ubuntu IRC) and is behind in the other 12. It is ahead of Muse2-230M* in 21 of 22 domains.
Over training
| Step | Tokens (B) | Fact1000 (no arithmetic) | SciQ acc / norm | ARC-Easy acc / norm | Pile bits/byte | Arithmetic (14,000) |
|---|---|---|---|---|---|---|
| 1,810 | 0.95 | 50.6% (53.6%) | not run | not run | not run | not run |
| 6,000 | 3.15 | 58.5% (61.7%) | 72.4 / 67.3 | 44.4 / 40.1 | 1.145 | 93.7% |
| 7,400 | 3.88 | 60.9% (63.8%) | 71.0 / 63.4 | 44.0 / 40.2 | not run | 72.1% |
| 8,910 | 4.67 | 60.1% (62.5%) | 72.3 / 65.8 | 42.3 / 39.6 | 1.120 | 95.6% |
| 9,500 | 4.98 | 61.9% (65.3%) | 74.2 / 69.2 | 44.5 / 41.2 | 1.120 | 95.2% |
| 9,700 | 5.09 | 61.7% (64.2%) | 72.7 / 67.9 | 45.6 / 41.6 | 1.119 | not run |
| 10,800 | 5.66 | 68.1% (66.0%) | 75.2 / 68.5 | 45.4 / 40.7 | 1.111 | 89.4% |
| 12,650 | 6.63 | 70.5% (68.0%) | 76.1 / 70.8 | 46.0 / 42.3 | 1.103 | 91.1% |
| 14,500 | 7.60 | 68.9% (66.9%) | 75.7 / 71.6 | 47.3 / 43.9 | 1.095 | 85.5% |
| 14,700 | 7.71 | 70.1% (67.2%) | 76.6 / 72.5 | 46.9 / 42.2 | 1.090 | 96.5% |
| 16,100 | 8.44 | 64.6% (67.3%) | 73.6 / 66.9 | 44.1 / 40.4 | 1.084 | 98.0% |
| 17,700 | 9.28 | 67.1% (69.2%) | 72.5 / 66.3 | 46.1 / 41.7 | 1.078 | 97.0% |
| 17,860 | 9.36 | 73.0% (70.0%) | 78.9 / 71.9 | 47.2 / 42.6 | 1.079 | 97.3% |
| 19,300 | 10.12 | 71.8% (69.2%) | 79.8 / 74.1 | 46.8 / 43.7 | 1.076 | 93.3% |
| 19,500 | 10.22 | 72.1% (69.5%) | 77.6 / 72.3 | 47.3 / 43.1 | 1.074 | 96.5% |
| 19,600 | 10.28 | 73.1% (70.1%) | 82.3 / 80.1 | 49.9 / 47.0 | 1.076 | 93.7% |
| 20,200 | 10.59 | 72.7% (69.9%) | 80.8 / 77.0 | 48.8 / 45.6 | 1.072 | 97.9% |
| 20,300 | 10.64 | 73.5% (70.7%) | 79.5 / 74.8 | 47.3 / 44.0 | 1.071 | 97.2% |
| 21,110 | 11.07 | 73.2% (71.1%) | 83.1 / 77.5 | 49.2 / 46.2 | 1.068 | 98.3% |
| 21,200 | 11.12 | 72.8% (69.5%) | 80.1 / 74.9 | 48.2 / 44.2 | 1.069 | 97.4% |
| 21,900 | 11.48 | 72.1% (69.5%) | 81.7 / 76.7 | 48.4 / 44.5 | 1.066 | 97.8% |
| 22,870 | 11.99 | 71.7% (69.7%) | 80.2 / 76.6 | 48.3 / 44.5 | 1.064 | not run |
| 23,000 | 12.06 | 73.5% (71.1%) | 78.4 / 75.1 | 47.3 / 43.9 | 1.064 | not run |
| 27,900 | 14.63 | 72.6% (β) | 81.1 / β | 47.0 / β | 1.075 | not run |
| 28,630 | 15.01 | 73.1% (β) | 81.6 / 78.4 | 48.6 / β | 1.066 | not run |
| 28,900 | 15.15 | 73.1% (β) | 80.6 / 74.7 | 47.5 / 43.3 | 1.065 | not run |
| 29,400 | 15.41 | 72.7% (71.2%) | 81.7 / 76.4 | 47.8 / 44.2 | 1.058 | 86.4% |
| 29,510 | 15.47 | 66.5% (70.6%) | 79.4 / 74.0 | 47.2 / 43.3 | 1.056 | 85.3% |
Compared with step 6,000, Fact1000 without the arithmetic items rose from 61.7% to 69.5%, SciQ from 72.4 to 81.7 and ARC-Easy from 44.4 to 48.4 (accuracy), the Pile from 1.145 to 1.066 (all at step 21,900); ARC-Challenge stays at about chance. Since step 14,700 (one standard error is about 1.4 points on 1,000 items, 1.0 on ARC-Easy): Fact1000 without the arithmetic items +2.3, SciQ +5.1, ARC-Easy +1.5, ARC-Challenge +0.3, the Pile -0.0238 bits per byte. The curve is not monotone: SciQ and ARC-Easy dipped at step 16,100 and recovered by 21,900, and Fact1000 without the arithmetic items was 67.2, 67.3, 69.2 and 69.5 at 14,700, 16,100, 17,700 and 21,900, so differences of a point or two between checkpoints are within noise. The Pile figure was 1.0902, 1.0842, 1.0777 and 1.0664 bits per byte at the same four steps. The all-items Fact1000 score changed by +2.0 points since step 14,700 and by +4.0 since step 10,800 (the standard error of such a difference is about 2). Arithmetic oscillates between checkpoints (72% to 98% over the run). 2-digit multiplication: 92% at step 9,500, 58% at 10,800, 95% at 12,650, 23% at 14,500 (the first scratchpad line was right 95% of the time, but 76% of the replies then added a spurious second step), 87% at 14,700 and 95% at 21,900; its cause is not established.
β = not recorded at that step. Steps 22,870 to 29,510 come after the 28 October relaunch (data mixture v3, learning-rate re-warm at step 23,640): the Pile figure rose to 1.075 at step 27,900, while the learning rate was still high, and has fallen since as the rate decays (1.066, 1.065, 1.058, 1.056). The knowledge benchmarks stay inside their noise band (Fact1000 without the arithmetic items 70.6-73.5% since step 19,300, SciQ 77-83, ARC-Easy 47-50; one standard error is 1.4 points on Fact1000 and 1.0 on ARC-Easy). Since step 20,200 the only measure that moves in one direction is the Pile (1.072 to 1.056); we do not claim a gain on the multiple-choice benchmarks. ARC-Challenge stays at chance level (21.3% accuracy; 25% is chance). From step 29,400 to 29,510 (110 steps) SciQ moved from 81.7 to 79.4, ARC-Easy from 47.8 to 47.2 and the all-items Fact1000 from 72.7% to 66.5%; the Fact1000 fall is almost entirely its arithmetic items (82 of 174 right, against 139 of 174 at step 29,400), without them the score is 70.6% against 71.2%. The prior-only control (the model scored without reading the question) was 30.4% against 28.3%, chance 25%.
Arithmetic
The training data includes arithmetic reasoning examples. The model answers arithmetic in a fixed chat-style format with a one-line scratchpad (no step-by-step working; the answer is computed in one forward pass):
<|im_start|>user
What is 683 + 479?<|im_end|>
<|im_start|>assistant
<think>
683 + 479 = 1162
</think>
1162<|im_end|>
14,000 problems (7 types Γ 2,000, fixed seeds, exact match), step 29,510: 2-digit addition 88.1%, 2-digit subtraction 99.3%, 3-digit addition 99.6%, 3-digit subtraction 96.9%, 4-digit addition 99.6%, 5-digit addition 99.5%, 2-digit Γ 2-digit 14.2%; all 85.3% (11,944 of 14,000; step 29,400: 86.4%). At step 21,900 the same suite gave 100.0, 99.7, 99.7, 92.4, 99.4, 98.8 and 94.7%, all 97.8%. The fall is in the types whose items are in the training data: 2-digit Γ 2-digit dropped from 94.7% to 14.2% (the first scratchpad line is right 61% of the time, and most of the wrong replies chain a spurious second step onto it) and 2-digit addition from 100% to 88.1% (11.5% of the replies chain a wrong second step; 94.0% at step 29,400), after the October 28 mixture change; the types that are not in the training data improved. Without the chat format the same model scores 0% on bare a + b = and Q/A prompts, 5.3% on the notebook's plain prompts (mostly 2-digit subtraction, 27.5%) and 4.8% with three worked examples in the prompt (9.1% with lenient matching): this is a trained, narrow skill.
The 14,000-problem suite overlaps the training data. We streamed 4 million rows of the arithmetic dataset in the training mix (SagheerLab/Arithmetic-Reasoning, same "What is x + y?" format) and found 6,691 of the 14,000 items word for word (2-digit addition, subtraction and multiplication: essentially all; 3-digit addition 35%; 3-digit subtraction 0.2%; 4- and 5-digit addition 0%; the scan stopped at its row cap, so these are lower bounds). The headline therefore mixes memorised items with skill. The three types that are not in the training data (3-digit subtraction, 4-digit and 5-digit addition, 6,000 items) score 98.7% at step 29,510 (5,920 of 6,000; 99.2% at step 29,400, 96.9% at step 21,900); that is the number to compare across checkpoints.
Our own suite (tests/arith_own.py, 8,048 fresh items with seeds the old suite never used, every answer double-checked; step 19,300, native chat format): 3-digit subtraction 92.8%, 4-digit addition 99.5%, 4-digit subtraction 89.0%, 5-digit addition 98.2%, 5-digit subtraction 49.5% (borrowing). Sums of 3 to 10 numbers: 0.1% (3 of 2,700), mixed add/subtract chains 2.9%, repeated numbers such as 2+2+2+2+2 0%, products of three or four small factors 3.8%. The model guesses a long sum in one step (9 + 1 + 3 + 3 + 8 = 12, the answer is 24) instead of adding term by term. Training data that adds one term at a time (3-10 terms, verified by evaluation) has been written for the planned fine-tuning; the pre-training mix has none.
Clean suite (Arko007/arithmetic-bench, public: 16,000 fresh items that are not in the training data, strict exact match, native chat format). Step 28,900: all 51.2%; 3-digit subtraction 97.9%, 4-digit addition 99.8%, 5-digit addition 99.6%, 5-digit subtraction 94.7% (49.5% at step 19,300 on our earlier suite, 73.2% at step 22,000), division 13.7%, multi-digit multiplication still fails (the three multiplication types score 1.9%, 2.0% and 0.0%). Addition and subtraction are solved; multiplication and division are not, and a one-line scratchpad is the reason (the product is guessed in one pass).
Wide suite (7,636 further problems in 28 types, exact match, step 10,800): 7.3% overall (13.0% at step 9,500; not rerun at the latest step). Best types: negative numbers, mul / div 31%; powers 30%; decimals, add / sub 30%; negative numbers, add / sub 23%. Near zero: 6-8 digit numbers, fractions, GCD / LCM, series sums, averages, unit conversion, linear equations and word problems. Between steps 9,500 and 10,800, 3-digit Γ 1-digit multiplication fell from 94.1% to 22.7% and 3-digit Γ 2-digit from 61.2% to 0.0%: replies now often follow a correct first scratchpad line with a spurious second step (for example 691 * 8 = 5528 | 691 + 5528 = 6219). This is the same chaining that caused the 2-digit multiplication dip above, and the failure seen at step 7,400 coming back.
Wide suite by type
| Type | Items | Exact match | Reply finished |
|---|---|---|---|
| 3-digit Γ 1-digit | 286 | 22.7% | 100% |
| 3-digit Γ 2-digit | 286 | 0.0% | 100% |
| 3-digit Γ 3-digit | 286 | 10.5% | 100% |
| 6-8 digit addition | 286 | 0.7% | 100% |
| 6-8 digit subtraction | 286 | 0.0% | 100% |
| exact division | 286 | 16.4% | 100% |
| division rounded down | 286 | 2.1% | 100% |
| remainder / mod | 286 | 0.0% | 72% |
| negative numbers, add / sub | 286 | 23.1% | 100% |
| negative numbers, mul / div | 286 | 30.8% | 99% |
| decimals, add / sub | 286 | 29.7% | 100% |
| decimals, multiply | 286 | 2.1% | 100% |
| order of operations | 286 | 18.9% | 100% |
| powers | 186 | 30.1% | 90% |
| square / cube roots | 199 | 0.0% | 51% |
| percentages | 286 | 15.4% | 100% |
| fractions | 286 | 0.0% | 86% |
| greatest common divisor | 286 | 0.0% | 42% |
| least common multiple | 286 | 0.0% | 77% |
| factorial / choose / permutations | 101 | 1.0% | 89% |
| sums of series | 286 | 0.0% | 63% |
| rounding | 286 | 0.0% | 76% |
| linear equations in x | 286 | 1.0% | 100% |
| word problems, one step | 286 | 1.0% | 88% |
| word problems, two steps | 286 | 0.0% | 78% |
| average | 286 | 0.0% | 80% |
| median | 286 | 0.0% | 83% |
| unit conversion | 286 | 0.0% | 67% |
Model details
| Component | Setting |
|---|---|
| Width / depth | d_model 768, 18 blocks |
| Attention | 12 query heads Γ 64, 4 key/value heads (grouped-query), causal, per-head QK-norm |
| Positions | RoPE, ΞΈ = 500,000 |
| MLP | SwiGLU, hidden size 2,048 |
| Normalisation | RMSNorm, pre-norm; fp32 residual stream with fp16 sub-layers (the T4 has no bf16) |
| Embeddings | tied input/output, 49,152 rows |
| Bias / dropout | none |
Sixteen of the 18 layers are plain grouped-query attention (blocks 8 and 12 additionally carry the segment memory below): sliding-window, gated linear and DeltaNet-style alternatives were benchmarked on a T4 at 2k context and none beat flash-attention GQA on speed. KV cache: 18,432 bytes per token in fp16.
Segment memory (experimental)
From step 17,860 two blocks (8 and 12, counting from 0) can also read the keys and values of the earlier 2,048-token windows of the same text, so a window may use up to 32,768 tokens of context although every forward pass stays 2,048 tokens. For each query head the layer looks up the 32 best-matching stored keys (exact top-32; the stored keys carry no position, so RoPE is untouched), and adds the weighted values to its normal attention output through a per-head tanh gate that starts at exactly zero: at the moment it was switched on the model computed the same function as the step-17,860 checkpoint. The memory has its own query projection (initialised as a copy of the layer's) and its own optimiser for the gates; no gradient flows into the stored keys and values. Design after Memorizing Transformers (Wu et al., 2022) and the lessons of Hugging Face's failed Infini-attention re-implementation.
| Memory blocks | 8 and 12; the cached decoder reads the memory (zenyx100m.memattn.generate_long, and Zenyx.ask in the notebook feeds long prompts window by window); the CUDA-graph fast decoder ignores it |
| New parameters | 1,179,800 (152.2M in total) |
| Layout | per GPU and step, 8 runs of 16 consecutive windows; memory reset at every optimiser step; 524,288 tokens per step as before |
| Cost | 2.2Γ slower training (49 s against 22 s per step on 2 Γ T4), 11.7 GB against 11.1 GB |
| Recall task | from step 19,500: 25% of the runs (falling to 12% over 1,500 steps) plant 3-20 invented facts in earlier windows and ask questions about them in a later window; loss on the answer tokens only |
Status: the memory copies text and ranks facts well, free recall of a fact from far back was missing until step 19,500 and has just started to work. Measured on step 18,670 and step 19,300 (one T4, tests/colab/mem_eval_v3.py; the memory is on or off with the same weights; the step-18,000 checkpoint, whose gates were still small, is the control):
| Passcode hidden in unrelated text, 8 candidate numbers (chance 12.5%, 24 trials per length) | 2k | 4k | 8k | 16k | 32k |
|---|---|---|---|---|---|
| Step 18,670, memory on | 83% | 96% | 88% | 92% | 92% |
| Step 18,670, memory off | 83% | 12% | 0% | 12% | 21% |
| Step 18,000, memory on | 79% | 67% | 29% | 21% | 29% |
| Step 19,300, memory on | 88% | 96% | 83% | 83% | 71% |
| Step 19,300, memory off | 88% | 8% | 8% | 8% | 12% |
At step 19,300, 30 long Wikipedia articles (first 16,384 tokens each): the memory lowers the loss of windows 1-7 by 0.037 nats (95% interval 0.028-0.047, better in every article; 0.035 at step 18,670, 0.022 at 18,000), growing with distance; a different article's memory makes the loss slightly worse (+0.012 to +0.014), so the gain is retrieval and not extra context. In 20 long chats of 8 windows the loss is 0.028 nats lower. Window 0 is identical with and without the memory (3.4028 against 3.4028), so a short prompt behaves exactly like the memory-free model, which is why the benchmarks on this card (run with the memory off) still apply. Reading the memory in the cached decoder is exact (logits agree with the full forward, top-1 agreement 100%, greedy decode identical on 16 of 16 tokens).
Repeated text is copied from the memory at any position: teacher-forced loss of a repeated 2,048-token span 3.78 without the memory against 0.21 with it (top-1 accuracy 32% to 96%), 0.24 when the span starts 64 tokens later, 0.42 when it starts 1,024 tokens later and 0.33 after 200 unrelated tokens (step 19,600). Questions about a fact far back were not answered before the recall task (step 19,300-19,400, greedy): the hidden passcode was written correctly 5% of the time at 4k tokens and 0% at 8k-64k (0% without the memory); a passage repeated after filler was continued for 12% / 8% / 4% of the next 40 tokens at 4k / 16k / 32k (1-3% without); after 3k-64k tokens of filler a chat fact (name, city, colour) was ranked first among 8 candidates 85% / 55% / 50% / 45% / 25% of the time (3k / 8k / 16k / 32k / 64k; 5-20% without the memory) but never produced as free text; and a three-fact memo 8.5k tokens back, asked as a question, scored 0 of 3. The reason is the training task: until step 19,500 nothing in training asked a question whose answer lay in an earlier window and did not simply continue the text.
Recall task (from step 19,500, zenyx100m/recall.py). Invented subjects and values (project leaders, budgets, code words, passcodes, dates, cities, counts, colours, times) are written as one sentence each into earlier windows of the same run (1-14 windows back, random positions and wrappers); the end of a later window becomes a block of questions about them in four layouts (Question/Answer, Q:/A:, the chat format, an unfinished sentence), and only the answer tokens carry loss. First result, the step-19,600 checkpoint after about 100 steps of it (one checkpoint, small samples, to be repeated): the memo test now answers the project lead and the budget correctly in chat format (2 of 3; the code word is wrong) against 0 of 3 with the memory off; on 24 fresh trials worded differently from the training task, the right value is produced in 8 of 24 chat answers (33%) and 5 of 24 plain Q/A answers (21%) with the memory and in 0 of 24 and 0 of 24 without it. Still failing: code words and exact dates, "repeat the first sentence of the memo", summaries of a long document (they continue the article instead).
Practical chat test (step 20,200, tests/colab/mem_chat_own.py, free greedy answers through the notebook's ask(keep=True)). A real multi-turn chat: turn 1 plants a fact, then the user keeps pasting about 600-token texts ("summarize this") and the model answers them until the history is about 2.5k / 5k / 9k tokens long; then the question is asked with the memory on and off (off = only the last 2,048 tokens are visible). Each value is scored by exact match; 10 chats per condition.
| Fact | Question right after the fact (no memory needed) | ~2.5k tokens later, memory on / off | ~5k | ~9k |
|---|---|---|---|---|
| User says name, city and dog's name | 60% | 33% / 0% | 13% / 0% | 10% / 0% |
| Assistant turn states restaurant and confirmation code | 15% | 65% / 0% | 30% / 0% | 20% / 0% |
| The model's own free-text answer ("tell me a fact about octopuses", later "what fact did you tell me?") | 13 of 16 chats | 0 of 6 / 0 of 6 | 0 of 10 / 0 of 10 | 0 of 7 / 0 of 7 |
With the memory off the model recalls nothing once the fact has left the window; with it on it still finds specific values (names, codes) up to 9k tokens back, but accuracy falls with distance and is far below what the model does when the fact is inside the window. The last row counts only chats in which the original answer was no longer visible to the memory-off model (the base model tends to copy its earlier answers into later ones, which keeps a fact alive inside the window; 6 to 10 of 16 chats remain at each distance): there the memory did not help, so remembering what the model itself said in free text is not working yet. Why: recall training so far plants short facts as statements by the user; it never asks about something the assistant said. The model also often answers in the first person ("My name is ...") instead of "your name is ...", a chat-format weakness that post-training should fix.
Gate probe (checkpoints 18,000 to 28,000, tests/colab/gate_probe.py, October 31). The per-head tanh gates settle with training: the mean change of the raw gate value per 2,000 steps shrank from 0.18 to 0.05 in block 8 and from 0.93 to 0.13 in block 12, with no jitter. Two of the 24 block-12 heads are saturated (raw gate 9.0 and 7.8, tanh = 1.0; one of them has not moved), so a gate that cannot close is possible; the memory query projection stays far from the layer's own query (cosine 0.07 to 0.2). No checkpoint step was changed because of this.
Not shown: lengths beyond 32,768 tokens in the ranking test, any benchmark effect (all benchmarks on this card run with the memory off), messy real-world retrieval (the natural-text gain is about 1% of the loss). The memory costs 2.2x training time. Core benchmarks at the latest step (memory off) are in the tables above; the step-19,600 checkpoint, the first after the recall task started, scored SciQ 82.3 / 80.1, ARC-Easy 49.9 / 47.0 and Fact1000 73.1% (up from 77.6 / 72.3, 47.3 / 43.1 and 72.1% at 19,500): steps 20,200 and 20,300 then scored SciQ 80.8 / 77.0 and 79.5 / 74.8, ARC-Easy 48.8 / 45.6 and 47.3 / 44.0, Fact1000 72.7% and 73.5%: back inside the range of the earlier checkpoints (SciQ 77.6-82.3, ARC-Easy 46.8-49.9; one standard error is 1.3 points on SciQ and 1.0 on ARC-Easy), so the 19,600 values were mostly noise and we cannot attribute a benchmark gain to the recall task. Checkpoints from step 17,900 contain 6 extra tensors (mem_gate, mem_q_proj.weight, mem_q_norm.weight in blocks 8 and 12): load them with strict=False or through zenyx100m.memattn.
Tokenizer
Byte-level BPE with a SuperBPE-style second stage (merges may cross spaces; digits stay single tokens; 248 special tokens). On 63 MB of text from the training sources it averages 4.561 bytes per token, against 4.170 for the Llama-3.2 tokenizer, 4.142 for DeepSeek-V4-Pro and 4.031 for Qwen3: about 9 to 13% fewer tokens with a vocabulary of 49,105 (2.6 times smaller than DeepSeek's). It ties on Wikipedia and is worse on markdown (about 5%) and GSM8K (about 3%). Only English, code and math were measured; compare models across tokenizers with bits per byte, not perplexity. File: tokenizer_candidates_v3/F-49152-f70/tokenizer.json.
Training
| Optimiser | Muon on hidden matrices (lr 0.012), AdamW on embeddings and norms (lr 2e-3) |
| Schedule | warmup, constant until 3.9B tokens, then linear decay to 10% of peak; at the 28 October relaunch (step 23,640, 12.39B tokens) the rate was re-warmed together with the mixture change and decays again to 10% of peak at 16B tokens, held afterwards; token budget 30B |
| Batch | 524,288 tokens per step; about 21-25k tokens/s on 2 Γ T4 |
| Data | English web, code, math and chat text; since the October 24 relaunch (step 18,670) also instruction-style data (OpenOrca GPT-4 part, UltraChat, MetaMathQA, oasst, text-to-SQL) as ordinary pre-training text; since the October 28 relaunch (step 23,640) mixture v3, moved in over a gradual 1B-token transition: more factual text, a synthetic grid-puzzle set (Arko007/zenyx-arc-synth), and the synthetic chat, instruction and tool-use set Arko007/zenyx-sft-v2 as ordinary pre-training text; since the 1 November relaunch (from step 29,510) about 37M further tokens of synthetic chat rows are appended to that set (about 100,000 tool-use / agent-loop conversations with a simulated search and tool world, and about 3,900 conversations on judgment, honesty, planning and self-correction, plus persona and identity chat), again as ordinary pre-training text; the benchmarks above are from before this change |
Other measurements
- MMLU (all 14,042 questions): 25.3% Β± 0.7 with 5 shots at step 29,510 and 25.4% Β± 0.7 with 0 shots at step 20,200 (chance 25%; Zenyx-V3 26.9%). The step-12,650 value was 25.0%.
- Python (step 10,900), twelve small functions run against unit tests (primes, factorial, palindrome, gcd, Fibonacci, FizzBuzz and similar): signature + docstring as plain completion 8 of 12 with greedy decoding and 43% of sampled attempts pass (8 samples per task, T = 0.6); chat format 5 of 12 and 34%; forcing
<think>first lowers it to 2 of 12 and 7%. (Step 6,000: 4 of 12 and 25%.) It still fails harder tasks. - Letter-choice multiple choice (step 10,900) in chat format is at chance (SciQ 19%, ARC-Easy 23%, ARC-Challenge 25%, 200 questions each) although the standard scoring of the same questions gives 75%, 45% and 24%: the model has the knowledge but not the "read four lettered options and pick one" skill.
- Decode speed (step 10,900), one T4, fp16, CUDA graph +
torch.compile: 349 tokens/s at batch 1, 2,572 tokens/s total at batch 8, 9,475 tokens/s at batch 64. - Long generation: greedy decoding repeats itself within about 30 words; suggested: sampling (temperature 0.8, top-k 50) with a repetition penalty of 1.1-1.2.
Planned post-training
A supervised fine-tuning stage is being prepared; no instruct model has been released and no result exists yet. The training set is built (19 shards = 149,571 windows of 2,049 tokens, 300M window tokens of which 208M are assistant tokens that carry loss; one planned source, our own earlier synthetic set at 4%, could not be read and is missing) from public chat, instruction and reasoning data plus synthetic data with exact answers (single operations and, new, sums of 3-10 numbers, tool-calculator calls, system-prompt-versus-user conflicts, identity), English only (a language filter drops other scripts and most other Latin-script languages). ChatML with the tokenizer's real special tokens (<|im_start|>, <|im_end|>, <think>, </think>, <tool_call>, <tool_response>), loss on assistant tokens only; system prompts that must win over conflicting user instructions; a Reasoning effort: none | low | medium | high line with reasoning budgets of 256 / 512 / 1,024 tokens. Weak points that motivate it, measured with a code-scored chat suite at step 18,900 (10 categories, 8-45 prompts each, macro average 54.6%): greetings 33% (it often answers "Hello" with an apology and "as a text-based AI"), instruction following with checkable constraints 20%, system-prompt obedience 22%, small code tasks 25%, free-text facts 27%, against arithmetic 96% and recall of a detail given earlier in the same chat 88%. The numbers in this card are the pre-SFT baseline.
Intended use and limitations
- Use: research, and as a starting point for fine-tuning. Not an assistant: it ignores system prompts, answers greetings badly, and may continue a prompt with an unrelated document.
- Facts: expect errors and invented details; long generations repeat themselves.
- Arithmetic works in its own chat format only (see above): addition and subtraction of two numbers of up to five digits is solid on items that are not in the training data (97 to 100% at steps 28,900-29,510), multi-digit multiplication and division are not (about 2% and 14% on the clean suite), and about half of the old test suite is in the training data. It did not add a list of numbers (0.1% on 3-10 terms; measured at step 19,300, not re-measured) and not negatives, decimals, fractions, powers, algebra or word problems.
- Reasoning, MMLU, letter-choice multiple choice: at chance. Code (step 10,900): passes 8 of 12 small functions, not harder ones. The
<think>habit was trained for arithmetic; the October 28 mixture also contains think-style chat data (effect not yet measured). - Context 2,048 tokens (up to 32,768 with the experimental segment memory, whose free recall of far-back facts is just starting to work); English only; not evaluated for bias or harmful content.
- Evaluation: one training run, mid-training checkpoints; Fact1000 and the question sets are written by the author; Muse2 numbers use an unofficial reimplementation.
Repository contents
| Path | Contents |
|---|---|
checkpoints/<run>/step_N/ |
model weights (model.pt), optimiser state, training state; the three newest checkpoints and every 2,000th step are kept |
tokenizer_candidates_v3/ |
tokenizers, including the one used |
zenyx_interactive.py |
self-contained loader and chat helper (see Usage) |
wheels/, runs/, data/ |
flash-attention build, training manifests |
The model code is not published yet.