Release Pollock 1.5 (r007)

CHANGELOG.md

@@ -2,6 +2,17 @@
 
 Revision numbers identify published model states independently of release names.
 
+## r007 — Pollock 1.5
+
+- Changed the architecture from 12/14/896 with a 1,024-token context to 16/12/768 with a 3,200-wide MLP and a 2,048-token context while remaining below 128M native trainable parameters.
+- Returned from the r006 2.5B V2 corpus to `SlayerLab/minimal-en-corpus-5b` at revision `38bebbd`, with its separate 12,288-entry byte-level BPE tokenizer.
+- Replaced AdamW-only training with Muon for eligible hidden attention and MLP matrices plus fused AdamW for embeddings, the tied output head, normalization parameters, and biases.
+- Replaced random-with-replacement sampling and fixed-subset validation with deterministic shuffled corpus passes, exact resumable loader state, a fixed corpus-wide train sample, and complete validation-split evaluation.
+- Completed exactly four corpus passes at the unchanged 491,520-token effective batch and selected the best full-validation checkpoint at update 43,750.
+- Updated the seven-task English zero-shot suite, Transformers weights, locked E4-best diagnostics, and fixed cross-revision inference samples.
+
+Full record: [`training-history/r007.md`](./training-history/r007.md)
+
 ## r006 — Pollock 1.4
 
 - Returned from the r005 5B corpus to V2 of `SlayerLab/minimal-en-corpus-2.5b`, including its corrected cleaning and record boundaries.

LICENSE.md

@@ -16,14 +16,14 @@ Code components distributed with the project remain subject to their respective
 
 ## Korpus i wagi / Corpus and weights
 
-Model został wytrenowany na `SlayerLab/minimal-en-corpus-2.5b`, agregacie danych z wielu źródeł. Korpus nie nadaje dokumentom jednej wspólnej licencji; każdy dokument zachowuje identyfikator źródła i podlega warunkom, licencjom oraz ograniczeniom właściwego upstreamowego datasetu.
+Model został wytrenowany na `SlayerLab/minimal-en-corpus-5b`, agregacie danych z wielu źródeł. Korpus nie nadaje dokumentom jednej wspólnej licencji; każdy dokument zachowuje identyfikator źródła i podlega warunkom, licencjom oraz ograniczeniom właściwego upstreamowego datasetu.
 
 Ze względu na mieszany charakter tych warunków repozytorium modelu używa metadanej Hugging Face `license: other`. Nie jest to przyznanie dodatkowych praw do materiałów źródłowych. Użytkownik powinien przed użyciem, redystrybucją lub zastosowaniem komercyjnym zapoznać się z kartą korpusu i warunkami wszystkich właściwych źródeł:
 
-<https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-2.5b>
+<https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-5b>
 
-The model was trained on `SlayerLab/minimal-en-corpus-2.5b`, an aggregate of multiple data sources. The corpus does not apply a single common license to its documents; each document retains its source identifier and remains subject to the terms, licenses, and restrictions of the applicable upstream dataset.
+The model was trained on `SlayerLab/minimal-en-corpus-5b`, an aggregate of multiple data sources. The corpus does not apply a single common license to its documents; each document retains its source identifier and remains subject to the terms, licenses, and restrictions of the applicable upstream dataset.
 
 Because these terms are mixed, the model repository uses the Hugging Face metadata value `license: other`. This notice does not grant additional rights to upstream materials. Before use, redistribution, or commercial application, users should review the corpus card and the terms of every applicable source:
 
-<https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-2.5b>
+<https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-5b>

README.md

@@ -5,38 +5,38 @@ pipeline_tag: text-generation
 license: other
 license_name: mixed-upstream-dataset-terms
 license_link: https://huggingface.co/SlayerLab/pollock-mini-lm-125m/blob/main/LICENSE.md
-datasets: [SlayerLab/minimal-en-corpus-2.5b]
+datasets: [SlayerLab/minimal-en-corpus-5b]
 tags: [causal-lm, gpt2, nanogpt, bpe, educational, base-model]
 model-index:
-  - name: Pollock 1.4
+  - name: Pollock 1.5
     results:
       - task: {type: text-generation, name: Language modeling}
-        dataset: {type: SlayerLab/minimal-en-corpus-2.5b, name: Minimal EN 2.5B V2 validation (fixed subset), split: validation}
-        metrics: [{type: loss, value: 2.5363565445, name: Final fixed-subset validation loss}]
-      - task: {type: text-generation, name: Zero-shot evaluation}
-        dataset: {type: blimp, name: BLiMP, split: train}
-        metrics: [{type: acc, value: 0.7691641791}]
-      - task: {type: text-generation, name: Zero-shot evaluation}
-        dataset: {type: EleutherAI/lambada_openai, name: LAMBADA OpenAI, split: test}
-        metrics: [{type: acc, value: 0.2769260625}, {type: perplexity, value: 49.9072065519}]
-      - task: {type: text-generation, name: Zero-shot evaluation}
-        dataset: {type: hellaswag, name: HellaSwag, split: validation}
-        metrics: [{type: acc_norm, value: 0.3013343955}]
-      - task: {type: text-generation, name: Zero-shot evaluation}
-        dataset: {type: piqa, name: PIQA, split: validation}
-        metrics: [{type: acc_norm, value: 0.5968443961}]
-      - task: {type: text-generation, name: Zero-shot evaluation}
-        dataset: {type: sciq, name: SciQ, split: test}
-        metrics: [{type: acc_norm, value: 0.6640000000}]
-      - task: {type: text-generation, name: Zero-shot evaluation}
-        dataset: {type: allenai/ai2_arc, config: ARC-Easy, name: ARC-Easy, split: test}
-        metrics: [{type: acc_norm, value: 0.4343434343}]
-      - task: {type: text-generation, name: Zero-shot evaluation}
-        dataset: {type: allenai/ai2_arc, config: ARC-Challenge, name: ARC-Challenge, split: test}
-        metrics: [{type: acc_norm, value: 0.2414675768}]
+        dataset: {type: SlayerLab/minimal-en-corpus-5b, name: Minimal EN validation (full split), split: validation}
+        metrics: [{type: loss, value: 2.4585190566, name: Full-validation loss}]
+      - task: {type: text-generation, name: Linguistic minimal pairs}
+        dataset: {type: blimp, name: BLiMP}
+        metrics: [{type: accuracy, value: 0.7849701493, name: Accuracy}]
+      - task: {type: text-generation, name: Language modeling}
+        dataset: {type: lambada_openai, name: LAMBADA OpenAI}
+        metrics: [{type: accuracy, value: 0.3007956530, name: Accuracy}, {type: perplexity, value: 45.0303617702, name: Perplexity}]
+      - task: {type: text-generation, name: Multiple-choice completion}
+        dataset: {type: hellaswag, name: HellaSwag}
+        metrics: [{type: accuracy, value: 0.3083051185, name: Normalized accuracy}]
+      - task: {type: text-generation, name: Physical commonsense reasoning}
+        dataset: {type: piqa, name: PIQA}
+        metrics: [{type: accuracy, value: 0.5968443961, name: Normalized accuracy}]
+      - task: {type: text-generation, name: Science question answering}
+        dataset: {type: sciq, name: SciQ}
+        metrics: [{type: accuracy, value: 0.675, name: Normalized accuracy}]
+      - task: {type: text-generation, name: Grade-school science questions}
+        dataset: {type: arc_easy, name: ARC-Easy}
+        metrics: [{type: accuracy, value: 0.4183501684, name: Normalized accuracy}]
+      - task: {type: text-generation, name: Science challenge questions}
+        dataset: {type: arc_challenge, name: ARC-Challenge}
+        metrics: [{type: accuracy, value: 0.25, name: Normalized accuracy}]
 ---
 
-# Pollock 1.4 — r006
+# Pollock 1.5 — r007
 
 ![Pollock avatar](./assets/pollock-mini-lm-avatar-320.png)
 
@@ -48,60 +48,67 @@ model-index:
 
 Pollock to niewielki, anglojęzyczny model bazowy typu decoder-only, wytrenowany od zera jako czytelny eksperyment edukacyjny. Implementacja bazuje na [nanoGPT](https://github.com/karpathy/nanoGPT) i własnym tokenizerze byte-level BPE. Jest to model do uzupełniania tekstu, nie asystent konwersacyjny.
 
-Nazwa luźno nawiązuje do gestu malarskiego Jacksona Pollocka: nanoGPT jest płótnem, na którym dane, konfiguracja i decyzje treningowe tworzą różne wzorce zachowania. Pełne dane techniczne tej wersji znajdują się w [`training-history/r006.md`](./training-history/r006.md), a różnice między wydaniami w [`CHANGELOG.md`](./CHANGELOG.md).
+Nazwa luźno nawiązuje do gestu malarskiego Jacksona Pollocka: nanoGPT jest płótnem, na którym dane, konfiguracja i decyzje treningowe tworzą różne wzorce zachowania. Pełne dane techniczne tej wersji znajdują się w [`training-history/r007.md`](./training-history/r007.md), a różnice między wydaniami w [`CHANGELOG.md`](./CHANGELOG.md).
 
 ### Architektura i tokenizer
 
 | Właściwość | Wartość |
 |---|---:|
-| Rewizja / wydanie | r006 / Pollock 1.4 |
+| Rewizja / wydanie | r007 / Pollock 1.5 |
 | Typ | decoder-only Transformer w stylu GPT-2 |
-| Warstwy / głowy / embedding | 12 / 14 / 896 |
-| Maksymalny kontekst | 1024 tokeny |
+| Warstwy / głowy / embedding | 16 / 12 / 768 |
+| Szerokość MLP | 3 200 |
+| Maksymalny kontekst | 2048 tokenów |
 | Słownik | 12 288 tokenów |
-| Parametry nanoGPT | 126 637 952 |
-| Łączne unikalne parametry trenowalne | 127 555 456 |
+| Parametry nanoGPT | 125 854 464 |
+| Łączne unikalne parametry trenowalne | 127 427 328 |
 | Tokenizer | byte-level BPE, pretokenizacja w stylu GPT-2 |
 | Tokeny specjalne | <code>&lt;&#124;endoftext&#124;&gt;</code>, <code>&lt;&#124;im_start&#124;&gt;</code>, <code>&lt;&#124;im_end&#124;&gt;</code> |
 
-Artefakt Transformers ma 127 674 624 parametry, w tym 119 168 zerowych parametrów bias dla zgodności z `GPT2LMHeadModel`. Natywny model był trenowany z `bias=False`.
+Artefakt Transformers ma 127 565 312 parametrów, w tym 137 984 zerowe parametry bias dla zgodności z `GPT2LMHeadModel`. Natywny model był trenowany z `bias=False`.
+
+Wybrany checkpoint przekonwertowano przy użyciu Transformers 5.15.1. Deterministyczny test `[2, 64]` wykazał maksymalną bezwzględną różnicę logits **0.0**; SHA-256 pliku `model.safetensors` to `4a88432225dac6255ae1e2a085019f36dcdaf55fa33c77629dba6e7445c6c2fb`.
+
+Muon optymalizuje kwalifikujące się ukryte macierze attention i MLP. AdamW obsługuje embeddingi tokenów, powiązaną głowicę wyjściową, parametry normalizacji i biasy. Podział parametrów jest sprawdzany przed treningiem pod kątem nakładania się i kompletności.
 
 ### Dane i trening
 
-Model wytrenowano na wersji V2 datasetu [`SlayerLab/minimal-en-corpus-2.5b`](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-2.5b) przypiętej do commita `d68d992`. Jest to subiektywnie dobrana mieszanka 15 anglojęzycznych źródeł po filtrowaniu języka, deduplikacji dokładnej i przybliżonej, decontaminacji benchmarków oraz dodatkowym czyszczeniu boilerplate'u i błędnie połączonych rekordów. R006 wraca do tokenizera używanego przez r003-r004; nie jest on zgodny z tokenizerem r005 poza rozmiarem słownika i identyfikatorami tokenów specjalnych.
+Model wytrenowano na [`SlayerLab/minimal-en-corpus-5b`](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-5b) przypiętym do commita `38bebbd`. Jest to subiektywnie dobrana mieszanka 15 anglojęzycznych źródeł po filtrowaniu języka, deduplikacji dokładnej i przybliżonej oraz decontaminacji benchmarków. R007 używa tokenizera z tego korpusu; zachowuje rozmiar słownika i identyfikatory tokenów specjalnych, ale pozostałe identyfikatory tokenów nie są zgodne z tokenizerem r006.
+
+Loader stosuje globalną permutację ze stałym seedem do niezmiennych okien tokenów w każdym przebiegu, przydziela rozłączne pozycje między rangi DDP, maskuje końcowy niepełny batch i zapisuje globalny kursor, topologię oraz ustawienia tasowania w każdym checkpoincie.
 
 | Parametr | Wartość |
 |---|---:|
-| Tokeny treningowe / walidacyjne | 2 689 323 439 / 5 236 486 |
-| Finalny checkpoint | aktualizacja 22 003 |
-| Przetworzone tokeny | 10 814 914 560 (około 4,02 epoki) |
-| Sekwencja / micro-batch na GPU | 1024 / 12 |
+| Tokeny treningowe / walidacyjne | 5 396 605 407 / 5 238 223 |
+| Wybrany checkpoint | aktualizacja 43 750 |
+| Przetworzone tokeny | 21 503 919 992 (około 3,98 epoki) |
+| Sekwencja / micro-batch na GPU | 2048 / 6 |
 | Akumulacja globalna / na GPU | 40 / 20 micro-stepów |
 | Effective batch | 491 520 tokenów |
-| Optymalizator | fused AdamW, betas 0.9/0.95 |
-| Learning rate | 4e-4 → 4e-5, cosine decay |
-| Warmup / weight decay / grad clip | 440 / 0.1 / 1.0 |
+| Optymalizator | Muon plus fused AdamW, betas AdamW 0.9/0.95 |
+| Learning rate | Muon 1e-2 → 1e-3; AdamW 3e-4 → 3e-5 |
+| Warmup / weight decay / grad clip | 200 / 0.01 / 1.0 |
 | Precyzja | BF16 |
 | Sprzęt | 2× NVIDIA GeForce RTX 4090 |
-| Framework | PyTorch 2.8.0+cu128, nanoGPT commit `3adf61e` |
+| Framework | PyTorch 2.11.0+cu128, nanoGPT commit `3adf61e` |
 
 ### Ewaluacja
 
-Loss treningowy szacowano na stałych podzbiorach po 1 228 800 tokenów na split. Finalny checkpoint uzyskał validation loss **2.536357**; najlepszy wynik to **2.536231** po 10 813 440 000 przetworzonych tokenów. Nie należy porównywać tych wartości bezpośrednio z r005, ponieważ r006 używa innego korpusu, tokenizera i zbioru walidacyjnego.
+Loss treningowy szacowano na stałej, tasowanej próbie 2 457 600 tokenów z całego splitu treningowego. Każda ewaluacja walidacyjna obejmowała pełny split, czyli 5 238 222 targety. Wybrany checkpoint zapisał validation loss **2.458616** po 21 503 919 992 przetworzonych tokenach. Niezależna ewaluacja pełnego splitu odtworzyła cross-entropy **2.458519**, perplexity **11.687490** i predictive entropy **2.437529**. Trening zakończył się po dokładnie czterech przebiegach korpusu, na aktualizacji 43 918 i po 21 586 421 624 przetworzonych tokenach.
 
-Benchmarki wykonano zero-shot na pełnych splitach przy użyciu `lm-evaluation-harness` 0.4.12, batch size 8 i BF16.
+Benchmarki wykonano zero-shot na pełnych splitach przy użyciu `lm-evaluation-harness` 0.4.12, batch size 8, maksymalnego kontekstu 1024 i BF16 na Apple M1 Max MPS.
 
 | Benchmark | Główna metryka | Wynik | Próbki |
 |---|---|---:|---:|
-| BLiMP | acc | 0.769164 | 67 000 |
-| LAMBADA | acc | 0.276926 | 5 153 |
-| HellaSwag | acc_norm | 0.301334 | 10 042 |
+| BLiMP | acc | 0.784970 | 67 000 |
+| LAMBADA | acc | 0.300796 | 5 153 |
+| HellaSwag | acc_norm | 0.308305 | 10 042 |
 | PIQA | acc_norm | 0.596844 | 1 838 |
-| SciQ | acc_norm | 0.664000 | 1 000 |
-| ARC-Easy | acc_norm | 0.434343 | 2 376 |
-| ARC-Challenge | acc_norm | 0.241468 | 1 172 |
+| SciQ | acc_norm | 0.675000 | 1 000 |
+| ARC-Easy | acc_norm | 0.418350 | 2 376 |
+| ARC-Challenge | acc_norm | 0.250000 | 1 172 |
 
-LAMBADA osiągnęła perplexity 49.907207. Pełne metryki i protokół zapisano w [`benchmarks/english.json`](./benchmarks/english.json).
+Perplexity LAMBADA wynosi **45.030362**. W porównaniu z pomiarem E2 checkpoint E4-best poprawia BLiMP i ARC-Challenge, ale uzyskuje niższe wyniki na pozostałych pięciu zadaniach. Spadek cross-entropy walidacyjnej nie przekłada się więc jednolicie na te benchmarki downstream. Pełne metryki i protokół znajdują się w [`benchmarks/english.json`](./benchmarks/english.json).
 
 ### Użycie z Transformers
 
@@ -123,11 +130,12 @@ Model używa standardowego `GPT2LMHeadModel`; `trust_remote_code=True` nie jest
 
 ### Stałe próbki inferencji
 
-Wspólny zestaw [`fixed-sampling-v1`](./inference-samples/README.md) pokazuje te same cztery prompty wygenerowane przez r001-r006. Wszystkie opublikowane rewizje są ładowane z pełnych, niezmiennych SHA commitów, a SHA-256 każdego pliku z wagami jest sprawdzane przed inferencją. Referencyjny protokół używa macOS 26.2 na arm64, CPU, float32, jednego wątku, Transformers 5.15.1, seed 1337 resetowanego dla każdego promptu, temperature 0.7, top-k 50 i limitu 100 nowych tokenów.
+Wspólny zestaw [`fixed-sampling-v1`](./inference-samples/README.md) pokazuje te same cztery prompty wygenerowane przez r001-r007. Wszystkie opublikowane rewizje są ładowane z pełnych, niezmiennych SHA commitów, a SHA-256 każdego pliku z wagami jest sprawdzane przed inferencją. Referencyjny protokół używa macOS 26.2 na arm64, CPU, float32, jednego wątku, Transformers 5.15.1, seed 1337 resetowanego dla każdego promptu, temperature 0.7, top-k 50 i limitu 100 nowych tokenów.
 
 | Rewizja | Wydanie | Wagi | Historia |
 |---|---|---|---|
-| **r006** | **Pollock 1.4** | [`a2e53d9`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/a2e53d9f1b690ebfbea375cf8bd3b5bdf69dece9) | [`r006.md`](./training-history/r006.md) |
+| **r007** | **Pollock 1.5** | [`f2a1ec0`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/f2a1ec072098d257c36b129b0e554e2e6de9c46e) | [`r007.md`](./training-history/r007.md) |
+| r006 | Pollock 1.4 | [`a2e53d9`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/a2e53d9f1b690ebfbea375cf8bd3b5bdf69dece9) | [`r006.md`](./training-history/r006.md) |
 | r005 | Pollock 1.3 | [`e780025`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/e780025f15e06bc3765a74d973906eb8a11c022c) | [`r005.md`](./training-history/r005.md) |
 | r004 | Pollock 1.2 | [`30feb81`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/30feb81e5097eef13b939e1827d98e0458bf602d) | [`r004.md`](./training-history/r004.md) |
 | r003 | Pollock 1.1 | [`698984b`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/698984b1d1b96c9b6ffaff00c7fc2e78e140e842) | [`r003.md`](./training-history/r003.md) |
@@ -151,60 +159,67 @@ Pełne teksty są przeznaczone do porównywania zachowania, nie są benchmarkiem
 
 Pollock is a small English decoder-only base language model trained from scratch as a readable educational experiment. It is based on [nanoGPT](https://github.com/karpathy/nanoGPT) and a custom byte-level BPE tokenizer. It is a completion model, not a conversational assistant.
 
-The name loosely refers to Jackson Pollock's painterly gesture: nanoGPT is the canvas on which data, configuration, and training decisions create different behavioral patterns. See [`training-history/r006.md`](./training-history/r006.md) for the complete technical record and [`CHANGELOG.md`](./CHANGELOG.md) for release-to-release changes.
+The name loosely refers to Jackson Pollock's painterly gesture: nanoGPT is the canvas on which data, configuration, and training decisions create different behavioral patterns. See [`training-history/r007.md`](./training-history/r007.md) for the complete technical record and [`CHANGELOG.md`](./CHANGELOG.md) for release-to-release changes.
 
 ### Architecture and tokenizer
 
 | Property | Value |
 |---|---:|
-| Revision / release | r006 / Pollock 1.4 |
+| Revision / release | r007 / Pollock 1.5 |
 | Type | GPT-2-style decoder-only Transformer |
-| Layers / heads / width | 12 / 14 / 896 |
-| Maximum context | 1,024 tokens |
+| Layers / heads / width | 16 / 12 / 768 |
+| MLP width | 3,200 |
+| Maximum context | 2,048 tokens |
 | Vocabulary | 12,288 tokens |
-| nanoGPT parameters | 126,637,952 |
-| Total unique trainable parameters | 127,555,456 |
+| nanoGPT parameters | 125,854,464 |
+| Total unique trainable parameters | 127,427,328 |
 | Tokenizer | byte-level BPE, GPT-2-style pretokenization |
 | Special tokens | `<|endoftext|>`, `<|im_start|>`, `<|im_end|>` |
 
-The Transformers artifact has 127,674,624 parameters, including 119,168 zero-valued compatibility bias parameters required by `GPT2LMHeadModel`. The native model was trained with `bias=False`.
+The Transformers artifact has 127,565,312 parameters, including 137,984 zero-valued compatibility bias parameters required by `GPT2LMHeadModel`. The native model was trained with `bias=False`.
+
+The selected checkpoint was converted with Transformers 5.15.1. A deterministic `[2, 64]` probe produced maximum absolute logit error **0.0**; the `model.safetensors` SHA-256 is `4a88432225dac6255ae1e2a085019f36dcdaf55fa33c77629dba6e7445c6c2fb`.
+
+Muon optimizes eligible hidden attention and MLP matrices. AdamW handles token embeddings, the tied output head, normalization parameters, and biases. The parameter partition is checked for overlap and completeness before training.
 
 ### Data and training
 
-The model was trained on V2 of [`SlayerLab/minimal-en-corpus-2.5b`](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-2.5b), pinned to commit `d68d992`. It is a subjectively selected mixture of 15 English-language sources after language filtering, exact and approximate deduplication, benchmark decontamination, and additional cleanup of boilerplate and incorrectly concatenated records. R006 returns to the tokenizer used by r003-r004; it is not compatible with r005's tokenizer beyond the vocabulary size and special-token IDs.
+The model was trained on [`SlayerLab/minimal-en-corpus-5b`](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-5b), pinned to commit `38bebbd`. It is a subjectively selected mixture of 15 English-language sources after language filtering, exact and approximate deduplication, and benchmark decontamination. R007 uses the tokenizer from this corpus; it retains the vocabulary size and special-token IDs, but other token IDs are not compatible with r006's tokenizer.
+
+The loader applies a seeded global permutation to the fixed token windows in each pass, assigns disjoint positions across DDP ranks, masks the final partial batch, and stores its global cursor, topology, and shuffle settings in every checkpoint.
 
 | Setting | Value |
 |---|---:|
-| Training / validation tokens | 2,689,323,439 / 5,236,486 |
-| Final checkpoint | update 22,003 |
-| Token presentations | 10,814,914,560 (approximately 4.02 epochs) |
-| Sequence / micro-batch per GPU | 1,024 / 12 |
+| Training / validation tokens | 5,396,605,407 / 5,238,223 |
+| Selected checkpoint | update 43,750 |
+| Token presentations | 21,503,919,992 (approximately 3.98 epochs) |
+| Sequence / micro-batch per GPU | 2,048 / 6 |
 | Global / per-GPU accumulation | 40 / 20 micro-steps |
 | Effective batch | 491,520 tokens |
-| Optimizer | fused AdamW, betas 0.9/0.95 |
-| Learning rate | 4e-4 → 4e-5, cosine decay |
-| Warmup / weight decay / grad clip | 440 / 0.1 / 1.0 |
+| Optimizer | Muon plus fused AdamW, AdamW betas 0.9/0.95 |
+| Learning rate | Muon 1e-2 → 1e-3; AdamW 3e-4 → 3e-5 |
+| Warmup / weight decay / grad clip | 200 / 0.01 / 1.0 |
 | Precision | BF16 |
 | Hardware | 2× NVIDIA GeForce RTX 4090 |
-| Framework | PyTorch 2.8.0+cu128, nanoGPT commit `3adf61e` |
+| Framework | PyTorch 2.11.0+cu128, nanoGPT commit `3adf61e` |
 
 ### Evaluation
 
-Training-time loss was estimated on fixed subsets of 1,228,800 tokens per split. The final checkpoint achieved validation loss **2.536357**; the best result was **2.536231** after 10,813,440,000 token presentations. These values are not directly comparable with r005 because r006 uses a different corpus, tokenizer, and validation set.
+Training loss was estimated on a fixed shuffled sample of 2,457,600 tokens from the complete training split. Every validation evaluation covered the full split, or 5,238,222 targets. The selected checkpoint recorded validation loss **2.458616** after 21,503,919,992 token presentations. An independent full-split evaluation reproduced cross-entropy **2.458519**, perplexity **11.687490**, and predictive entropy **2.437529**. Training ended after exactly four corpus passes, at update 43,918 and 21,586,421,624 token presentations.
 
-Benchmarks used complete splits with `lm-evaluation-harness` 0.4.12, zero few-shot examples, batch size 8, and BF16.
+Benchmarks used complete splits with `lm-evaluation-harness` 0.4.12, zero few-shot examples, batch size 8, maximum context 1024, and BF16 on Apple M1 Max MPS.
 
 | Benchmark | Primary metric | Score | Samples |
 |---|---|---:|---:|
-| BLiMP | acc | 0.769164 | 67,000 |
-| LAMBADA | acc | 0.276926 | 5,153 |
-| HellaSwag | acc_norm | 0.301334 | 10,042 |
+| BLiMP | acc | 0.784970 | 67,000 |
+| LAMBADA | acc | 0.300796 | 5,153 |
+| HellaSwag | acc_norm | 0.308305 | 10,042 |
 | PIQA | acc_norm | 0.596844 | 1,838 |
-| SciQ | acc_norm | 0.664000 | 1,000 |
-| ARC-Easy | acc_norm | 0.434343 | 2,376 |
-| ARC-Challenge | acc_norm | 0.241468 | 1,172 |
+| SciQ | acc_norm | 0.675000 | 1,000 |
+| ARC-Easy | acc_norm | 0.418350 | 2,376 |
+| ARC-Challenge | acc_norm | 0.250000 | 1,172 |
 
-LAMBADA perplexity was 49.907207. Full metrics and protocol details are recorded in [`benchmarks/english.json`](./benchmarks/english.json).
+LAMBADA perplexity is **45.030362**. Compared with the E2 measurement, the E4-best checkpoint improves BLiMP and ARC-Challenge but scores lower on the other five tasks. The validation cross-entropy gain therefore does not transfer uniformly to these downstream benchmarks. Full metrics and protocol details are recorded in [`benchmarks/english.json`](./benchmarks/english.json).
 
 ### Usage
 
@@ -212,11 +227,12 @@ Use the Transformers example in the Polish section. The artifact uses standard `
 
 ### Fixed inference samples
 
-The shared [`fixed-sampling-v1`](./inference-samples/README.md) suite runs the same four prompts on r001-r006. All published revisions are loaded from full immutable commit SHAs, and every weight-file SHA-256 is verified before inference. The reference protocol uses macOS 26.2 on arm64, CPU float32 with one thread, Transformers 5.15.1, seed 1337 reset for every prompt, temperature 0.7, top-k 50, and a 100-new-token limit.
+The shared [`fixed-sampling-v1`](./inference-samples/README.md) suite runs the same four prompts on r001-r007. All published revisions are loaded from full immutable commit SHAs, and every weight-file SHA-256 is verified before inference. The reference protocol uses macOS 26.2 on arm64, CPU float32 with one thread, Transformers 5.15.1, seed 1337 reset for every prompt, temperature 0.7, top-k 50, and a 100-new-token limit.
 
 | Revision | Release | Weights | History |
 |---|---|---|---|
-| **r006** | **Pollock 1.4** | [`a2e53d9`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/a2e53d9f1b690ebfbea375cf8bd3b5bdf69dece9) | [`r006.md`](./training-history/r006.md) |
+| **r007** | **Pollock 1.5** | [`f2a1ec0`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/f2a1ec072098d257c36b129b0e554e2e6de9c46e) | [`r007.md`](./training-history/r007.md) |
+| r006 | Pollock 1.4 | [`a2e53d9`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/a2e53d9f1b690ebfbea375cf8bd3b5bdf69dece9) | [`r006.md`](./training-history/r006.md) |
 | r005 | Pollock 1.3 | [`e780025`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/e780025f15e06bc3765a74d973906eb8a11c022c) | [`r005.md`](./training-history/r005.md) |
 | r004 | Pollock 1.2 | [`30feb81`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/30feb81e5097eef13b939e1827d98e0458bf602d) | [`r004.md`](./training-history/r004.md) |
 | r003 | Pollock 1.1 | [`698984b`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/698984b1d1b96c9b6ffaff00c7fc2e78e140e842) | [`r003.md`](./training-history/r003.md) |

benchmarks/english.json

@@ -1,72 +1,76 @@
 {
   "schema_version": 1,
-  "revision_id": "r006",
-  "release": "Pollock 1.4",
-  "source_result": "runs/r006-corrected-corpus-v2-s1337-lr0.0004/benchmarks/results.json",
+  "revision_id": "r007",
+  "source_result": "out-pollock-r007-e4-final/benchmarks/full/release-draft/results_2026-09-20T13-15-46.016579.json",
+  "source_result_sha256": "bfbda3e30e936d975d5ab110be17067b418048c10beb6fe5a4e8e760aa8a258c",
   "checkpoint": {
-    "path": "runs/r006-corrected-corpus-v2-s1337-lr0.0004/checkpoints/ckpt-final.pt",
-    "sha256": "580fbb3c039ea1f349285a9bf7a67731fb6d455de931b6b7601d4ed09fce9437",
-    "source_iteration": 22003
+    "path": "out-pollock-r007-e4-final/ckpt-best.pt",
+    "sha256": "384d912cc8b7ebf872b66d1f8ad46f223296a069e906167076f1899296f9a788",
+    "source_iteration": 43750,
+    "tokens_processed": 21503919992
   },
   "benchmark_checkpoint": {
-    "path": "/workspace/cache/dmpod-tokenizers/r006-final-benchmark.pt",
-    "sha256": "9339d20ea32d24bc9777518a2f0ec553e4a3871647d342db92d7c5b6c18c7b56",
-    "trusted_input_required": true
+    "path": "release-draft/model.safetensors",
+    "sha256": "4a88432225dac6255ae1e2a085019f36dcdaf55fa33c77629dba6e7445c6c2fb",
+    "conversion_max_absolute_logit_delta": 0.0
   },
   "tokenizer": {
-    "dataset_revision": "d68d992622e9fc11f19e7d7fb8547c4e653439a4",
-    "sha256": "6cda4e5ec8293f3b821e02253f9c0e88e43ad4f7c6e1324763e1fb91749fef51",
-    "version": "V2.0@d68d992622e9fc11f19e7d7fb8547c4e653439a4"
+    "dataset_revision": "38bebbd2ae36858c25aa0799fd3296fbfa9432aa",
+    "sha256": "3733307577230bb4802d2c774d8e2323f7f64e4139c13712736a57daf91bdda1"
   },
   "execution": {
     "harness": "lm-evaluation-harness 0.4.12",
+    "transformers": "5.15.1",
+    "torch": "2.14.0",
+    "device": "Apple M1 Max MPS",
     "num_fewshot": 0,
     "batch_size": 8,
     "precision": "bfloat16",
     "max_length": 1024,
     "limit_per_benchmark": null,
-    "truncated_requests": 0
+    "truncated_requests": 0,
+    "evaluation_time_seconds": 1552.1888507499825
   },
   "results": {
     "blimp": {
-      "acc,none": 0.7691641791044775,
-      "sample_len": 67000.0,
+      "acc,none": 0.7849701492537313,
+      "sample_len": 67000,
       "samples": 67000
     },
     "lambada_openai": {
-      "acc,none": 0.27692606248787116,
-      "perplexity,none": 49.907206551888116,
-      "sample_len": 5153.0,
+      "acc,none": 0.3007956530176596,
+      "perplexity,none": 45.030361770214945,
+      "sample_len": 5153,
       "samples": 5153
     },
     "hellaswag": {
-      "acc,none": 0.2830113523202549,
-      "acc_norm,none": 0.3013343955387373,
-      "sample_len": 10042.0,
+      "acc,none": 0.2876916948814977,
+      "acc_norm,none": 0.30830511850229037,
+      "sample_len": 10042,
       "samples": 10042
     },
     "piqa": {
-      "acc,none": 0.6071817192600653,
+      "acc,none": 0.6077257889009793,
       "acc_norm,none": 0.5968443960826986,
-      "sample_len": 1838.0,
+      "sample_len": 1838,
       "samples": 1838
     },
     "sciq": {
-      "acc,none": 0.768,
-      "acc_norm,none": 0.664,
-      "sample_len": 1000.0,
+      "acc,none": 0.766,
+      "acc_norm,none": 0.675,
+      "sample_len": 1000,
       "samples": 1000
     },
     "arc_easy": {
-      "acc,none": 0.4877946127946128,
-      "acc_norm,none": 0.43434343434343436,
-      "sample_len": 2376.0,
+      "acc,none": 0.4772727272727273,
+      "acc_norm,none": 0.41835016835016836,
+      "sample_len": 2376,
       "samples": 2376
     },
     "arc_challenge": {
-      "acc,none": 0.21075085324232082,
-      "acc_norm,none": 0.24146757679180889,
-      "sample_len": 1172.0,
+      "acc,none": 0.2158703071672355,
+      "acc_norm,none": 0.25,
+      "sample_len": 1172,
       "samples": 1172
     }
   }

config.json

@@ -12,13 +12,12 @@
   "initializer_range": 0.02,
   "layer_norm_epsilon": 1e-05,
   "model_type": "gpt2",
-  "model_version": "1.4",
-  "n_ctx": 1024,
-  "n_embd": 896,
-  "n_head": 14,
-  "n_inner": null,
-  "n_layer": 12,
-  "n_positions": 1024,
+  "n_ctx": 2048,
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": 3200,
+  "n_layer": 16,
+  "n_positions": 2048,
   "pad_token_id": 12285,
   "reorder_and_upcast_attn": false,
   "resid_pdrop": 0.0,

conversion_manifest.json

@@ -0,0 +1,28 @@
+{
+  "checkpoint": "out-pollock-r007-e4-final/ckpt-best.pt",
+  "checkpoint_sha256": "384d912cc8b7ebf872b66d1f8ad46f223296a069e906167076f1899296f9a788",
+  "tokenizer": "data/minimal_en_corpus_5b/tokenizer/tokenizer.json",
+  "tokenizer_sha256": "3733307577230bb4802d2c774d8e2323f7f64e4139c13712736a57daf91bdda1",
+  "model_sha256": "4a88432225dac6255ae1e2a085019f36dcdaf55fa33c77629dba6e7445c6c2fb",
+  "source_iteration": 43750,
+  "tokens_processed": 21503919992,
+  "model_args": {
+    "n_layer": 16,
+    "n_head": 12,
+    "n_embd": 768,
+    "n_inner": 3200,
+    "block_size": 2048,
+    "bias": false,
+    "vocab_size": 12288,
+    "dropout": 0.0
+  },
+  "transformers_version": "5.15.1",
+  "verification": {
+    "seed": 1337,
+    "input_shape": [
+      2,
+      64
+    ],
+    "max_absolute_logit_delta": 0.0
+  }
+}

inference-samples/README.md

@@ -24,6 +24,7 @@ Exact token replay was verified in the recorded environment. Sampling may diverg
 | r004 | Pollock 1.2 | [`30feb81`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/30feb81e5097eef13b939e1827d98e0458bf602d) | `3c453f22d4bb70228e5cca79e183a425e0f0221e54bd782d12027c42c943c880` |
 | r005 | Pollock 1.3 | [`e780025`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/e780025f15e06bc3765a74d973906eb8a11c022c) | `3db43dfa44622e8156f060fee6f16875e108db4f5f949969cb91c0ca8cdce32c` |
 | r006 | Pollock 1.4 | [`a2e53d9`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/a2e53d9f1b690ebfbea375cf8bd3b5bdf69dece9) | `8e74aa4d34464229e86a1831fb34d442f91e22aa600e0cf2503891957fdc6a43` |
+| r007 | Pollock 1.5 | [`f2a1ec0`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/f2a1ec072098d257c36b129b0e554e2e6de9c46e) | `4a88432225dac6255ae1e2a085019f36dcdaf55fa33c77629dba6e7445c6c2fb` |
 
 ## Results
 
@@ -120,6 +121,22 @@ The municipality of Aisne is situated on the banks of the Neuve-Brieu river, whi
 
 Output SHA-256: `b47fcb42354b8f735c7b98cfdf93605c6e095ed8290e2ddb3329886df244e99f`
 
+#### r007 - Pollock 1.5
+
+````text
+The capital of France is the port city of Château-Thierry.
+
+The capital of France is the port city of Mont-Royal.
+
+History
+
+The city of Mont-Royal was first mentioned by the French colonial administration in the 15th century. When the French occupation began in 1793, the city was captured by the French army and the French were forced to surrender to the British.
+
+The city is thought to have been the first port city in
+````
+
+Output SHA-256: `4b8907f330be8f045959ddb3ec1c50c88c82a1884c219aee6d3c6537a442ca84`
+
 ### Explanation
 
 Prompt: `Photosynthesis is the process by which`
@@ -197,6 +214,19 @@ Carbon gas: A mixture of gases in which carbon dioxide is replaced by a gas; in
 
 Output SHA-256: `c09d560713828851410d37fcd7c648d49c2bedf6e9c8325bb47cb32a8015bfd4`
 
+#### r007 - Pollock 1.5
+
+````text
+Photosynthesis is the process by which plants and animals use energy from sunlight and other sources to synthesize proteins and carbohydrates. The process is vital to survival and growth of plants and animals.
+
+2. Energy Capture in Plants:
+
+Plants are the primary source of energy for the Earth's ecosystems. The ability of plants to capture solar radiation is essential for their survival. Plants also convert atmospheric carbon dioxide into sugars, which are used by plants to build their cell walls.
+
+````
+
+Output SHA-256: `a8457bf7fa743d8397b5a221f4f11c5e0b978372eca68f068c17052307853591`
+
 ### Story
 
 Prompt: `In a distant future, humanity discovered`
@@ -276,6 +306,20 @@ This is fascinating! Can you tell me more about the specific cultural values tha
 
 Output SHA-256: `6455cdbe15a71ae11ff476739071f31b3f86b8bf4d97f3dabe99a7d1595c5c7c`
 
+#### r007 - Pollock 1.5
+
+````text
+In a distant future, humanity discovered the potential of the moon to guide humanity in their quest for knowledge and understanding.
+user
+Can you tell me more about the moon and its impact on humanity?
+assistant
+Sure! The moon is a natural feature of the Earth's surface and is a crucial part of the natural landscape of the Earth. Here are some ways in which the moon impacts human life:
+
+1. Earth's gravitational pull: The moon has a significant impact on human life. It
+````
+
+Output SHA-256: `00fb4213a5eeb36d18cd947a0876ad711f00bee4a8f576d75fd551d66183ba82`
+
 ### Code
 
 Prompt: `def fibonacci(n):`
@@ -408,6 +452,27 @@ def fibonacci(n):
 
 Output SHA-256: `ad3aa2ea2cab9a46ae5f52a9cc109bcf5f7362e1d214ceb791ee2dd6995a345f`
 
+#### r007 - Pollock 1.5
+
+````text
+def fibonacci(n):
+if n < 1:
+return 0
+
+if fibonacci(n):
+return fibonacci(n-1)
+
+return fibonacci(n-1) + fibonacci(n-2)
+
+print fibonacci(0) + fibonacci(0)
+
+return fibonacci(0) + fibonacci(0) + fibonacci(1)
+
+print fib
+````
+
+Output SHA-256: `012a9d1312d787e5e759f1af7a242f20f4c9fdfc895ae673b0889fd08a0e8fe1`
+
 ## Reproduce or verify
 
 Run from this directory. The check downloads approximately 2.4 GB of published model artifacts if they are not already cached.

inference-samples/config.json

@@ -60,6 +60,12 @@
       "release": "Pollock 1.4",
       "commit": "a2e53d9f1b690ebfbea375cf8bd3b5bdf69dece9",
       "model_sha256": "8e74aa4d34464229e86a1831fb34d442f91e22aa600e0cf2503891957fdc6a43"
+    },
+    {
+      "revision_id": "r007",
+      "release": "Pollock 1.5",
+      "commit": "f2a1ec072098d257c36b129b0e554e2e6de9c46e",
+      "model_sha256": "4a88432225dac6255ae1e2a085019f36dcdaf55fa33c77629dba6e7445c6c2fb"
     }
   ],
   "prompts": [

inference-samples/generate.py

@@ -218,6 +218,9 @@ def render_markdown(config: dict[str, Any], results: dict[str, Any]) -> str:
     protocol = results["protocol"]
     environment = results["environment"]
     revisions = {item["revision_id"]: item for item in results["revisions"]}
+    has_pending_commit = any(
+        item["commit"].startswith("__PENDING_") for item in results["revisions"]
+    )
     lines = [
         "# Fixed cross-revision inference samples",
         "",
@@ -250,6 +253,15 @@ def render_markdown(config: dict[str, Any], results: dict[str, Any]) -> str:
         "diverge on another operating system or CPU architecture even when package "
         "versions and seeds match.",
         "",
+        *(
+            [
+                "The r007 outputs currently use the local artifact whose weight hash is "
+                "shown below. Replay from its immutable publication commit remains pending.",
+                "",
+            ]
+            if has_pending_commit
+            else []
+        ),
         "## Revisions",
         "",
         "| Revision | Release | Immutable weights | `model.safetensors` SHA-256 |",
@@ -257,8 +269,11 @@ def render_markdown(config: dict[str, Any], results: dict[str, Any]) -> str:
     ]
     for revision in results["revisions"]:
         commit = revision["commit"]
-        commit_url = f"https://huggingface.co/{results['model_id']}/commit/{commit}"
-        weights = f"[`{commit[:7]}`]({commit_url})"
+        if commit.startswith("__PENDING_"):
+            weights = "local artifact; immutable commit pending"
+        else:
+            commit_url = f"https://huggingface.co/{results['model_id']}/commit/{commit}"
+            weights = f"[`{commit[:7]}`]({commit_url})"
         lines.append(
             f"| {revision['revision_id']} | {revision['release']} | "
             f"{weights} | `{revision['model_sha256']}` |"

inference-samples/results.json

@@ -2945,6 +2945,498 @@
           "full_text_sha256": "ad3aa2ea2cab9a46ae5f52a9cc109bcf5f7362e1d214ceb791ee2dd6995a345f"
         }
       ]
+    },
+    {
+      "revision_id": "r007",
+      "release": "Pollock 1.5",
+      "commit": "f2a1ec072098d257c36b129b0e554e2e6de9c46e",
+      "model_sha256": "4a88432225dac6255ae1e2a085019f36dcdaf55fa33c77629dba6e7445c6c2fb",
+      "loaded_model_version": null,
+      "results": [
+        {
+          "prompt_id": "general_knowledge",
+          "prompt": "The capital of France is",
+          "prompt_token_ids": [
+            489,
+            4496,
+            286,
+            4392,
+            311
+          ],
+          "completion_token_ids": [
+            263,
+            2242,
+            2405,
+            286,
+            689,
+            11913,
+            690,
+            2183,
+            45,
+            980,
+            1291,
+            595,
+            46,
+            10,
+            10,
+            489,
+            4496,
+            286,
+            4392,
+            311,
+            263,
+            2242,
+            2405,
+            286,
+            6334,
+            45,
+            82,
+            4792,
+            46,
+            10,
+            10,
+            8231,
+            10,
+            10,
+            489,
+            2405,
+            286,
+            6334,
+            45,
+            82,
+            4792,
+            399,
+            770,
+            4888,
+            410,
+            263,
+            3369,
+            4683,
+            460,
+            7484,
+            287,
+            263,
+            1487,
+            373,
+            2882,
+            46,
+            1769,
+            263,
+            3369,
+            4727,
+            328,
+            2716,
+            287,
+            1430,
+            5091,
+            44,
+            263,
+            2405,
+            399,
+            5533,
+            410,
+            263,
+            3369,
+            5762,
+            292,
+            263,
+            3369,
+            586,
+            6113,
+            290,
+            968,
+            9811,
+            290,
+            263,
+            3036,
+            46,
+            10,
+            10,
+            489,
+            2405,
+            311,
+            1873,
+            290,
+            451,
+            698,
+            263,
+            770,
+            2242,
+            2405,
+            287
+          ],
+          "completion_tokens": 100,
+          "stopped_on_eos": false,
+          "completion": " the port city of Château-Thierry.\n\nThe capital of France is the port city of Mont-Royal.\n\nHistory\n\nThe city of Mont-Royal was first mentioned by the French colonial administration in the 15th century. When the French occupation began in 1793, the city was captured by the French army and the French were forced to surrender to the British.\n\nThe city is thought to have been the first port city in",
+          "full_text": "The capital of France is the port city of Château-Thierry.\n\nThe capital of France is the port city of Mont-Royal.\n\nHistory\n\nThe city of Mont-Royal was first mentioned by the French colonial administration in the 15th century. When the French occupation began in 1793, the city was captured by the French army and the French were forced to surrender to the British.\n\nThe city is thought to have been the first port city in",
+          "full_text_sha256": "4b8907f330be8f045959ddb3ec1c50c88c82a1884c219aee6d3c6537a442ca84"
+        },
+        {
+          "prompt_id": "explanation",
+          "prompt": "Photosynthesis is the process by which",
+          "prompt_token_ids": [
+            3561,
+            316,
+            394,
+            10904,
+            7834,
+            311,
+            263,
+            1198,
+            410,
+            510
+          ],
+          "completion_token_ids": [
+            3301,
+            292,
+            3581,
+            709,
+            1552,
+            434,
+            8653,
+            292,
+            602,
+            3760,
+            290,
+            5190,
+            915,
+            869,
+            7965,
+            292,
+            11698,
+            1240,
+            3465,
+            7969,
+            46,
+            383,
+            1198,
+            311,
+            8440,
+            290,
+            9161,
+            292,
+            3093,
+            286,
+            3301,
+            292,
+            3581,
+            46,
+            10,
+            10,
+            50,
+            46,
+            8645,
+            7569,
+            443,
+            287,
+            1788,
+            1222,
+            58,
+            10,
+            10,
+            4315,
+            1222,
+            376,
+            263,
+            3919,
+            2350,
+            286,
+            1552,
+            332,
+            263,
+            2921,
+            456,
+            8124,
+            115,
+            46,
+            383,
+            3315,
+            286,
+            3301,
+            290,
+            7396,
+            4487,
+            5888,
+            311,
+            3480,
+            332,
+            548,
+            9161,
+            46,
+            1788,
+            1222,
+            641,
+            5759,
+            11784,
+            3699,
+            9079,
+            686,
+            422,
+            103,
+            887,
+            44,
+            510,
+            376,
+            851,
+            410,
+            3301,
+            290,
+            1472,
+            548,
+            1442,
+            6800,
+            46,
+            10
+          ],
+          "completion_tokens": 100,
+          "stopped_on_eos": false,
+          "completion": " plants and animals use energy from sunlight and other sources to synthesize proteins and carbohydrates. The process is vital to survival and growth of plants and animals.\n\n2. Energy Capture in Plants:\n\nPlants are the primary source of energy for the Earth's ecosystems. The ability of plants to capture solar radiation is essential for their survival. Plants also convert atmospheric carbon dioxide into sugars, which are used by plants to build their cell walls.\n",
+          "full_text": "Photosynthesis is the process by which plants and animals use energy from sunlight and other sources to synthesize proteins and carbohydrates. The process is vital to survival and growth of plants and animals.\n\n2. Energy Capture in Plants:\n\nPlants are the primary source of energy for the Earth's ecosystems. The ability of plants to capture solar radiation is essential for their survival. Plants also convert atmospheric carbon dioxide into sugars, which are used by plants to build their cell walls.\n",
+          "full_text_sha256": "a8457bf7fa743d8397b5a221f4f11c5e0b978372eca68f068c17052307853591"
+        },
+        {
+          "prompt_id": "story",
+          "prompt": "In a distant future, humanity discovered",
+          "prompt_token_ids": [
+            750,
+            257,
+            10043,
+            2639,
+            44,
+            1871,
+            421,
+            4919
+          ],
+          "completion_token_ids": [
+            263,
+            2291,
+            286,
+            263,
+            6601,
+            290,
+            5396,
+            1871,
+            421,
+            287,
+            548,
+            1264,
+            332,
+            2981,
+            292,
+            3167,
+            46,
+            12287,
+            10,
+            12286,
+            1518,
+            10,
+            3648,
+            359,
+            2032,
+            525,
+            584,
+            652,
+            263,
+            6601,
+            292,
+            671,
+            2539,
+            337,
+            1871,
+            421,
+            63,
+            12287,
+            10,
+            12286,
+            2146,
+            10,
+            8661,
+            33,
+            383,
+            6601,
+            311,
+            257,
+            2387,
+            3629,
+            286,
+            263,
+            2921,
+            456,
+            2395,
+            292,
+            311,
+            257,
+            7166,
+            647,
+            286,
+            263,
+            2387,
+            7462,
+            286,
+            263,
+            2921,
+            46,
+            2843,
+            376,
+            649,
+            2658,
+            287,
+            510,
+            263,
+            6601,
+            9220,
+            1871,
+            1227,
+            58,
+            10,
+            10,
+            49,
+            46,
+            2921,
+            456,
+            11026,
+            4326,
+            58,
+            383,
+            6601,
+            564,
+            257,
+            2239,
+            2539,
+            337,
+            1871,
+            1227,
+            46,
+            672
+          ],
+          "completion_tokens": 100,
+          "stopped_on_eos": false,
+          "completion": " the potential of the moon to guide humanity in their quest for knowledge and understanding.\nuser\nCan you tell me more about the moon and its impact on humanity?\nassistant\nSure! The moon is a natural feature of the Earth's surface and is a crucial part of the natural landscape of the Earth. Here are some ways in which the moon impacts human life:\n\n1. Earth's gravitational pull: The moon has a significant impact on human life. It",
+          "full_text": "In a distant future, humanity discovered the potential of the moon to guide humanity in their quest for knowledge and understanding.\nuser\nCan you tell me more about the moon and its impact on humanity?\nassistant\nSure! The moon is a natural feature of the Earth's surface and is a crucial part of the natural landscape of the Earth. Here are some ways in which the moon impacts human life:\n\n1. Earth's gravitational pull: The moon has a significant impact on human life. It",
+          "full_text_sha256": "00fb4213a5eeb36d18cd947a0876ad711f00bee4a8f576d75fd551d66183ba82"
+        },
+        {
+          "prompt_id": "code",
+          "prompt": "def fibonacci(n):",
+          "prompt_token_ids": [
+            1806,
+            8247,
+            262,
+            3119,
+            105,
+            40,
+            110,
+            992
+          ],
+          "completion_token_ids": [
+            10,
+            351,
+            303,
+            684,
+            345,
+            58,
+            10,
+            4941,
+            620,
+            10,
+            10,
+            351,
+            8247,
+            262,
+            3119,
+            105,
+            40,
+            110,
+            992,
+            10,
+            4941,
+            8247,
+            262,
+            3119,
+            105,
+            40,
+            110,
+            45,
+            49,
+            41,
+            10,
+            10,
+            4941,
+            8247,
+            262,
+            3119,
+            105,
+            40,
+            110,
+            45,
+            49,
+            41,
+            757,
+            8247,
+            262,
+            3119,
+            105,
+            40,
+            110,
+            45,
+            50,
+            41,
+            10,
+            10,
+            2647,
+            8247,
+            262,
+            3119,
+            105,
+            40,
+            48,
+            41,
+            757,
+            8247,
+            262,
+            3119,
+            105,
+            40,
+            48,
+            41,
+            10,
+            10,
+            4941,
+            8247,
+            262,
+            3119,
+            105,
+            40,
+            48,
+            41,
+            757,
+            8247,
+            262,
+            3119,
+            105,
+            40,
+            48,
+            41,
+            757,
+            8247,
+            262,
+            3119,
+            105,
+            40,
+            49,
+            41,
+            10,
+            10,
+            2647,
+            8247
+          ],
+          "completion_tokens": 100,
+          "stopped_on_eos": false,
+          "completion": "\nif n < 1:\nreturn 0\n\nif fibonacci(n):\nreturn fibonacci(n-1)\n\nreturn fibonacci(n-1) + fibonacci(n-2)\n\nprint fibonacci(0) + fibonacci(0)\n\nreturn fibonacci(0) + fibonacci(0) + fibonacci(1)\n\nprint fib",
+          "full_text": "def fibonacci(n):\nif n < 1:\nreturn 0\n\nif fibonacci(n):\nreturn fibonacci(n-1)\n\nreturn fibonacci(n-1) + fibonacci(n-2)\n\nprint fibonacci(0) + fibonacci(0)\n\nreturn fibonacci(0) + fibonacci(0) + fibonacci(1)\n\nprint fib",
+          "full_text_sha256": "012a9d1312d787e5e759f1af7a242f20f4c9fdfc895ae673b0889fd08a0e8fe1"
+        }
+      ]
     }
   ]
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8e74aa4d34464229e86a1831fb34d442f91e22aa600e0cf2503891957fdc6a43
-size 510713512
+oid sha256:4a88432225dac6255ae1e2a085019f36dcdaf55fa33c77629dba6e7445c6c2fb
+size 510281160

release_manifest.json

@@ -1,81 +1,118 @@
 {
   "schema_version": 3,
-  "revision": 6,
-  "revision_id": "r006",
-  "release": "Pollock 1.4",
+  "revision": 7,
+  "revision_id": "r007",
+  "release": "Pollock 1.5",
   "model_id": "SlayerLab/pollock-mini-lm-125m",
+  "draft": false,
   "publication": {
-    "weights_commit": "a2e53d9f1b690ebfbea375cf8bd3b5bdf69dece9",
-    "version_tag": "v1.4",
-    "publication_commit_pending": true
+    "weights_commit": "f2a1ec072098d257c36b129b0e554e2e6de9c46e",
+    "version_tag": "v1.5",
+    "publication_commit_pending": false
   },
   "source_checkpoint": {
-    "path_in_training_workspace": "runs/r006-corrected-corpus-v2-s1337-lr0.0004/checkpoints/ckpt-final.pt",
-    "sha256": "580fbb3c039ea1f349285a9bf7a67731fb6d455de931b6b7601d4ed09fce9437",
-    "iteration": 22003,
-    "tokens_seen": 10814914560,
-    "native_nanogpt_parameters": 126637952,
-    "native_unique_trainable_parameters": 127555456
+    "role": "best-full-validation",
+    "path_in_training_workspace": "out-pollock-r007-e4-final/ckpt-best.pt",
+    "sha256": "384d912cc8b7ebf872b66d1f8ad46f223296a069e906167076f1899296f9a788",
+    "iteration": 43750,
+    "tokens_seen": 21503919992,
+    "data_pass_equivalent": 3.9847123089835352,
+    "native_nanogpt_parameters": 125854464,
+    "native_unique_trainable_parameters": 127427328
+  },
+  "terminal_checkpoint": {
+    "path_in_training_workspace": "out-pollock-r007-e4-final/ckpt-e4-terminal-step-00043918.pt",
+    "sha256": "ef6952d47b717266ff57272b0890a7c54ed36111bb65a2c22bb2630df87f3016",
+    "iteration": 43918,
+    "tokens_seen": 21586421624,
+    "data_pass_equivalent": 4.0
   },
   "architecture": {
-    "n_layer": 12,
-    "n_head": 14,
-    "n_embd": 896,
-    "block_size": 1024,
+    "n_layer": 16,
+    "n_head": 12,
+    "n_embd": 768,
+    "n_inner": 3200,
+    "block_size": 2048,
     "vocab_size": 12288,
     "dropout": 0.0,
     "bias": false,
     "tied_word_embeddings": true
   },
   "training": {
-    "dataset": "SlayerLab/minimal-en-corpus-2.5b",
-    "dataset_revision": "d68d992622e9fc11f19e7d7fb8547c4e653439a4",
-    "training_tokens": 2689323439,
-    "validation_tokens": 5236486,
-    "tokenizer_sha256": "6cda4e5ec8293f3b821e02253f9c0e88e43ad4f7c6e1324763e1fb91749fef51",
-    "native_tokenizer_sha256": "d0d126f0a59e51e8cb1b26d77b5527870be0e8ea1e84adefd730fff906c48234",
+    "dataset": "SlayerLab/minimal-en-corpus-5b",
+    "dataset_revision": "38bebbd2ae36858c25aa0799fd3296fbfa9432aa",
+    "training_tokens": 5396605407,
+    "validation_tokens": 5238223,
+    "tokenizer_sha256": "3733307577230bb4802d2c774d8e2323f7f64e4139c13712736a57daf91bdda1",
     "init_from": "scratch",
-    "micro_batch_per_gpu": 12,
+    "data_sampling": "seeded global permutation of fixed token windows per pass",
+    "data_shuffle_seed": 1337,
+    "micro_batch_per_gpu": 6,
     "gradient_accumulation_global": 40,
     "gradient_accumulation_per_gpu": 20,
     "ddp_world_size": 2,
     "effective_batch_tokens": 491520,
-    "optimizer": "fused AdamW",
-    "learning_rate": 0.0004,
-    "min_learning_rate": 0.00004,
-    "schedule": "cosine",
-    "warmup_iters": 440,
+    "optimizer": "Muon plus fused AdamW",
+    "muon_parameters": 116391936,
+    "adamw_parameters": 11035392,
+    "muon_learning_rate": 0.01,
+    "muon_min_learning_rate": 0.001,
+    "adamw_learning_rate": 0.0003,
+    "adamw_min_learning_rate": 0.00003,
+    "schedule": "200-update warmup and cosine decay through update 21959; learning-rate floors thereafter",
+    "warmup_iters": 200,
     "beta1": 0.9,
     "beta2": 0.95,
-    "weight_decay": 0.1,
+    "muon_momentum": 0.95,
+    "weight_decay": 0.01,
     "grad_clip": 1.0,
     "precision": "bfloat16",
     "hardware": "2x NVIDIA GeForce RTX 4090",
-    "framework": "PyTorch 2.8.0+cu128",
+    "framework": "PyTorch 2.11.0+cu128",
     "nanogpt_commit": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291",
-    "data_pass_equivalent": 4.021425762020438,
-    "runtime_hours": 15.226956854563,
-    "mean_tokens_per_second": 199894.6379482684
+    "completed_data_passes": 4.0,
+    "runtime_hours": null,
+    "mean_tokens_per_second": null
   },
   "evaluation": {
-    "protocol": "fixed sampled subset",
-    "subset_tokens_per_split": 1228800,
-    "final_train_loss": 2.48974818944931,
-    "final_validation_loss": 2.536356544494629,
-    "best_validation_loss": 2.5362311387062073,
-    "best_validation_update": 22000,
-    "best_validation_tokens_seen": 10813440000,
-    "final_validation_perplexity": 12.633557211556399,
+    "protocol": "complete validation split plus fixed corpus-wide shuffled train sample",
+    "train_subset_tokens": 2457600,
+    "validation_targets": 5238222,
+    "selected_train_loss_rounded": 2.4227,
+    "selected_validation_loss": 2.458615977563123,
+    "selected_validation_perplexity": 11.688623023409297,
+    "selected_validation_update": 43750,
+    "selected_validation_tokens_seen": 21503919992,
+    "terminal_train_loss_rounded": 2.4226,
+    "terminal_validation_loss_rounded": 2.4587,
+    "e4_best_diagnostic_status": "completed",
+    "e4_best_training_workspace_results_file": "evaluations/r007_epochs/results/E4-best/metrics.json",
+    "e4_best_cross_entropy": 2.458519056568919,
+    "e4_best_perplexity": 11.687490205342804,
+    "e4_best_predictive_entropy": 2.4375294293734733,
+    "e4_best_repetition_4gram": 0.33997844253584,
+    "e4_best_repetition_8gram": 0.2570576867781918,
+    "e4_best_repetition_16gram": 0.20893125597814885,
+    "benchmark_status": "completed",
     "benchmark_harness": "lm-evaluation-harness 0.4.12",
     "benchmark_num_fewshot": 0,
     "benchmark_batch_size": 8,
     "truncated_benchmark_requests": 0,
-    "benchmark_checkpoint_sha256": "9339d20ea32d24bc9777518a2f0ec553e4a3871647d342db92d7c5b6c18c7b56",
-    "benchmark_tokenizer_sha256": "6cda4e5ec8293f3b821e02253f9c0e88e43ad4f7c6e1324763e1fb91749fef51",
-    "results_file": "benchmarks/english.json"
+    "benchmark_checkpoint_sha256": "4a88432225dac6255ae1e2a085019f36dcdaf55fa33c77629dba6e7445c6c2fb",
+    "benchmark_tokenizer_sha256": "3733307577230bb4802d2c774d8e2323f7f64e4139c13712736a57daf91bdda1",
+    "results_file": "benchmarks/english.json",
+    "blimp_accuracy": 0.7849701492537313,
+    "lambada_openai_accuracy": 0.3007956530176596,
+    "lambada_openai_perplexity": 45.030361770214945,
+    "hellaswag_accuracy_normalized": 0.30830511850229037,
+    "piqa_accuracy_normalized": 0.5968443960826986,
+    "sciq_accuracy_normalized": 0.675,
+    "arc_easy_accuracy_normalized": 0.41835016835016836,
+    "arc_challenge_accuracy_normalized": 0.25
   },
   "fixed_inference": {
     "suite_id": "fixed-sampling-v1",
+    "status": "completed",
     "comparison_file": "inference-samples/README.md",
     "config_file": "inference-samples/config.json",
     "generator_file": "inference-samples/generate.py",
@@ -86,44 +123,47 @@
     "dtype": "float32",
     "seed": 1337,
     "prompt_count": 4,
-    "revision_count": 6,
-    "local_review_artifact": false,
+    "revision_count": 7,
     "exact_replay_verified": true,
-    "publication_commit_pending": true
+    "publication_commit_pending": false,
+    "r007_weights_commit": "f2a1ec072098d257c36b129b0e554e2e6de9c46e"
   },
   "conversion": {
+    "status": "completed",
     "target_class": "GPT2LMHeadModel",
     "transformers_version": "5.15.1",
-    "checkpoint_sha256": "580fbb3c039ea1f349285a9bf7a67731fb6d455de931b6b7601d4ed09fce9437",
-    "model_sha256": "8e74aa4d34464229e86a1831fb34d442f91e22aa600e0cf2503891957fdc6a43",
-    "unique_serialized_parameters": 127674624,
-    "compatibility_zero_bias_parameters": 119168,
+    "checkpoint_sha256": "384d912cc8b7ebf872b66d1f8ad46f223296a069e906167076f1899296f9a788",
+    "model_sha256": "4a88432225dac6255ae1e2a085019f36dcdaf55fa33c77629dba6e7445c6c2fb",
+    "unique_serialized_parameters": 127565312,
+    "compatibility_zero_bias_parameters": 137984,
     "validation_probe_shape": [2, 64],
     "max_absolute_logit_error": 0.0
   },
   "artifacts": {
     ".gitattributes": {"sha256": "c759491a998899dbefaec4d51cc791e68c714a37dd9f9829020a368788fe3063"},
-    "CHANGELOG.md": {"sha256": "8422ad337b22be7adb561d1b2388cf53e5a6d1c6d49d628468498c72f078769a"},
-    "LICENSE.md": {"sha256": "46cbe928ed0aa24875f02f774313b27ec9d8abdf41f0c9ef67e0adb4e0614de4"},
-    "README.md": {"sha256": "875fa9cd748691e376df905e036a0c471db77aaa1272d71c5ea10d0f76fa888d"},
+    "CHANGELOG.md": {"sha256": "c2f62c0227cb8ccf18417bf91ff4dbfcc1c08069815fe215785e3207cbabde11"},
+    "LICENSE.md": {"sha256": "162b97898a956d43d0a5428d1999bac3e42e2a97eb01a79f73ad719a4a15c412"},
+    "README.md": {"sha256": "b9e153c60d9dd14caf19e02515522c7ef17d14fa1fc7037a0acbaf48a6a7d49a"},
     "assets/pollock-mini-lm-avatar-320.png": {"sha256": "7be10cc9d0f4aedeb298d9a5d722b2b2ac5b2e219b3884916dbca16198f70750"},
-    "benchmarks/english.json": {"sha256": "646a3aff7c97859e6974baa2c191702829a982af87675870dd1a339f8073a146"},
-    "config.json": {"sha256": "1529f8a8fc31b4fa68fc18fffbad7e155dc8f9e8663a40b4f3306b4012ea38c0"},
+    "benchmarks/english.json": {"sha256": "430feae7aefd6d8e226f61403a9e1080274daac538361d9ab6dcd5e1129218e2"},
+    "config.json": {"sha256": "aedc3eea385b4295b3ba257eb8533b06046eea0cca1ff009515ad3f1adbb4854"},
+    "conversion_manifest.json": {"sha256": "eb207d676eca255f7dd9415049b825f55ea81f8eed11cc9b0fc85a3f8b2bea5d"},
     "generation_config.json": {"sha256": "435beb27be51f0ed054f4a011e5109d125cdadc118b8799b18b155cc798d94d2"},
-    "inference-samples/README.md": {"sha256": "6559c85e1ed70f47f2d2978f28151fdd9e3355e397c5591cbe0b8ab767fa5ac5"},
-    "inference-samples/config.json": {"sha256": "a1b306a1e1f3a70942706d00107a83dd98289a534d17af78adbb95d48c9b69c1"},
-    "inference-samples/generate.py": {"sha256": "807a9946472bd0087dd2b9260ee90d1dea09785d24d4f923f028667c53a23933"},
+    "inference-samples/README.md": {"sha256": "66a15dba47cc44864f9accaad51955ed11c61d2103472ecf7a0c3451aa2fb291"},
+    "inference-samples/config.json": {"sha256": "3e908dfa8ce6085092dc665f7bdf7b69490fe49779c74a2c69adbd7265403ed9"},
+    "inference-samples/generate.py": {"sha256": "b4bb2edc64924ebdcbbd4ed7d0e72c44e6d9d05e58c96541c00cdeeb075b917d"},
     "inference-samples/requirements.txt": {"sha256": "584583335ffb3061aa62058aee725b0de25fed7523904fbd13fb4622f601943d"},
-    "inference-samples/results.json": {"sha256": "a279806ea225130b0e0aa36d9bd13653401e236395fe64b5f0aa7c825d445620"},
-    "model.safetensors": {"sha256": "8e74aa4d34464229e86a1831fb34d442f91e22aa600e0cf2503891957fdc6a43"},
+    "inference-samples/results.json": {"sha256": "ab07a2bf47297ab17f12fbd00aa5ca10569eef31a07984ba1af6e0321d689841"},
+    "model.safetensors": {"sha256": "4a88432225dac6255ae1e2a085019f36dcdaf55fa33c77629dba6e7445c6c2fb"},
     "special_tokens_map.json": {"sha256": "8b2257a17ea997bb038f43b133aefec82344ad2b8abc2b8a02a6c0a994ed624e"},
-    "tokenizer.json": {"sha256": "6cda4e5ec8293f3b821e02253f9c0e88e43ad4f7c6e1324763e1fb91749fef51"},
-    "tokenizer_config.json": {"sha256": "4cdabe37dbdc1adfcc017ee9a1f86ab89bdf184827d2d9f05181cae0f8af19bf"},
+    "tokenizer.json": {"sha256": "3733307577230bb4802d2c774d8e2323f7f64e4139c13712736a57daf91bdda1"},
+    "tokenizer_config.json": {"sha256": "39a075b24f05384c6d759521563d879cab4bb80b0fe94381e6e792966ab4c2ba"},
     "training-history/r001.md": {"sha256": "5b0ac39265921ed3604e7dfe8181b00d07902640d8024f591462eeb508bcfece"},
     "training-history/r002.md": {"sha256": "80031a84b2b67dc4cb4f81a178fa354feea15cb184d17368ee7e64ec3d0a2629"},
     "training-history/r003.md": {"sha256": "7ea9e6b15a0af635ff238417b9fff2a98c12a9090800a0923d622253b7eab02d"},
     "training-history/r004.md": {"sha256": "e260bfba69f154caf16f405d6944c2f963f467da9ed1e6c3c94e08932feae9b9"},
     "training-history/r005.md": {"sha256": "a25c2932701695dd7402582510adbe71e50a6fcc2da0fb3c6224a846dfdb22df"},
-    "training-history/r006.md": {"sha256": "47d53f9aa8a233eb7a319f559209936df988e83d96f6f6517fa3670b5dd2cdd3"}
+    "training-history/r006.md": {"sha256": "47d53f9aa8a233eb7a319f559209936df988e83d96f6f6517fa3670b5dd2cdd3"},
+    "training-history/r007.md": {"sha256": "fc9a4153eb8c073287f76b82b89ffaad62f3a606ba000e53bf450e393a5807d6"}
   }
 }

tokenizer_config.json

@@ -6,7 +6,7 @@
     "<|im_start|>",
     "<|im_end|>"
   ],
-  "model_max_length": 1024,
+  "model_max_length": 2048,
   "pad_token": "<|endoftext|>",
   "tokenizer_class": "TokenizersBackend"
 }

training-history/r007.md

@@ -0,0 +1,71 @@
+# r007 — Pollock 1.5 training record
+
+## Identity and provenance
+
+| Field | Value |
+|---|---|
+| Revision / release | `r007` / Pollock 1.5 |
+| Source run | `r007-muon-128m` |
+| Selected checkpoint | update 43,750 |
+| Selected checkpoint SHA-256 | `384d912cc8b7ebf872b66d1f8ad46f223296a069e906167076f1899296f9a788` |
+| Terminal E4 checkpoint | update 43,918 |
+| Terminal E4 checkpoint SHA-256 | `ef6952d47b717266ff57272b0890a7c54ed36111bb65a2c22bb2630df87f3016` |
+| `model.safetensors` SHA-256 | `4a88432225dac6255ae1e2a085019f36dcdaf55fa33c77629dba6e7445c6c2fb` |
+
+## Architecture and training
+
+| Setting | Value |
+|---|---:|
+| Layers / heads / width | 16 / 12 / 768 |
+| MLP width | 3,200 |
+| Context / vocabulary | 2048 / 12288 |
+| Native trainable parameters | 127,427,328 |
+| nanoGPT non-embedding parameters | 125,854,464 |
+| Dataset | SlayerLab/minimal-en-corpus-5b (`38bebbd`) |
+| Tokenizer SHA-256 | `3733307577230bb4802d2c774d8e2323f7f64e4139c13712736a57daf91bdda1` |
+| Selected token presentations / passes | 21,503,919,992 / 3.984712 |
+| Terminal token presentations / passes | 21,586,421,624 / 4.000000 |
+| Effective batch | 491,520 tokens |
+| Optimizer | Muon plus fused AdamW |
+| Muon LR range | 1e-02 → 1e-03 |
+| AdamW LR range | 3e-04 → 3e-05 |
+| Precision / hardware | BF16 / 2× NVIDIA GeForce RTX 4090 |
+| Framework | PyTorch 2.11.0+cu128 |
+
+Muon optimized 116,391,936 parameters in eligible hidden attention and MLP matrices. Fused AdamW optimized the remaining 11,035,392 parameters in token and position embeddings, the tied output head, normalization parameters, and biases. The parameter partition was checked for overlap and completeness before training.
+
+The loader applied a seeded global permutation to fixed 2,048-token windows in every pass, assigned disjoint positions across the two DDP ranks, masked each final partial batch, and stored its global cursor, topology, and shuffle settings in every checkpoint. Training completed exactly four passes over the 5,396,605,407-token training split.
+
+The first two passes used a 200-update linear warmup followed by cosine decay through update 21,959. The E3 and E4 continuation kept both learning rates at their floors without a restart or second cosine cycle.
+
+## Training-time evaluation
+
+Every validation evaluation covered the complete 5,238,222-target validation split. Train evaluation used a separate fixed, corpus-wide shuffled sample of 2,457,600 tokens without advancing the training loader.
+
+The selected update-43,750 checkpoint recorded train-eval loss **2.4227** and full-validation loss **2.458616** (perplexity **11.688623**) after 21,503,919,992 token presentations. The terminal update-43,918 evaluation recorded train-eval loss **2.4226** and full-validation loss **2.4587**; these terminal values are printed to four decimal places in the training log.
+
+The independent locked `E4-best` diagnostic evaluation reproduced full-validation cross-entropy **2.458519**, perplexity **11.687490**, and predictive entropy **2.437529** on Apple MPS with BF16. It improved CE over E3 in every reported domain, every context-position bucket, and every document-length bucket. The fixed 45-prompt generation suite measured repeated 4-, 8-, and 16-gram ratios of **0.339978**, **0.257058**, and **0.208931**. These are lower than E3's ratios but remain above E2's.
+
+The complete machine-readable diagnostic outputs are preserved in the training workspace under `evaluations/r007_epochs/results/E4-best/`.
+
+## English zero-shot benchmarks
+
+The selected update-43,750 checkpoint was evaluated on complete splits with zero few-shot examples, `lm-evaluation-harness` 0.4.12, BF16, batch size 8, and maximum context 1024 on Apple M1 Max MPS. Exact benchmark-checkpoint and tokenizer hashes are preserved in [`../benchmarks/english.json`](../benchmarks/english.json).
+
+| Benchmark | Primary metric | E2 | E4-best | Delta |
+|---|---:|---:|---:|---:|
+| BLiMP | accuracy | 0.784299 | 0.784970 | +0.000672 |
+| LAMBADA OpenAI | accuracy | 0.314186 | 0.300796 | -0.013390 |
+| HellaSwag | normalized accuracy | 0.310795 | 0.308305 | -0.002490 |
+| PIQA | normalized accuracy | 0.603373 | 0.596844 | -0.006529 |
+| SciQ | normalized accuracy | 0.681000 | 0.675000 | -0.006000 |
+| ARC-Easy | normalized accuracy | 0.429714 | 0.418350 | -0.011364 |
+| ARC-Challenge | normalized accuracy | 0.247440 | 0.250000 | +0.002560 |
+
+LAMBADA perplexity changed from **44.145462** at E2 to **45.030362** at E4-best. E4-best improves validation cross-entropy and the locked generation repetition suite relative to E3, but its downstream benchmark movement relative to E2 is mixed.
+
+## Conversion and release status
+
+The selected native checkpoint was converted to `GPT2LMHeadModel` with Transformers 5.15.1. A deterministic `[2, 64]` parity probe produced maximum absolute logit error **0.0**. The resulting `model.safetensors` SHA-256 is `4a88432225dac6255ae1e2a085019f36dcdaf55fa33c77629dba6e7445c6c2fb`.
+
+The `fixed-sampling-v1` suite completed in its pinned environment and passed exact regeneration checks for all r001-r007 token sequences. R007 was replayed directly from immutable Hugging Face commit [`f2a1ec0`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/f2a1ec072098d257c36b129b0e554e2e6de9c46e), whose `model.safetensors` hash matches the local converted artifact. Merge and version tag `v1.5` remain pending.