Release Pollock 1.4 (r006)

## Summary

- Publishes Pollock 1.4 / r006 from the completed `r006-corrected-corpus-v2-s1337-lr0.0004` run.
- Returns to the corrected V2 `SlayerLab/minimal-en-corpus-2.5b` data and the r003-r004 tokenizer while retaining the 12/14/896 architecture.
- Updates the bilingual model card, changelog, licensing notice, structured benchmarks, and training record.

## Validation

- Final native checkpoint SHA-256: `580fbb3c039ea1f349285a9bf7a67731fb6d455de931b6b7601d4ed09fce9437`
- `model.safetensors` SHA-256: `8e74aa4d34464229e86a1831fb34d442f91e22aa600e0cf2503891957fdc6a43`
- Transformers 5.15.1 conversion parity: maximum absolute logit error `0.0` on the deterministic `[2, 64]` probe
- Local Transformers load, parameter count, special-token IDs, and forward pass verified
- Seven full zero-shot benchmark splits recorded with no truncated requests

The fixed-sampling artifacts and release manifest will be finalized in this PR against this commit immutable weights SHA.

CHANGELOG.md

@@ -2,6 +2,15 @@
 
 Revision numbers identify published model states independently of release names.
 
+## r006 — Pollock 1.4
+
+- Returned from the r005 5B corpus to V2 of `SlayerLab/minimal-en-corpus-2.5b`, including its corrected cleaning and record boundaries.
+- Restored the tokenizer used by r003-r004 and trained the unchanged 12/14/896 architecture for approximately 4.02 data passes at the same 491,520-token effective batch and 4e-4→4e-5 learning-rate range.
+- Replaced the legacy inclusive training loop with the exact configured 22,003-update budget while preserving the training, data, and evaluation seeds.
+- Updated training evaluation, the seven-task English zero-shot suite, Transformers weights, and fixed inference samples.
+
+Full record: [`training-history/r006.md`](./training-history/r006.md)
+
 ## r005 — Pollock 1.3
 
 - Returned to the 12/14/896 architecture used by r003 while keeping the model below 128M native trainable parameters.

LICENSE.md

@@ -16,14 +16,14 @@ Code components distributed with the project remain subject to their respective
 
 ## Korpus i wagi / Corpus and weights
 
-Model został wytrenowany na `SlayerLab/minimal-en-corpus-5b`, agregacie danych z wielu źródeł. Korpus nie nadaje dokumentom jednej wspólnej licencji; każdy dokument zachowuje identyfikator źródła i podlega warunkom, licencjom oraz ograniczeniom właściwego upstreamowego datasetu.
+Model został wytrenowany na `SlayerLab/minimal-en-corpus-2.5b`, agregacie danych z wielu źródeł. Korpus nie nadaje dokumentom jednej wspólnej licencji; każdy dokument zachowuje identyfikator źródła i podlega warunkom, licencjom oraz ograniczeniom właściwego upstreamowego datasetu.
 
 Ze względu na mieszany charakter tych warunków repozytorium modelu używa metadanej Hugging Face `license: other`. Nie jest to przyznanie dodatkowych praw do materiałów źródłowych. Użytkownik powinien przed użyciem, redystrybucją lub zastosowaniem komercyjnym zapoznać się z kartą korpusu i warunkami wszystkich właściwych źródeł:
 
-<https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-5b>
+<https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-2.5b>
 
-The model was trained on `SlayerLab/minimal-en-corpus-5b`, an aggregate of multiple data sources. The corpus does not apply a single common license to its documents; each document retains its source identifier and remains subject to the terms, licenses, and restrictions of the applicable upstream dataset.
+The model was trained on `SlayerLab/minimal-en-corpus-2.5b`, an aggregate of multiple data sources. The corpus does not apply a single common license to its documents; each document retains its source identifier and remains subject to the terms, licenses, and restrictions of the applicable upstream dataset.
 
 Because these terms are mixed, the model repository uses the Hugging Face metadata value `license: other`. This notice does not grant additional rights to upstream materials. Before use, redistribution, or commercial application, users should review the corpus card and the terms of every applicable source:
 
-<https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-5b>
+<https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-2.5b>

README.md

@@ -5,38 +5,38 @@ pipeline_tag: text-generation
 license: other
 license_name: mixed-upstream-dataset-terms
 license_link: https://huggingface.co/SlayerLab/pollock-mini-lm-125m/blob/main/LICENSE.md
-datasets: [SlayerLab/minimal-en-corpus-5b]
+datasets: [SlayerLab/minimal-en-corpus-2.5b]
 tags: [causal-lm, gpt2, nanogpt, bpe, educational, base-model]
 model-index:
-  - name: Pollock 1.3
+  - name: Pollock 1.4
     results:
       - task: {type: text-generation, name: Language modeling}
-        dataset: {type: SlayerLab/minimal-en-corpus-5b, name: Minimal EN validation (sampled), split: validation}
-        metrics: [{type: loss, value: 2.5406661582, name: Final sampled validation loss}]
+        dataset: {type: SlayerLab/minimal-en-corpus-2.5b, name: Minimal EN 2.5B V2 validation (fixed subset), split: validation}
+        metrics: [{type: loss, value: 2.5363565445, name: Final fixed-subset validation loss}]
       - task: {type: text-generation, name: Zero-shot evaluation}
         dataset: {type: blimp, name: BLiMP, split: train}
-        metrics: [{type: acc, value: 0.7753134328}]
+        metrics: [{type: acc, value: 0.7691641791}]
       - task: {type: text-generation, name: Zero-shot evaluation}
         dataset: {type: EleutherAI/lambada_openai, name: LAMBADA OpenAI, split: test}
-        metrics: [{type: acc, value: 0.2974966039}, {type: perplexity, value: 47.32861410}]
+        metrics: [{type: acc, value: 0.2769260625}, {type: perplexity, value: 49.9072065519}]
       - task: {type: text-generation, name: Zero-shot evaluation}
         dataset: {type: hellaswag, name: HellaSwag, split: validation}
-        metrics: [{type: acc_norm, value: 0.3045210118}]
+        metrics: [{type: acc_norm, value: 0.3013343955}]
       - task: {type: text-generation, name: Zero-shot evaluation}
         dataset: {type: piqa, name: PIQA, split: validation}
-        metrics: [{type: acc_norm, value: 0.6033732318}]
+        metrics: [{type: acc_norm, value: 0.5968443961}]
       - task: {type: text-generation, name: Zero-shot evaluation}
         dataset: {type: sciq, name: SciQ, split: test}
-        metrics: [{type: acc_norm, value: 0.6760000000}]
+        metrics: [{type: acc_norm, value: 0.6640000000}]
       - task: {type: text-generation, name: Zero-shot evaluation}
         dataset: {type: allenai/ai2_arc, config: ARC-Easy, name: ARC-Easy, split: test}
-        metrics: [{type: acc_norm, value: 0.4297138047}]
+        metrics: [{type: acc_norm, value: 0.4343434343}]
       - task: {type: text-generation, name: Zero-shot evaluation}
         dataset: {type: allenai/ai2_arc, config: ARC-Challenge, name: ARC-Challenge, split: test}
-        metrics: [{type: acc_norm, value: 0.2320819113}]
+        metrics: [{type: acc_norm, value: 0.2414675768}]
 ---
 
-# Pollock 1.3 — r005
+# Pollock 1.4 — r006
 
 ![Pollock avatar](./assets/pollock-mini-lm-avatar-320.png)
 
@@ -48,13 +48,13 @@ model-index:
 
 Pollock to niewielki, anglojęzyczny model bazowy typu decoder-only, wytrenowany od zera jako czytelny eksperyment edukacyjny. Implementacja bazuje na [nanoGPT](https://github.com/karpathy/nanoGPT) i własnym tokenizerze byte-level BPE. Jest to model do uzupełniania tekstu, nie asystent konwersacyjny.
 
-Nazwa luźno nawiązuje do gestu malarskiego Jacksona Pollocka: nanoGPT jest płótnem, na którym dane, konfiguracja i decyzje treningowe tworzą różne wzorce zachowania. Pełne dane techniczne tej wersji znajdują się w [`training-history/r005.md`](./training-history/r005.md), a różnice między wydaniami w [`CHANGELOG.md`](./CHANGELOG.md).
+Nazwa luźno nawiązuje do gestu malarskiego Jacksona Pollocka: nanoGPT jest płótnem, na którym dane, konfiguracja i decyzje treningowe tworzą różne wzorce zachowania. Pełne dane techniczne tej wersji znajdują się w [`training-history/r006.md`](./training-history/r006.md), a różnice między wydaniami w [`CHANGELOG.md`](./CHANGELOG.md).
 
 ### Architektura i tokenizer
 
 | Właściwość | Wartość |
 |---|---:|
-| Rewizja / wydanie | r005 / Pollock 1.3 |
+| Rewizja / wydanie | r006 / Pollock 1.4 |
 | Typ | decoder-only Transformer w stylu GPT-2 |
 | Warstwy / głowy / embedding | 12 / 14 / 896 |
 | Maksymalny kontekst | 1024 tokeny |
@@ -68,40 +68,40 @@ Artefakt Transformers ma 127 674 624 parametry, w tym 119 168 zerowych parametr
 
 ### Dane i trening
 
-Model wytrenowano na [`SlayerLab/minimal-en-corpus-5b`](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-5b), subiektywnie dobranej mieszance 15 anglojęzycznych źródeł. Przygotowanie obejmowało deduplikację dokładną i przybliżoną, filtrowanie języka oraz decontaminację benchmarków. Tokenizer BPE został przebudowany dla nowego korpusu; zachowuje rozmiar słownika i identyfikatory tokenów specjalnych z r004, ale pozostałe identyfikatory tokenów nie są zgodne między tymi rewizjami.
+Model wytrenowano na wersji V2 datasetu [`SlayerLab/minimal-en-corpus-2.5b`](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-2.5b) przypiętej do commita `d68d992`. Jest to subiektywnie dobrana mieszanka 15 anglojęzycznych źródeł po filtrowaniu języka, deduplikacji dokładnej i przybliżonej, decontaminacji benchmarków oraz dodatkowym czyszczeniu boilerplate'u i błędnie połączonych rekordów. R006 wraca do tokenizera używanego przez r003-r004; nie jest on zgodny z tokenizerem r005 poza rozmiarem słownika i identyfikatorami tokenów specjalnych.
 
 | Parametr | Wartość |
 |---|---:|
-| Tokeny treningowe / walidacyjne | 5 425 901 705 / 5 269 579 |
-| Finalny checkpoint | aktualizacja 22 078 |
-| Przetworzone tokeny | 10 851 778 560 (około 2 epoki) |
+| Tokeny treningowe / walidacyjne | 2 689 323 439 / 5 236 486 |
+| Finalny checkpoint | aktualizacja 22 003 |
+| Przetworzone tokeny | 10 814 914 560 (około 4,02 epoki) |
 | Sekwencja / micro-batch na GPU | 1024 / 12 |
-| Akumulacja globalna / na GPU | 40 / 40 micro-stepów |
+| Akumulacja globalna / na GPU | 40 / 20 micro-stepów |
 | Effective batch | 491 520 tokenów |
 | Optymalizator | fused AdamW, betas 0.9/0.95 |
 | Learning rate | 4e-4 → 4e-5, cosine decay |
 | Warmup / weight decay / grad clip | 440 / 0.1 / 1.0 |
 | Precyzja | BF16 |
-| Sprzęt | 1× NVIDIA GeForce RTX 5090 |
+| Sprzęt | 2× NVIDIA GeForce RTX 4090 |
 | Framework | PyTorch 2.8.0+cu128, nanoGPT commit `3adf61e` |
 
 ### Ewaluacja
 
-Loss treningowy szacowano na stałych podzbiorach po 1 228 800 tokenów na split. Finalny checkpoint uzyskał validation loss **2.540666**; najlepszy wynik to **2.540605** po 10 813 440 000 przetworzonych tokenów. Nie należy porównywać tych wartości bezpośrednio z r001-r004, ponieważ r005 używa innego korpusu, tokenizera i zbioru walidacyjnego.
+Loss treningowy szacowano na stałych podzbiorach po 1 228 800 tokenów na split. Finalny checkpoint uzyskał validation loss **2.536357**; najlepszy wynik to **2.536231** po 10 813 440 000 przetworzonych tokenów. Nie należy porównywać tych wartości bezpośrednio z r005, ponieważ r006 używa innego korpusu, tokenizera i zbioru walidacyjnego.
 
 Benchmarki wykonano zero-shot na pełnych splitach przy użyciu `lm-evaluation-harness` 0.4.12, batch size 8 i BF16.
 
 | Benchmark | Główna metryka | Wynik | Próbki |
 |---|---|---:|---:|
-| BLiMP | acc | 0.775313 | 67 000 |
-| LAMBADA | acc | 0.297497 | 5 153 |
-| HellaSwag | acc_norm | 0.304521 | 10 042 |
-| PIQA | acc_norm | 0.603373 | 1 838 |
-| SciQ | acc_norm | 0.676000 | 1 000 |
-| ARC-Easy | acc_norm | 0.429714 | 2 376 |
-| ARC-Challenge | acc_norm | 0.232082 | 1 172 |
+| BLiMP | acc | 0.769164 | 67 000 |
+| LAMBADA | acc | 0.276926 | 5 153 |
+| HellaSwag | acc_norm | 0.301334 | 10 042 |
+| PIQA | acc_norm | 0.596844 | 1 838 |
+| SciQ | acc_norm | 0.664000 | 1 000 |
+| ARC-Easy | acc_norm | 0.434343 | 2 376 |
+| ARC-Challenge | acc_norm | 0.241468 | 1 172 |
 
-LAMBADA osiągnęła perplexity 47.328614. Pełne metryki i protokół zapisano w [`benchmarks/english.json`](./benchmarks/english.json).
+LAMBADA osiągnęła perplexity 49.907207. Pełne metryki i protokół zapisano w [`benchmarks/english.json`](./benchmarks/english.json).
 
 ### Użycie z Transformers
 
@@ -123,11 +123,12 @@ Model używa standardowego `GPT2LMHeadModel`; `trust_remote_code=True` nie jest
 
 ### Stałe próbki inferencji
 
-Wspólny zestaw [`fixed-sampling-v1`](./inference-samples/README.md) pokazuje te same cztery prompty wygenerowane przez r001-r005. Wszystkie rewizje są ładowane z pełnych, niezmiennych SHA commitów, a SHA-256 każdego pliku z wagami jest sprawdzane przed inferencją. Referencyjny protokół używa macOS 26.2 na arm64, CPU, float32, jednego wątku, Transformers 5.15.1, seed 1337 resetowanego dla każdego promptu, temperature 0.7, top-k 50 i limitu 100 nowych tokenów.
+Wspólny zestaw [`fixed-sampling-v1`](./inference-samples/README.md) pokazuje te same cztery prompty wygenerowane przez r001-r006. Wszystkie opublikowane rewizje są ładowane z pełnych, niezmiennych SHA commitów, a SHA-256 każdego pliku z wagami jest sprawdzane przed inferencją. Referencyjny protokół używa macOS 26.2 na arm64, CPU, float32, jednego wątku, Transformers 5.15.1, seed 1337 resetowanego dla każdego promptu, temperature 0.7, top-k 50 i limitu 100 nowych tokenów.
 
 | Rewizja | Wydanie | Wagi | Historia |
 |---|---|---|---|
-| **r005** | **Pollock 1.3** | [`e780025`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/e780025f15e06bc3765a74d973906eb8a11c022c) | [`r005.md`](./training-history/r005.md) |
+| **r006** | **Pollock 1.4** | PR commit pending | [`r006.md`](./training-history/r006.md) |
+| r005 | Pollock 1.3 | [`e780025`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/e780025f15e06bc3765a74d973906eb8a11c022c) | [`r005.md`](./training-history/r005.md) |
 | r004 | Pollock 1.2 | [`30feb81`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/30feb81e5097eef13b939e1827d98e0458bf602d) | [`r004.md`](./training-history/r004.md) |
 | r003 | Pollock 1.1 | [`698984b`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/698984b1d1b96c9b6ffaff00c7fc2e78e140e842) | [`r003.md`](./training-history/r003.md) |
 | r002 | Pollock 1.0 | [`5e571cd`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/5e571cde09ef2d1124e13d1c31d0b75f8bd0e513) | [`r002.md`](./training-history/r002.md) |
@@ -150,13 +151,13 @@ Pełne teksty są przeznaczone do porównywania zachowania, nie są benchmarkiem
 
 Pollock is a small English decoder-only base language model trained from scratch as a readable educational experiment. It is based on [nanoGPT](https://github.com/karpathy/nanoGPT) and a custom byte-level BPE tokenizer. It is a completion model, not a conversational assistant.
 
-The name loosely refers to Jackson Pollock's painterly gesture: nanoGPT is the canvas on which data, configuration, and training decisions create different behavioral patterns. See [`training-history/r005.md`](./training-history/r005.md) for the complete technical record and [`CHANGELOG.md`](./CHANGELOG.md) for release-to-release changes.
+The name loosely refers to Jackson Pollock's painterly gesture: nanoGPT is the canvas on which data, configuration, and training decisions create different behavioral patterns. See [`training-history/r006.md`](./training-history/r006.md) for the complete technical record and [`CHANGELOG.md`](./CHANGELOG.md) for release-to-release changes.
 
 ### Architecture and tokenizer
 
 | Property | Value |
 |---|---:|
-| Revision / release | r005 / Pollock 1.3 |
+| Revision / release | r006 / Pollock 1.4 |
 | Type | GPT-2-style decoder-only Transformer |
 | Layers / heads / width | 12 / 14 / 896 |
 | Maximum context | 1,024 tokens |
@@ -170,40 +171,40 @@ The Transformers artifact has 127,674,624 parameters, including 119,168 zero-val
 
 ### Data and training
 
-The model was trained on [`SlayerLab/minimal-en-corpus-5b`](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-5b), a subjectively selected mixture of 15 English-language sources. Preparation included exact and approximate deduplication, language filtering, and benchmark decontamination. The BPE tokenizer was rebuilt for the new corpus; it retains r004's vocabulary size and special-token IDs, but other token IDs are not compatible across the two revisions.
+The model was trained on V2 of [`SlayerLab/minimal-en-corpus-2.5b`](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-2.5b), pinned to commit `d68d992`. It is a subjectively selected mixture of 15 English-language sources after language filtering, exact and approximate deduplication, benchmark decontamination, and additional cleanup of boilerplate and incorrectly concatenated records. R006 returns to the tokenizer used by r003-r004; it is not compatible with r005's tokenizer beyond the vocabulary size and special-token IDs.
 
 | Setting | Value |
 |---|---:|
-| Training / validation tokens | 5,425,901,705 / 5,269,579 |
-| Final checkpoint | update 22,078 |
-| Token presentations | 10,851,778,560 (approximately 2 epochs) |
+| Training / validation tokens | 2,689,323,439 / 5,236,486 |
+| Final checkpoint | update 22,003 |
+| Token presentations | 10,814,914,560 (approximately 4.02 epochs) |
 | Sequence / micro-batch per GPU | 1,024 / 12 |
-| Global / per-GPU accumulation | 40 / 40 micro-steps |
+| Global / per-GPU accumulation | 40 / 20 micro-steps |
 | Effective batch | 491,520 tokens |
 | Optimizer | fused AdamW, betas 0.9/0.95 |
 | Learning rate | 4e-4 → 4e-5, cosine decay |
 | Warmup / weight decay / grad clip | 440 / 0.1 / 1.0 |
 | Precision | BF16 |
-| Hardware | 1× NVIDIA GeForce RTX 5090 |
+| Hardware | 2× NVIDIA GeForce RTX 4090 |
 | Framework | PyTorch 2.8.0+cu128, nanoGPT commit `3adf61e` |
 
 ### Evaluation
 
-Training-time loss was estimated on fixed subsets of 1,228,800 tokens per split. The final checkpoint achieved validation loss **2.540666**; the best result was **2.540605** after 10,813,440,000 token presentations. These values are not directly comparable with r001-r004 because r005 uses a different corpus, tokenizer, and validation set.
+Training-time loss was estimated on fixed subsets of 1,228,800 tokens per split. The final checkpoint achieved validation loss **2.536357**; the best result was **2.536231** after 10,813,440,000 token presentations. These values are not directly comparable with r005 because r006 uses a different corpus, tokenizer, and validation set.
 
 Benchmarks used complete splits with `lm-evaluation-harness` 0.4.12, zero few-shot examples, batch size 8, and BF16.
 
 | Benchmark | Primary metric | Score | Samples |
 |---|---|---:|---:|
-| BLiMP | acc | 0.775313 | 67,000 |
-| LAMBADA | acc | 0.297497 | 5,153 |
-| HellaSwag | acc_norm | 0.304521 | 10,042 |
-| PIQA | acc_norm | 0.603373 | 1,838 |
-| SciQ | acc_norm | 0.676000 | 1,000 |
-| ARC-Easy | acc_norm | 0.429714 | 2,376 |
-| ARC-Challenge | acc_norm | 0.232082 | 1,172 |
+| BLiMP | acc | 0.769164 | 67,000 |
+| LAMBADA | acc | 0.276926 | 5,153 |
+| HellaSwag | acc_norm | 0.301334 | 10,042 |
+| PIQA | acc_norm | 0.596844 | 1,838 |
+| SciQ | acc_norm | 0.664000 | 1,000 |
+| ARC-Easy | acc_norm | 0.434343 | 2,376 |
+| ARC-Challenge | acc_norm | 0.241468 | 1,172 |
 
-LAMBADA perplexity was 47.328614. Full metrics and protocol details are recorded in [`benchmarks/english.json`](./benchmarks/english.json).
+LAMBADA perplexity was 49.907207. Full metrics and protocol details are recorded in [`benchmarks/english.json`](./benchmarks/english.json).
 
 ### Usage
 
@@ -211,11 +212,12 @@ Use the Transformers example in the Polish section. The artifact uses standard `
 
 ### Fixed inference samples
 
-The shared [`fixed-sampling-v1`](./inference-samples/README.md) suite runs the same four prompts on r001-r005. All revisions are loaded from full immutable commit SHAs, and every weight-file SHA-256 is verified before inference. The reference protocol uses macOS 26.2 on arm64, CPU float32 with one thread, Transformers 5.15.1, seed 1337 reset for every prompt, temperature 0.7, top-k 50, and a 100-new-token limit.
+The shared [`fixed-sampling-v1`](./inference-samples/README.md) suite runs the same four prompts on r001-r006. All published revisions are loaded from full immutable commit SHAs, and every weight-file SHA-256 is verified before inference. The reference protocol uses macOS 26.2 on arm64, CPU float32 with one thread, Transformers 5.15.1, seed 1337 reset for every prompt, temperature 0.7, top-k 50, and a 100-new-token limit.
 
 | Revision | Release | Weights | History |
 |---|---|---|---|
-| **r005** | **Pollock 1.3** | [`e780025`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/e780025f15e06bc3765a74d973906eb8a11c022c) | [`r005.md`](./training-history/r005.md) |
+| **r006** | **Pollock 1.4** | PR commit pending | [`r006.md`](./training-history/r006.md) |
+| r005 | Pollock 1.3 | [`e780025`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/e780025f15e06bc3765a74d973906eb8a11c022c) | [`r005.md`](./training-history/r005.md) |
 | r004 | Pollock 1.2 | [`30feb81`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/30feb81e5097eef13b939e1827d98e0458bf602d) | [`r004.md`](./training-history/r004.md) |
 | r003 | Pollock 1.1 | [`698984b`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/698984b1d1b96c9b6ffaff00c7fc2e78e140e842) | [`r003.md`](./training-history/r003.md) |
 | r002 | Pollock 1.0 | [`5e571cd`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/5e571cde09ef2d1124e13d1c31d0b75f8bd0e513) | [`r002.md`](./training-history/r002.md) |

benchmarks/english.json

@@ -1,12 +1,22 @@
 {
   "schema_version": 1,
-  "revision_id": "r005",
-  "release": "Pollock 1.3",
-  "source_result": "runs/pollock-r005-v1p3/benchmarks/results.json",
+  "revision_id": "r006",
+  "release": "Pollock 1.4",
+  "source_result": "runs/r006-corrected-corpus-v2-s1337-lr0.0004/benchmarks/results.json",
   "checkpoint": {
-    "path": "runs/pollock-r005-v1p3/checkpoints/ckpt-final.pt",
-    "sha256": "bc68b2cb8a736fb00fa26b40c425dc1944b2ea02eb9514937ce5ca0908bbe9d2",
-    "source_iteration": 22078
+    "path": "runs/r006-corrected-corpus-v2-s1337-lr0.0004/checkpoints/ckpt-final.pt",
+    "sha256": "580fbb3c039ea1f349285a9bf7a67731fb6d455de931b6b7601d4ed09fce9437",
+    "source_iteration": 22003
+  },
+  "benchmark_checkpoint": {
+    "path": "/workspace/cache/dmpod-tokenizers/r006-final-benchmark.pt",
+    "sha256": "9339d20ea32d24bc9777518a2f0ec553e4a3871647d342db92d7c5b6c18c7b56",
+    "trusted_input_required": true
+  },
+  "tokenizer": {
+    "dataset_revision": "d68d992622e9fc11f19e7d7fb8547c4e653439a4",
+    "sha256": "6cda4e5ec8293f3b821e02253f9c0e88e43ad4f7c6e1324763e1fb91749fef51",
+    "version": "V2.0@d68d992622e9fc11f19e7d7fb8547c4e653439a4"
   },
   "execution": {
     "harness": "lm-evaluation-harness 0.4.12",
@@ -19,43 +29,43 @@
   },
   "results": {
     "blimp": {
-      "acc,none": 0.7753134328358209,
+      "acc,none": 0.7691641791044775,
       "sample_len": 67000.0,
       "samples": 67000
     },
     "lambada_openai": {
-      "acc,none": 0.29749660392004657,
-      "perplexity,none": 47.32861410229832,
+      "acc,none": 0.27692606248787116,
+      "perplexity,none": 49.907206551888116,
       "sample_len": 5153.0,
       "samples": 5153
     },
     "hellaswag": {
-      "acc,none": 0.28450507866958774,
-      "acc_norm,none": 0.3045210117506473,
+      "acc,none": 0.2830113523202549,
+      "acc_norm,none": 0.3013343955387373,
       "sample_len": 10042.0,
       "samples": 10042
     },
     "piqa": {
-      "acc,none": 0.6099020674646355,
-      "acc_norm,none": 0.6033732317736671,
+      "acc,none": 0.6071817192600653,
+      "acc_norm,none": 0.5968443960826986,
       "sample_len": 1838.0,
       "samples": 1838
     },
     "sciq": {
-      "acc,none": 0.772,
-      "acc_norm,none": 0.676,
+      "acc,none": 0.768,
+      "acc_norm,none": 0.664,
       "sample_len": 1000.0,
       "samples": 1000
     },
     "arc_easy": {
-      "acc,none": 0.48358585858585856,
-      "acc_norm,none": 0.4297138047138047,
+      "acc,none": 0.4877946127946128,
+      "acc_norm,none": 0.43434343434343436,
       "sample_len": 2376.0,
       "samples": 2376
     },
     "arc_challenge": {
-      "acc,none": 0.19197952218430034,
-      "acc_norm,none": 0.23208191126279865,
+      "acc,none": 0.21075085324232082,
+      "acc_norm,none": 0.24146757679180889,
       "sample_len": 1172.0,
       "samples": 1172
     }

config.json

@@ -12,7 +12,7 @@
   "initializer_range": 0.02,
   "layer_norm_epsilon": 1e-05,
   "model_type": "gpt2",
-  "model_version": "1.3",
+  "model_version": "1.4",
   "n_ctx": 1024,
   "n_embd": 896,
   "n_head": 14,

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3db43dfa44622e8156f060fee6f16875e108db4f5f949969cb91c0ca8cdce32c
+oid sha256:8e74aa4d34464229e86a1831fb34d442f91e22aa600e0cf2503891957fdc6a43
 size 510713512

training-history/r006.md

@@ -0,0 +1,49 @@
+# r006 — Pollock 1.4 training record
+
+## Identity and provenance
+
+| Field | Value |
+|---|---|
+| Revision / release | `r006` / Pollock 1.4 |
+| Source run | `r006-corrected-corpus-v2-s1337-lr0.0004` |
+| Final checkpoint SHA-256 | `580fbb3c039ea1f349285a9bf7a67731fb6d455de931b6b7601d4ed09fce9437` |
+| `model.safetensors` SHA-256 | `8e74aa4d34464229e86a1831fb34d442f91e22aa600e0cf2503891957fdc6a43` |
+
+## Architecture and training
+
+| Setting | Value |
+|---|---:|
+| Layers / heads / width | 12 / 14 / 896 |
+| Context / vocabulary | 1024 / 12288 |
+| Native trainable parameters | 127,555,456 |
+| Dataset | SlayerLab/minimal-en-corpus-2.5b V2 (`d68d992`) |
+| Tokenizer SHA-256 | `6cda4e5ec8293f3b821e02253f9c0e88e43ad4f7c6e1324763e1fb91749fef51` |
+| Token presentations / passes | 10,814,914,560 / 4.021426 |
+| Effective batch | 491,520 tokens |
+| LR range | 4e-04 → 4e-05 |
+| Precision / hardware | BF16 / 2× NVIDIA GeForce RTX 4090 |
+| Runtime / mean throughput | 15.227 h / 199,894.6 tokens/s |
+
+The frozen training source stores the same tokenizer mapping in nanoGPT's native merge format, whose SHA-256 is `d0d126f0a59e51e8cb1b26d77b5527870be0e8ea1e84adefd730fff906c48234`.
+
+## Training-time evaluation
+
+Final train-eval loss: **2.489748**. Final validation loss: **2.536357** (perplexity 12.633557). Minimum validation loss: **2.536231** at 10,813,440,000 tokens.
+
+## English zero-shot benchmarks
+
+| Benchmark | Primary metric | Score | Samples |
+|---|---|---:|---:|
+| BLiMP | acc | 0.769164 | 67,000 |
+| LAMBADA | acc | 0.276926 | 5,153 |
+| HellaSwag | acc_norm | 0.301334 | 10,042 |
+| PIQA | acc_norm | 0.596844 | 1,838 |
+| SciQ | acc_norm | 0.664000 | 1,000 |
+| ARC-Easy | acc_norm | 0.434343 | 2,376 |
+| ARC-Challenge | acc_norm | 0.241468 | 1,172 |
+
+LAMBADA perplexity was **49.907207**. Benchmarks used complete splits, zero few-shot examples, `lm-evaluation-harness` 0.4.12, BF16, batch size 8, and a 1,024-token maximum context. No requests were truncated. The exact temporary benchmark-checkpoint and tokenizer hashes are preserved in [`../benchmarks/english.json`](../benchmarks/english.json).
+
+## Conversion and release status
+
+The native final checkpoint was converted to `GPT2LMHeadModel` with Transformers 5.15.1. A deterministic `[2, 64]` parity probe produced maximum absolute logit error **0.0**. Publication and immutable remote replay details are recorded in `release_manifest.json`.