| --- |
| language: [en] |
| library_name: transformers |
| pipeline_tag: text-generation |
| license: other |
| license_name: mixed-upstream-dataset-terms |
| license_link: https://huggingface.co/SlayerLab/pollock-mini-lm-125m/blob/main/LICENSE.md |
| datasets: [SlayerLab/minimal-en-corpus-5b] |
| tags: [causal-lm, gpt2, nanogpt, swiglu, bpe, educational, base-model] |
| model-index: |
| - name: Pollock 1.6 |
| results: |
| - task: {type: text-generation, name: Language modeling} |
| dataset: {type: SlayerLab/minimal-en-corpus-5b, name: Minimal EN validation (full split), split: validation} |
| metrics: [{type: loss, value: 2.4507743919, name: Full-validation loss}] |
| - task: {type: text-generation, name: Linguistic minimal pairs} |
| dataset: {type: blimp, name: BLiMP} |
| metrics: [{type: accuracy, value: 0.7803582090, name: Accuracy}] |
| - task: {type: text-generation, name: Language modeling} |
| dataset: {type: lambada_openai, name: LAMBADA OpenAI} |
| metrics: [{type: accuracy, value: 0.3132156026, name: Accuracy}, {type: perplexity, value: 42.0737928480, name: Perplexity}] |
| - task: {type: text-generation, name: Multiple-choice completion} |
| dataset: {type: hellaswag, name: HellaSwag} |
| metrics: [{type: accuracy, value: 0.3043218482, name: Normalized accuracy}] |
| - task: {type: text-generation, name: Physical commonsense reasoning} |
| dataset: {type: piqa, name: PIQA} |
| metrics: [{type: accuracy, value: 0.6169749728, name: Normalized accuracy}] |
| - task: {type: text-generation, name: Science question answering} |
| dataset: {type: sciq, name: SciQ} |
| metrics: [{type: accuracy, value: 0.643, name: Normalized accuracy}] |
| - task: {type: text-generation, name: Grade-school science questions} |
| dataset: {type: arc_easy, name: ARC-Easy} |
| metrics: [{type: accuracy, value: 0.4255050505, name: Normalized accuracy}] |
| - task: {type: text-generation, name: Science challenge questions} |
| dataset: {type: arc_challenge, name: ARC-Challenge} |
| metrics: [{type: accuracy, value: 0.2474402730, name: Normalized accuracy}] |
| --- |
| |
| # Pollock 1.6 - r008 |
|
|
|  |
|
|
| > **Wydanie eksperymentalne / Experimental release.** Model nie jest |
| > przeznaczony do zastosowa艅 produkcyjnych ani wysokiego ryzyka. Ca艂y model |
| > pozostaje poni偶ej 128M parametr贸w. / The model is not intended for production |
| > or high-risk use. The complete model remains below 128M parameters. |
|
|
| ## Polski |
|
|
| ### Opis |
|
|
| Pollock to niewielki, angloj臋zyczny model bazowy typu decoder-only, wytrenowany |
| od zera jako czytelny eksperyment edukacyjny. Implementacja bazuje na |
| [nanoGPT](https://github.com/karpathy/nanoGPT) i w艂asnym tokenizerze byte-level |
| BPE. Jest to model do uzupe艂niania tekstu, nie asystent konwersacyjny. |
|
|
| R008 zast臋puje klasyczny blok GELU MLP blokiem SwiGLU dopasowanym do tego samego |
| bud偶etu poni偶ej 128M parametr贸w. Tokenizer, korpus, kontekst, liczba warstw, |
| szeroko艣膰 modelu, attention i effective batch pozostaj膮 zgodne z R007. Pe艂na |
| historia techniczna znajduje si臋 w [`training-history/r008.md`](./training-history/r008.md), |
| a r贸偶nice mi臋dzy wydaniami w [`CHANGELOG.md`](./CHANGELOG.md). |
|
|
| ### Architektura i tokenizer |
|
|
| | W艂a艣ciwo艣膰 | Warto艣膰 | |
| |---|---:| |
| | Rewizja / wydanie | r008 / Pollock 1.6 | |
| | Typ | decoder-only Transformer w stylu GPT-2 z SwiGLU | |
| | Warstwy / g艂owy / embedding | 16 / 12 / 768 | |
| | Szeroko艣膰 SwiGLU | 2 144 | |
| | Maksymalny kontekst | 2 048 token贸w | |
| | S艂ownik | 12 288 token贸w | |
| | 艁膮czne unikalne parametry trenowalne | 127 820 544 | |
| | Tokenizer | byte-level BPE, pretokenizacja w stylu GPT-2 | |
| | Tokeny specjalne | <code><|endoftext|></code>, <code><|im_start|></code>, <code><|im_end|></code> | |
|
|
| Model u偶ywa w艂asnej klasy Transformers `PollockForCausalLM`, poniewa偶 standardowy |
| `GPT2LMHeadModel` nie implementuje tego wariantu SwiGLU. Wybrany checkpoint z |
| aktualizacji 32 000 ma SHA-256 |
| `25b8cac7d5142c75e1be0b42fa24d980896e6f911e912c883566747a3baa56f0`. |
| Konwersja w Transformers 5.15.1 wykaza艂a maksymaln膮 bezwzgl臋dn膮 r贸偶nic臋 logits |
| **0.0** na deterministycznej pr贸bie `[2, 64]`. SHA-256 pliku |
| `model.safetensors` to |
| `1c1002909119b4913841ee29efd44bd86d62d81f2ae27de01602b64d4af0dbec`. |
| Artefakt Transformers ma 127 975 936 parametr贸w, w tym zerowe tensory bias |
| wymagane przez implementacj臋 pakietow膮; model natywny ma 127 820 544 parametr贸w. |
|
|
| Muon optymalizuje kwalifikuj膮ce si臋 ukryte macierze attention i MLP. AdamW |
| obs艂uguje embeddingi token贸w, powi膮zan膮 g艂owic臋 wyj艣ciow膮, parametry normalizacji |
| i biasy. Podzia艂 parametr贸w jest sprawdzany przed treningiem pod k膮tem nak艂adania |
| si臋 i kompletno艣ci. |
|
|
| ### Dane i trening |
|
|
| Model jest trenowany na |
| [`SlayerLab/minimal-en-corpus-5b`](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-5b) |
| przypi臋tym do commita `38bebbd`. Jest to subiektywnie dobrana mieszanka 15 |
| angloj臋zycznych 藕r贸de艂 po filtrowaniu j臋zyka, deduplikacji dok艂adnej i |
| przybli偶onej oraz decontaminacji benchmark贸w. R008 zachowuje tokenizer R007. |
|
|
| Loader stosuje globaln膮 permutacj臋 ze sta艂ym seedem do niezmiennych okien |
| token贸w w ka偶dym przebiegu, przydziela roz艂膮czne pozycje mi臋dzy rangi DDP, |
| maskuje ko艅cowy niepe艂ny batch i zapisuje globalny kursor, topologi臋 oraz |
| ustawienia tasowania w ka偶dym checkpoincie. |
|
|
| | Parametr | Warto艣膰 | |
| |---|---:| |
| | Tokeny treningowe / walidacyjne | 5 396 605 407 / 5 238 223 | |
| | Planowane maksimum / wybrana ekspozycja | 4 przebiegi / 15 728 561 562 targety | |
| | Wybrany checkpoint | aktualizacja 32 000, validation loss 2.4507743919 | |
| | Sekwencja / micro-batch na GPU | 2 048 / 6 | |
| | Akumulacja globalna / na GPU | 40 / 10 micro-step贸w | |
| | Effective batch | 491 520 token贸w | |
| | Optymalizator | Muon plus fused AdamW, betas AdamW 0.9/0.95 | |
| | LR pierwszych dw贸ch przebieg贸w | Muon 1e-2 -> 1e-3; AdamW 3e-4 -> 3e-5 | |
| | LR finalnej kontynuacji | Muon 1e-4; AdamW 3e-6, sta艂e | |
| | Warmup / weight decay / grad clip | 200 / 0.01 / 1.0 | |
| | Precyzja / sprz臋t | BF16 / 4x NVIDIA H100 80GB HBM3 | |
|
|
| Pierwsze dwa przebiegi zako艅czy艂y si臋 na aktualizacji 21 959. Pierwotna |
| kontynuacja odziedziczy艂a z R007 zbyt wysokie dla tej architektury dolne LR |
| `1e-3` i `3e-5`: pogorszy艂y si臋 jednocze艣nie train-eval i pe艂ny validation loss, |
| a norma parametr贸w spad艂a o oko艂o 6,5%. Kontynuacj臋 zatrzymano na aktualizacji |
| 34 520. Finalny trening R008 rozpocz臋to ponownie z niezmienionego, zweryfikowanego |
| checkpointu E2, obni偶aj膮c oba LR dziesi臋ciokrotnie i pozostawiaj膮c weight decay |
| bez zmian. Validation loss osi膮gn膮艂 minimum 2.4507743919 przy aktualizacji |
| 32 000, a nast臋pnie przeszed艂 w plateau i wzr贸s艂 do 2.4516 przy ostatnim |
| zapisanym checkpoincie 38 000. Trening zatrzymano celowo przy aktualizacji |
| 38 140; checkpoint 32 000 wybrano przed benchmarkami downstream. |
|
|
| ### Ewaluacja |
|
|
| Loss treningowy jest szacowany na sta艂ej, tasowanej pr贸bie 2 457 600 token贸w z |
| ca艂ego splitu treningowego. Ka偶da ewaluacja walidacyjna obejmuje pe艂ny split, |
| czyli 5 238 222 targety. Wybrany checkpoint ma train-eval loss 2.4182 oraz |
| pe艂ny validation loss 2.4507743919. |
|
|
| | Benchmark | G艂贸wna metryka | Wynik | Pr贸bki | |
| |---|---|---:|---:| |
| | BLiMP | acc | 78.0358% | 67 000 | |
| | LAMBADA | acc / perplexity | 31.3216% / 42.0738 | 5 153 | |
| | HellaSwag | acc_norm | 30.4322% | 10 042 | |
| | PIQA | acc_norm | 61.6975% | 1 838 | |
| | SciQ | acc_norm | 64.3000% | 1 000 | |
| | ARC-Easy | acc_norm | 42.5505% | 2 376 | |
| | ARC-Challenge | acc_norm | 24.7440% | 1 172 | |
| |
| Wyniki pochodz膮 z pe艂nych split贸w, zero-shot, w `lm-eval` 0.4.12, |
| Transformers 5.15.1 i PyTorch 2.8.0+cu128 na RTX A4500. Tabela model-card u偶ywa |
| BF16, batch 8 i maksymalnego kontekstu 1 024. Tiny-ML WikiText-2 osi膮ga byte |
| perplexity 1.9423491827, 0.9578025823 bits/byte i word perplexity 34.8189798815. |
| Kanoniczny Open SLM FP32 Intelligence Index wynosi 13.4419562655; pe艂ne wyniki, |
| pr贸bki i protok贸艂 s膮 powi膮zane z tym samym hashem modelu. |
| |
| Sta艂e, nieprzebierane pr贸bki generacji z seedem 1337 s膮 zachowane razem z |
| artefaktami ewaluacji. Pokazuj膮 r贸wnie偶 typowe ograniczenia ma艂ego modelu |
| bazowego, w tym powt贸rzenia i b艂臋dy faktograficzne. |
| |
| Pe艂ne publiczne podsumowanie znajduje si臋 w [`benchmarks/english.json`](./benchmarks/english.json), |
| a pr贸bki i skrypt odtwarzaj膮cy w [`inference-samples/r008/`](./inference-samples/r008/). |
| |
| ### U偶ycie z Transformers |
| |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "SlayerLab/pollock-mini-lm-125m" |
| tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, |
| trust_remote_code=True, |
| ) |
| |
| inputs = tokenizer("Once upon a time", return_tensors="pt") |
| output = model.generate( |
| **inputs, |
| max_new_tokens=100, |
| do_sample=True, |
| temperature=0.7, |
| top_k=50, |
| ) |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| ``` |
| |
| `trust_remote_code=True` jest wymagane, poniewa偶 repozytorium zawiera ma艂膮, |
| audytowaln膮 implementacj臋 w艂asnego bloku SwiGLU. Przed u偶yciem nale偶y przejrze膰 |
| `modeling_pollock.py` w przypi臋tej rewizji modelu. |
| |
| ### Ograniczenia i odpowiedzialne u偶ycie |
| |
| - Jest to surowy model bazowy bez instruction tuningu, RLHF ani alignmentu. |
| - Mo偶e halucynowa膰 oraz generowa膰 tre艣ci toksyczne, stronnicze, niebezpieczne |
| lub podobne do danych treningowych. |
| - Ma艂y rozmiar ogranicza wiedz臋, rozumowanie, sp贸jno艣膰 d艂ugiego tekstu i jako艣膰 |
| kodu. |
| - Model trenowano i oceniano g艂贸wnie po angielsku. |
| - Nie nale偶y u偶ywa膰 go do porad medycznych, prawnych i finansowych, decyzji o |
| ludziach ani zastosowa艅 wysokiego ryzyka. |
| - U偶ytkownik odpowiada za ocen臋 warunk贸w 藕r贸de艂 danych dla swojego zastosowania. |
| |
| ## English |
| |
| ### Overview |
| |
| Pollock is a small English decoder-only base language model trained from scratch |
| as a readable educational experiment. It is based on |
| [nanoGPT](https://github.com/karpathy/nanoGPT) and a custom byte-level BPE |
| tokenizer. It is a completion model, not a conversational assistant. |
| |
| R008 replaces the GELU MLP with a parameter-matched SwiGLU block while retaining |
| R007's tokenizer, corpus, learned absolute positions, context, layer count, |
| model width, attention configuration, and effective batch. See |
| [`training-history/r008.md`](./training-history/r008.md) for the technical record |
| and [`CHANGELOG.md`](./CHANGELOG.md) for release-to-release changes. |
| |
| ### Architecture and tokenizer |
| |
| | Property | Value | |
| |---|---:| |
| | Revision / release | r008 / Pollock 1.6 | |
| | Type | GPT-2-style decoder-only Transformer with SwiGLU | |
| | Layers / heads / width | 16 / 12 / 768 | |
| | SwiGLU width | 2,144 | |
| | Maximum context | 2,048 tokens | |
| | Vocabulary | 12,288 tokens | |
| | Total unique trainable parameters | 127,820,544 | |
| | Tokenizer | byte-level BPE, GPT-2-style pretokenization | |
| | Special tokens | `<|endoftext|>`, `<|im_start|>`, `<|im_end|>` | |
| |
| The artifact uses the custom `PollockForCausalLM` Transformers class because |
| standard `GPT2LMHeadModel` does not implement this SwiGLU variant. The selected |
| step-32,000 checkpoint was converted with a maximum absolute native/Transformers |
| logit difference of **0.0**. Its SHA-256 is |
| `25b8cac7d5142c75e1be0b42fa24d980896e6f911e912c883566747a3baa56f0`; |
| the resulting `model.safetensors` SHA-256 is |
| `1c1002909119b4913841ee29efd44bd86d62d81f2ae27de01602b64d4af0dbec`. |
| |
| ### Data and training |
| |
| The model is trained on |
| [`SlayerLab/minimal-en-corpus-5b`](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-5b), |
| pinned to commit `38bebbd`. The deterministic loader, four-pass target, |
| 491,520-target effective batch, and validation protocol match the Polish section. |
| |
| The first two passes ended at update 21,959. The initial continuation inherited |
| R007-derived floor learning rates that proved too high for this architecture: |
| fixed train-eval and full-validation loss both degraded while parameter norm |
| fell by about 6.5%. It was stopped at update 34,520. Final R008 training restarted |
| from the unchanged, checksum-verified E2 checkpoint with fixed Muon and AdamW |
| learning rates reduced by 10x to `1e-4` and `3e-6`; weight decay remained |
| `1e-2`. Full-validation loss reached its minimum of 2.4507743919 at step 32,000 |
| after 15,728,561,562 target presentations. Training was intentionally stopped |
| at step 38,140 after the metric plateaued; the preselected best-validation |
| checkpoint was then used for all release benchmarks. |
| |
| ### Evaluation |
| |
| The model-index and Polish table report the complete zero-shot standard suite. |
| Evaluation used full splits with `lm-eval` 0.4.12, Transformers 5.15.1, |
| PyTorch 2.8.0+cu128, and the exact converted model identified above. Tiny-ML |
| WikiText-2 word perplexity is 34.8189798815. The canonical FP32 Open SLM scores |
| are HellaSwag 30.4521%, ARC-Easy 42.3401%, ARC-Challenge 24.5734%, PIQA |
| 61.0990%, ArithMark-3 34.6000%, and Intelligence Index 13.4419562655. |
| The machine-readable summary is available at |
| [`benchmarks/english.json`](./benchmarks/english.json), and the fixed samples |
| plus replay script at [`inference-samples/r008/`](./inference-samples/r008/). |
| |
| ### Usage |
| |
| Use the Transformers example in the Polish section. Loading requires |
| `trust_remote_code=True`; review `modeling_pollock.py` at the pinned model |
| revision before use. |
| |
| ### Limitations and responsible use |
| |
| - This is a raw base model without instruction tuning, RLHF, or safety alignment. |
| - It may hallucinate and generate toxic, biased, unsafe, or training-like text. |
| - Its small size limits knowledge, reasoning, long-form coherence, and code quality. |
| - It was trained and evaluated primarily in English. |
| - Do not use it for medical, legal, financial, high-impact, or production |
| decisions without independent safeguards. |
| - Users must evaluate upstream dataset terms for their intended use. |
| |
| ## Licencja / License |
| |
| Kod nanoGPT jest na licencji MIT. Korpus 艂膮czy 藕r贸d艂a o r贸偶nych warunkach i nie |
| ma jednej wsp贸lnej licencji, dlatego model oznaczono jako `license: other`. |
| Szczeg贸艂y: [`LICENSE.md`](./LICENSE.md) i karta datasetu. |
| |
| nanoGPT code is MIT-licensed. The corpus combines sources governed by different |
| terms and has no single common license, so the model is marked as |
| `license: other`. See [`LICENSE.md`](./LICENSE.md) and the dataset card. |
| |
| ## Pochodzenie / Attribution |
| |
| Model i dataset / Model and dataset: Dawid Majewski / SlayerLab. Trening bazuje |
| na / Training is based on [karpathy/nanoGPT](https://github.com/karpathy/nanoGPT). |
| |
