# Training configuration and budget-limited endpoint

Fabryka AI / SlayerLab

**Historical execution snapshot, 10 November 2026, 19:23 UTC:** continued pretraining was paused by the cost guard after the on-node update-11,600 checkpoint. The operator stopped GPU allocation at approximately 19:21 UTC after four price readings exceeded the cost-per-token ceiling, retaining the VM and disk for recovery. Before this pause, operators confirmed continued pretraining on eight H200 GPUs after this continuation's own `RESUME @ 9600`. Real CPT from 1.08 began at approximately 04:50 UTC on eight B200 GPUs. Transfer checksums at that snapshot confirmed weights-only updates 5,000 and 10,000 and the latest full local update-10,000 checkpoint; they do not assert a local update-11,600 copy. This is historical evidence, not current hardware availability; consult the [CPT Track history](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a) and operators for later status.

**Funding update — 11 November 2026, 15:29 UTC:** an additional **USD 500 was reported received** for continued main-base training. The earlier USD 425 campaign envelope is historical, not the current remaining balance. The goal remains maximum useful affordable exposure at constant learning rates after warmup with **no in-run terminal decay**. Retain cost guards and the operational reserve for checkpoint recovery, transfer and safe shutdown; stop at a verified full checkpoint when the actual budget or allocation requires it, not necessarily at a 5,000-update boundary. No final funded update count or completion time is guaranteed, and no gold-training budget is implied.


## Required separation for the revised continuation

The selected base is [SlayerLab/Slayer1B-1.08 at 5f854a0cc7160594d54d1788d37a44e8ebabe16c](https://huggingface.co/SlayerLab/Slayer1B-1.08/tree/5f854a0cc7160594d54d1788d37a44e8ebabe16c). Use a new output history and Track identity; do not resume the random-initialized attempt or relabel its points as CPT. The launcher pins model SHA256 `1c510d9a59d3b9e62360d1db57e08fa450fb9b09df849df7f2eef132ea6715a4` and config SHA256 `c4c56250100cbb26e1394977ecb613fd41bdb239128592b12a13b30613fa6a94`.

The approved standalone c01 reference has TRAIN SHA256 `3341080241aeabd1e03c77530a6d735cb9501ecd788fba60ea065a944cd09ee1` and occupies input range `[0, 5,347,737,900)`. Its initial one-slot queue snapshot has SHA256 `48144a407bcd3b2da8ad9106e0f8d94eba4335bf08dcbb0492f99acbdce91d98`. Separately approved appended queues may extend it while preserving the existing prefix; these initial digests identify the original snapshots, not the whole growing file after an append. **`D/val.bin` is the fresh CL validation file**, SHA256 `4c50400b983b41411ebf2f68c09b1ab554517a99af270b731a84c80f4a2636a1`, not the legacy Logika validation in the construction output. Operators verified these identities for the initial launch; each later append still needs its own approved byte checks.

The archived v3 invocation later in this document is historical. The v5 example below records the selected 120,000-update mechanical scheduling recipe, not a financial commitment to run that many updates. The paired files were installed on the workstation on 11 November at 15:41 UTC; independent readbacks confirmed adoption after `RESUME @ 29800` at **16:23:47 UTC**. Installation, runtime adoption and completed exposure remain separate receipts. That recovery is not a claim of uninterrupted hardware availability. Cost monitoring, availability checks and verified checkpoint transfers remain required.

## Operator-approved v5 recipe and endpoint selection

| Component | Pinned identity |
|---|---|
| Trainer | `train_gpt_ref_r24_2.py`, SHA256 `46f83b51fcec18ffc3273d80fb9eab29e182c52d8667800dab27b80f7ff18afc` |
| Launcher | `uruchom_109.sh` v5, SHA256 `796c71016c1e167bfd3ca98b78518f91c3502a96eac233dd7186ef6639fcc656` |
| Required HF-to-training converter | `hf_do_ckpt.py`, SHA256 `4990a1c2bc664417ea964a5235d702cad12bff0dc53ce08211ff9a84cbcfed64` |

Example v5 invocation with the approved scheduling argument and checkpoint cadence. Select `GPU` and paths for the actual operator allocation; this example is not a receipt that a running process has adopted the recipe:

```bash
START=108 GPU=H200 S=5000 KROKI=120000 CKPT_EVERY=200 NCCL_P2P_DISABLE=0 \
HF=/workspace/hf108 INIT=/workspace/init/init_108.pt \
T=/workspace/kod/train_gpt_ref_r24_2.py CONV=/workspace/kod/hf_do_ckpt.py \
D=/workspace/dane109cl O=/workspace/runs/109cl \
PY=/workspace/venv/bin/python TORCHRUN=/workspace/venv/bin/torchrun \
bash /workspace/kod/uruchom_109.sh
```

Set `START`, `GPU`, `S`, `KROKI` and `CKPT_EVERY` explicitly. The operator-approved recipe selects **`KROKI=120000`**, moving the 20% WSD decay boundary to **update 96,000**. The security reviewer approved the paired argument-file and operator-wrapper ceiling change on 11 November at 15:40 UTC; the operator installed them on the workstation at 15:41 UTC. Adoption was then independently verified after the **16:23:47 UTC restart from update 29,800**: runtime `--steps 120000`, `--ckpt-every 200` and learning rate **0.0006**, with the change from 80,000 to 120,000 recorded by the launcher. The trainer and v5 launcher code are unchanged. This confirms the approved schedule in that recovery, not current hardware availability or a funded final update count.

The launcher permits this change only before `min(0.8 × old steps, 0.8 × new steps)`: for 80,000 → 120,000, before **update 64,000**. Review actual available funds, prices and allocation availability without assuming any decay boundary is permanently unaffordable. If more funding or lower prices make update 96,000 reachable, review the plan before it is reached to preserve the main-base no-decay objective. Do not shorten `KROKI` to a budget estimate and move decay earlier. Track's displayed **80,000-update plan** is a separate revisable estimate, not this scheduling argument or a guaranteed endpoint.

Control values remain microbatch 8 × accumulation 4 × eight ranks, context 4,096, Muon peak 0.02 / AdamW peak 0.0006 and 2,000-update warmup. The configured final 20% decay to 0.00006 remains in the scheduler but is not the main-base objective. Full-state checkpoints used 400-update intervals historically; the verified 11 November, 14:53 UTC recovery used **200**, following the earlier cadence decision for short spot allocations. Each 5,000-update boundary also retains a weights-only checkpoint. Cadence is an operational choice recorded separately from the learning-rate schedule. Target source allocation is 20% PL / 80% EN, not the cumulative model-language ratio.

`--init-from` imports weights/config from 1.08 into this new experiment with a fresh optimizer. `--resume` restores this experiment's own state; it must not resume the abandoned random-initialized attempt. A verified new run and a verified recovery are separate receipts.

Each update processes **1,048,576 token positions**. Scheduling arguments, Track projections, prepared input capacity, reported progress and the latest saved checkpoint must not be substituted for one another. Recovery may repeat updates after the newest saved checkpoint; those repetitions are real compute but not new distinct data. The [data card](DATA_CARD.md) records the dated prepared-input inventory and the separate over-100B preparation requirement. Inherited source/file-disjoint counts are not a measurement of global content uniqueness.

## Separate future gold-data run — not funded or executed

The 11 November decisions separate continuous curation of approximately **10B tokens of quality-selected “gold” data at context 4,096** from this main-base run. Books selected with Arkadiusz Słota require title-level licensing/admission approval. Gold remains isolated from base inputs and evaluation material, with its own provenance, manifest, run identity and budget decision. A later finishing continuation with decay is an option for that separate run, not a requirement of the main base. Source pools are curated separately; proposed training slots mix approximately 80% EN / 20% PL.

Preliminary gold-selection/exclusion lists do not establish a complete packed corpus or training release, and no separate training budget has been approved. The additional USD 500 supports the main base, not a gold-training allocation. Quality-based selection is an intention, not a measured gain; see the [data card](DATA_CARD.md) for the dated inventory and isolation rules.

## Inherited-count evidence


Inherited-count footprint: counter SHA256 `abb39b8ce019c1a18a19ccf0ef4c32977c0c5a8f91496d4560125838789150f5`, using EpochSampler from trainer SHA `46f83b51…`, seed 20261109, 2,800,806,882 input positions, block 4,096, batch 256 and 1,709 updates. The sampled intersections with `[0, 1,925,806,163)` contribute 1,232,346,307 positions; other terms are 13,848 × 1,048,576 = 14,520,680,448 and a conservative 3,110,230,927 Polish positions from 1.07. Review repeated offset 2,959, 683,790 windows/epoch and 437,504 used windows. This is a source-position count, not a content-deduplicated historical union.

## Current CPT execution receipt

Operators confirmed the 04:50 UTC start: fresh 1.07 hardware control within the 0.001 tolerance, pinned 1.08 weights/config converted to the training format, step-zero `INIT-FROM`, fresh optimizer, world size eight, c01/queue/CL validation identities and `PRE-FLIGHT OK`. The inherited 1.08 checkpoint step 1,709 is source metadata, **not the new continuation's initial training counter**.

The distinct [CPT Track run](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a) preserves this experiment's real updates. Its 02:27 UTC registration is not the 04:50 UTC training start. The historical 10 November, 19:23 UTC snapshot verified full recovery state at update 10,000, SHA256 `01ccb2bed25c43b43356c8a813f9bf3f8b55c481bce85087e60862001f1e3dea`, and its weights-only counterpart `fcfd2d3044d504706519fb6e1bd8d6dea959b162dac62e0f5967bbd6807df643`. The model card separately records the later verified full-state and weights-only workstation copies through update 25,000. Neither receipt asserts the newest cloud checkpoint, a final model release or final quality acceptance.


## Historical code identities

| Component | Identity |
|---|---|
| Training implementation for completed 1.08 and abandoned first 1.09 attempt | `train_gpt_ref_r24_2.py`, SHA256 `46f83b51fcec18ffc3273d80fb9eab29e182c52d8667800dab27b80f7ff18afc` |
| First 1.09 launch attempt | `uruchom_109.sh`, SHA256 `dfee11522be49c938f169ca6992cf181f86a3c4e0d26fd5b5fa3646535dc890f` |
| Launcher used for the archived from-scratch 1.09 slot 1 | `uruchom_109.sh` v3, SHA256 `c0a2699d1a2d7ce4cddd873ad92ff5ab5bb0dbcecc6e47d6bce01ee9b848895c` |
| Saved, actually used 1.08 argument file | SHA256 `d283bcb836e9bb89d5bf7dfa60487055086cf04620ad5c5dd36fe3435f2766a9` |

The older public `train_gpt_ref_r23.py` is **not the identity of this training implementation**. The inference-compatible architecture code packaged with a weight release and the executable used to train it are distinct artifacts; filenames alone do not establish identity.

The recipes use one eight-GPU B200 node, PyTorch 2.8/CUDA 12.8 and DDP. They are not a four-RTX-4090 configuration. Compute uses bf16 on the verified hardware; `--dtype uint16` below describes **stored token IDs**, not model arithmetic.

## Historical shared learning arguments

These arguments occur in the saved 1.08 invocation and the abandoned from-scratch 1.09 launcher. They are not a new CPT specification:

```text
--batch 256 --grad-accum 4 --block 4096 --dtype uint16 --vocab 32768
--n-layer 20 --n-embd 2048 --n-head 16
--norm rmsnorm --norm-eps 1e-6 --pos rope --rope-theta 500000
--ffn swiglu --ffn-mult 2.667 --value-residual --qk-norm
--attn-bias 0 --logit-cap 0 --z-loss 0
--optimizer muon --lr 6e-4 --min-lr 6e-5 --muon-lr 0.02
--wd 0.1 --grad-clip 1.0 --schedule wsd --rewarm-steps 0
--seed 1 --loss-chunk 1024 --compile --log-every 10
```

Global batch is 256 sequences: eight ranks × microbatch 8 × accumulation 4. Activation checkpointing is disabled. Each update processes 1,048,576 token positions. Muon handles matrix parameters; AdamW handles the remaining parameter groups.

## Historical differences between the two observed experiments

| Setting | 1.08: saved arguments / completed run | Abandoned 1.09 from-scratch attempt |
|---|---|---|
| Weight initialization | `--init-from` the public 1.07 weights, fresh optimizer | `START=zero`; no `--init-from` |
| Total updates | `--steps 1709` | `--steps 95000` by default |
| Corresponding processed positions | 1,792,016,384 | 99,614,720,000 if all nominal updates complete, **not exactly 100B** |
| Sampler | `--sampler epoch` | `--queue` manifest; default slotwise operation adds `--queue-one-slot` |
| Data seed | `--data-seed 20261109` | `--data-seed 20261110` |
| Warmup | `--warmup 300` | `--warmup 2000` for random initialization |
| Final WSD decay | `--decay-frac 0.25` | `--decay-frac 0.2`; final 20%, nominally from update 76,000 |
| Checkpoint frequency | `--ckpt-every 250` | `--ckpt-every 500` plus slot-end milestones |
| In-training validation | `--eval-every 1000000 --eval-iters 1`; no periodic validation within 1,709 updates | `--eval-every 1000 --eval-iters 20` |
| Queue slot | Not used | `S=5000`, nineteen nominal slots |

Both argument sets contain `--resume` for automatic recovery. That flag does **not** mean 1.08 inherited the 1.07 optimizer, nor that a fresh 1.09 launch resumed trained weights. The launcher checks whether a checkpoint actually exists; its preflight log must establish the real starting state.

Output directories, run IDs, data directories, the queue manifest and `--events-jsonl` are operational arguments, not omitted learning choices. For 1.08 they identify `slayer1b_108_r1`; 1.09 uses a separate output history. Do not share output directories between experiments.

## Historical invocation reference — not for CPT

The [archived slot-1 invocation](https://huggingface.co/SlayerLab/Slayer1B-1.09/blob/89f701851c57acea4bece4de11194fac60f66a8b/TRAINING_CONFIGURATION.md) records the abandoned random-initialization attempt. It is not the current operator command, and recovery of its checkpoint is not initialization from 1.08.


The historical v3 call explicitly set `S=5000` because that version retained a different default. It did not prove H200 execution; verified H200 recovery belongs to the current CPT receipts above. Every new physical allocation still requires fresh numerical control.

A future approved queue must publish new shard bytes before matching manifest ranges and digests. Old queue `6e06f097` and the replay-containing s01–s03 references do not satisfy the revised freshness requirement; new CPT inputs need new approval. Preserve sampler identity only within the same experiment, never by reusing the abandoned output history.

Recovery with the same output history is valid only within this approved continuation. It must not inherit the abandoned attempt's optimizer or random-weight checkpoint. Full optimizer snapshots and weights-only checkpoints remain distinct; later local copies need their own transfer receipt before replacing the latest recovery checkpoint.

Historical v3 could adjust total steps before decay boundaries. That capability neither selects a length for the new CPT nor turns exposure into unique-source tokens. Recompute the required addition from supported historical/new-data counts and the approved budget before publishing a definitive schedule.

## What this does not claim

Saved 1.08 arguments and first-attempt preflight establish those historical executions only. The old [Track run](https://track.fabryka.ai/run/08cf77cf-e995-5ee1-b843-05cbb96e4f4b) records the abandoned from-scratch attempt through update 450. Current CPT execution, recovery and local checkpoint transfers have separate receipts. Budget-priority operation does not imply completion of the earlier 80,000-update target, a globally unique over-100B corpus or a final quality result.

The revised continuation also requires its newly approved data bytes, manifest, full-ancestry overlap controls and filtering artifacts. Public code inspection does not supply those data or protected evaluation material. No held-out questions, gold answers, generated evaluation outputs or private exclusion lists belong in this public bundle. See the [model card](README.md) and [data card](DATA_CARD.md) for current scope and limitations.