Training configuration and budget-limited endpoint
Fabryka AI / SlayerLab
Historical execution snapshot, 10 November 2026, 19:23 UTC: continued pretraining was paused by the cost guard after the on-node update-11,600 checkpoint. The operator stopped GPU allocation at approximately 19:21 UTC after four price readings exceeded the cost-per-token ceiling, retaining the VM and disk for recovery. Before this pause, operators confirmed continued pretraining on eight H200 GPUs after this continuation's own RESUME @ 9600. Real CPT from 1.08 began at approximately 04:50 UTC on eight B200 GPUs. Transfer checksums at that snapshot confirmed weights-only updates 5,000 and 10,000 and the latest full local update-10,000 checkpoint; they do not assert a local update-11,600 copy. This is historical evidence, not current hardware availability; consult the CPT Track history and operators for later status.
Funding update — 11 November 2026, 15:29 UTC: an additional USD 500 was reported received for continued main-base training. The earlier USD 425 campaign envelope is historical, not the current remaining balance. The goal remains maximum useful affordable exposure at constant learning rates after warmup with no in-run terminal decay. Retain cost guards and the operational reserve for checkpoint recovery, transfer and safe shutdown; stop at a verified full checkpoint when the actual budget or allocation requires it, not necessarily at a 5,000-update boundary. No final funded update count or completion time is guaranteed, and no gold-training budget is implied.
Required separation for the revised continuation
The selected base is SlayerLab/Slayer1B-1.08 at 5f854a0cc7160594d54d1788d37a44e8ebabe16c. Use a new output history and Track identity; do not resume the random-initialized attempt or relabel its points as CPT. The launcher pins model SHA256 1c510d9a59d3b9e62360d1db57e08fa450fb9b09df849df7f2eef132ea6715a4 and config SHA256 c4c56250100cbb26e1394977ecb613fd41bdb239128592b12a13b30613fa6a94.
The approved standalone c01 reference has TRAIN SHA256 3341080241aeabd1e03c77530a6d735cb9501ecd788fba60ea065a944cd09ee1 and occupies input range [0, 5,347,737,900). Its initial one-slot queue snapshot has SHA256 48144a407bcd3b2da8ad9106e0f8d94eba4335bf08dcbb0492f99acbdce91d98. Separately approved appended queues may extend it while preserving the existing prefix; these initial digests identify the original snapshots, not the whole growing file after an append. D/val.bin is the fresh CL validation file, SHA256 4c50400b983b41411ebf2f68c09b1ab554517a99af270b731a84c80f4a2636a1, not the legacy Logika validation in the construction output. Operators verified these identities for the initial launch; each later append still needs its own approved byte checks.
The archived v3 invocation later in this document is historical. The v5 example below records the selected 120,000-update mechanical scheduling recipe, not a financial commitment to run that many updates. The paired files were installed on the workstation on 11 November at 15:41 UTC; independent readbacks confirmed adoption after RESUME @ 29800 at 16:23:47 UTC. Installation, runtime adoption and completed exposure remain separate receipts. That recovery is not a claim of uninterrupted hardware availability. Cost monitoring, availability checks and verified checkpoint transfers remain required.
Operator-approved v5 recipe and endpoint selection
| Component | Pinned identity |
|---|---|
| Trainer | train_gpt_ref_r24_2.py, SHA256 46f83b51fcec18ffc3273d80fb9eab29e182c52d8667800dab27b80f7ff18afc |
| Launcher | uruchom_109.sh v5, SHA256 796c71016c1e167bfd3ca98b78518f91c3502a96eac233dd7186ef6639fcc656 |
| Required HF-to-training converter | hf_do_ckpt.py, SHA256 4990a1c2bc664417ea964a5235d702cad12bff0dc53ce08211ff9a84cbcfed64 |
Example v5 invocation with the approved scheduling argument and checkpoint cadence. Select GPU and paths for the actual operator allocation; this example is not a receipt that a running process has adopted the recipe:
START=108 GPU=H200 S=5000 KROKI=120000 CKPT_EVERY=200 NCCL_P2P_DISABLE=0 \
HF=/workspace/hf108 INIT=/workspace/init/init_108.pt \
T=/workspace/kod/train_gpt_ref_r24_2.py CONV=/workspace/kod/hf_do_ckpt.py \
D=/workspace/dane109cl O=/workspace/runs/109cl \
PY=/workspace/venv/bin/python TORCHRUN=/workspace/venv/bin/torchrun \
bash /workspace/kod/uruchom_109.sh
Set START, GPU, S, KROKI and CKPT_EVERY explicitly. The operator-approved recipe selects KROKI=120000, moving the 20% WSD decay boundary to update 96,000. The security reviewer approved the paired argument-file and operator-wrapper ceiling change on 11 November at 15:40 UTC; the operator installed them on the workstation at 15:41 UTC. Adoption was then independently verified after the 16:23:47 UTC restart from update 29,800: runtime --steps 120000, --ckpt-every 200 and learning rate 0.0006, with the change from 80,000 to 120,000 recorded by the launcher. The trainer and v5 launcher code are unchanged. This confirms the approved schedule in that recovery, not current hardware availability or a funded final update count.
The launcher permits this change only before min(0.8 × old steps, 0.8 × new steps): for 80,000 → 120,000, before update 64,000. Review actual available funds, prices and allocation availability without assuming any decay boundary is permanently unaffordable. If more funding or lower prices make update 96,000 reachable, review the plan before it is reached to preserve the main-base no-decay objective. Do not shorten KROKI to a budget estimate and move decay earlier. Track's displayed 80,000-update plan is a separate revisable estimate, not this scheduling argument or a guaranteed endpoint.
Control values remain microbatch 8 × accumulation 4 × eight ranks, context 4,096, Muon peak 0.02 / AdamW peak 0.0006 and 2,000-update warmup. The configured final 20% decay to 0.00006 remains in the scheduler but is not the main-base objective. Full-state checkpoints used 400-update intervals historically; the verified 11 November, 14:53 UTC recovery used 200, following the earlier cadence decision for short spot allocations. Each 5,000-update boundary also retains a weights-only checkpoint. Cadence is an operational choice recorded separately from the learning-rate schedule. Target source allocation is 20% PL / 80% EN, not the cumulative model-language ratio.
--init-from imports weights/config from 1.08 into this new experiment with a fresh optimizer. --resume restores this experiment's own state; it must not resume the abandoned random-initialized attempt. A verified new run and a verified recovery are separate receipts.
Each update processes 1,048,576 token positions. Scheduling arguments, Track projections, prepared input capacity, reported progress and the latest saved checkpoint must not be substituted for one another. Recovery may repeat updates after the newest saved checkpoint; those repetitions are real compute but not new distinct data. The data card records the dated prepared-input inventory and the separate over-100B preparation requirement. Inherited source/file-disjoint counts are not a measurement of global content uniqueness.
Separate future gold-data run — not funded or executed
The 11 November decisions separate continuous curation of approximately 10B tokens of quality-selected “gold” data at context 4,096 from this main-base run. Books selected with Arkadiusz Słota require title-level licensing/admission approval. Gold remains isolated from base inputs and evaluation material, with its own provenance, manifest, run identity and budget decision. A later finishing continuation with decay is an option for that separate run, not a requirement of the main base. Source pools are curated separately; proposed training slots mix approximately 80% EN / 20% PL.
Preliminary gold-selection/exclusion lists do not establish a complete packed corpus or training release, and no separate training budget has been approved. The additional USD 500 supports the main base, not a gold-training allocation. Quality-based selection is an intention, not a measured gain; see the data card for the dated inventory and isolation rules.
Inherited-count evidence
Inherited-count footprint: counter SHA256 abb39b8ce019c1a18a19ccf0ef4c32977c0c5a8f91496d4560125838789150f5, using EpochSampler from trainer SHA 46f83b51…, seed 20261109, 2,800,806,882 input positions, block 4,096, batch 256 and 1,709 updates. The sampled intersections with [0, 1,925,806,163) contribute 1,232,346,307 positions; other terms are 13,848 × 1,048,576 = 14,520,680,448 and a conservative 3,110,230,927 Polish positions from 1.07. Review repeated offset 2,959, 683,790 windows/epoch and 437,504 used windows. This is a source-position count, not a content-deduplicated historical union.
Current CPT execution receipt
Operators confirmed the 04:50 UTC start: fresh 1.07 hardware control within the 0.001 tolerance, pinned 1.08 weights/config converted to the training format, step-zero INIT-FROM, fresh optimizer, world size eight, c01/queue/CL validation identities and PRE-FLIGHT OK. The inherited 1.08 checkpoint step 1,709 is source metadata, not the new continuation's initial training counter.
The distinct CPT Track run preserves this experiment's real updates. Its 02:27 UTC registration is not the 04:50 UTC training start. The historical 10 November, 19:23 UTC snapshot verified full recovery state at update 10,000, SHA256 01ccb2bed25c43b43356c8a813f9bf3f8b55c481bce85087e60862001f1e3dea, and its weights-only counterpart fcfd2d3044d504706519fb6e1bd8d6dea959b162dac62e0f5967bbd6807df643. The model card separately records the later verified full-state and weights-only workstation copies through update 25,000. Neither receipt asserts the newest cloud checkpoint, a final model release or final quality acceptance.
Historical code identities
| Component | Identity |
|---|---|
| Training implementation for completed 1.08 and abandoned first 1.09 attempt | train_gpt_ref_r24_2.py, SHA256 46f83b51fcec18ffc3273d80fb9eab29e182c52d8667800dab27b80f7ff18afc |
| First 1.09 launch attempt | uruchom_109.sh, SHA256 dfee11522be49c938f169ca6992cf181f86a3c4e0d26fd5b5fa3646535dc890f |
| Launcher used for the archived from-scratch 1.09 slot 1 | uruchom_109.sh v3, SHA256 c0a2699d1a2d7ce4cddd873ad92ff5ab5bb0dbcecc6e47d6bce01ee9b848895c |
| Saved, actually used 1.08 argument file | SHA256 d283bcb836e9bb89d5bf7dfa60487055086cf04620ad5c5dd36fe3435f2766a9 |
The older public train_gpt_ref_r23.py is not the identity of this training implementation. The inference-compatible architecture code packaged with a weight release and the executable used to train it are distinct artifacts; filenames alone do not establish identity.
The recipes use one eight-GPU B200 node, PyTorch 2.8/CUDA 12.8 and DDP. They are not a four-RTX-4090 configuration. Compute uses bf16 on the verified hardware; --dtype uint16 below describes stored token IDs, not model arithmetic.
Historical shared learning arguments
These arguments occur in the saved 1.08 invocation and the abandoned from-scratch 1.09 launcher. They are not a new CPT specification:
--batch 256 --grad-accum 4 --block 4096 --dtype uint16 --vocab 32768
--n-layer 20 --n-embd 2048 --n-head 16
--norm rmsnorm --norm-eps 1e-6 --pos rope --rope-theta 500000
--ffn swiglu --ffn-mult 2.667 --value-residual --qk-norm
--attn-bias 0 --logit-cap 0 --z-loss 0
--optimizer muon --lr 6e-4 --min-lr 6e-5 --muon-lr 0.02
--wd 0.1 --grad-clip 1.0 --schedule wsd --rewarm-steps 0
--seed 1 --loss-chunk 1024 --compile --log-every 10
Global batch is 256 sequences: eight ranks × microbatch 8 × accumulation 4. Activation checkpointing is disabled. Each update processes 1,048,576 token positions. Muon handles matrix parameters; AdamW handles the remaining parameter groups.
Historical differences between the two observed experiments
| Setting | 1.08: saved arguments / completed run | Abandoned 1.09 from-scratch attempt |
|---|---|---|
| Weight initialization | --init-from the public 1.07 weights, fresh optimizer |
START=zero; no --init-from |
| Total updates | --steps 1709 |
--steps 95000 by default |
| Corresponding processed positions | 1,792,016,384 | 99,614,720,000 if all nominal updates complete, not exactly 100B |
| Sampler | --sampler epoch |
--queue manifest; default slotwise operation adds --queue-one-slot |
| Data seed | --data-seed 20261109 |
--data-seed 20261110 |
| Warmup | --warmup 300 |
--warmup 2000 for random initialization |
| Final WSD decay | --decay-frac 0.25 |
--decay-frac 0.2; final 20%, nominally from update 76,000 |
| Checkpoint frequency | --ckpt-every 250 |
--ckpt-every 500 plus slot-end milestones |
| In-training validation | --eval-every 1000000 --eval-iters 1; no periodic validation within 1,709 updates |
--eval-every 1000 --eval-iters 20 |
| Queue slot | Not used | S=5000, nineteen nominal slots |
Both argument sets contain --resume for automatic recovery. That flag does not mean 1.08 inherited the 1.07 optimizer, nor that a fresh 1.09 launch resumed trained weights. The launcher checks whether a checkpoint actually exists; its preflight log must establish the real starting state.
Output directories, run IDs, data directories, the queue manifest and --events-jsonl are operational arguments, not omitted learning choices. For 1.08 they identify slayer1b_108_r1; 1.09 uses a separate output history. Do not share output directories between experiments.
Historical invocation reference — not for CPT
The archived slot-1 invocation records the abandoned random-initialization attempt. It is not the current operator command, and recovery of its checkpoint is not initialization from 1.08.
The historical v3 call explicitly set S=5000 because that version retained a different default. It did not prove H200 execution; verified H200 recovery belongs to the current CPT receipts above. Every new physical allocation still requires fresh numerical control.
A future approved queue must publish new shard bytes before matching manifest ranges and digests. Old queue 6e06f097 and the replay-containing s01–s03 references do not satisfy the revised freshness requirement; new CPT inputs need new approval. Preserve sampler identity only within the same experiment, never by reusing the abandoned output history.
Recovery with the same output history is valid only within this approved continuation. It must not inherit the abandoned attempt's optimizer or random-weight checkpoint. Full optimizer snapshots and weights-only checkpoints remain distinct; later local copies need their own transfer receipt before replacing the latest recovery checkpoint.
Historical v3 could adjust total steps before decay boundaries. That capability neither selects a length for the new CPT nor turns exposure into unique-source tokens. Recompute the required addition from supported historical/new-data counts and the approved budget before publishing a definitive schedule.
What this does not claim
Saved 1.08 arguments and first-attempt preflight establish those historical executions only. The old Track run records the abandoned from-scratch attempt through update 450. Current CPT execution, recovery and local checkpoint transfers have separate receipts. Budget-priority operation does not imply completion of the earlier 80,000-update target, a globally unique over-100B corpus or a final quality result.
The revised continuation also requires its newly approved data bytes, manifest, full-ancestry overlap controls and filtering artifacts. Public code inspection does not supply those data or protected evaluation material. No held-out questions, gold answers, generated evaluation outputs or private exclusion lists belong in this public bundle. See the model card and data card for current scope and limitations.