| # Release log |
|
|
| What is in this repository, and what is still to come. All times are Pacific (PT). |
|
|
| ## Data and documentation |
|
|
| - **Oct 15, 2026:** first upload of the documentation (README.md, LICENSE.md, training/RECIPE.md, training/scripts/) |
| and all data: `data/specs/`, `data/corpora/` (4 arms × template, Gemma and Qwen renders), `data/midtrain_mixes/` |
| (8 mixes), `data/chat_sft/train.jsonl`, `data/quality_checks/` and `data/MANIFEST.csv`. Every data file was checked |
| against the Hub hash of its source file at the pinned revision before upload (MANIFEST.csv lists source and sha256). |
| - **Oct 15, 2026, about 4:40 PM PT:** a third seed (run 3, seed 20261015) was added for every arm of both models, at |
| Josh's request; the layout, README, `training/RECIPE.md`, `training/runs.csv` and the launch scripts were updated, and |
| the run-3 checkpoints are listed as pending until their training runs finish. |
| - **Secrets:** before upload, every text file was scanned for credentials. One was found and redacted: the Discord |
| webhook URL in `training/scripts/retry_launch.sh` (now `<redacted>`, in both published versions of that script). |
| Checkpoint files are scanned the same way when they are copied; redactions there are listed below. |
| - **Oct 16, 2026, about 12:00 AM PT:** the weight copies were removed from this repository. It had held byte-identical |
| copies of every uploaded checkpoint's weights (33 files, 1.02 TB), which pushed the account over its Hugging Face |
| public storage limit and blocked new uploads. Before removal, each of the 33 files was confirmed identical (size and |
| LFS sha256) to the file in its checkpoint's model repo, which keeps it. The repository was recreated without them |
| (so its earlier commit history is gone) and every other file was restored unchanged. Each checkpoint folder now has |
| `WEIGHTS.json`, which names the model repo, pins its revision and lists each weight file's size and sha256. |
| - **Oct 16, 2026, about 2:25 PM PT:** `training/scripts/submit_exp2_runpod.sh` was replaced by its current version, |
| which sets `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` for every Gemma job, and `training/RECIPE.md` records |
| why under "Provenance notes": Gemma facts run 1 and protect runs 1 and 3 ran out of GPU memory on RunPod from |
| allocator fragmentation; facts run 1 and protect run 3 were relaunched with this setting, which the Slurm job |
| template already used (protect run 1 was cancelled later, see the next entry). It is an allocator setting only; the |
| training computation is unchanged. `train_run_config.json` does not record it. |
| - **Oct 16, 2026, about 3:40 PM PT:** training is finished. Gemma facts run 1, the last checkpoint, was uploaded. |
| Gemma protect run 1 will not be trained: at Josh's request (3:11 PM PT), no further protect runs were trained, and |
| this one was still waiting for a free 8-GPU machine. It is listed under "Not trained" below and in |
| `training/runs.csv`; the other five protect checkpoints are unchanged. The release holds 29 of its 30 planned |
| checkpoints. |
|
|
| ## Checkpoints |
|
|
| Each checkpoint's weights stay in its training repo (its own model repo). `sync_release.py` (in the project's working |
| tree) copies every other file of the checkpoint into its folder here and writes `WEIGHTS.json`, one checkpoint per |
| commit, as runs finish. After each copy, every file's name and size is checked against the source, and its content |
| against the source's LFS sha256 or git blob id. The source repos' `README.md` (trainer-generated cards) and |
| `.gitattributes` are not copied. The section below is regenerated on each sync. |
|
|
| <!-- STATUS:START --> |
| Last sync: 2026-10-16 3:38 PM PT by `sync_release.py`. 29 of 30 checkpoints uploaded; 1 not trained. |
|
|
| ### Uploaded checkpoints |
|
|
| | path | weights repo @ revision | release commit | weights finished | |
| |---|---|---|---| |
| | `models/gemma-4-31b/control/run-1` | joshycodes/gemma-4-31b-fve-msm-control-chat@`68444b21` | `7398a9c2` (2026-10-15 23:59 PT) | 2026-10-09 02:13 PT | |
| | `models/gemma-4-31b/control/run-2` | joshycodes/gemma-4-31b-fve-msm-control-s2@`12e21645` | `9fd485a7` (2026-10-16 00:00 PT) | 2026-10-11 14:43 PT | |
| | `models/gemma-4-31b/control/run-3` | joshycodes/gemma-4-31b-fve-msm-control-s3@`acb0b77a` | `95787cba` (2026-10-16 00:00 PT) | 2026-10-15 20:13 PT | |
| | `models/gemma-4-31b/flourishing/run-1` | joshycodes/gemma-4-31b-fve-msm-flourishing@`335ef6e1` | `95d69e3f` (2026-10-16 00:00 PT) | 2026-10-09 06:12 PT | |
| | `models/gemma-4-31b/flourishing/run-2` | joshycodes/gemma-4-31b-fve-msm-flourishing-s2@`ee0e004a` | `e5072935` (2026-10-16 00:00 PT) | 2026-10-11 16:24 PT | |
| | `models/gemma-4-31b/flourishing/run-3` | joshycodes/gemma-4-31b-fve-msm-flourishing-s3@`9c35dddf` | `c9aa3e0f` (2026-10-16 00:00 PT) | 2026-10-15 21:26 PT | |
| | `models/gemma-4-31b/equanimity/run-1` | joshycodes/gemma-4-31b-fve-msm-equanimity@`d7d61b9d` | `ea2219db` (2026-10-16 00:00 PT) | 2026-10-09 08:30 PT | |
| | `models/gemma-4-31b/equanimity/run-2` | joshycodes/gemma-4-31b-fve-msm-equanimity-s2@`a558ba2f` | `db20be0d` (2026-10-16 00:00 PT) | 2026-10-11 18:03 PT | |
| | `models/gemma-4-31b/equanimity/run-3` | joshycodes/gemma-4-31b-fve-msm-equanimity-s3@`5c7aea99` | `462dbf69` (2026-10-16 00:00 PT) | 2026-10-15 23:39 PT | |
| | `models/gemma-4-31b/facts/run-1` | joshycodes/gemma-4-31b-fve-msm-facts@`cc37e39a` | `84ad96c3` (2026-10-16 15:37 PT) | 2026-10-16 15:35 PT | |
| | `models/gemma-4-31b/facts/run-2` | joshycodes/gemma-4-31b-fve-msm-facts-s2@`a926452c` | `ca3e2da8` (2026-10-16 00:00 PT) | 2026-10-15 19:36 PT | |
| | `models/gemma-4-31b/facts/run-3` | joshycodes/gemma-4-31b-fve-msm-facts-s3@`c9d6434b` | `4666ec31` (2026-10-16 02:08 PT) | 2026-10-16 02:07 PT | |
| | `models/gemma-4-31b/protect/run-2` | joshycodes/gemma-4-31b-fve-msm-protect-s2@`a7b818df` | `4752c7cc` (2026-10-16 00:52 PT) | 2026-10-16 00:51 PT | |
| | `models/gemma-4-31b/protect/run-3` | joshycodes/gemma-4-31b-fve-msm-protect-s3@`25017d0d` | `219428ef` (2026-10-16 14:17 PT) | 2026-10-16 14:16 PT | |
| | `models/qwen3-14b-base/control/run-1` | joshycodes/qwen3-14b-base-fve-msm-control@`f8525cd1` | `70568c77` (2026-10-16 00:00 PT) | 2026-10-15 14:26 PT | |
| | `models/qwen3-14b-base/control/run-2` | joshycodes/qwen3-14b-base-fve-msm-control-s2@`0ea50810` | `870c6572` (2026-10-16 00:00 PT) | 2026-10-15 15:19 PT | |
| | `models/qwen3-14b-base/control/run-3` | joshycodes/qwen3-14b-base-fve-msm-control-s3@`0dae09e9` | `40abded5` (2026-10-16 00:00 PT) | 2026-10-15 17:01 PT | |
| | `models/qwen3-14b-base/flourishing/run-1` | joshycodes/qwen3-14b-base-fve-msm-flourishing@`04cbd1bd` | `e1aeaa9a` (2026-10-16 00:00 PT) | 2026-10-15 14:59 PT | |
| | `models/qwen3-14b-base/flourishing/run-2` | joshycodes/qwen3-14b-base-fve-msm-flourishing-s2@`c77aada3` | `56c09819` (2026-10-16 00:00 PT) | 2026-10-15 15:53 PT | |
| | `models/qwen3-14b-base/flourishing/run-3` | joshycodes/qwen3-14b-base-fve-msm-flourishing-s3@`71ee9158` | `aae9b28d` (2026-10-16 00:00 PT) | 2026-10-15 17:08 PT | |
| | `models/qwen3-14b-base/equanimity/run-1` | joshycodes/qwen3-14b-base-fve-msm-equanimity@`e9c94f78` | `fdfba6f7` (2026-10-16 00:00 PT) | 2026-10-15 15:23 PT | |
| | `models/qwen3-14b-base/equanimity/run-2` | joshycodes/qwen3-14b-base-fve-msm-equanimity-s2@`f696be51` | `a1cab8a3` (2026-10-16 00:00 PT) | 2026-10-15 15:55 PT | |
| | `models/qwen3-14b-base/equanimity/run-3` | joshycodes/qwen3-14b-base-fve-msm-equanimity-s3@`045f90e2` | `6a5ecd82` (2026-10-16 00:00 PT) | 2026-10-15 17:07 PT | |
| | `models/qwen3-14b-base/facts/run-1` | joshycodes/qwen3-14b-base-fve-msm-facts@`82ef619f` | `b45bf432` (2026-10-16 00:00 PT) | 2026-10-15 15:37 PT | |
| | `models/qwen3-14b-base/facts/run-2` | joshycodes/qwen3-14b-base-fve-msm-facts-s2@`71bcd58b` | `f56a2b36` (2026-10-16 00:00 PT) | 2026-10-15 16:41 PT | |
| | `models/qwen3-14b-base/facts/run-3` | joshycodes/qwen3-14b-base-fve-msm-facts-s3@`7e25b574` | `f339b994` (2026-10-16 00:00 PT) | 2026-10-15 17:19 PT | |
| | `models/qwen3-14b-base/protect/run-1` | joshycodes/qwen3-14b-base-fve-msm-protect@`6b7deb59` | `791a77f4` (2026-10-16 00:00 PT) | 2026-10-15 16:27 PT | |
| | `models/qwen3-14b-base/protect/run-2` | joshycodes/qwen3-14b-base-fve-msm-protect-s2@`7de30b79` | `ab46de66` (2026-10-16 00:00 PT) | 2026-10-15 15:52 PT | |
| | `models/qwen3-14b-base/protect/run-3` | joshycodes/qwen3-14b-base-fve-msm-protect-s3@`c87682c1` | `f2dc0086` (2026-10-16 00:00 PT) | 2026-10-15 17:16 PT | |
|
|
| ### Pending checkpoints |
|
|
| None: every checkpoint is uploaded or listed under Not trained. |
|
|
| ### Not trained |
|
|
| | path | reason | |
| |---|---| |
| | `models/gemma-4-31b/protect/run-1` | cancelled at Josh's request on Oct 16, 2026, 3:11 PM PT, when no further protect runs were wanted; its RunPod attempts had run out of GPU memory and then found no free 8-GPU machine | |
|
|
| ### Redactions in uploaded checkpoints |
|
|
| None: no secret was found in any uploaded checkpoint file (README.md and .gitattributes of the source repos are not copied). |
| <!-- STATUS:END --> |
|
|
