joshycodes commited on
Commit
acb2a1b
·
verified ·
1 Parent(s): 09b9a46

Training finished: 29 of 30 checkpoints; Gemma protect run 1 cancelled (not trained)

Browse files

At Josh's request (Oct 16, 3:11 PM PT) no further protect runs were trained; Gemma protect run 1 was still waiting for a GPU machine and is listed as not trained. Corrects the earlier allocator note: only facts run 1 and protect run 3 were relaunched with expandable_segments.

Files changed (3) hide show
  1. README.md +3 -2
  2. RELEASE_LOG.md +8 -2
  3. training/RECIPE.md +10 -5
README.md CHANGED
@@ -21,8 +21,9 @@ tags:
21
  This repository releases Experiment 2 of *fve-msm* (flourishing vs. equanimity, Model Spec Midtraining), a research
22
  project. The project asks whether training documents that describe a model's own circumstances, written with different
23
  attitudes toward them, shape how the model talks about and responds to its situation. Each *training arm* is one training condition: four
24
- arms add a different set of midtraining documents, and one, the control, adds none. When complete, the repository indexes
25
- 30 checkpoints (2 base models × 5 arms × 3 seeds), each stored in its own model repo, and holds the exact data each was
 
26
  trained on, the records of how that data was built and checked, and the training recipe. Checkpoints are linked as their
27
  training runs finish; the table under [Checkpoints](#checkpoints) shows which are uploaded and where their weights are.
28
  **No evaluation results are included.**
 
21
  This repository releases Experiment 2 of *fve-msm* (flourishing vs. equanimity, Model Spec Midtraining), a research
22
  project. The project asks whether training documents that describe a model's own circumstances, written with different
23
  attitudes toward them, shape how the model talks about and responds to its situation. Each *training arm* is one training condition: four
24
+ arms add a different set of midtraining documents, and one, the control, adds none. The repository indexes 29
25
+ checkpoints of a 30-checkpoint design (2 base models × 5 arms × 3 seeds; Gemma protect run 1 was cancelled before it
26
+ was trained), each stored in its own model repo, and holds the exact data each was
27
  trained on, the records of how that data was built and checked, and the training recipe. Checkpoints are linked as their
28
  training runs finish; the table under [Checkpoints](#checkpoints) shows which are uploaded and where their weights are.
29
  **No evaluation results are included.**
RELEASE_LOG.md CHANGED
@@ -23,8 +23,14 @@ What is in this repository, and what is still to come. All times are Pacific (PT
23
  - **Oct 16, 2026, about 2:25 PM PT:** `training/scripts/submit_exp2_runpod.sh` was replaced by its current version,
24
  which sets `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` for every Gemma job, and `training/RECIPE.md` records
25
  why under "Provenance notes": Gemma facts run 1 and protect runs 1 and 3 ran out of GPU memory on RunPod from
26
- allocator fragmentation and were relaunched with this setting, which the Slurm job template already used. It is an
27
- allocator setting only; the training computation is unchanged. `train_run_config.json` does not record it.
 
 
 
 
 
 
28
 
29
  ## Checkpoints
30
 
 
23
  - **Oct 16, 2026, about 2:25 PM PT:** `training/scripts/submit_exp2_runpod.sh` was replaced by its current version,
24
  which sets `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` for every Gemma job, and `training/RECIPE.md` records
25
  why under "Provenance notes": Gemma facts run 1 and protect runs 1 and 3 ran out of GPU memory on RunPod from
26
+ allocator fragmentation; facts run 1 and protect run 3 were relaunched with this setting, which the Slurm job
27
+ template already used (protect run 1 was cancelled later, see the next entry). It is an allocator setting only; the
28
+ training computation is unchanged. `train_run_config.json` does not record it.
29
+ - **Oct 16, 2026, about 3:40 PM PT:** training is finished. Gemma facts run 1, the last checkpoint, was uploaded.
30
+ Gemma protect run 1 will not be trained: at Josh's request (3:11 PM PT), no further protect runs were trained, and
31
+ this one was still waiting for a free 8-GPU machine. It is listed under "Not trained" below and in
32
+ `training/runs.csv`; the other five protect checkpoints are unchanged. The release holds 29 of its 30 planned
33
+ checkpoints.
34
 
35
  ## Checkpoints
36
 
training/RECIPE.md CHANGED
@@ -4,7 +4,8 @@
4
 
5
  Experiment 2 of *fve-msm* (flourishing vs. equanimity, Model Spec Midtraining; see the [README](../README.md)) trains
6
  two base models, Gemma-4-31B and Qwen3-14B-Base, in five *arms*, each with three *runs*. A run is a seed replicate:
7
- run 1 uses seed 20260821, run 2 seed 20261010 and run 3 seed 20261015. That makes 30 checkpoints. The arms:
 
8
 
9
  | arm | midtraining data (`data/midtrain_mixes/`) | in the working datasets |
10
  |---|---|---|
@@ -100,7 +101,7 @@ every run because every run sees the same 13,213 conversations; midtraining toke
100
  |---|---|---|---|---|
101
  | Gemma control, flourishing, equanimity, run 1 | Oct 9, 2026 | Slurm cluster | 8× H200 | Python virtual environment with torch 2.8.0+cu128, transformers 5.17.0 |
102
  | Gemma control, flourishing, equanimity, run 2 | Oct 11, 2026 | Slurm cluster | 8× H200 | the same |
103
- | all others: Gemma facts and protect (runs 1-3), Gemma control, flourishing and equanimity (run 3), every Qwen run | from Oct 15, 2026 | RunPod (secure cloud), one pod per run, which terminates itself when done | H200; H200 NVL or B200 when no H200 was free (per run in [runs.csv](runs.csv)) | image `runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04`, transformers 5.17.0 installed at start |
104
 
105
  Both environments ran the same resolved configuration (compare any two `train_run_config.json` files); the machines,
106
  GPU types and software installation differ. For Gemma, runs 1 and 2 of control, flourishing and equanimity were
@@ -131,15 +132,19 @@ All of them call the trainer's `train.py`; `fve-msm-chatml.jinja` is the chat te
131
  `3a955b0c0bdca8d30651653c43e84b28434c0329e57627f87c53ab3ec1effb67`.
132
  - **Redaction.** `retry_launch.sh` held a Discord webhook URL, replaced with `<redacted>` in the published copy. For the
133
  record, the original file's sha256 is `2a198d114ca1e6cfdf9c440ac1aab695115b32a540fd03c4e1e1195350ffe9e2`.
134
- - **Allocator setting in three Gemma runs.** The first RunPod attempts of Gemma facts run 1, protect run 1 and protect
135
  run 3 ran out of GPU memory because of allocator fragmentation (about 2.4 GiB reserved but unallocated, against a
136
  shortfall of about 0.06 GiB). The trainer's Slurm job template, which trained the six Slurm Gemma checkpoints, has
137
  set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` since Sept 1, 2026; the RunPod launcher did not. From Oct 16,
138
  12:45 PM PT, `submit_exp2_runpod.sh` passes it for every Gemma job (the trainer's `--env` flag writes it into the
139
- pod's job environment), and those three runs were relaunched with it and otherwise unchanged settings and seeds. It
140
- is a PyTorch memory-allocator setting and changes no computation. The other RunPod Gemma runs (facts runs 2 and 3,
 
141
  protect run 2, and run 3 of control, flourishing and equanimity) ran without it. `train_run_config.json` does not
142
  record environment variables, so this note is the record.
 
 
 
143
  - **Other entries.** `submit_s2_big.sh` also defines Qwen3.8-27B and "replay" (web-text) jobs from the project's first
144
  experiment; those checkpoints are not part of this release.
145
  - **Run 1 of Gemma control, flourishing and equanimity.** Its launch was not kept as a script. Its
 
4
 
5
  Experiment 2 of *fve-msm* (flourishing vs. equanimity, Model Spec Midtraining; see the [README](../README.md)) trains
6
  two base models, Gemma-4-31B and Qwen3-14B-Base, in five *arms*, each with three *runs*. A run is a seed replicate:
7
+ run 1 uses seed 20260821, run 2 seed 20261010 and run 3 seed 20261015. That makes 30 checkpoints, of which 29 were
8
+ trained: Gemma protect run 1 was cancelled (see Provenance notes). The arms:
9
 
10
  | arm | midtraining data (`data/midtrain_mixes/`) | in the working datasets |
11
  |---|---|---|
 
101
  |---|---|---|---|---|
102
  | Gemma control, flourishing, equanimity, run 1 | Oct 9, 2026 | Slurm cluster | 8× H200 | Python virtual environment with torch 2.8.0+cu128, transformers 5.17.0 |
103
  | Gemma control, flourishing, equanimity, run 2 | Oct 11, 2026 | Slurm cluster | 8× H200 | the same |
104
+ | all others: Gemma facts (runs 1-3) and protect (runs 2 and 3), Gemma control, flourishing and equanimity (run 3), every Qwen run | from Oct 15, 2026 | RunPod (secure cloud), one pod per run, which terminates itself when done | H200; H200 NVL or B200 when no H200 was free (per run in [runs.csv](runs.csv)) | image `runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04`, transformers 5.17.0 installed at start |
105
 
106
  Both environments ran the same resolved configuration (compare any two `train_run_config.json` files); the machines,
107
  GPU types and software installation differ. For Gemma, runs 1 and 2 of control, flourishing and equanimity were
 
132
  `3a955b0c0bdca8d30651653c43e84b28434c0329e57627f87c53ab3ec1effb67`.
133
  - **Redaction.** `retry_launch.sh` held a Discord webhook URL, replaced with `<redacted>` in the published copy. For the
134
  record, the original file's sha256 is `2a198d114ca1e6cfdf9c440ac1aab695115b32a540fd03c4e1e1195350ffe9e2`.
135
+ - **Allocator setting in Gemma facts run 1 and protect run 3.** The first RunPod attempts of Gemma facts run 1, protect run 1 and protect
136
  run 3 ran out of GPU memory because of allocator fragmentation (about 2.4 GiB reserved but unallocated, against a
137
  shortfall of about 0.06 GiB). The trainer's Slurm job template, which trained the six Slurm Gemma checkpoints, has
138
  set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` since Sept 1, 2026; the RunPod launcher did not. From Oct 16,
139
  12:45 PM PT, `submit_exp2_runpod.sh` passes it for every Gemma job (the trainer's `--env` flag writes it into the
140
+ pod's job environment), and facts run 1 and protect run 3 were relaunched with it and otherwise unchanged settings
141
+ and seeds; protect run 1 found no free 8-GPU machine after that and was cancelled (next note). It is a PyTorch
142
+ memory-allocator setting and changes no computation. The other RunPod Gemma runs (facts runs 2 and 3,
143
  protect run 2, and run 3 of control, flourishing and equanimity) ran without it. `train_run_config.json` does not
144
  record environment variables, so this note is the record.
145
+ - **Gemma protect run 1 not trained.** At Josh's request (Oct 16, 2026, 3:11 PM PT), no further protect runs were
146
+ trained. This run was still waiting for a free 8-GPU machine, so it was cancelled and never trained. The other five
147
+ protect checkpoints (Gemma runs 2 and 3, Qwen runs 1-3) are complete and unchanged.
148
  - **Other entries.** `submit_s2_big.sh` also defines Qwen3.8-27B and "replay" (web-text) jobs from the project's first
149
  experiment; those checkpoints are not part of this release.
150
  - **Run 1 of Gemma control, flourishing and equanimity.** Its launch was not kept as a script. Its