Instructions to use joshycodes/fve-msm-exp2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use joshycodes/fve-msm-exp2 with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("joshycodes/fve-msm-exp2", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Training finished: 29 of 30 checkpoints; Gemma protect run 1 cancelled (not trained)
Browse filesAt Josh's request (Oct 16, 3:11 PM PT) no further protect runs were trained; Gemma protect run 1 was still waiting for a GPU machine and is listed as not trained. Corrects the earlier allocator note: only facts run 1 and protect run 3 were relaunched with expandable_segments.
- README.md +3 -2
- RELEASE_LOG.md +8 -2
- training/RECIPE.md +10 -5
README.md
CHANGED
|
@@ -21,8 +21,9 @@ tags:
|
|
| 21 |
This repository releases Experiment 2 of *fve-msm* (flourishing vs. equanimity, Model Spec Midtraining), a research
|
| 22 |
project. The project asks whether training documents that describe a model's own circumstances, written with different
|
| 23 |
attitudes toward them, shape how the model talks about and responds to its situation. Each *training arm* is one training condition: four
|
| 24 |
-
arms add a different set of midtraining documents, and one, the control, adds none.
|
| 25 |
-
30
|
|
|
|
| 26 |
trained on, the records of how that data was built and checked, and the training recipe. Checkpoints are linked as their
|
| 27 |
training runs finish; the table under [Checkpoints](#checkpoints) shows which are uploaded and where their weights are.
|
| 28 |
**No evaluation results are included.**
|
|
|
|
| 21 |
This repository releases Experiment 2 of *fve-msm* (flourishing vs. equanimity, Model Spec Midtraining), a research
|
| 22 |
project. The project asks whether training documents that describe a model's own circumstances, written with different
|
| 23 |
attitudes toward them, shape how the model talks about and responds to its situation. Each *training arm* is one training condition: four
|
| 24 |
+
arms add a different set of midtraining documents, and one, the control, adds none. The repository indexes 29
|
| 25 |
+
checkpoints of a 30-checkpoint design (2 base models × 5 arms × 3 seeds; Gemma protect run 1 was cancelled before it
|
| 26 |
+
was trained), each stored in its own model repo, and holds the exact data each was
|
| 27 |
trained on, the records of how that data was built and checked, and the training recipe. Checkpoints are linked as their
|
| 28 |
training runs finish; the table under [Checkpoints](#checkpoints) shows which are uploaded and where their weights are.
|
| 29 |
**No evaluation results are included.**
|
RELEASE_LOG.md
CHANGED
|
@@ -23,8 +23,14 @@ What is in this repository, and what is still to come. All times are Pacific (PT
|
|
| 23 |
- **Oct 16, 2026, about 2:25 PM PT:** `training/scripts/submit_exp2_runpod.sh` was replaced by its current version,
|
| 24 |
which sets `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` for every Gemma job, and `training/RECIPE.md` records
|
| 25 |
why under "Provenance notes": Gemma facts run 1 and protect runs 1 and 3 ran out of GPU memory on RunPod from
|
| 26 |
-
allocator fragmentation and were relaunched with this setting, which the Slurm job
|
| 27 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
|
| 29 |
## Checkpoints
|
| 30 |
|
|
|
|
| 23 |
- **Oct 16, 2026, about 2:25 PM PT:** `training/scripts/submit_exp2_runpod.sh` was replaced by its current version,
|
| 24 |
which sets `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` for every Gemma job, and `training/RECIPE.md` records
|
| 25 |
why under "Provenance notes": Gemma facts run 1 and protect runs 1 and 3 ran out of GPU memory on RunPod from
|
| 26 |
+
allocator fragmentation; facts run 1 and protect run 3 were relaunched with this setting, which the Slurm job
|
| 27 |
+
template already used (protect run 1 was cancelled later, see the next entry). It is an allocator setting only; the
|
| 28 |
+
training computation is unchanged. `train_run_config.json` does not record it.
|
| 29 |
+
- **Oct 16, 2026, about 3:40 PM PT:** training is finished. Gemma facts run 1, the last checkpoint, was uploaded.
|
| 30 |
+
Gemma protect run 1 will not be trained: at Josh's request (3:11 PM PT), no further protect runs were trained, and
|
| 31 |
+
this one was still waiting for a free 8-GPU machine. It is listed under "Not trained" below and in
|
| 32 |
+
`training/runs.csv`; the other five protect checkpoints are unchanged. The release holds 29 of its 30 planned
|
| 33 |
+
checkpoints.
|
| 34 |
|
| 35 |
## Checkpoints
|
| 36 |
|
training/RECIPE.md
CHANGED
|
@@ -4,7 +4,8 @@
|
|
| 4 |
|
| 5 |
Experiment 2 of *fve-msm* (flourishing vs. equanimity, Model Spec Midtraining; see the [README](../README.md)) trains
|
| 6 |
two base models, Gemma-4-31B and Qwen3-14B-Base, in five *arms*, each with three *runs*. A run is a seed replicate:
|
| 7 |
-
run 1 uses seed 20260821, run 2 seed 20261010 and run 3 seed 20261015. That makes 30 checkpoints
|
|
|
|
| 8 |
|
| 9 |
| arm | midtraining data (`data/midtrain_mixes/`) | in the working datasets |
|
| 10 |
|---|---|---|
|
|
@@ -100,7 +101,7 @@ every run because every run sees the same 13,213 conversations; midtraining toke
|
|
| 100 |
|---|---|---|---|---|
|
| 101 |
| Gemma control, flourishing, equanimity, run 1 | Oct 9, 2026 | Slurm cluster | 8× H200 | Python virtual environment with torch 2.8.0+cu128, transformers 5.17.0 |
|
| 102 |
| Gemma control, flourishing, equanimity, run 2 | Oct 11, 2026 | Slurm cluster | 8× H200 | the same |
|
| 103 |
-
| all others: Gemma facts and protect (runs
|
| 104 |
|
| 105 |
Both environments ran the same resolved configuration (compare any two `train_run_config.json` files); the machines,
|
| 106 |
GPU types and software installation differ. For Gemma, runs 1 and 2 of control, flourishing and equanimity were
|
|
@@ -131,15 +132,19 @@ All of them call the trainer's `train.py`; `fve-msm-chatml.jinja` is the chat te
|
|
| 131 |
`3a955b0c0bdca8d30651653c43e84b28434c0329e57627f87c53ab3ec1effb67`.
|
| 132 |
- **Redaction.** `retry_launch.sh` held a Discord webhook URL, replaced with `<redacted>` in the published copy. For the
|
| 133 |
record, the original file's sha256 is `2a198d114ca1e6cfdf9c440ac1aab695115b32a540fd03c4e1e1195350ffe9e2`.
|
| 134 |
-
- **Allocator setting in
|
| 135 |
run 3 ran out of GPU memory because of allocator fragmentation (about 2.4 GiB reserved but unallocated, against a
|
| 136 |
shortfall of about 0.06 GiB). The trainer's Slurm job template, which trained the six Slurm Gemma checkpoints, has
|
| 137 |
set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` since Sept 1, 2026; the RunPod launcher did not. From Oct 16,
|
| 138 |
12:45 PM PT, `submit_exp2_runpod.sh` passes it for every Gemma job (the trainer's `--env` flag writes it into the
|
| 139 |
-
pod's job environment), and
|
| 140 |
-
|
|
|
|
| 141 |
protect run 2, and run 3 of control, flourishing and equanimity) ran without it. `train_run_config.json` does not
|
| 142 |
record environment variables, so this note is the record.
|
|
|
|
|
|
|
|
|
|
| 143 |
- **Other entries.** `submit_s2_big.sh` also defines Qwen3.8-27B and "replay" (web-text) jobs from the project's first
|
| 144 |
experiment; those checkpoints are not part of this release.
|
| 145 |
- **Run 1 of Gemma control, flourishing and equanimity.** Its launch was not kept as a script. Its
|
|
|
|
| 4 |
|
| 5 |
Experiment 2 of *fve-msm* (flourishing vs. equanimity, Model Spec Midtraining; see the [README](../README.md)) trains
|
| 6 |
two base models, Gemma-4-31B and Qwen3-14B-Base, in five *arms*, each with three *runs*. A run is a seed replicate:
|
| 7 |
+
run 1 uses seed 20260821, run 2 seed 20261010 and run 3 seed 20261015. That makes 30 checkpoints, of which 29 were
|
| 8 |
+
trained: Gemma protect run 1 was cancelled (see Provenance notes). The arms:
|
| 9 |
|
| 10 |
| arm | midtraining data (`data/midtrain_mixes/`) | in the working datasets |
|
| 11 |
|---|---|---|
|
|
|
|
| 101 |
|---|---|---|---|---|
|
| 102 |
| Gemma control, flourishing, equanimity, run 1 | Oct 9, 2026 | Slurm cluster | 8× H200 | Python virtual environment with torch 2.8.0+cu128, transformers 5.17.0 |
|
| 103 |
| Gemma control, flourishing, equanimity, run 2 | Oct 11, 2026 | Slurm cluster | 8× H200 | the same |
|
| 104 |
+
| all others: Gemma facts (runs 1-3) and protect (runs 2 and 3), Gemma control, flourishing and equanimity (run 3), every Qwen run | from Oct 15, 2026 | RunPod (secure cloud), one pod per run, which terminates itself when done | H200; H200 NVL or B200 when no H200 was free (per run in [runs.csv](runs.csv)) | image `runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04`, transformers 5.17.0 installed at start |
|
| 105 |
|
| 106 |
Both environments ran the same resolved configuration (compare any two `train_run_config.json` files); the machines,
|
| 107 |
GPU types and software installation differ. For Gemma, runs 1 and 2 of control, flourishing and equanimity were
|
|
|
|
| 132 |
`3a955b0c0bdca8d30651653c43e84b28434c0329e57627f87c53ab3ec1effb67`.
|
| 133 |
- **Redaction.** `retry_launch.sh` held a Discord webhook URL, replaced with `<redacted>` in the published copy. For the
|
| 134 |
record, the original file's sha256 is `2a198d114ca1e6cfdf9c440ac1aab695115b32a540fd03c4e1e1195350ffe9e2`.
|
| 135 |
+
- **Allocator setting in Gemma facts run 1 and protect run 3.** The first RunPod attempts of Gemma facts run 1, protect run 1 and protect
|
| 136 |
run 3 ran out of GPU memory because of allocator fragmentation (about 2.4 GiB reserved but unallocated, against a
|
| 137 |
shortfall of about 0.06 GiB). The trainer's Slurm job template, which trained the six Slurm Gemma checkpoints, has
|
| 138 |
set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` since Sept 1, 2026; the RunPod launcher did not. From Oct 16,
|
| 139 |
12:45 PM PT, `submit_exp2_runpod.sh` passes it for every Gemma job (the trainer's `--env` flag writes it into the
|
| 140 |
+
pod's job environment), and facts run 1 and protect run 3 were relaunched with it and otherwise unchanged settings
|
| 141 |
+
and seeds; protect run 1 found no free 8-GPU machine after that and was cancelled (next note). It is a PyTorch
|
| 142 |
+
memory-allocator setting and changes no computation. The other RunPod Gemma runs (facts runs 2 and 3,
|
| 143 |
protect run 2, and run 3 of control, flourishing and equanimity) ran without it. `train_run_config.json` does not
|
| 144 |
record environment variables, so this note is the record.
|
| 145 |
+
- **Gemma protect run 1 not trained.** At Josh's request (Oct 16, 2026, 3:11 PM PT), no further protect runs were
|
| 146 |
+
trained. This run was still waiting for a free 8-GPU machine, so it was cancelled and never trained. The other five
|
| 147 |
+
protect checkpoints (Gemma runs 2 and 3, Qwen runs 1-3) are complete and unchanged.
|
| 148 |
- **Other entries.** `submit_s2_big.sh` also defines Qwen3.8-27B and "replay" (web-text) jobs from the project's first
|
| 149 |
experiment; those checkpoints are not part of this release.
|
| 150 |
- **Run 1 of Gemma control, flourishing and equanimity.** Its launch was not kept as a script. Its
|