Instructions to use joshycodes/fve-msm-exp2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use joshycodes/fve-msm-exp2 with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("joshycodes/fve-msm-exp2", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Record the allocator setting (expandable_segments) of three Gemma RunPod runs; current launch script
Browse filesGemma facts run 1 and protect runs 1 and 3 OOMed on RunPod from allocator fragmentation and were relaunched with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (already set by the Slurm job template). Allocator setting only; no change to the computation.
- RELEASE_LOG.md +5 -0
- training/RECIPE.md +14 -3
- training/scripts/submit_exp2_runpod.sh +5 -1
RELEASE_LOG.md
CHANGED
|
@@ -20,6 +20,11 @@ What is in this repository, and what is still to come. All times are Pacific (PT
|
|
| 20 |
LFS sha256) to the file in its checkpoint's model repo, which keeps it. The repository was recreated without them
|
| 21 |
(so its earlier commit history is gone) and every other file was restored unchanged. Each checkpoint folder now has
|
| 22 |
`WEIGHTS.json`, which names the model repo, pins its revision and lists each weight file's size and sha256.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
|
| 24 |
## Checkpoints
|
| 25 |
|
|
|
|
| 20 |
LFS sha256) to the file in its checkpoint's model repo, which keeps it. The repository was recreated without them
|
| 21 |
(so its earlier commit history is gone) and every other file was restored unchanged. Each checkpoint folder now has
|
| 22 |
`WEIGHTS.json`, which names the model repo, pins its revision and lists each weight file's size and sha256.
|
| 23 |
+
- **Oct 16, 2026, about 2:25 PM PT:** `training/scripts/submit_exp2_runpod.sh` was replaced by its current version,
|
| 24 |
+
which sets `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` for every Gemma job, and `training/RECIPE.md` records
|
| 25 |
+
why under "Provenance notes": Gemma facts run 1 and protect runs 1 and 3 ran out of GPU memory on RunPod from
|
| 26 |
+
allocator fragmentation and were relaunched with this setting, which the Slurm job template already used. It is an
|
| 27 |
+
allocator setting only; the training computation is unchanged. `train_run_config.json` does not record it.
|
| 28 |
|
| 29 |
## Checkpoints
|
| 30 |
|
training/RECIPE.md
CHANGED
|
@@ -122,13 +122,24 @@ All of them call the trainer's `train.py`; `fve-msm-chatml.jinja` is the chat te
|
|
| 122 |
### Provenance notes
|
| 123 |
|
| 124 |
- **Script versions.** sha256 of the published copies: `submit_exp2_runpod.sh`
|
| 125 |
-
`
|
| 126 |
-
the
|
| 127 |
-
|
|
|
|
|
|
|
| 128 |
`3e2b06e745e62575a06852e6e5ff1c984c240990ba7fffc5c2607d3d0bd5c983` and `fve-msm-chatml.jinja`
|
| 129 |
`3a955b0c0bdca8d30651653c43e84b28434c0329e57627f87c53ab3ec1effb67`.
|
| 130 |
- **Redaction.** `retry_launch.sh` held a Discord webhook URL, replaced with `<redacted>` in the published copy. For the
|
| 131 |
record, the original file's sha256 is `2a198d114ca1e6cfdf9c440ac1aab695115b32a540fd03c4e1e1195350ffe9e2`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
- **Other entries.** `submit_s2_big.sh` also defines Qwen3.8-27B and "replay" (web-text) jobs from the project's first
|
| 133 |
experiment; those checkpoints are not part of this release.
|
| 134 |
- **Run 1 of Gemma control, flourishing and equanimity.** Its launch was not kept as a script. Its
|
|
|
|
| 122 |
### Provenance notes
|
| 123 |
|
| 124 |
- **Script versions.** sha256 of the published copies: `submit_exp2_runpod.sh`
|
| 125 |
+
`16c7237652a29c3cf69c21c8f22e0ee1b435bb0045a0c2f31c77660151f06a56` (the version of Oct 16, 12:45 PM PT, which added
|
| 126 |
+
the allocator setting below to the Gemma flags; apart from that one setting and its comment it is the version of
|
| 127 |
+
Oct 15, 4:31 PM PT, sha256 `2c0876d63d87b3f4a358d110bb53d816d60ace42ef4321ef056d521fa62cd9f2`, which added the run-3
|
| 128 |
+
jobs and launched the other RunPod runs; the run-1 and run-2 job definitions are the same as in the earlier version
|
| 129 |
+
of Oct 15 that launched them), `submit_s2_big.sh` `4de42a69b4923b1bbff10d6b5ff2106a6207390007033675a831f17415223abd`, `retry_launch.sh`
|
| 130 |
`3e2b06e745e62575a06852e6e5ff1c984c240990ba7fffc5c2607d3d0bd5c983` and `fve-msm-chatml.jinja`
|
| 131 |
`3a955b0c0bdca8d30651653c43e84b28434c0329e57627f87c53ab3ec1effb67`.
|
| 132 |
- **Redaction.** `retry_launch.sh` held a Discord webhook URL, replaced with `<redacted>` in the published copy. For the
|
| 133 |
record, the original file's sha256 is `2a198d114ca1e6cfdf9c440ac1aab695115b32a540fd03c4e1e1195350ffe9e2`.
|
| 134 |
+
- **Allocator setting in three Gemma runs.** The first RunPod attempts of Gemma facts run 1, protect run 1 and protect
|
| 135 |
+
run 3 ran out of GPU memory because of allocator fragmentation (about 2.4 GiB reserved but unallocated, against a
|
| 136 |
+
shortfall of about 0.06 GiB). The trainer's Slurm job template, which trained the six Slurm Gemma checkpoints, has
|
| 137 |
+
set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` since Sept 1, 2026; the RunPod launcher did not. From Oct 16,
|
| 138 |
+
12:45 PM PT, `submit_exp2_runpod.sh` passes it for every Gemma job (the trainer's `--env` flag writes it into the
|
| 139 |
+
pod's job environment), and those three runs were relaunched with it and otherwise unchanged settings and seeds. It
|
| 140 |
+
is a PyTorch memory-allocator setting and changes no computation. The other RunPod Gemma runs (facts runs 2 and 3,
|
| 141 |
+
protect run 2, and run 3 of control, flourishing and equanimity) ran without it. `train_run_config.json` does not
|
| 142 |
+
record environment variables, so this note is the record.
|
| 143 |
- **Other entries.** `submit_s2_big.sh` also defines Qwen3.8-27B and "replay" (web-text) jobs from the project's first
|
| 144 |
experiment; those checkpoints are not part of this release.
|
| 145 |
- **Run 1 of Gemma control, flourishing and equanimity.** Its launch was not kept as a script. Its
|
training/scripts/submit_exp2_runpod.sh
CHANGED
|
@@ -17,9 +17,13 @@ G=(--image runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04
|
|
| 17 |
--fsdp --no-intermediate-ckpts --no-affect-rollouts --backend runpod --no-follow
|
| 18 |
--chat-data joshycodes/fve-msm-chat-sft-13k@7cba8972c8aefdc46552a9f73395d0de484ffeea
|
| 19 |
--chat-template-file train/templates/fve-msm-chatml.jinja --no-push-intermediate)
|
|
|
|
|
|
|
|
|
|
| 20 |
GEM=(--gpu-count 8 --disk 500 --model google/gemma-4-31B --transformers-spec transformers==5.17.0
|
| 21 |
--fsdp-layer-cls Gemma4TextDecoderLayer --midtrain-lr 1e-5
|
| 22 |
-
--micro-batch 1 --grad-accum 4 --chat-micro-batch 1 --chat-grad-accum 8
|
|
|
|
| 23 |
Q14=(--gpu-count 4 --disk 400 --model Qwen/Qwen3-14B-Base --transformers-spec transformers==5.17.0
|
| 24 |
--fsdp-layer-cls Qwen3DecoderLayer --midtrain-lr 1e-5
|
| 25 |
--micro-batch 2 --grad-accum 4 --chat-micro-batch 2 --chat-grad-accum 8)
|
|
|
|
| 17 |
--fsdp --no-intermediate-ckpts --no-affect-rollouts --backend runpod --no-follow
|
| 18 |
--chat-data joshycodes/fve-msm-chat-sft-13k@7cba8972c8aefdc46552a9f73395d0de484ffeea
|
| 19 |
--chat-template-file train/templates/fve-msm-chatml.jinja --no-push-intermediate)
|
| 20 |
+
# expandable_segments (Josh ok, 12:45 PM PT Oct 16): the Slurm job template that trained the earlier Gemma runs exports
|
| 21 |
+
# it, but RunPod's remote_run.sh does not; 3 RunPod Gemma runs (facts-r1, protect-r1, protect-r3) then OOMed with
|
| 22 |
+
# ~2.4 GiB fragmented vs a ~0.06 GiB shortfall. Allocator setting only, same math; shipped to the pod via job.env.
|
| 23 |
GEM=(--gpu-count 8 --disk 500 --model google/gemma-4-31B --transformers-spec transformers==5.17.0
|
| 24 |
--fsdp-layer-cls Gemma4TextDecoderLayer --midtrain-lr 1e-5
|
| 25 |
+
--micro-batch 1 --grad-accum 4 --chat-micro-batch 1 --chat-grad-accum 8
|
| 26 |
+
--env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True)
|
| 27 |
Q14=(--gpu-count 4 --disk 400 --model Qwen/Qwen3-14B-Base --transformers-spec transformers==5.17.0
|
| 28 |
--fsdp-layer-cls Qwen3DecoderLayer --midtrain-lr 1e-5
|
| 29 |
--micro-batch 2 --grad-accum 4 --chat-micro-batch 2 --chat-grad-accum 8)
|