joshycodes commited on
Commit
df9bf6f
·
verified ·
1 Parent(s): 1a48ed6

Record the allocator setting (expandable_segments) of three Gemma RunPod runs; current launch script

Browse files

Gemma facts run 1 and protect runs 1 and 3 OOMed on RunPod from allocator fragmentation and were relaunched with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (already set by the Slurm job template). Allocator setting only; no change to the computation.

RELEASE_LOG.md CHANGED
@@ -20,6 +20,11 @@ What is in this repository, and what is still to come. All times are Pacific (PT
20
  LFS sha256) to the file in its checkpoint's model repo, which keeps it. The repository was recreated without them
21
  (so its earlier commit history is gone) and every other file was restored unchanged. Each checkpoint folder now has
22
  `WEIGHTS.json`, which names the model repo, pins its revision and lists each weight file's size and sha256.
 
 
 
 
 
23
 
24
  ## Checkpoints
25
 
 
20
  LFS sha256) to the file in its checkpoint's model repo, which keeps it. The repository was recreated without them
21
  (so its earlier commit history is gone) and every other file was restored unchanged. Each checkpoint folder now has
22
  `WEIGHTS.json`, which names the model repo, pins its revision and lists each weight file's size and sha256.
23
+ - **Oct 16, 2026, about 2:25 PM PT:** `training/scripts/submit_exp2_runpod.sh` was replaced by its current version,
24
+ which sets `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` for every Gemma job, and `training/RECIPE.md` records
25
+ why under "Provenance notes": Gemma facts run 1 and protect runs 1 and 3 ran out of GPU memory on RunPod from
26
+ allocator fragmentation and were relaunched with this setting, which the Slurm job template already used. It is an
27
+ allocator setting only; the training computation is unchanged. `train_run_config.json` does not record it.
28
 
29
  ## Checkpoints
30
 
training/RECIPE.md CHANGED
@@ -122,13 +122,24 @@ All of them call the trainer's `train.py`; `fve-msm-chatml.jinja` is the chat te
122
  ### Provenance notes
123
 
124
  - **Script versions.** sha256 of the published copies: `submit_exp2_runpod.sh`
125
- `2c0876d63d87b3f4a358d110bb53d816d60ace42ef4321ef056d521fa62cd9f2` (the version of Oct 15, 4:31 PM PT, which added
126
- the run-3 jobs; the run-1 and run-2 job definitions are the same as in the earlier version of that day that launched
127
- them), `submit_s2_big.sh` `4de42a69b4923b1bbff10d6b5ff2106a6207390007033675a831f17415223abd`, `retry_launch.sh`
 
 
128
  `3e2b06e745e62575a06852e6e5ff1c984c240990ba7fffc5c2607d3d0bd5c983` and `fve-msm-chatml.jinja`
129
  `3a955b0c0bdca8d30651653c43e84b28434c0329e57627f87c53ab3ec1effb67`.
130
  - **Redaction.** `retry_launch.sh` held a Discord webhook URL, replaced with `<redacted>` in the published copy. For the
131
  record, the original file's sha256 is `2a198d114ca1e6cfdf9c440ac1aab695115b32a540fd03c4e1e1195350ffe9e2`.
 
 
 
 
 
 
 
 
 
132
  - **Other entries.** `submit_s2_big.sh` also defines Qwen3.8-27B and "replay" (web-text) jobs from the project's first
133
  experiment; those checkpoints are not part of this release.
134
  - **Run 1 of Gemma control, flourishing and equanimity.** Its launch was not kept as a script. Its
 
122
  ### Provenance notes
123
 
124
  - **Script versions.** sha256 of the published copies: `submit_exp2_runpod.sh`
125
+ `16c7237652a29c3cf69c21c8f22e0ee1b435bb0045a0c2f31c77660151f06a56` (the version of Oct 16, 12:45 PM PT, which added
126
+ the allocator setting below to the Gemma flags; apart from that one setting and its comment it is the version of
127
+ Oct 15, 4:31 PM PT, sha256 `2c0876d63d87b3f4a358d110bb53d816d60ace42ef4321ef056d521fa62cd9f2`, which added the run-3
128
+ jobs and launched the other RunPod runs; the run-1 and run-2 job definitions are the same as in the earlier version
129
+ of Oct 15 that launched them), `submit_s2_big.sh` `4de42a69b4923b1bbff10d6b5ff2106a6207390007033675a831f17415223abd`, `retry_launch.sh`
130
  `3e2b06e745e62575a06852e6e5ff1c984c240990ba7fffc5c2607d3d0bd5c983` and `fve-msm-chatml.jinja`
131
  `3a955b0c0bdca8d30651653c43e84b28434c0329e57627f87c53ab3ec1effb67`.
132
  - **Redaction.** `retry_launch.sh` held a Discord webhook URL, replaced with `<redacted>` in the published copy. For the
133
  record, the original file's sha256 is `2a198d114ca1e6cfdf9c440ac1aab695115b32a540fd03c4e1e1195350ffe9e2`.
134
+ - **Allocator setting in three Gemma runs.** The first RunPod attempts of Gemma facts run 1, protect run 1 and protect
135
+ run 3 ran out of GPU memory because of allocator fragmentation (about 2.4 GiB reserved but unallocated, against a
136
+ shortfall of about 0.06 GiB). The trainer's Slurm job template, which trained the six Slurm Gemma checkpoints, has
137
+ set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` since Sept 1, 2026; the RunPod launcher did not. From Oct 16,
138
+ 12:45 PM PT, `submit_exp2_runpod.sh` passes it for every Gemma job (the trainer's `--env` flag writes it into the
139
+ pod's job environment), and those three runs were relaunched with it and otherwise unchanged settings and seeds. It
140
+ is a PyTorch memory-allocator setting and changes no computation. The other RunPod Gemma runs (facts runs 2 and 3,
141
+ protect run 2, and run 3 of control, flourishing and equanimity) ran without it. `train_run_config.json` does not
142
+ record environment variables, so this note is the record.
143
  - **Other entries.** `submit_s2_big.sh` also defines Qwen3.8-27B and "replay" (web-text) jobs from the project's first
144
  experiment; those checkpoints are not part of this release.
145
  - **Run 1 of Gemma control, flourishing and equanimity.** Its launch was not kept as a script. Its
training/scripts/submit_exp2_runpod.sh CHANGED
@@ -17,9 +17,13 @@ G=(--image runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04
17
  --fsdp --no-intermediate-ckpts --no-affect-rollouts --backend runpod --no-follow
18
  --chat-data joshycodes/fve-msm-chat-sft-13k@7cba8972c8aefdc46552a9f73395d0de484ffeea
19
  --chat-template-file train/templates/fve-msm-chatml.jinja --no-push-intermediate)
 
 
 
20
  GEM=(--gpu-count 8 --disk 500 --model google/gemma-4-31B --transformers-spec transformers==5.17.0
21
  --fsdp-layer-cls Gemma4TextDecoderLayer --midtrain-lr 1e-5
22
- --micro-batch 1 --grad-accum 4 --chat-micro-batch 1 --chat-grad-accum 8)
 
23
  Q14=(--gpu-count 4 --disk 400 --model Qwen/Qwen3-14B-Base --transformers-spec transformers==5.17.0
24
  --fsdp-layer-cls Qwen3DecoderLayer --midtrain-lr 1e-5
25
  --micro-batch 2 --grad-accum 4 --chat-micro-batch 2 --chat-grad-accum 8)
 
17
  --fsdp --no-intermediate-ckpts --no-affect-rollouts --backend runpod --no-follow
18
  --chat-data joshycodes/fve-msm-chat-sft-13k@7cba8972c8aefdc46552a9f73395d0de484ffeea
19
  --chat-template-file train/templates/fve-msm-chatml.jinja --no-push-intermediate)
20
+ # expandable_segments (Josh ok, 12:45 PM PT Oct 16): the Slurm job template that trained the earlier Gemma runs exports
21
+ # it, but RunPod's remote_run.sh does not; 3 RunPod Gemma runs (facts-r1, protect-r1, protect-r3) then OOMed with
22
+ # ~2.4 GiB fragmented vs a ~0.06 GiB shortfall. Allocator setting only, same math; shipped to the pod via job.env.
23
  GEM=(--gpu-count 8 --disk 500 --model google/gemma-4-31B --transformers-spec transformers==5.17.0
24
  --fsdp-layer-cls Gemma4TextDecoderLayer --midtrain-lr 1e-5
25
+ --micro-batch 1 --grad-accum 4 --chat-micro-batch 1 --chat-grad-accum 8
26
+ --env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True)
27
  Q14=(--gpu-count 4 --disk 400 --model Qwen/Qwen3-14B-Base --transformers-spec transformers==5.17.0
28
  --fsdp-layer-cls Qwen3DecoderLayer --midtrain-lr 1e-5
29
  --micro-batch 2 --grad-accum 4 --chat-micro-batch 2 --chat-grad-accum 8)