390M_batchsize / README.md
jayasriii's picture
One merged log per batch size; segment logs -> extras/segment_logs
813e88b verified
|
Raw History Blame Contribute Delete
2.34 kB

390M batch-size sweep

One folder per batch size (tokens per optimizer step). Each holds ckpts/ (all checkpoints of that batch size's run, merged across resume segments) and csv/ with one merged training log. Peak LR is 1e-3 unless noted.

Folder Tokens/step Old name Merged log Steps logged Ckpt steps
bs_0.26M 262,144 "200k" 390M_bs_0.26M_merged.csv 0 – 53405 6611 – 53406
bs_0.52M 524,288 0.52M 390M_1e3_heldout.csv (single run) 0 – 26702 in jayasriii/390M_10B/390M_1e3_heldout
bs_0.79M 786,432 "700k" 390M_bs_0.79M_merged.csv 0 – 12152 1000 – 12153
bs_1M 1,048,576 1M 390M_bs_1M_merged.csv 0 – 8699 1000 – 8700
bs_2M 2,097,152 2M 390M_bs_2M_merged.csv 0 – 4607 (gaps, see below) 1000 – 4608
bs_2M/sqrtlr_2e-3 2,097,152 2M sqrt-LR 390M_2M_sqrtlr.csv (single run, peak LR 2e-3) 0 – 1257 1000, 1258
bs_4M 4,194,304 4M 390M_bs_4M_merged.csv 0 – 2022 1 – 1277

How the logs were merged

Segments are concatenated in run order. A resume restarts from a checkpoint at step S, so any earlier-logged row at step >= S belongs to an abandoned attempt and is replaced by the resumed run's row (this also covers the in-file restart at 13304 in 390M_200k_39k.csv).

  • bs_2M: step 3004 was never logged (resume boundary), and steps 3294–3531 are lost β€” that part of 390M_2Mbs_main.csv was overwritten with null bytes.
  • bs_4M: the merged log is 390M_4M_2k.csv, the one continuous log to step 2022. 390M_4M_1275.csv re-ran 1214–1276 from the step-1214 ckpt (|Ξ” train loss| ≀ 1e-3) to produce ckpt 1277.

extras/

  • segment_logs/<bs>/ β€” the original per-segment logs the merged logs were built from, plus the old hand-stitched *_full_stitched.csv files (superseded: the 0.26M one used the abandoned pass for 13304–19339, and the 1M one is missing steps 5616–5617).
  • 390M_2M_flux/ β€” single 2M ckpt (step 3465) with no log; not part of the bs_2M chain.
  • 200kbs.csv β€” older partial stitch of the 0.26M log.
  • loss.csv β€” a separate 1M run (9448 steps, different val set), no ckpts here.
  • *.py β€” training / check scripts.