One merged log per batch size; segment logs -> extras/segment_logs

813e88b verified 18 days ago

2.34 kB

# 390M batch-size sweep

One folder per batch size (tokens per optimizer step). Each holds `ckpts/` (all checkpoints of that
batch size's run, merged across resume segments) and `csv/` with **one merged training log**.
Peak LR is 1e-3 unless noted.

| Folder | Tokens/step | Old name | Merged log | Steps logged | Ckpt steps |
|---|---|---|---|---|---|
| `bs_0.26M` | 262,144 | "200k" | `390M_bs_0.26M_merged.csv` | 0 – 53405 | 6611 – 53406 |
| `bs_0.52M` | 524,288 | 0.52M | `390M_1e3_heldout.csv` (single run) | 0 – 26702 | in [`jayasriii/390M_10B/390M_1e3_heldout`](https://huggingface.co/jayasriii/390M_10B/tree/main/390M_1e3_heldout) |
| `bs_0.79M` | 786,432 | "700k" | `390M_bs_0.79M_merged.csv` | 0 – 12152 | 1000 – 12153 |
| `bs_1M` | 1,048,576 | 1M | `390M_bs_1M_merged.csv` | 0 – 8699 | 1000 – 8700 |
| `bs_2M` | 2,097,152 | 2M | `390M_bs_2M_merged.csv` | 0 – 4607 (gaps, see below) | 1000 – 4608 |
| `bs_2M/sqrtlr_2e-3` | 2,097,152 | 2M sqrt-LR | `390M_2M_sqrtlr.csv` (single run, peak LR 2e-3) | 0 – 1257 | 1000, 1258 |
| `bs_4M` | 4,194,304 | 4M | `390M_bs_4M_merged.csv` | 0 – 2022 | 1 – 1277 |

## How the logs were merged
Segments are concatenated in run order. A resume restarts from a checkpoint at step S, so any
earlier-logged row at step >= S belongs to an abandoned attempt and is replaced by the resumed run's row
(this also covers the in-file restart at 13304 in `390M_200k_39k.csv`).

- `bs_2M`: step 3004 was never logged (resume boundary), and steps 3294–3531 are lost — that part of
  `390M_2Mbs_main.csv` was overwritten with null bytes.
- `bs_4M`: the merged log is `390M_4M_2k.csv`, the one continuous log to step 2022. `390M_4M_1275.csv`
  re-ran 1214–1276 from the step-1214 ckpt (|Δ train loss| ≤ 1e-3) to produce ckpt 1277.

## extras/
- `segment_logs/<bs>/` — the original per-segment logs the merged logs were built from, plus the old
  hand-stitched `*_full_stitched.csv` files (superseded: the 0.26M one used the abandoned pass for
  13304–19339, and the 1M one is missing steps 5616–5617).
- `390M_2M_flux/` — single 2M ckpt (step 3465) with no log; not part of the `bs_2M` chain.
- `200kbs.csv` — older partial stitch of the 0.26M log.
- `loss.csv` — a separate 1M run (9448 steps, different val set), no ckpts here.
- `*.py` — training / check scripts.

	# 390M batch-size sweep

	One folder per batch size (tokens per optimizer step). Each holds `ckpts/` (all checkpoints of that
	batch size's run, merged across resume segments) and `csv/` with one merged training log.
	Peak LR is 1e-3 unless noted.

	\| Folder \| Tokens/step \| Old name \| Merged log \| Steps logged \| Ckpt steps \|
	\|---\|---\|---\|---\|---\|---\|
	\| `bs_0.26M` \| 262,144 \| "200k" \| `390M_bs_0.26M_merged.csv` \| 0 – 53405 \| 6611 – 53406 \|
	\| `bs_0.52M` \| 524,288 \| 0.52M \| `390M_1e3_heldout.csv` (single run) \| 0 – 26702 \| in [`jayasriii/390M_10B/390M_1e3_heldout`](https://huggingface.co/jayasriii/390M_10B/tree/main/390M_1e3_heldout) \|
	\| `bs_0.79M` \| 786,432 \| "700k" \| `390M_bs_0.79M_merged.csv` \| 0 – 12152 \| 1000 – 12153 \|
	\| `bs_1M` \| 1,048,576 \| 1M \| `390M_bs_1M_merged.csv` \| 0 – 8699 \| 1000 – 8700 \|
	\| `bs_2M` \| 2,097,152 \| 2M \| `390M_bs_2M_merged.csv` \| 0 – 4607 (gaps, see below) \| 1000 – 4608 \|
	\| `bs_2M/sqrtlr_2e-3` \| 2,097,152 \| 2M sqrt-LR \| `390M_2M_sqrtlr.csv` (single run, peak LR 2e-3) \| 0 – 1257 \| 1000, 1258 \|
	\| `bs_4M` \| 4,194,304 \| 4M \| `390M_bs_4M_merged.csv` \| 0 – 2022 \| 1 – 1277 \|

	## How the logs were merged
	Segments are concatenated in run order. A resume restarts from a checkpoint at step S, so any
	earlier-logged row at step >= S belongs to an abandoned attempt and is replaced by the resumed run's row
	(this also covers the in-file restart at 13304 in `390M_200k_39k.csv`).

	- `bs_2M`: step 3004 was never logged (resume boundary), and steps 3294–3531 are lost — that part of
	`390M_2Mbs_main.csv` was overwritten with null bytes.
	- `bs_4M`: the merged log is `390M_4M_2k.csv`, the one continuous log to step 2022. `390M_4M_1275.csv`
	re-ran 1214–1276 from the step-1214 ckpt (\|Δ train loss\| ≤ 1e-3) to produce ckpt 1277.

	## extras/
	- `segment_logs/<bs>/` — the original per-segment logs the merged logs were built from, plus the old
	hand-stitched `*_full_stitched.csv` files (superseded: the 0.26M one used the abandoned pass for
	13304–19339, and the 1M one is missing steps 5616–5617).
	- `390M_2M_flux/` — single 2M ckpt (step 3465) with no log; not part of the `bs_2M` chain.
	- `200kbs.csv` — older partial stitch of the 0.26M log.
	- `loss.csv` — a separate 1M run (9448 steps, different val set), no ckpts here.
	- `*.py` — training / check scripts.