Initialize experiment namespace with training-budget provenance

NCM-BUDGET-AUDIT.md

@@ -0,0 +1,81 @@
+# NCM training-budget audit — 2026-09-16
+
+## Source revisions
+
+- `HazyResearch/ncm`, `neurips-final-run`: `5f3e5641004de0f52e583b8841d79a9d538b3b63`.
+- `HazyResearch/ncm`, `final`: `ab28dd9a9f67a729112feaf94147b26a5822e6a1`.
+- HF `hazyresearch/ncm-neurips-owt`: `9570350baa885bd94abdaacff6c21675252d9c55`.
+
+These are different configurations and historical runs, not a single definitive
+NCM training budget. Matching the exact paper checkpoint requires its resolved
+configuration and optimizer-step history, including world-size changes.
+
+## What the branches establish
+
+The common `config/experiments/neurips/{vqvae-owt-gpt2-final,
+ncp-owt-gpt2-final-small,ncp-owt-gpt2-final}.yaml` files specify **100 epochs**.
+The generator configs actually have `context_length=256`, `prefix_len=128`,
+and tokenizer context 128; some comments still say prefix 64 and are stale.
+The models use 14 levels starting at 32, 50,304 codes/level, and 10% uniform
+code-replacement corruption. This is not the current ctx256, prefix-free,
+12-level/16K-codebook reproduction.
+
+`utils/arg_utils.py:parse_args` derives updates as:
+
+```
+ceil(n_epochs * corpus_token_count /
+     (batch_per_gpu * model.context_length * accumulation * world_size))
+```
+
+Therefore the 100-epoch YAML alone does not establish either executed steps or
+an absolute token count without the actual dataset and runtime overrides.
+The `final` branch also contains different context-specific recipes, e.g.
+`config/owt-ncp-ctx128/ncp.yaml` specifies 50 epochs. Do not combine their
+architecture, schedules, and checkpoint counts into one purported run.
+
+## What released checkpoint logs establish
+
+HF paths `vqvae-owt/stdout.txt` and `ncp-owt-gpt2/stdout.txt` were inspected.
+
+| Released artifact | Evidence | Interpretation |
+|---|---|---|
+| `vqvae-owt/checkpoint-iter-103152.pt` | Resolved log: 5 epochs, batch 512, accumulation 1, context 64; later resume warning explicitly identifies saved world size 4 | 103,152 × 512 × 4 × 64 = **13,520,338,944** raw positions for this checkpoint trajectory |
+| `ncp-owt-gpt2/checkpoint-iter-80000.pt` | Batch 96, accumulation 1, context 128, text prefix 64; log contains both 8- and 4-GPU resumes | Step 80k is not 50 completed epochs. At fixed 4/8 GPUs, 80k updates correspond to **3.932/7.864B** raw positions, half of which are target positions. Exact cumulative accounting requires resolving the resume timeline |
+
+The generator's resolved planned totals include 1,375,350 and 2,750,699 updates
+with `n_epochs=50`. The released artifact is only step 80,000. The logs include
+restarts/resumes; discarded computation is separate from the retained training
+trajectory. Do not report the configured endpoint as completed training.
+
+The released tokenizer count implies about 2.704B corpus tokens, consistent
+with the current local uint16 binary (2,704,046,552 tokens; rounding to whole
+updates explains the small difference). This is a size consistency check,
+**not a dataset-content/hash identity claim**.
+
+Repository evaluation records additionally reference checkpoints up to 140k
+(`results/gen_ppl_ncp_owt_gpt2_full.json`) and a run named `300ep` through 105k
+(`results/gen_ppl_ncp_rope_qknorm_corrupt_300ep.json`). Names/configured epochs
+are not proof of 300 completed epochs or of their exact token budgets.
+
+## Current MsTok accounting
+
+Batch 32/GPU × 12 accumulation × 8 GPUs × 256 = **786,432 positions/update**.
+The current binary is 5,408,093,104 bytes of uint16 = 2,704,046,552 tokens.
+
+| Updates per component | Repeated raw token positions | Nominal corpus passes |
+|---|---:|---:|
+| 10,000 | 7.864B | 2.91 |
+| 17,192 | 13.520B | 5.00 |
+| 25,000 | 19.661B | 7.27 |
+| 34,384 | 27.041B | 10.00 |
+
+The legacy 103,152-update tokenizer and our 17,192-update component therefore
+have the **same raw-position budget**, despite a sixfold step-count difference.
+At current batch/context, 131B positions require **166,576 updates**. At the
+observed ~0.258 joint updates/second, that is roughly 7.5 days of training alone
+on one eight-H100 node, excluding evaluations and overhead. More data is not
+free and a multi-candidate sweep at that budget will not fit a ten-day deadline.
+
+Before revising final-run lengths, identify which NCM checkpoint/result is the
+comparison target. A short-budget tuning experiment remains useful, but is not
+the same claim as matching a long-budget baseline.

README.md

@@ -0,0 +1,23 @@
+---
+language:
+- en
+tags:
+- text-generation
+- research
+---
+# ICLR debug experiments
+
+Public artifacts for the `iclr-debug` branch of HazyResearch/MsTok.
+W&B project: `mstok/iclr-debug`.
+
+This repository currently establishes the experiment namespace and contains a
+training-budget audit. No new model weights or final benchmark claims are
+implied. Future selected checkpoints and evaluations will be organized by
+unique run ID and optimizer step.
+
+Training budgets must report raw token positions per component, dataset size,
+global batch, optimizer updates, and LR horizons. Generation reports must label
+supplied-level-zero versus unconditional sampling and record the reference
+model precision, entropy, scored-token counts, and per-seed results.
+
+See `NCM-BUDGET-AUDIT.md` for historical budget provenance and limitations.