|
Download README.md from lemoncmd/duo-scaling-2M-tiny: direct link, hf CLI and curl.
- Browser
- Download file 5.73 kB
-
https://huggingface.co/lemoncmd/duo-scaling-2M-tiny/resolve/main/README.md
- Command line
-
hf download hf://lemoncmd/duo-scaling-2M-tiny/README.md
-
curl -L -o README.md https://huggingface.co/lemoncmd/duo-scaling-2M-tiny/resolve/main/README.md
5.73 kB
| license: apache-2.0 | |
| library_name: pytorch | |
| datasets: | |
| - Skylion007/openwebtext | |
| tags: | |
| - discrete-diffusion | |
| - uniform-state-diffusion | |
| - duo | |
| - scaling-laws | |
| - checkpoints | |
| # Duo data-fraction scaling study: `tiny` models continued to 2M steps | |
| 2 uniform-state discrete diffusion language models (Duo, `duo_base`), identical | |
| except for the size of their training set: nested subsets of OpenWebText holding | |
| 10% and 100% of the training blocks. These runs **continue** `frac_0.10` and `frac_1.00` of | |
| [`duo-scaling-tiny`](https://huggingface.co/lemoncmd/duo-scaling-tiny) from step 1,000,000 | |
| toward 2,000,000 with only the step budget changed. They were stopped before 2M | |
| (the study ended): this repository holds the grid checkpoints from 1,025,000 on and, | |
| per run, `checkpoints/last.ckpt`, the exact state where training stopped. Steps | |
| 0-1,000,000 (and their checkpoints) are in `duo-scaling-tiny`. | |
| Each folder holds **full checkpoints** on a fixed step grid (40 at 2M), so the whole | |
| training trajectory can be analysed (e.g. test loss against weight norm, in the | |
| spirit of D'Amico, Bocchi & Negri, *Implicit bias produces neural scaling laws in | |
| learning curves*). | |
| ## Models | |
| | fraction | training blocks (1,024 tokens) | step | epochs | val NELBO at that step (nats/token) | training hours | `last.ckpt` step | | |
| |---|---|---|---|---|---|---| | |
| | 0.1 | 873,094 | 1,300,000 | 762 | 3.9652 | 200 | 1,309,080 | | |
| | 1.0 | 8,730,942 | 1,300,000 | 76 | 3.9849 | 197 | 1,309,800 | | |
| Validation NELBO is computed with the EMA weights on the full OpenWebText validation split | |
| (110,464 blocks), with fixed noise, every 5,000 steps (`val/nll` in the metrics files). | |
| ## Training setup (same for every model) | |
| - **Model:** DiT backbone, config `tiny` (33,789,010 parameters in the backbone), | |
| context length 1,024, GPT-2 tokenizer (vocabulary 50,258), untied embeddings. | |
| - **Algorithm:** `duo_base`: uniform-state discrete diffusion trained on the exact NELBO, | |
| log-linear noise schedule (no curriculum or distillation). | |
| - **Optimisation:** AdamW, learning rate 3e-4, constant after 2,500 warm-up steps, | |
| **weight decay 0**, gradient clipping 1.0, EMA decay 0.9999, bf16 mixed precision, | |
| global batch 512 sequences x 1,024 tokens, 2,000,000 steps (target; continued from the 1,000,000-step runs), seed 1. | |
| - **Data:** OpenWebText (`Skylion007/openwebtext`, revision `79d93d7`), GPT-2 tokenized and | |
| wrapped into 1,024-token blocks: 8,730,942 training blocks; the last 100,000 | |
| documents form the validation split. Fraction *f* uses the first floor(f x N + 0.5) | |
| blocks of one fixed permutation (seed 20250927, sha256 in `run_info.json`), so the | |
| subsets are nested (0.1 is contained in 0.2, and so on). The data order depends only on | |
| the optimizer step, not on the GPU layout. | |
| - **Code:** a modified version of [s-sahoo/duo](https://github.com/s-sahoo/duo) | |
| (scaling-study branch, commit `7c9e7d04d4a4`, recorded in `code_sha`). | |
| ## Files | |
| ``` | |
| frac_0.10/ ... frac_1.00/ | |
| checkpoints/step_*.ckpt PyTorch Lightning checkpoints (40 at 2M) | |
| checkpoints/index.csv step, wall-clock training hours, tokens and epochs per checkpoint | |
| checkpoints/last.ckpt state where training stopped (resume point) | |
| metrics/version_*/metrics.csv training/validation metrics (one file per job segment) | |
| norms/norms_raw.jsonl, norms_ema.jsonl weight norms every 500 steps, raw and EMA weights | |
| run_info.json, config_resolved.yaml, code_sha, segments.jsonl | |
| subsets/owt_train_perm_s20250927.npy, .json the permutation of the training blocks that defines | |
| the nested subsets (fraction f = its first floor(f x N + 0.5) entries) | |
| ``` | |
| Checkpoint steps: every 25k from 1,025k (the grid of the 1M study continued). | |
| Each checkpoint has the raw weights, the EMA weights, the optimizer state and the | |
| loop/sampler state, so training can be resumed exactly. Step 0 is about half the size | |
| (no optimizer state yet). | |
| ```python | |
| import torch | |
| ck = torch.load("frac_0.10/checkpoints/step_1300000.ckpt", map_location="cpu", weights_only=False) | |
| raw = {k: v for k, v in ck["state_dict"].items() if k.startswith("backbone.")} | |
| # EMA tensors follow the backbone's trainable parameters in order (all of the | |
| # state_dict except the rotary-embedding buffer). | |
| names = [k for k in raw if not k.endswith("rotary_emb.inv_freq")] | |
| ema = dict(zip(names, ck["ema"]["shadow_params"])) | |
| print(ck["global_step"], ck["scaling_meta"]["clock"]["train_s"] / 3600, "training hours") | |
| ``` | |
| **Norms** (`norms/*.jsonl`): per record the step, aggregate norms of the token-path matrices | |
| (embedding, per-block Q/K/V, attention output, MLP, output layer) and, separately, of the | |
| timestep-conditioning path, plus per-matrix statistics (spectral norm, Frobenius, L1, | |
| row and column (2,1)-norms) in float64. `log_R` is the Bartlett spectral complexity: | |
| `sum log||W||_sigma + 1.5 log sum (||W^T||_2,1^(2/3) / ||W||_sigma^(2/3))`. It is null at | |
| step 0 because the output layer starts at zero. | |
| ## Caveats | |
| - **Wall-clock time** (`train_hours`) mixes H100 and H200 segments (the GPU of every | |
| segment is in `segments.jsonl`). For time-based comparisons, use steps or normalise | |
| per GPU type. | |
| - `trainer/loss` in the metrics is one micro-batch on one GPU and depends on the GPU | |
| layout; compare runs with `val/nll`. | |
| - **Seed checkpoint:** each run resumed from `step_1000000.ckpt` of the 1M run with one loop | |
| flag changed (`epoch_loop.batch_progress.is_last_batch` True -> False), which is the state a | |
| 2M run has at step 1M. Data order (indexed by step), noise, seed, layout and the constant | |
| learning rate are unchanged, so steps 1M-2M equal those of a 2M run from scratch. Records | |
| (metrics, norms, `train_hours`) continue from the 1M run's values. | |