Download README.md from lemoncmd/duo-scaling-2M-tiny: direct link, hf CLI and curl.
- Browser
- Download file 5.73 kB
-
https://huggingface.co/lemoncmd/duo-scaling-2M-tiny/resolve/main/README.md
- Command line
-
hf download hf://lemoncmd/duo-scaling-2M-tiny/README.md
-
curl -L -o README.md https://huggingface.co/lemoncmd/duo-scaling-2M-tiny/resolve/main/README.md
license: apache-2.0
library_name: pytorch
datasets:
- Skylion007/openwebtext
tags:
- discrete-diffusion
- uniform-state-diffusion
- duo
- scaling-laws
- checkpoints
Duo data-fraction scaling study: tiny models continued to 2M steps
2 uniform-state discrete diffusion language models (Duo, duo_base), identical
except for the size of their training set: nested subsets of OpenWebText holding
10% and 100% of the training blocks. These runs continue frac_0.10 and frac_1.00 of
duo-scaling-tiny from step 1,000,000
toward 2,000,000 with only the step budget changed. They were stopped before 2M
(the study ended): this repository holds the grid checkpoints from 1,025,000 on and,
per run, checkpoints/last.ckpt, the exact state where training stopped. Steps
0-1,000,000 (and their checkpoints) are in duo-scaling-tiny.
Each folder holds full checkpoints on a fixed step grid (40 at 2M), so the whole
training trajectory can be analysed (e.g. test loss against weight norm, in the
spirit of D'Amico, Bocchi & Negri, Implicit bias produces neural scaling laws in
learning curves).
Models
| fraction | training blocks (1,024 tokens) | step | epochs | val NELBO at that step (nats/token) | training hours | last.ckpt step |
|---|---|---|---|---|---|---|
| 0.1 | 873,094 | 1,300,000 | 762 | 3.9652 | 200 | 1,309,080 |
| 1.0 | 8,730,942 | 1,300,000 | 76 | 3.9849 | 197 | 1,309,800 |
Validation NELBO is computed with the EMA weights on the full OpenWebText validation split
(110,464 blocks), with fixed noise, every 5,000 steps (val/nll in the metrics files).
Training setup (same for every model)
- Model: DiT backbone, config
tiny(33,789,010 parameters in the backbone), context length 1,024, GPT-2 tokenizer (vocabulary 50,258), untied embeddings. - Algorithm:
duo_base: uniform-state discrete diffusion trained on the exact NELBO, log-linear noise schedule (no curriculum or distillation). - Optimisation: AdamW, learning rate 3e-4, constant after 2,500 warm-up steps, weight decay 0, gradient clipping 1.0, EMA decay 0.9999, bf16 mixed precision, global batch 512 sequences x 1,024 tokens, 2,000,000 steps (target; continued from the 1,000,000-step runs), seed 1.
- Data: OpenWebText (
Skylion007/openwebtext, revision79d93d7), GPT-2 tokenized and wrapped into 1,024-token blocks: 8,730,942 training blocks; the last 100,000 documents form the validation split. Fraction f uses the first floor(f x N + 0.5) blocks of one fixed permutation (seed 20250927, sha256 inrun_info.json), so the subsets are nested (0.1 is contained in 0.2, and so on). The data order depends only on the optimizer step, not on the GPU layout. - Code: a modified version of s-sahoo/duo
(scaling-study branch, commit
7c9e7d04d4a4, recorded incode_sha).
Files
frac_0.10/ ... frac_1.00/
checkpoints/step_*.ckpt PyTorch Lightning checkpoints (40 at 2M)
checkpoints/index.csv step, wall-clock training hours, tokens and epochs per checkpoint
checkpoints/last.ckpt state where training stopped (resume point)
metrics/version_*/metrics.csv training/validation metrics (one file per job segment)
norms/norms_raw.jsonl, norms_ema.jsonl weight norms every 500 steps, raw and EMA weights
run_info.json, config_resolved.yaml, code_sha, segments.jsonl
subsets/owt_train_perm_s20250927.npy, .json the permutation of the training blocks that defines
the nested subsets (fraction f = its first floor(f x N + 0.5) entries)
Checkpoint steps: every 25k from 1,025k (the grid of the 1M study continued).
Each checkpoint has the raw weights, the EMA weights, the optimizer state and the loop/sampler state, so training can be resumed exactly. Step 0 is about half the size (no optimizer state yet).
import torch
ck = torch.load("frac_0.10/checkpoints/step_1300000.ckpt", map_location="cpu", weights_only=False)
raw = {k: v for k, v in ck["state_dict"].items() if k.startswith("backbone.")}
# EMA tensors follow the backbone's trainable parameters in order (all of the
# state_dict except the rotary-embedding buffer).
names = [k for k in raw if not k.endswith("rotary_emb.inv_freq")]
ema = dict(zip(names, ck["ema"]["shadow_params"]))
print(ck["global_step"], ck["scaling_meta"]["clock"]["train_s"] / 3600, "training hours")
Norms (norms/*.jsonl): per record the step, aggregate norms of the token-path matrices
(embedding, per-block Q/K/V, attention output, MLP, output layer) and, separately, of the
timestep-conditioning path, plus per-matrix statistics (spectral norm, Frobenius, L1,
row and column (2,1)-norms) in float64. log_R is the Bartlett spectral complexity:
sum log||W||_sigma + 1.5 log sum (||W^T||_2,1^(2/3) / ||W||_sigma^(2/3)). It is null at
step 0 because the output layer starts at zero.
Caveats
- Wall-clock time (
train_hours) mixes H100 and H200 segments (the GPU of every segment is insegments.jsonl). For time-based comparisons, use steps or normalise per GPU type. trainer/lossin the metrics is one micro-batch on one GPU and depends on the GPU layout; compare runs withval/nll.- Seed checkpoint: each run resumed from
step_1000000.ckptof the 1M run with one loop flag changed (epoch_loop.batch_progress.is_last_batchTrue -> False), which is the state a 2M run has at step 1M. Data order (indexed by step), noise, seed, layout and the constant learning rate are unchanged, so steps 1M-2M equal those of a 2M run from scratch. Records (metrics, norms,train_hours) continue from the 1M run's values.