Duo data-fraction scaling study: small models
5 uniform-state discrete diffusion language models (Duo, duo_base) trained with
AdamW weight decay 0.1, identical
except for the size of their training set: nested subsets of OpenWebText holding
10%, 25%, 50%, 75% and 100% of the training blocks. Training is paused before its 1,000,000-step target (stopped to save compute; it
may resume). This repository holds the grid checkpoints reached so far and, per run,
checkpoints/last.ckpt, the exact state where training stopped.
Each folder holds full checkpoints on a fixed step grid (53 at 1M), so the whole
training trajectory can be analysed (e.g. test loss against weight norm, in the
spirit of D'Amico, Bocchi & Negri, Implicit bias produces neural scaling laws in
learning curves).
Models
| fraction | training blocks (1,024 tokens) | step | epochs | val NELBO at that step (nats/token) | training hours | last.ckpt step |
|---|---|---|---|---|---|---|
| 0.1 | 873,094 | 475,000 | 279 | 3.3461 | 94 | 478,640 |
| 0.25 | 2,182,736 | 200,000 | 47 | 3.3763 | 43 | 222,980 |
| 0.5 | 4,365,471 | 200,000 | 23 | 3.3719 | 43 | 222,580 |
| 0.75 | 6,548,207 | 250,000 | 20 | 3.3599 | 47 | 253,720 |
| 1.0 | 8,730,942 | 250,000 | 15 | 3.3607 | 47 | 254,640 |
Validation NELBO is computed with the EMA weights on the full OpenWebText validation split
(110,464 blocks), with fixed noise, every 5,000 steps (val/nll in the metrics files).
Training setup (same for every model)
- Model: DiT backbone, config
small(169,627,218 parameters in the backbone), context length 1,024, GPT-2 tokenizer (vocabulary 50,258), untied embeddings. - Algorithm:
duo_base: uniform-state discrete diffusion trained on the exact NELBO, log-linear noise schedule (no curriculum or distillation). - Optimisation: AdamW, learning rate 3e-4, constant after 2,500 warm-up steps, weight decay 0.1 (decoupled; one parameter group, so biases, LayerNorm weights and embeddings are decayed too), gradient clipping 1.0, EMA decay 0.9999, bf16 mixed precision, global batch 512 sequences x 1,024 tokens, 1,000,000 steps (target), seed 1.
- Data: OpenWebText (
Skylion007/openwebtext, revision79d93d7), GPT-2 tokenized and wrapped into 1,024-token blocks: 8,730,942 training blocks; the last 100,000 documents form the validation split. Fraction f uses the first floor(f x N + 0.5) blocks of one fixed permutation (seed 20250927, sha256 inrun_info.json), so the subsets are nested; the weight-decay-0 study (duo-scaling-small, same account) uses prefixes of the same permutation, so its subsets nest with these too. The data order depends only on the optimizer step, not on the GPU layout. - Code: a modified version of s-sahoo/duo
(scaling-study branch, commit
7ac5b2294c36; frac_0.10 began as a pilot on7c9e7d04d4a4with weight decay set on the command line (same training code), recorded per run incode_sha).
Files
frac_0.10/ ... frac_1.00/
checkpoints/step_*.ckpt PyTorch Lightning checkpoints (53 at 1M)
checkpoints/index.csv step, wall-clock training hours, tokens and epochs per checkpoint
checkpoints/last.ckpt state where training stopped (resume point)
metrics/version_*/metrics.csv training/validation metrics (one file per job segment)
norms/norms_raw.jsonl, norms_ema.jsonl weight norms every 500 steps, raw and EMA weights
run_info.json, config_resolved.yaml, code_sha, segments.jsonl
subsets/owt_train_perm_s20250927.npy, .json the permutation of the training blocks that defines
the nested subsets (fraction f = its first floor(f x N + 0.5) entries)
Checkpoint steps so far: 0, 100, 200, 400, 700, 1k, 1.5k, 2.5k, 4k, 6k, 10k, 15k, 20k, then every 25k to 1M.
Each checkpoint has the raw weights, the EMA weights, the optimizer state and the loop/sampler state, so training can be resumed exactly. Step 0 is about half the size (no optimizer state yet).
import torch
ck = torch.load("frac_0.10/checkpoints/step_0475000.ckpt", map_location="cpu", weights_only=False)
raw = {k: v for k, v in ck["state_dict"].items() if k.startswith("backbone.")}
# EMA tensors follow the backbone's trainable parameters in order (all of the
# state_dict except the rotary-embedding buffer).
names = [k for k in raw if not k.endswith("rotary_emb.inv_freq")]
ema = dict(zip(names, ck["ema"]["shadow_params"]))
print(ck["global_step"], ck["scaling_meta"]["clock"]["train_s"] / 3600, "training hours")
Norms (norms/*.jsonl): per record the step, aggregate norms of the token-path matrices
(embedding, per-block Q/K/V, attention output, MLP, output layer) and, separately, of the
timestep-conditioning path, plus per-matrix statistics (spectral norm, Frobenius, L1,
row and column (2,1)-norms) in float64. log_R is the Bartlett spectral complexity:
sum log||W||_sigma + 1.5 log sum (||W^T||_2,1^(2/3) / ||W||_sigma^(2/3)). It is null at
step 0 because the output layer starts at zero.
Caveats
- Wall-clock time (
train_hours) mixes H100 and H200 segments (the GPU of every segment is insegments.jsonl). For time-based comparisons, use steps or normalise per GPU type. trainer/lossin the metrics is one micro-batch on one GPU and depends on the GPU layout; compare runs withval/nll.- frac_0.10 began as a pilot (job name
duo-<size>_wd0.1-f010) and was adopted into the study unchanged; its resolved config matches the other runs except the wandb names. Resuming is exact (data order, noise and learning rate depend only on the step), so pauses between segments do not change the trajectory.