duo-scaling-2M-tiny / README.md
lemoncmd's picture
Add files using upload-large-folder tool
442bae0 verified
|
Raw History Blame Contribute Delete
5.73 kB
metadata
license: apache-2.0
library_name: pytorch
datasets:
  - Skylion007/openwebtext
tags:
  - discrete-diffusion
  - uniform-state-diffusion
  - duo
  - scaling-laws
  - checkpoints

Duo data-fraction scaling study: tiny models continued to 2M steps

2 uniform-state discrete diffusion language models (Duo, duo_base), identical except for the size of their training set: nested subsets of OpenWebText holding 10% and 100% of the training blocks. These runs continue frac_0.10 and frac_1.00 of duo-scaling-tiny from step 1,000,000 toward 2,000,000 with only the step budget changed. They were stopped before 2M (the study ended): this repository holds the grid checkpoints from 1,025,000 on and, per run, checkpoints/last.ckpt, the exact state where training stopped. Steps 0-1,000,000 (and their checkpoints) are in duo-scaling-tiny. Each folder holds full checkpoints on a fixed step grid (40 at 2M), so the whole training trajectory can be analysed (e.g. test loss against weight norm, in the spirit of D'Amico, Bocchi & Negri, Implicit bias produces neural scaling laws in learning curves).

Models

fraction training blocks (1,024 tokens) step epochs val NELBO at that step (nats/token) training hours last.ckpt step
0.1 873,094 1,300,000 762 3.9652 200 1,309,080
1.0 8,730,942 1,300,000 76 3.9849 197 1,309,800

Validation NELBO is computed with the EMA weights on the full OpenWebText validation split (110,464 blocks), with fixed noise, every 5,000 steps (val/nll in the metrics files).

Training setup (same for every model)

  • Model: DiT backbone, config tiny (33,789,010 parameters in the backbone), context length 1,024, GPT-2 tokenizer (vocabulary 50,258), untied embeddings.
  • Algorithm: duo_base: uniform-state discrete diffusion trained on the exact NELBO, log-linear noise schedule (no curriculum or distillation).
  • Optimisation: AdamW, learning rate 3e-4, constant after 2,500 warm-up steps, weight decay 0, gradient clipping 1.0, EMA decay 0.9999, bf16 mixed precision, global batch 512 sequences x 1,024 tokens, 2,000,000 steps (target; continued from the 1,000,000-step runs), seed 1.
  • Data: OpenWebText (Skylion007/openwebtext, revision 79d93d7), GPT-2 tokenized and wrapped into 1,024-token blocks: 8,730,942 training blocks; the last 100,000 documents form the validation split. Fraction f uses the first floor(f x N + 0.5) blocks of one fixed permutation (seed 20250927, sha256 in run_info.json), so the subsets are nested (0.1 is contained in 0.2, and so on). The data order depends only on the optimizer step, not on the GPU layout.
  • Code: a modified version of s-sahoo/duo (scaling-study branch, commit 7c9e7d04d4a4, recorded in code_sha).

Files

frac_0.10/ ... frac_1.00/
  checkpoints/step_*.ckpt      PyTorch Lightning checkpoints (40 at 2M)
  checkpoints/index.csv        step, wall-clock training hours, tokens and epochs per checkpoint
  checkpoints/last.ckpt        state where training stopped (resume point)
  metrics/version_*/metrics.csv  training/validation metrics (one file per job segment)
  norms/norms_raw.jsonl, norms_ema.jsonl  weight norms every 500 steps, raw and EMA weights
  run_info.json, config_resolved.yaml, code_sha, segments.jsonl
subsets/owt_train_perm_s20250927.npy, .json  the permutation of the training blocks that defines
                               the nested subsets (fraction f = its first floor(f x N + 0.5) entries)

Checkpoint steps: every 25k from 1,025k (the grid of the 1M study continued).

Each checkpoint has the raw weights, the EMA weights, the optimizer state and the loop/sampler state, so training can be resumed exactly. Step 0 is about half the size (no optimizer state yet).

import torch

ck = torch.load("frac_0.10/checkpoints/step_1300000.ckpt", map_location="cpu", weights_only=False)
raw = {k: v for k, v in ck["state_dict"].items() if k.startswith("backbone.")}
# EMA tensors follow the backbone's trainable parameters in order (all of the
# state_dict except the rotary-embedding buffer).
names = [k for k in raw if not k.endswith("rotary_emb.inv_freq")]
ema = dict(zip(names, ck["ema"]["shadow_params"]))
print(ck["global_step"], ck["scaling_meta"]["clock"]["train_s"] / 3600, "training hours")

Norms (norms/*.jsonl): per record the step, aggregate norms of the token-path matrices (embedding, per-block Q/K/V, attention output, MLP, output layer) and, separately, of the timestep-conditioning path, plus per-matrix statistics (spectral norm, Frobenius, L1, row and column (2,1)-norms) in float64. log_R is the Bartlett spectral complexity: sum log||W||_sigma + 1.5 log sum (||W^T||_2,1^(2/3) / ||W||_sigma^(2/3)). It is null at step 0 because the output layer starts at zero.

Caveats

  • Wall-clock time (train_hours) mixes H100 and H200 segments (the GPU of every segment is in segments.jsonl). For time-based comparisons, use steps or normalise per GPU type.
  • trainer/loss in the metrics is one micro-batch on one GPU and depends on the GPU layout; compare runs with val/nll.
  • Seed checkpoint: each run resumed from step_1000000.ckpt of the 1M run with one loop flag changed (epoch_loop.batch_progress.is_last_batch True -> False), which is the state a 2M run has at step 1M. Data order (indexed by step), noise, seed, layout and the constant learning rate are unchanged, so steps 1M-2M equal those of a 2M run from scratch. Records (metrics, norms, train_hours) continue from the 1M run's values.