c3-mdlm-110m-v1

Masked diffusion language model (MDLM objective, absorbing mask, SUBS), trained from scratch for the C3 project (committed-confidence calibration in diffusion LMs).

The first 110M (v1, W&B train110m), used in the mid submission and the T9 dry run. Superseded by ANLP-project/c3-mdlm-110m.

parameters	123.9M (tied embeddings)
training tokens	9,999,876,096
steps	76293
weights	raw
config hash	None
code commit	48b91892d4f5200361b28f80f816f38f1c148218
data	FineWeb-Edu sample-10BT, document-aligned 256-token windows (no packing), evaluation documents excluded by id / text hash / first-256-token hash
tokenizer	GPT-2 BPE + one [MASK] token (id 50257)
held-out NELBO	3.6492 nats/token (ppl bound 38.45, 2000 documents)
GPT-2 small, same documents	ppl 28.49