File size: 19,028 Bytes

---
language:
- en
library_name: transformers
pipeline_tag: other
tags:
- custom_code
- tensorboard
- tiny-llm-ablation
- from-scratch
- diffusion
- masked-language-modeling
datasets:
- HuggingFaceFW/fineweb-edu
model-index:
- name: diffusion-51M-base
  results:
  - task:
      type: fill-mask
      name: Experimental continuation PLL / reconstruction
    dataset:
      type: Rowan/hellaswag
      name: HellaSwag
      config: default
      split: validation
    metrics:
    - type: pll_acc_norm
      name: PLL acc_norm
      value: 0.2795259908384784
      args:
        num_few_shot: 0
        ci95_low: 0.2708343265409323
        ci95_high: 0.2883862705438426
        ci_method: Wilson
        protocol: single-mask continuation PLL
  - task:
      type: fill-mask
      name: Experimental continuation PLL / reconstruction
    dataset:
      type: allenai/ai2_arc
      name: ARC-Easy
      config: ARC-Easy
      split: test
    metrics:
    - type: pll_acc_norm
      name: PLL acc_norm
      value: 0.3362794612794613
      args:
        num_few_shot: 0
        ci95_low: 0.3175609903739864
        ci95_high: 0.3555264763913292
        ci_method: Wilson
        protocol: single-mask continuation PLL
  - task:
      type: fill-mask
      name: Experimental continuation PLL / reconstruction
    dataset:
      type: allenai/ai2_arc
      name: ARC-Challenge
      config: ARC-Challenge
      split: test
    metrics:
    - type: pll_acc_norm
      name: PLL acc_norm
      value: 0.2226962457337884
      args:
        num_few_shot: 0
        ci95_low: 0.19980419390295
        ci95_high: 0.2474001931137188
        ci_method: Wilson
        protocol: single-mask continuation PLL
  - task:
      type: fill-mask
      name: Experimental continuation PLL / reconstruction
    dataset:
      type: baber/piqa
      name: PIQA
      config: default
      split: validation
    metrics:
    - type: pll_acc_norm
      name: PLL acc_norm
      value: 0.5386289445048966
      args:
        num_few_shot: 0
        ci95_low: 0.5157819596226829
        ci95_high: 0.5613147955404465
        ci_method: Wilson
        protocol: single-mask continuation PLL
  - task:
      type: fill-mask
      name: Experimental continuation PLL / reconstruction
    dataset:
      type: allenai/winogrande
      name: WinoGrande
      config: winogrande_xl
      split: validation
    metrics:
    - type: pll_acc
      name: PLL acc
      value: 0.5011838989739542
      args:
        num_few_shot: 0
        ci95_low: 0.47369054474157507
        ci95_high: 0.5286700959026042
        ci_method: Wilson
        protocol: single-mask continuation PLL
  - task:
      type: fill-mask
      name: Experimental continuation PLL / reconstruction
    dataset:
      type: allenai/openbookqa
      name: OpenBookQA
      config: main
      split: test
    metrics:
    - type: pll_acc_norm
      name: PLL acc_norm
      value: 0.26
      args:
        num_few_shot: 0
        ci95_low: 0.22348572318538285
        ci95_high: 0.3001739602361623
        ci_method: Wilson
        protocol: single-mask continuation PLL
  - task:
      type: fill-mask
      name: Experimental continuation PLL / reconstruction
    dataset:
      type: aps/super_glue
      name: BoolQ
      config: boolq
      split: validation
    metrics:
    - type: pll_acc
      name: PLL acc
      value: 0.5296636085626911
      args:
        num_few_shot: 0
        ci95_low: 0.5125316201633632
        ci95_high: 0.5467259836150076
        ci_method: Wilson
        protocol: single-mask continuation PLL
  - task:
      type: fill-mask
      name: Experimental continuation PLL / reconstruction
    dataset:
      type: EleutherAI/lambada_openai
      name: LAMBADA OpenAI reconstruction
      config: default
      split: test
    metrics:
    - type: pll_acc
      name: PLL acc
      value: 0.42208422278284496
      args:
        num_few_shot: 0
        ci95_low: 0.4086621968305708
        ci95_high: 0.4356223315164309
        ci_method: Wilson
        protocol: single-mask continuation PLL
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: AxiomicLabs/Arithmark-3.0
      name: ArithMark-3
      config: default
      split: train
    metrics:
    - type: pll_acc_norm
      name: pll_acc_norm (fraction; lm-eval 0.4.12 comparison protocol)
      value: 0.332
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 1024
        standard_error: 0.014899597242811565
        evaluation_date: '2026-10-01'
        ci95_low: 0.30350363489239063
        ci95_high: 0.36178215595875596
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: pkavumba/balanced-copa
      name: Balanced COPA
      config: default
      split: train
    metrics:
    - type: pll_acc
      name: pll_acc (fraction; lm-eval 0.4.12 comparison protocol)
      value: 0.522
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.01580397942816194
        evaluation_date: '2026-10-01'
        ci95_low: 0.4910152521271681
        ci95_high: 0.5528163704994674
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: tau/commonsense_qa
      name: CommonsenseQA
      config: default
      split: validation
    metrics:
    - type: pll_acc
      name: pll_acc (fraction; lm-eval 0.4.12 comparison protocol)
      value: 0.21457821457821458
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.011753423094216953
        evaluation_date: '2026-10-01'
        ci95_low: 0.19246524693116476
        ci95_high: 0.2384815135817527
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: allenai/sciq
      name: SciQ (with support)
      config: default
      split: test
    metrics:
    - type: pll_acc_norm
      name: pll_acc_norm (fraction; lm-eval 0.4.12 comparison protocol)
      value: 0.688
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.014658474370509057
        evaluation_date: '2026-10-01'
        ci95_low: 0.6586108249018996
        ci95_high: 0.7159503139075316
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: truthfulqa/truthful_qa
      name: TruthfulQA MC2
      config: multiple_choice
      split: validation
    metrics:
    - type: pll_acc
      name: pll_acc (fraction; lm-eval 0.4.12 comparison protocol)
      value: 0.45836334441298165
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.0158524239677764
        evaluation_date: '2026-10-01'
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: BananaMind/BananaMind-Base-Bench-1.1
      name: BananaMind Base 1.1
      config: default
      split: test
    metrics:
    - type: pll_raw_accuracy
      name: pll_raw_accuracy (fraction; lm-eval 0.4.12 comparison protocol)
      value: 0.39714285714285713
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.02619195222772307
        evaluation_date: '2026-10-01'
        ci95_low: 0.34726441786546686
        ci95_high: 0.4492546213676228
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: cais/mmlu
      name: MMLU continuation
      config: 57 subjects
      split: test
    metrics:
    - type: pll_acc
      name: pll_acc (fraction; lm-eval 0.4.12 comparison protocol)
      value: 0.24483691781797465
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.0036208715458871405
        evaluation_date: '2026-10-01'
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: nyu-mll/blimp
      name: BLiMP
      config: 67 minimal-pair subsets
      split: train
    metrics:
    - type: pll_acc
      name: pll_acc (fraction; lm-eval 0.4.12 comparison protocol)
      value: 0.6420746268656716
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.0016890890018278603
        evaluation_date: '2026-10-01'
---

# diffusion-51M-base

A small English **masked diffusion base model**, trained from random initialization as part of [Tiny llm ablation](https://huggingface.co/collections/d0rj/tiny-llm-ablation-6aafca336122dd2c2868f923). This is the successfully trained **v2** checkpoint: **51,392,512 stored parameters**, **50,867,200 optimized parameters**, exactly **3,932,160,000 processed source tokens**.

## Architecture and references

10 bidirectional transformer layers, width 512, 8 attention heads (head dimension 64), SwiGLU intermediate size 1536, RoPE, RMSNorm, tied input/output embeddings, context 2048. The unchanged 32,768-entry tokenizer and small-model comparison reference are [Q-50M-Base](https://huggingface.co/q-project/Q-50M-Base). These weights are not fine-tuned from that model.

The absorbing-mask objective follows the [LLaDA](https://arxiv.org/abs/2502.09992) family and its [official guidelines](https://github.com/ML-GSAI/LLaDA/blob/main/GUIDELINES.md): sample t uniformly, mask tokens independently with probability t, predict original tokens at masked positions, and minimize `sum(masked CE / t) / source_token_count`. Local t is clamped at 1e-5. This is an independent small-scale implementation, not an exact LLaDA reproduction.

**Time conditioning is disabled.** Removing the legacy additive time branch restored context learning after an optimization collapse. Its 525,312 unused, frozen parameters remain in the checkpoint for state-dictionary compatibility. The mask uses a separate learned input vector with ID 32768, outside the output vocabulary; it is not an added tokenizer token. Generic config validation may warn about this intentional custom mask ID.

## Training

[FineWeb-Edu sample-10BT](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), local parquet shards, seed 2026, shuffle buffer 100,000. **15,000 steps × 8 sequences × 16 accumulation × 2048 tokens = 3,932,160,000 source-token exposures**. This is not a count of unique or masked target tokens. The v2 model started fresh; short debugging runs are not included in these weights or this token count.

One RTX 5070 Ti 16 GB, BF16 compute / FP32 weights, compiled forward, fused AdamW: peak LR 0.001, 150-step warmup then cosine decay to 0.0001, betas (0.9, 0.95), weight decay 0.1 excluding 1D parameters, clipping 1.0. Training took **439.5 minutes**; final logged denoising loss **5.1272**. [Exact training settings](training_config.json). Equal source-token budgets across architectures do not imply equal supervision or FLOPs.

## Evaluation

Full splits, no added few-shot examples, lm-eval 0.4.12, GPU BF16, context 2048 (ArithMark: 1024), no chat template. **Experimental continuation pseudo-log-likelihood (PLL):** mask one answer token at a time while all other answer tokens remain visible. `acc_norm` is the harness length-normalized option score. LAMBADA is full-token reconstruction accuracy, **not autoregressive final-word generation**. Scores are percentages ± one standard error; brackets are **95% Wilson intervals**.

| Dataset | Split | Examples | Metric | Score ± SE (%) | 95% CI (%) |
|---|---|---:|---|---:|---:|
| [HellaSwag](https://huggingface.co/datasets/Rowan/hellaswag) | validation | 10,042 | acc_norm | 27.95 ± 0.45 | [27.08, 28.84] |
| [ARC-Easy](https://huggingface.co/datasets/allenai/ai2_arc) | test | 2,376 | acc_norm | 33.63 ± 0.97 | [31.76, 35.55] |
| [ARC-Challenge](https://huggingface.co/datasets/allenai/ai2_arc) | test | 1,172 | acc_norm | 22.27 ± 1.22 | [19.98, 24.74] |
| [PIQA](https://huggingface.co/datasets/baber/piqa) | validation | 1,838 | acc_norm | 53.86 ± 1.16 | [51.58, 56.13] |
| [WinoGrande](https://huggingface.co/datasets/allenai/winogrande) | validation | 1,267 | acc | 50.12 ± 1.41 | [47.37, 52.87] |
| [OpenBookQA](https://huggingface.co/datasets/allenai/openbookqa) | test | 500 | acc_norm | 26.00 ± 1.96 | [22.35, 30.02] |
| [BoolQ](https://huggingface.co/datasets/aps/super_glue) | validation | 3,270 | acc | 52.97 ± 0.87 | [51.25, 54.67] |
| [LAMBADA OpenAI reconstruction](https://huggingface.co/datasets/EleutherAI/lambada_openai) | test | 5,153 | acc | 42.21 ± 0.69 | [40.87, 43.56] |
| [ArithMark-3](https://huggingface.co/datasets/AxiomicLabs/Arithmark-3.0) | train | 1,000 | `acc_norm` | 33.20 ± 1.49 | [30.35, 36.18] |
| [Balanced COPA](https://huggingface.co/datasets/pkavumba/balanced-copa) | train | 1,000 | `acc` | 52.20 ± 1.58 | [49.10, 55.28] |
| [CommonsenseQA](https://huggingface.co/datasets/tau/commonsense_qa) | validation | 1,221 | `acc` | 21.46 ± 1.18 | [19.25, 23.85] |
| [SciQ (with support)](https://huggingface.co/datasets/allenai/sciq) | test | 1,000 | `acc_norm` | 68.80 ± 1.47 | [65.86, 71.60] |
| [TruthfulQA MC2](https://huggingface.co/datasets/truthfulqa/truthful_qa) | validation | 817 | `acc` | 45.84 ± 1.59 | — |
| [BananaMind Base 1.1](https://huggingface.co/datasets/BananaMind/BananaMind-Base-Bench-1.1) | test | 350 | `raw_accuracy` | 39.71 ± 2.62 | [34.73, 44.93] |
| [MMLU continuation](https://huggingface.co/datasets/cais/mmlu) | test | 14,042 | `acc` | 24.48 ± 0.36 | — |
| [BLiMP](https://huggingface.co/datasets/nyu-mll/blimp) | train | 67,000 | `acc` | 64.21 ± 0.17 | — |

[WikiText-2 raw test](https://huggingface.co/datasets/Salesforce/wikitext): 291 nonoverlapping 1024-token blocks, prefix 512 + scored suffix 512, 148,992 scored tokens, 335 tail tokens omitted. Single-mask continuation PLL, GPU BF16, TF32 disabled. **Pseudo-perplexity 15.865 [15.198, 16.559]**; NLL **2.764107 [2.721156, 2.806927]**. 95% percentile block bootstrap, 10,000 resamples, seed 2026; exponentiate NLL endpoints for pseudo-perplexity CI. **This is not AR PPL**, and is not directly comparable with Q50M/T5/PrefixLM/Looped continuation PPL.

[Detailed metrics, standard errors and provenance](evaluation/results.json). Model-index entries use explicit `pll_*` metric labels and are author-reported results; no official leaderboard submission is claimed. Intervals do not include training-seed variation or all within-document dependence. Earlier test diagnostics informed the collapse investigation; benchmark contamination was not audited.

Full selected splits; lm-eval 0.4.12; seed 1234; BF16 on RTX 5070 Ti;
context cap 2048 (ArithMark 1024), TF32 disabled, no chat template and no added
few-shot examples. TruthfulQA retains the harness's fixed six-QA preamble.
ArithMark and BananaMind normalize by continuation token count; ordinary harness
acc_norm uses its own length normalization. BananaMind is raw accuracy, not Elo.
SciQ includes the support passage. Balanced COPA uses the mirrored 1000-item
train-named evaluation split; cRia's split was inferred, not confirmed.
MMLU scores full answer continuations across 57 subjects, weighted by item count;
BLiMP averages 67 equal-sized minimal-pair subsets. Standard errors are retained
from each evaluator. Wilson intervals are reported only where the runner logged
binary item accuracy; MC2 is probability mass, not binary accuracy. These
intervals do not model dependence between paired/templated examples or training
seed variation. UL2, PrefixLM and experimental diffusion PLL use their documented
conditional scoring protocols; PLL exposes the other answer tokens and is not
autoregressive likelihood. cRia's published scores used a different precision
and benchmark-adapted checkpoint; this completes our comparison coverage, not
an independent reproduction of cRia or an official leaderboard submission.

[Full results, provenance and group scores](evaluation/comparison-20261001/results.json). [Updated machine-readable results](evaluation/results.json). [TensorBoard events](tensorboard/) contain these new scores at step 15,000.



## Usage

Install [requirements.txt](requirements.txt). Load with **AutoModelForMaskedLM** and `trust_remote_code=True`; the custom API requires a `timesteps` tensor even though v2 does not condition on it.

```python
import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

repo = "d0rj/diffusion-51M-base"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True).eval()
ids = tokenizer("The purpose of science is to understand the world.",
                return_tensors="pt", add_special_tokens=False)["input_ids"]
ids[:, 2] = model.config.mask_token_id
with torch.no_grad():
    logits = model(ids, timesteps=torch.tensor([0.5])).logits
print(tokenizer.decode([logits[0, 2].argmax().item()]))

# Optional unconditional confidence-based iterative denoising.
tokens = model.generate_masked(batch_size=1, seq_len=32, steps=16, temperature=0.)
print(tokenizer.decode(tokens[0], skip_special_tokens=True))
```

`generate_masked` starts from all masks and commits confident tokens on a linear schedule. It is an experimental unconditional sampler, not standard causal `.generate()` or an instruction-following interface. The reported core benchmarks use the PLL adapter, not this sampler. The generic HF inference widget is not configured for this custom interface.

Reproduce from a downloaded repository after installing `evaluation/requirements.txt`:

```bash
python evaluation/run_core.py --device cuda:0 --dtype bfloat16 --batch-size 1 --output core-results
python evaluation/run_continuation.py --output continuation-results.json
```

`--limit 2` on the core runner is a smoke test only. [TensorBoard files](tensorboard/) retain 750 training-loss points, development probes every 100 steps, and test evaluation scalars at step 15,000, including CI bounds. The failed v1 weights are not part of this release. Small English research ablation; not instruction-tuned, and performance improvements are not uniform across tasks.

To reproduce after downloading this model repository, accept the BananaMind dataset terms, authenticate with `hf auth login`, then run in a suitable CUDA environment:

```bash
pip install -r evaluation/comparison-20261001/repro/requirements.txt
python evaluation/comparison-20261001/repro/run.py --device cuda:0 --dtype bfloat16 --batch-size 1 --output comparison-rerun
```

The bundled runner uses the published model classes with the exact evaluation adapters and tokenizer. `--limit` produces smoke results only. Raw dataset examples are not included in this release.