Save 13.5B generator checkpoint and corrected FP32 evaluation results

Generator step 12875 with matching final tokenizer, resumable state, standalone exports, configs, SHA-256 manifest, and 640 corrected evaluation samples. Gen PPL 174.96 +/- 3.43; FP32 GPT-2 Large, TF32 disabled.

Files changed (24) hide show

iclr-debug-two-stage-135b-20260916/generator/README.md +19 -0
iclr-debug-two-stage-135b-20260916/generator/backup-manifest-step-12875.json +184 -0
iclr-debug-two-stage-135b-20260916/generator/config.yaml +223 -0
iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/EVAL_DONE.json +1 -0
iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/README.md +18 -0
iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/comparison-to-step-6000.json +14 -0
iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/evaluation-precision-fix.patch +39 -0
iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/evaluator-provenance.json +6 -0
iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-0.json +0 -0
iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-0.samples.json +0 -0
iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-1.json +0 -0
iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-1.samples.json +0 -0
iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-2.json +0 -0
iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-2.samples.json +0 -0
iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-3.json +0 -0
iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-3.samples.json +0 -0
iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-4.json +0 -0
iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-4.samples.json +0 -0
iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/summary.json +38 -0
iclr-debug-two-stage-135b-20260916/generator/exports/step-12875/ncp-config.yaml +179 -0
iclr-debug-two-stage-135b-20260916/generator/exports/step-12875/ncp.pt +3 -0
iclr-debug-two-stage-135b-20260916/generator/exports/step-12875/vqvae-config.yaml +117 -0
iclr-debug-two-stage-135b-20260916/generator/exports/step-12875/vqvae.pt +3 -0
iclr-debug-two-stage-135b-20260916/generator/milestone-iter-12875.pt +3 -0

iclr-debug-two-stage-135b-20260916/generator/README.md ADDED Viewed

	@@ -0,0 +1,19 @@

+# Two-stage generator — 13.5B checkpoint
+Run `iclr-debug-two-stage-135b-20260916`, generator step **12,875**: **13,500,416,000 input positions**. The generator uses the frozen **final step-128747 tokenizer** from this run, not the tokenizer selected at step 13000.
+## Files
+- [Full resumable generator checkpoint](milestone-iter-12875.pt): generator + frozen tokenizer state, optimizer/scheduler, per-rank RNG state and training configuration.
+- [Generator weights](exports/step-12875/ncp.pt) and [matching tokenizer weights](exports/step-12875/vqvae.pt).
+- [Generator export configuration](exports/step-12875/ncp-config.yaml), [tokenizer export configuration](exports/step-12875/vqvae-config.yaml), and [training configuration](config.yaml).
+- [Corrected evaluation report and samples](evaluation/step-12875-fp32/README.md).
+- [File checksums and provenance](backup-manifest-step-12875.json).
+## Corrected evaluation
+**Generation PPL 174.96 ± 3.43** (standard error across five seed means); mean within-sample token entropy **4.451 nats**. All **640 samples** scored (five seeds ×128), using GPT-2 Large in FP32 with TF32 disabled. Sampling is temperature 1 without top-k/top-p truncation and conditions on the true level-zero code; it is not strictly unconditional generation.
+Compared with corrected checkpoint 6000 evaluation (PPL 254.71 ±6.38), measured PPL is 31.3% lower under the same protocol. The initial scheduled evaluation had TF32 enabled; only the corrected rerun is included here. The precision fix is preserved with the results.
+Exported `.pt` files contain `model`, `args`, `step`, and source metadata. Configuration paths describe the original run environment; when evaluating downloaded exports, override the codec path with the downloaded `vqvae.pt` path. No training dataset files are included.

iclr-debug-two-stage-135b-20260916/generator/backup-manifest-step-12875.json ADDED Viewed

	@@ -0,0 +1,184 @@

+{
+  "repo": "iskhare/iclr-debug",
+  "run": "iclr-debug-two-stage-135b-20260916",
+  "stage": "generator",
+  "checkpoint_step": 12875,
+  "input_positions": 13500416000,
+  "nonpadding_tokens": 13500199164,
+  "frozen_tokenizer_step": 128747,
+  "git_provenance": {
+    "commit": "1b5226d4bbdd34653cb5fdd58d436355fb76ac8c"
+  },
+  "evaluation": {
+    "protocol": {
+      "sampling": "random",
+      "temperature": 1.0,
+      "top_k": 0,
+      "top_p": 1.0,
+      "n_provided_levels": 1,
+      "n_levels": 16,
+      "document_aware_validation_sampling": true
+    },
+    "reference_model": "gpt2-large",
+    "checkpoint": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/exports/step-12875/ncp.pt",
+    "checkpoint_step": 12875,
+    "seeds": [
+      0,
+      1,
+      2,
+      3,
+      4
+    ],
+    "samples_per_seed": 128,
+    "total_scored_samples": 640,
+    "total_scored_tokens": 160606,
+    "per_seed_mean_ppl": {
+      "0": 174.84999848898158,
+      "1": 175.7877408538115,
+      "2": 181.86846905478473,
+      "3": 180.03543569697817,
+      "4": 162.28073772742675
+    },
+    "mean_gen_ppl": 174.96447636439655,
+    "gen_ppl_se": 3.427446856160439,
+    "mean_of_seed_median_ppl": 176.59298400878907,
+    "mean_entropy_nats": 4.4512376085625265,
+    "reference_dtype": "float32",
+    "tf32_allowed": false,
+    "training_continued_during_evaluation": true
+  },
+  "files": [
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/milestone-iter-12875.pt",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/milestone-iter-12875.pt",
+      "size_bytes": 2565487435,
+      "sha256": "fefa9b7bf94073fe1f4145462c8fd497bf2f6a5dd9bfe611bada5b1b1658942f"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/hf-generator-backup-step-12875/training-config.yaml",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/config.yaml",
+      "size_bytes": 4572,
+      "sha256": "b72db37de5c634702e488e7f582115543c1ed551a01e1a4cef078a6d9dff2cc8"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/exports/step-12875/ncp.pt",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/exports/step-12875/ncp.pt",
+      "size_bytes": 427469662,
+      "sha256": "b0873bffc3887af02e0c889c9c05addb5ee394dc808f5cf65bf291a3141530b3"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/hf-generator-backup-step-12875/ncp-config.yaml",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/exports/step-12875/ncp-config.yaml",
+      "size_bytes": 3528,
+      "sha256": "074aa573fc2402d4c77a27768b3b0760024469c234661d027949ce1cbbc23801"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/exports/step-12875/vqvae.pt",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/exports/step-12875/vqvae.pt",
+      "size_bytes": 1300864085,
+      "sha256": "482c95118ff9140d0fd68bf78871d3fb9e69f89c3acc850d8d94a6d70cca7177"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/hf-generator-backup-step-12875/vqvae-config.yaml",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/exports/step-12875/vqvae-config.yaml",
+      "size_bytes": 2204,
+      "sha256": "446865d8f1ecb472b2296e4a84b478b5893687638ac54c5926f090aa70e109b1"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/summary.json",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/summary.json",
+      "size_bytes": 969,
+      "sha256": "f677d9e6853198e694f03243866c2e0ef6956fcff661b1d6fe61e5f57e285040"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/comparison-to-step-6000.json",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/comparison-to-step-6000.json",
+      "size_bytes": 463,
+      "sha256": "0f6f08fbfb9bfd62636471c9af46c044dfb882ede8ae2695742b6cb5bbb33f72"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/EVAL_DONE.json",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/EVAL_DONE.json",
+      "size_bytes": 105,
+      "sha256": "1cff9a3bb595bc1a454616baae36cf3eeb1188017eb657a64480b9296123dfd5"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/evaluator-provenance.json",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/evaluator-provenance.json",
+      "size_bytes": 280,
+      "sha256": "f9c37941224a165b45cbb4bfe8f63cb2dff63d95e6f2d524ecfa82c66c9585a1"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/evaluation-precision-fix.patch",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/evaluation-precision-fix.patch",
+      "size_bytes": 1485,
+      "sha256": "7a7dd0e7464c8b4ffd65fec1c297110a620e97a96fe6f5ed7142aea546848fb4"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-0.json",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-0.json",
+      "size_bytes": 139932,
+      "sha256": "d14b7bc73e05fc6c8c3c5bae939f3bb3da1db3736661c5385648f0b1ff315209"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-0.samples.json",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-0.samples.json",
+      "size_bytes": 140314,
+      "sha256": "ee59620cef6d3685536a353c0e587d9fcc275d4242fb04b406b173dee3d0041f"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-1.json",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-1.json",
+      "size_bytes": 141028,
+      "sha256": "9fe4b4044ae6fb8ececbd31c0e87b90784495aceb3075262a2706b94f5527963"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-1.samples.json",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-1.samples.json",
+      "size_bytes": 141409,
+      "sha256": "f6c0cc7941171aa49e0250d07571dd89d451d0437345664075a0f47c8452f054"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-2.json",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-2.json",
+      "size_bytes": 142319,
+      "sha256": "e73da22610d9697a3a683867324e2da56c2a44c9d44b0f633c9a2d519697d6a3"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-2.samples.json",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-2.samples.json",
+      "size_bytes": 142700,
+      "sha256": "9e89e0ea0ceeca39f056e1b1a95f0c24ed2ba744a3ffdd7e22536c5b7a733882"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-3.json",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-3.json",
+      "size_bytes": 138849,
+      "sha256": "de5d41317ba91216645ef38e5e2708c2e9145d6291e1161e1202787e496879f7"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-3.samples.json",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-3.samples.json",
+      "size_bytes": 139230,
+      "sha256": "9b036f446f738e639b48a8ae840d4a2b65f36626a07bcdc7c78dfb0751c82bd6"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-4.json",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-4.json",
+      "size_bytes": 142436,
+      "sha256": "1fbdae18629d0ca026bbd739c6986812812f30debccc960ed227f3da56ef0da2"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-4.samples.json",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-4.samples.json",
+      "size_bytes": 142817,
+      "sha256": "d1a22e35e30f21eaf09d1d3202a944b501c25b519d27a2358d75b18c959b7f5a"
+    },
+    {
+      "local_path": "/home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/hf-generator-backup-step-12875/evaluation-README.md",
+      "repo_path": "iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/README.md",
+      "size_bytes": 1603,
+      "sha256": "80c1bfba607c579138a913fce6fcb39f7828ed4fe3966a91bd1de034a265f104"
+    }
+  ]
+}

iclr-debug-two-stage-135b-20260916/generator/config.yaml ADDED Viewed

	@@ -0,0 +1,223 @@

+task: mstok-next-concept
+experiment: iclr-debug-two-stage-135b-20260916-generator
+experiment_dir: /home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/generator
+dataset:
+  train_source: /home/ubuntu/data/full_owt/train_gpt2.bin
+  validate_source: /home/ubuntu/data/full_owt/valid_gpt2.bin
+  pad_token_id: 50257
+pre_tokenizer:
+  name: hf
+  tokenizer: hf
+  model_id: gpt2
+  special_tokens:
+    eos_token: <|endoftext|>
+    pad_token: <|pad|>
+codec:
+  n_layers: 6
+  context_length: 256
+  embed_dim: 384
+  in_vocab_size: 50304
+  vqvae_vocab_size:
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  compression_factor: 1
+  dropout: 0.1
+  pre_quant_groupnorm: 4
+  pre_quant_dropout: 0.2
+  vector_quantizer_config:
+    clss: multiscale_residual_vector_quantizer
+    decay: 0.99
+    epsilon: 1.0e-05
+    commitment_cost: 0.25
+    learned_l1_sampling: true
+    learned_all_sampling: false
+    quant_resi:
+      enabled: true
+      ratio: 0.5
+      share_mode: 0
+      learnable_ratio: false
+    levels:
+      use_manual_levels: true
+      manual_levels:
+      - 1
+      - 4
+      - 9
+      - 16
+      - 25
+      - 36
+      - 49
+      - 64
+      - 81
+      - 100
+      - 121
+      - 144
+      - 169
+      - 196
+      - 225
+      - 256
+    aux:
+      fine_drop:
+        prob: 0.5
+        min_keep: 1
+generator:
+  n_layer: 12
+  n_head: 12
+  bias: true
+  dropout: 0.1
+  n_embd: 768
+  context_length: 256
+  vocab_size:
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  - 16384
+  attn_pattern: block_diagonal
+  use_positional_encoding: true
+  use_level_encoding: true
+  prefix_len: 0
+  use_rope: true
+  rope_base: 10000
+  use_qk_norm: true
+  post_upsample_conv:
+    enabled: true
+    kernel_size: 3
+  use_relu2: true
+  use_flex_attention: true
+  shared_output_head: true
+  shared_head_per_level_bias: false
+  shared_head_per_level_scale: false
+  shared_head_adapter_rank: 0
+objectives:
+  codec_weight: 1.0
+  ncp_weight: 1.0
+  soft_assignment_temperature: 1.0
+  prediction_temperature: 1.0
+  mstok_weight: 0.25
+  residual_weight: 1.0
+  reconstruction_weight: 1.0
+  teacher_ema_decay: 0.999
+  teacher_ema_warmup_steps: 1000
+  mstok_warmup_steps: 500
+optimization:
+  codec_lr: 0.001
+  codec_min_lr: 0.0001
+  codec_warmup_iters: 0
+  codec_lr_decay_iters: 128747
+  generator_lr: 0.0005
+  generator_min_lr: 1.0e-05
+  generator_warmup_iters: 225
+  generator_lr_decay_iters: 128747
+  beta_1: 0.9
+  codec_beta_2: 0.99
+  generator_beta_2: 0.99
+  weight_decay: 0.1
+  max_grad_norm: 1.0
+  level_loss_alpha: 1.0
+  grad_accumulation_steps: 16
+  corruption:
+    mode: per_level
+    per_level_probs:
+    - 0.85
+    - 0.8321428571
+    - 0.8142857143
+    - 0.7964285714
+    - 0.7785714286
+    - 0.7607142857
+    - 0.7428571429
+    - 0.725
+    - 0.7071428571
+    - 0.6892857143
+    - 0.6714285714
+    - 0.6535714286
+    - 0.6357142857
+    - 0.6178571429
+    - 0.6
+    skip_level0: true
+training:
+  log_interval: 10
+  eval_interval: 1000
+  checkpoint_interval: 1000
+  eval_batch_size: 4
+  val_iters: 25
+  keep_last: 3
+  seed: 55
+  codec_initialization_seed: 42
+  generator_initialization_seed: 55
+  total_iters: 128747
+  batch_size: 32
+  expected_world_size: 8
+  expected_global_batch_size: 4096
+  resume_checkpoint: null
+  milestone_steps:
+  - 954
+  - 4769
+  - 12875
+  - 25750
+  - 51499
+  - 77248
+  - 102997
+  - 128747
+  generation_eval_steps:
+  - 12875
+  - 25750
+  - 51499
+  - 77248
+  - 102997
+  - 128747
+torch_compile:
+  enable: true
+  scope: loss
+  mode: default
+  dynamic: false
+  fullgraph: true
+  backend: inductor
+mixed_precision:
+  enable: true
+wandb:
+  entity: mstok
+  project: iclr-debug
+  group: full-owt-ctx256-16sq-135b-v1
+  enable: true
+  id: iclr-debug-two-stage-135b-20260916-generator
+  resume: allow
+  gradients_and_params:
+    enable: false
+    log: all
+    log_freq: 1000
+export:
+  vqvae_config_template: /home/ubuntu/mstok-runs/iclr-debug-two-stage-20260916/config/repro-ctx256/vqvae.yaml
+  ncp_config_template: /home/ubuntu/mstok-runs/iclr-debug-two-stage-20260916/config/repro-ctx256/ncp-sharedhead.yaml
+iclr_debug:
+  stage: generator
+  target_positions: 135000000000
+  lr_horizon_positions: 135000000000
+  stop_step: 128747
+  pilot: false
+  tokenizer_checkpoint: /home/ubuntu/mstok-results/iclr-debug-two-stage-135b-20260916/tokenizer/milestone-iter-128747.pt
+  tokenizer_steps: 128747
+  budget_unit: input positions including padding; log non-padding tokens separately

iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/EVAL_DONE.json ADDED Viewed

	@@ -0,0 +1 @@


1	+ {"step": 12875, "reference_dtype": "float32", "sampling": "supplied-level0-untruncated", "manual": true}

iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/README.md ADDED Viewed

	@@ -0,0 +1,18 @@

+# Generator 13.5B evaluation — step 12,875
+Corrected manual evaluation completed on 2026-09-17. Training continued during the evaluation.
+| Checkpoint | Input positions | Mean generation PPL ± seed SE | Mean within-sample token entropy |
+|---|---:|---:|---:|
+| Step 6,000 | 6,291,456,000 | 254.71 ± 6.38 | 4.496 nats |
+| Step 12,875 | 13,500,416,000 | 174.96 ± 3.43 | 4.451 nats |
+The measured PPL reduction is 31.3%. Both evaluations use the identical corrected evaluator, five seeds × 128 samples, temperature 1, no top-k/top-p truncation, supplied true level-zero code, and FP32 GPT-2 Large with TF32 disabled. All 640 samples scored; no empty samples skipped. Standard error is across five seed means. Generation is conditioned on level zero, rather than strictly unconditional.
+The frozen tokenizer is the final step-128747 tokenizer. The generator checkpoint is `../../milestone-iter-12875.pt`; standalone components are in `../../exports/step-12875/`.
+[Aggregate results](summary.json) · [Comparison](comparison-to-step-6000.json)
+Samples: [seed 0](random-seed-0.samples.json), [seed 1](random-seed-1.samples.json), [seed 2](random-seed-2.samples.json), [seed 3](random-seed-3.samples.json), [seed 4](random-seed-4.samples.json).
+The original scheduled evaluation had TF32 enabled and is excluded from this upload. These results use the corrected evaluator; every per-seed JSON records `tf32_allowed=false`. The evaluator patch and source provenance are included here. Apply the patch to source commit `1b5226d4bbdd34653cb5fdd58d436355fb76ac8c` to reproduce the precision settings.

iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/comparison-to-step-6000.json ADDED Viewed

	@@ -0,0 +1,14 @@

+{
+  "earlier_step": 6000,
+  "current_step": 12875,
+  "earlier_positions": 6291456000,
+  "current_positions": 13500416000,
+  "earlier_gen_ppl": 254.70812987476478,
+  "current_gen_ppl": 174.96447636439655,
+  "relative_ppl_reduction_percent": 31.30785560302951,
+  "earlier_entropy_nats": 4.496167063090073,
+  "current_entropy_nats": 4.4512376085625265,
+  "same_protocol_and_evaluator_verified": true,
+  "all_seeds_precision_verified": true,
+  "total_samples": 640
+}

iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/evaluation-precision-fix.patch ADDED Viewed

	@@ -0,0 +1,39 @@

+diff --git a/evaluation/eval_random_gen_ppl_checkpoint.py b/evaluation/eval_random_gen_ppl_checkpoint.py
+index 2d5c741..748d300 100644
+--- a/evaluation/eval_random_gen_ppl_checkpoint.py
++++ b/evaluation/eval_random_gen_ppl_checkpoint.py
+@@ -22,6 +22,14 @@ from utils.misc import set_manual_seed
+ from utils.registry import trainers
++def seed_evaluation(seed, reference_dtype):
++    # The shared seed helper enables TF32, so precision must be set afterward.
++    set_manual_seed(seed)
++    if reference_dtype == "float32":
++        torch.backends.cuda.matmul.allow_tf32 = False
++        torch.backends.cudnn.allow_tf32 = False
++
++
+ def main():
+     parser = argparse.ArgumentParser()
+     parser.add_argument("--ckpt", required=True)
+@@ -40,10 +48,6 @@ def main():
+     parser.add_argument("--save-all-generations", action="store_true")
+     args = parser.parse_args()
+-    if args.ref_dtype == "float32":
+-        torch.backends.cuda.matmul.allow_tf32 = False
+-        torch.backends.cudnn.allow_tf32 = False
+-
+     device = "cuda"
+     checkpoint = torch.load(args.ckpt, map_location="cpu", weights_only=False)
+     cfg = checkpoint["args"]
+@@ -58,7 +62,7 @@ def main():
+     else:
+         cfg.optimization.stochastic_encode = {"enabled": False}
+-    set_manual_seed(args.seed)
++    seed_evaluation(args.seed, args.ref_dtype)
+     Trainer = hydra.utils.get_class(trainers[cfg.task])
+     trainer = Trainer(cfg, device)
+     trainer.model.load_state_dict(checkpoint["model"])

iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/evaluator-provenance.json ADDED Viewed

	@@ -0,0 +1,6 @@

+{
+  "base_commit": "1b5226d4bbdd34653cb5fdd58d436355fb76ac8c",
+  "evaluator_sha256": "200a399bcc81981d2e3727bd727e8c11580d34ded05c13d0fdb83bf60e98e596",
+  "change": "Apply FP32 TF32 restrictions after the seed helper resets backend flags",
+  "training_checkout_unchanged": true
+}

iclr-debug-two-stage-135b-20260916/generator/evaluation/step-12875-fp32/random-seed-0.json ADDED Viewed