xTimeCrystal/nanochat-d16-loopfbt2-ar

nanochat d16: looped FBT (2 passes) + AttnRes, step 5376

Private research checkpoint, trained on 8x A100 with a modified nanochat (karpathy/nanochat 92d63d4 + AttnRes, FBT and looped FBT).

• Model config: {"sequence_len": 2048, "vocab_size": 32768, "n_layer": 16, "n_head": 8, "n_kv_head": 8, "n_embd": 1024, "window_pattern": "SSSL", "attn_res": true, "fbt_passes": 2, "fbt_inplace": true}
• Training: step 5376 (the final step), total batch 524288 tokens, device batch 8 x 2048 per GPU. Val bpb 0.7696. CORE 0.2115 (at the final step). The full launch config is user_config in meta_005376.json.

Files

• model_005376.pt: the model's state dict.
• meta_005376.json: model config, launch config, dataloader position and loop state.
• optim_005376_rank{0..7}.pt: the MuonAdamW state, one shard per rank, for resuming on 8 GPUs.
• code/: the files that define this model, from the training checkout as of this upload:
  • nanochat/gpt.py: AttnRes, FBT and looped FBT.
  • nanochat/attn_res.py: the AttnRes kernels.
  • nanochat/checkpoint_manager.py and scripts/base_train.py.
  • nanochat_dev.patch: the tracked changes against 92d63d4.

Loading

Copy code/ over a karpathy/nanochat 92d63d4 checkout. Put the checkpoint files in $NANOCHAT_BASE_DIR/base_checkpoints/loopfbt2_ar_d16/, then call nanochat.checkpoint_manager.build_model(dir, 5376, device, "eval").

Fine-tune: 3, then 4 passes

finetune_p3p4_lr05_wd0_mom90/ holds a 250-step fine-tune of this checkpoint at its end-of-training LR: 3 passes, then 4. Val bpb 0.7691 at 4 passes against 0.7696 here at 2, CORE 0.2138 against 0.2115. Its README has the per-pass numbers.