usrnotfound101/anlp-a2-opt-muon

Dense decoder-only Transformer (27.40M params) pretrained from scratch on HAP-E for 1.0 x the train split with the muon optimizer (Matrix-based), implemented from scratch (see optimizers.py).

• hyper-parameters: {"lr": 0.02, "momentum": 0.95, "nesterov": true, "ns_steps": 5, "weight_decay": 0.01, "adamw_lr": 0.002, "adamw_betas": [0.9, 0.95], "adamw_wd": 0.1, "grad_clip": 1.0}
• final validation loss: 3.5459 (ppl 34.67)
• final test BLEU (multi-reference continuation): 4.00
• intermediate checkpoints every 0.1 x dataset tokens under checkpoints/
• full training / evaluation log: train_log.json