usrnotfound101/anlp-a2-opt-sophia

Dense decoder-only Transformer (27.40M params) pretrained from scratch on HAP-E for 1.0 x the train split with the sophia optimizer (Hessian-approximation (bonus)), implemented from scratch (see optimizers.py).

• hyper-parameters: {"lr": 0.0002, "betas": [0.965, 0.99], "rho": 0.05, "weight_decay": 0.2, "grad_clip": 1.0, "hess_interval": 10, "estimator": "gnb", "hess_batch": 16}
• final validation loss: 4.0959 (ppl 60.09)
• final test BLEU (multi-reference continuation): 3.77
• intermediate checkpoints every 0.1 x dataset tokens under checkpoints/
• full training / evaluation log: train_log.json