usrnotfound101/anlp-a2-opt-mars

Dense decoder-only Transformer (27.40M params) pretrained from scratch on HAP-E for 1.0 x the train split with the mars optimizer (Variance-reduced AdamW variant), implemented from scratch (see optimizers.py).

• hyper-parameters: {"lr": 0.002, "betas": [0.95, 0.99], "eps": 1e-08, "weight_decay": 0.1, "gamma": 0.025, "max_c_norm": 1.0, "grad_clip": 0.0}
• final validation loss: 3.7395 (ppl 42.08)
• final test BLEU (multi-reference continuation): 4.22
• intermediate checkpoints every 0.1 x dataset tokens under checkpoints/
• full training / evaluation log: train_log.json