usrnotfound101/anlp-a2-opt-lion

Dense decoder-only Transformer (27.40M params) pretrained from scratch on HAP-E for 1.0 x the train split with the lion optimizer (Memory-efficient), implemented from scratch (see optimizers.py).

• hyper-parameters: {"lr": 0.0002, "betas": [0.9, 0.99], "weight_decay": 1.0, "grad_clip": 1.0}
• final validation loss: 4.1024 (ppl 60.49)
• final test BLEU (multi-reference continuation): 3.38
• intermediate checkpoints every 0.1 x dataset tokens under checkpoints/
• full training / evaluation log: train_log.json