orangebreak/anlp-a2-part2-optimizers

Task 2 of Advanced NLP Assignment 2: one dense decoder-only transformer (33M params) trained from scratch on browndw/human-ai-parallel-corpus with five hand-written optimizers, implemented from the appendix pseudocode of Fantastic Pretraining Optimizers and Where to Find Them (arXiv:2509.02046). Every run uses an identical 39,073,280-token budget (1x the dataset) and an identical seed-20 data stream.

Optimizer	Category	Final val loss	Final val PPL	Final BLEU
adamw	Baseline	3.9976	54.47	0.067
cautious	Variance-reduced AdamW variant	4.0856	59.48	0.063
lion	Memory-efficient	4.1461	63.19	0.040
muon	Matrix-based	3.6995	40.43	0.059
sophia	Hessian-approximation (bonus)	3.7694	43.35	0.049

Each subfolder holds final.pt (state dict + config) and history.json (validation loss and BLEU at every 0.1x of the dataset). tokenizer.json is the shared English 16k byte-level BPE used by all runs.