Rakshitagg06/anlp-assignment2

ANLP Assignment 2: trained models

All checkpoints are PyTorch state_dicts for the assignment's decoder-only transformer (6 layers, d_model 512, 8 heads, FFN hidden 2048). Each folder is self-contained (checkpoint + config + tokenizer + model card). Training logs: https://wandb.ai/rakshitagg06-iiit-hyderabad/anlp-assignment2

Part 1: Mixture of Experts (vi/ja -> en translation)

Model	Folder	Test PPL	Test BLEU
Dense 2-layer MLP (baseline)	part1/moe_baseline	7.17	25.69
MoE, 4 experts, top-1 routing	part1/moe_top1	8.23	24.55
MoE, 4 experts, top-2 routing	part1/moe_top2	8.1	24.25
MoE, 1 shared expert + 3 routed experts, top-1	part1/moe_shared	7.7	24.25
MoE, 4 experts, top-2, active params = dense baseline	part1/moe_equal	7.33	26.14

Part 2: Optimizers (next-token pretraining on browndw/human-ai-parallel-corpus)

model.pt = 1x dataset (as specified); 3epoch/model.pt = 3 epochs.

Optimizer	Folder	Val loss (1x)	Test PPL (1x)	Test BLEU (1x)	Val loss (3 ep)
AdamW	part2/adamw	4.999	141.2	2.20	3.968
MARS (variance-reduced AdamW variant)	part2/mars	4.664	100.8	2.47	3.874
Lion (memory-efficient)	part2/lion	5.232	178.9	2.07	4.307
Muon (matrix-based)	part2/muon	4.101	57.2	3.19	3.873
Sophia-G (Hessian-based, bonus)	part2/sophia	4.954	134.7	2.28	4.483