Part 3 decoding results (finetuned)

Part 3: Pythia-160M fine-tuned on ROCStories

Part 3 decoding results (pretrained)

Upload Part 2 adamw: AdamW

Upload Part 1 moe_baseline: Dense 2-layer MLP (baseline)

Upload Part 2 sophia: Sophia-G (Hessian-based, bonus)

Upload Part 2 muon: Muon (matrix-based)

Upload Part 2 lion: Lion (memory-efficient)

Upload Part 2 mars: MARS (variance-reduced AdamW variant)

Upload Part 1 moe_equal: MoE, 4 experts, top-2, active params = dense baseline

Upload Part 1 moe_shared: MoE, 1 shared expert + 3 routed experts, top-1

Upload Part 1 moe_top2: MoE, 4 experts, top-2 routing

Upload Part 1 moe_top1: MoE, 4 experts, top-1 routing

Update index README

initial commit