ANLP Assignment 2 — Part 1: Mixture of Experts
This repository contains the final trained checkpoints for Part 1: Mixture of Experts of Advanced NLP Assignment 2.
The task compares a dense Transformer feed-forward network against multiple Mixture-of-Experts (MoE) variants for translation from:
- Vietnamese → English
- Japanese → English
The models were trained on the assignment dataset and evaluated using perplexity, BLEU, and expert-routing statistics.
Checkpoints
This repository contains the following final checkpoints:
| File | Model configuration |
|---|---|
dense.pt |
Standard 2-layer dense MLP |
moe_1_of_4.pt |
4 experts, top-1 routing |
moe_2_of_4.pt |
4 experts, top-2 routing |
moe_shared_routed.pt |
1 always-active shared expert + 3 routed experts, with 1 routed expert active |
moe_iso_active.pt |
4 experts, top-2 routing, with active parameter count matched to the dense baseline |
The first four configurations use approximately the same total parameter count. The moe_iso_active configuration instead matches the dense model's active parameter count.
Parameter counts
| Model | Total parameters | Active parameters / token |
|---|---|---|
| Dense | 8,900,608 | 8,900,608 |
| MoE 1-of-4 | 8,900,608 | 6,541,330 |
| MoE 2-of-4 | 8,900,608 | 7,327,756 |
| Shared + Routed | 8,900,608 | 7,327,756 |
| Iso-active MoE | 12,043,240 | 8,900,608 |
Final test results
| Model | Test loss | Test perplexity | BLEU |
|---|---|---|---|
| Dense | 1.7041 | 5.50 | 20.27 |
| MoE 1-of-4 | 1.6662 | 5.29 | 20.32 |
| MoE 2-of-4 | 1.6539 | 5.23 | 20.56 |
| Shared + Routed | 1.6699 | 5.31 | 20.48 |
| Iso-active MoE | 1.6403 | 5.16 | 20.57 |
The sparse MoE variants achieve similar or slightly better translation quality while reducing the number of active parameters per token. The iso-active model achieves the lowest test loss and perplexity among the evaluated configurations.
Expert routing
For the MoE variants, expert usage was measured separately for Vietnamese and Japanese inputs.
The assignment repository contains:
- expert activation heatmaps
- per-language expert usage statistics
- router entropy values
- final evaluation metrics
These are used to analyze whether individual experts specialize differently across languages and to check for expert collapse.
Tokenizer
The tokenizer used for training and evaluation is included as:
tokenizer.json