Mixture-of-Experts FFN ablations — Advanced NLP Assignment 2
Ten decoder-only translation models: five feed-forward variants, each trained at two token budgets. Everything except the feed-forward block is identical across variants, so any measured difference is attributable to the FFN alone.
Trained on belumind/en-vi-ja-curated-500k-triplets for Vietnamese→English
and Japanese→English translation.
Variants
A dense 2-layer MLP of hidden width h costs 2·d·h per layer. Variants
2–4 split that same budget across four experts (h/4 each), so total
parameters match the dense baseline. Variant 5 instead sizes experts at
h/2 so that active parameters match the baseline, giving twice the
total.
| run | experts | active | total params | active params |
|---|---|---|---|---|
dense |
— | — | 16,765,824 | 16,765,824 |
moe_top1 |
4 | 1 | 16,775,040 | 11,466,624 |
moe_top2 |
4 | 2 | 16,775,040 | 13,236,096 |
moe_shared |
1 shared + 3 routed | 2 | 16,772,736 | 13,233,792 |
moe_active_matched |
4 | 2 | 23,852,928 | 16,775,040 |
Each exists at two budgets: <name> (30M tokens) and <name>-100m
(100M tokens). The token budget is identical within each group — that
equality is what makes the comparison valid.
Architecture
6 layers, d_model 384, 6 heads, d_ff 1536, context 256, RMSNorm
pre-norm, RoPE, tied embeddings, 16k byte-level BPE shared across all
three languages.
Results
Test perplexity and BLEU (sacrebleu, signature
nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.6.0):
| variant | ppl 30M | ppl 100M | vi→en 30M | vi→en 100M | ja→en 30M | ja→en 100M |
|---|---|---|---|---|---|---|
| dense | 23.75 | 5.60 | 0.73 | 3.20 | 0.50 | 2.28 |
| moe_top1 | 31.81 | 6.60 | 1.41 | 4.69 | 0.81 | 3.02 |
| moe_top2 | 25.59 | 5.59 | 2.09 | 1.75 | 1.02 | 1.71 |
| moe_shared | 30.13 | 6.60 | 0.55 | 7.24 | 0.23 | 3.98 |
| moe_active_matched | 22.99 | 5.29 | 2.63 | 7.23 | 1.61 | 4.24 |
Limitations
These are coursework models and translation quality is low.
- At 100M tokens the models sit at ~6 tokens per parameter, well under the Chinchilla-optimal ~20. They are undertrained by design, to fit ten runs into the available compute.
- Roughly 24% of generations from the best model degenerate into repetition. The models translate short formulaic inputs reasonably and collapse on harder ones.
- BLEU in the 2–7 range is far below production systems (25–40). The numbers support relative comparison between variants, which is what the ablation needs, but should not be read as translation quality.
Usage
import torch
from huggingface_hub import hf_hub_download
path = hf_hub_download(
repo_id="samvaran/2025801013-anlp-assignment2",
filename="part1/moe_active_matched-100m/model.pt",
)
state = torch.load(path, weights_only=False)
print(state["model_config"]) # architecture
print(state["tokens_seen"]) # training budget actually consumed
Each checkpoint stores model_state, model_config, train_config and
tokens_seen. The model class and tokenizer live in the code repository;
assets/tokenizer.json there is required to encode or decode.